adminfoo.net

ITcookbook

noreply@blogger.com (quux) — Sat, 14 Nov 2009 23:26:00 +0000

I've got a new project going, and it's called ITcookbook.

I've got a collaborator this time (Jeff Palmer, an accomplished BSD admin), who will kick me in the pants and incite me to, you know ... write! He's no slouch in the writing department, and he'll be bringing his own unique skills and viewpoints to the table.

We're both pretty excited about this project. In it, we're going to be building and growing a complete IT infrastructure, with full documentation of how we accomplish every little step along the way. In text, screenshots ... and video. Our goal is to serve up plenty of ready-to-use complete step-by-step recipes handy to any IT practitioner out there, at any experience level. We'll be showing successes and failures. We will be hammering hard on things like documentation, and tools, and communication with the boss, and how to make good decisions about the foundations and growth of your IT infrastructure.

I will be bringing lots of my old content from here and other places into the ITcookbook project. I'll be updating a lot of that old stuff to fit with the changes that have happened over the years. Jeff will be bringing his *nix experience to the table, because the IT infrastructure we build will not be Windows-only.

We invite you to come take a look!

In Defense of UAC

noreply@blogger.com (quux) — Tue, 29 Jan 2008 03:06:00 +0000

This is me, editorializing - not an activity I prefer to use this blog for. But occasionally something seems so nonsensical to me that I just have to speak up. And today it's UAC that has me speaking up.

I'm greatly disappointed by the number of true techies and semi-techies who have gone into a sort of 'hate UAC' mode. I understand their frustrations; UAC can slow down routine sysadmin activities and break into the flow of your day. But, like seatbelts in a car, it's one of those things that seems like a terrible imposition at first, then just blends into your routine to the point where you hardly think about it. You buckle up and you're safer because you did it. I was one of those guys who hated seatbelt laws when they first appeared. But I got used to it, and now that I've survived a rollover accident because of a seatbelt, I'm pretty glad I did!

I've been running Windows systems according to the least privilege principle ever since Windows NT 4. I call this nonadmin because it's less of a mouthful. And I've been forcing users to do it, and begging my co-administrators to do it, since 1996. But I can't lie: nonadmin was a huge pain in the NT4/W2000/XP/W2003 days; the RunAs utility was a major PITA. You had to use special workarounds to elevate essential Windows tools like the file Explorer. There were a great many programs which didn't work well (or at all!) in a nonadmin context. Running nonadmin was doable ... but only with great self discipline, and a fair amount of pain as you worked through the changes in habits and gained the knowledge required to get it done.

In a corporate setting, it is possible for the administrator to do all the work and leave the regular office users blissfully unaware of the fact that they are nonadmin. But when you're the administrator - as home users and many smallbiz users are - the time and effort required to go nonadmin are just more than most people are prepared to expend. So they ignored the issue. And it was easy enough to do; most folks were completely unaware that MS had already made them administrators during the OS install!

There's another group of power users who were also blissfully unaware of their admin status: application coders and testers. Because they didn't know (or did not care) that they were administrators, it was very easy to write programs in ways that required admin capabilities to run properly. And this perpetuated the problem. It became a catch-22: users were admins because devs were. And devs were admins because users were.

Microsoft needed to break out of this loop, and they knew it. And UAC was born. It's actually an interesting compromise: because of all those legacy applications which require admin privs, MS couldn't simply force all users to a full nonadmin mode, which would have been technically preferable. So UAC strikes a middle ground: you're still an adminsitrator by default. But ... not really.

Márton Anka explains this well, so I will simply summarize: all accounts you create during Vista's setup are still members of the Administrators group. But when these accounts log in, all normal operations are carried out with a Limited User's access level. However, when Vista sees you do something that requires admin privs, it presents the UAC dialog box before it elevates privs for that process only. Unless you are running in the account named Administrator, in which case you never see the UAC prompt. Also note that users created after Vista setup do default to a Limited User status; they can elevate privs via UAC but must type a username and password to do so.

So why do I defend all this, while so many other techies are railing against it? Well, remember my saying how hard it was to run nonadmin with just the RunAs tool? UAC now makes all those difficulties a thing of the past.

I run as a Limited User, so every UAC prompt means I need to enter the name of my admin user, and its password. And I saw quite a lot of UAC while installing Vista, and in the two weeks afterwards. I did get a little sick of it, to be honest. But I kept reminding myself of the pain of RunAs, and I soldiered on. Finally, after setting up all my apps, and making various little tweaks and twiddles here and there (as we all tend to do), and exploring the various new administrative features of Vista ... I noticed that I was hardly seeing the UAC prompt at all.

Fast forward to the present. Though I still keep an XP system handy, Vista is the OS I spend my day in. From here I do of course RDP/VNC/SSH to other systems. As a sysadmin of course I'm always poking my nose into things. But honestly, I think I've seen less than 5 UAC prompts in the last 7 days - and I've installed one new app and upgraded another in that time.

So all that RunAs futzing is gone now - I only need to use RunAs for the rare occasions when I need to have an administrator-level instance of cmd. Instead of me having to remember to elevate my privs, Vista does the remembering for me. And whenever I see the UAC prompt, I pause for a second and ask myself whether I expected this. If I didn't expect it, I can ask the other relevant questions, like:

Do I recognize the program that wants elevated privs?

Do I trust it?

Do I have a reasonable understanding of why it needs elevated privs? (Do I know what it will do with them?)

Does the action really need to have full unfettered access to my system and data?

If I can't give myself satisfactory answers to these questions, I can just say no. And that's the end of it, clean and simple.

UAC is also providing a lot of back-pressure on program developers and testers to organize their programs and associated data into non-critical areas of the system, and this is a good thing. It means that new programs will be more stable and less risky to run, because developers themselves will not want to be hit with constant UAC dialogs. In the long run, this may be the largest benefit UAC provides.

In summary - UAC makes it easier to run nonadmin, and that's a good thing. It provides incentive for developers to write better code, and that's an excellent thing. I'm a big fan of UAC, and I hope the naysayers will see the light. Just as the seatbelt-haters did.

Miss me?

noreply@blogger.com (quux) — Sat, 19 Jan 2008 09:00:00 +0000

Contritely, I have to admit that I just haven't been blogging ~~a lot~~ at all lately. Somehow I got to thinking of my blog entries as things which should require doing the research of getting all the facts exactly right, taking screenshots, and coming up with exactly the right wording - well .. it is a time sink. And my time has been going other places lately.

But 2008 has rolled around, and I do still have stuff to share. Also, I've discovered this so-called tumbleblog phenomenon. Like 'blogosphere', I'm not sure I like the word, but I do like the concept, which is basically: less organization, more quick hits. So I proudly present my very own tumbleblog: http://quux.tumblr.com/. It is mostly just a collection of links and pithy observations I make on the fly as I go about my techie day.

I should also come clean to the many, err, 'almost adminfoo ready' bits 'n pieces that I have stored away on my wiki at http://quux.wiki.zoho.com/. So, I have been writing; it just hasn't been making it to this here blog. My apologies for not pointing this out sooner.

Also, I'm taking steps which may free up the time to do more proper adminfoo blogging. No promises yet, but let's all keep our fingers crossed!

Windows Perfmon: The Top Ten Counters

noreply@blogger.com (quux) — Thu, 05 Apr 2007 04:05:00 +0000

One of the things I love about Windows is Performance Monitor a/k/a PerfMon. It's an amazing tool that goes far too often unused - and when it does get used, it is often misinterpreted. So today I'm going to take you on the nickel tour through PerfMon, and the ten counters most valuable to determining overall system health and activity.

To open PerfMon, just go to the Start Menu, choose Run and type perfmon.

Bottleneck analysis

The most common use of PerfMon is to answer the burning question: why is my system running slow?

With the five performance counters listed below, you can quickly get an overall impression of how healthy a system is - and where the problems are, if they exist. The idea here is to pick counters that will be at low or zero values when the system is healthy, and at high values when something is overloaded. A 'perfectly healthy' system would show all counters flatlined at zero. (Perfection is unattainable, so you'll probably never see all of these counters flatlined at zero in real life. The CPU will almost always have a few items in queue.)

Processor utilization

System\Processor Queue Length - number of threads queued and waiting for time on the CPU. Divide this by the number of CPUs in the system. If the answer is less than 10, the system is most likely running well.

Memory utilization

Memory\Pages Input/Sec - The best indicator of whether you are memory-bound, this counter shows the rate at which pages are read from disk to resolve hard page faults. In other words, the number of times the system was forced to retreive something from disk that should have been in RAM. Occasional spikes are fine, but this should generally flatline at zero.

Disk Utilization

PhysicalDisk\Current Disk Queue Length\driveletter - this is probably the single most valuable counter to watch. It shows how many read or write requests are waiting to execute to the disk. For single disks, it should idle at 2-3 or lower, with occasional spikes being okay. For RAID arrays, divide by the number of active spindles in the array; again try for 2-3 or lower. Because a shortage of RAM will tend to beat on the disk, look closely at the Memory\Pages Input/Sec counter if disk queue lengths are high.

Network Utilization

Network Interface\Output Queue Length\nic name - is the number of packets in queue waiting to be sent. If there is a sustained average of more than two packets in queue, you should be looking to resolve a network bottleneck.
Network Interface\Packets Received Errors\nic name - packet errors that kept the TCP/IP stack from delivering packets to higher layers. This value should stay low.

To highlight a particular counter's line on the graph, select that counter in the lower pane. Then click the lightbulb icon on the toolbar above the graph. This will make the line for that counter turn thick and white (or black on some systems - I never found out why this changes).

Pay close attention to the scale column! Perfmon attempts to automatically pick a scale that will magnify or reduce the counter enough to produce a meaningful line on the graph ... but it doesn't always get it right. As an example, Perfmon often chooses to multiply Disk Queue Length by 100. So, you might think the disk queue length is sustained at 10 (bad!) when in fact it's really at 1 (good). If you're not sure, highlight the counter in the lower pane, and watch the Last and Average values just below the graph. In the screenshot below, I modified all of the counters to a scale value of 1.0, then changed the graph's vertical axis to go from 0-10.

To change graph properties (like scale and vertical axis as discussed above), rightclick the graph and choose Properties. There are a number of things to customize here ... fiddle with it until you have a graph that looks good to you.

To get a more detailed explanation of any counter, rightclick anywhere in the perfmon graph and choose Add Counters. Select the counter and object that you are curious about, and click the Explain button.

This screenshot shows a very lightly-loaded XP system, with the Memory\Pages Input/Sec counter highlighted:

All we see here is the Proccessor Queue Length hovering between 1 and 4, and two short spikes of Pages Input/Sec. All other counters are flatlined at zero, which is easy to check by highlighting each of them and watching the values bar underneath the graph. This is a happy system - no problems here!

But if we saw any of the above counters averaging more than 2-4 for long periods of time (except Processor Queue Length: don't worry unless it's above 10 for long lengths of time), we'd be able to conclude that there was a problem with that subsystem. We could then drill down using more detailed counters to see exactly what was causing that subsystem to be overloaded. More detailed analysis is beyond the scope of this article, but if there's enough interest I could do a second article on that. Leave a comment if you're interested!

General activity counters

Well, the system is healthy - and that's good ... but how hard is it working? Is the processor workin' hard, or hardly workin'? How much RAM is in use, how many bytes are being written to or read from the disk or network? The following counters are a good overview of general activity of the system.

Processor utilization

Processor\% Processor Time\_Total - just a handy idea of how 'loaded' the CPU is at any given time. Don't confuse 100% processor utilization with a slow system though - processor queue length, mentioned above, is much better at determining this.

Memory utilization

Process\Working Set\_Total (or per specific process) - this basically shows how much memory is in the working set, or currently allocated RAM.
Memory\Available MBytes - amount of free RAM available to be used by new processes.

Disk Utilization

PhysicalDisk\Bytes/sec\_Total (or per process) - shows the number of bytes per second being written to or read from the disk.

Network Utilization

Network Interface\Bytes Total/Sec\nic name - Measures the number of bytes sent or received.

In the graph below, I added these five counters to my existing 'bottlenecks' graph, and changed the vertical axis to go from 0-100. I highlighted the Working Set\_Total counter, which is currently at about 123 megabytes for the system. Notice how it shows a thick line at the top of the graph - you could assume that it was pegged at 100, if you didn't read the values bar (123,052,03 divided by a million is approximately 123 megabytes).

And ... that's all for now. Hopefully this quick show-and-tell has given you enough information to use PerfMon more usefully in the future!

OS Vulnerabilities Compared

noreply@blogger.com (quux) — Fri, 30 Mar 2007 04:54:00 +0000

Matthew Vea at OmniNerd has put together a fascinating report detailing the vulnerabilities of about a dozen operating system variants. I’m in awe of the simple yet effective method he used to cut through the fog:

Install the OS as default-ly as possible. Scan it with nmap and Nessus during the installation.
At completion of installation, scan again.
Install relatively common listening services and scan again.
Install the latest ‘major patch’, and scan again.
Finally install all ‘minor patches’ published prior to Jan 1 2007, and scan again.

I very much encourage you to read the full report, but one thing I sorely missed was a summary chart so I could get a better sense of what all that verbiage really means. So I created one – you see it below.

Some important points about this summary chart:

I left out the ‘mid-install’ scan info. I’m assuming y’all have the sense not to build your critical machines whilst connected to attack-prone networks.
The study mentions local vulnerabilities in one or two places, but is primarily concerned with remote vulns. In the ‘vulns’ column I list only those remote-exploitable vulns found by Nessus.
I’m not 100% sure I have the numbers exactly right. In some places the report was confusingly worded. I think I have preserved the author’s intent and I really hope he’ll let me know if I fumbled the ball.
I list port names for a reason. It seemed to me that in at least some cases, the choice of services to install in the ‘services installed’ config was a bit arbitrary. I note that some server OS have a web server enabled, some do not. So I thought this was important to include!
ICMP is not counted as one of the open ports.
As best I can tell, no firewall is enabled in any of the tests. In some cases, default firewalls were explicitly shut off.

There’s a lot to be learned here. For now, I’m drawing no conclusions. But I welcome yours, in the comments!

I'm sorry the above image is so small - click it to see at readable size. I tried to use an actual html table, so you'd be able to cut/paste from it, but I'm learning that Blogger likes to mangle html in its own special ways, so for now we'll have to make do with this image.

The SHTF Plan

noreply@blogger.com (quux) — Tue, 27 Mar 2007 05:49:00 +0000

So I found a DarkReading article on What to Do When Your Security's Breached, and I thought about it a bit, and decided I'd like to formulate my own plan and present it here. This isn’t just a ‘breach response’ plan; it’s really a plan for anytime the fecal matter impacts the rotating blades. (SHTF! Get it?)

I have some points in common with the DarkReading article. Here I focus mainly on operational items, but occasionally touch on the management issues too. Even if you’re not the boss, quietly suggesting these things can make all the difference.

It’s important to realize that the following list should not be considered as if written in stone. It’s a set of guidelines; a plan of attack. You may be able to check off items 3–7 inside of 20 minutes, or even decide some items don’t apply to your situation. That’s fine – in the heat of battle, situations become fluid. Things change. Keep your flexibility, but keep these things in mind. Be ready, when it’s all over, to say you thought about these things, and to explain why you did or did not do them!

First confirm your fears. You don't want to be the boy who cried wolf; you need to have a firm basis for your theory that someone has hacked (or is hacking) your network. At this stage, don't worry about mitigation. Just be sure you have enough evidence to form a solid basis for your theory that maliscious activity is going on.
Document everything. Start a timestamped log (textfile or handwritten notes) and gather your evidence nondestructively. Throughout the following steps, keeping a timestamped log of who was doing what is going to be important for many reasons. When it’s all done, you’ll need a record of what was damaged, what was changed, what was offline and for how long. The records will also be valuable in possible legal actions, and for planning for your next incident.
Assemble the team. By which I really mean - get your boss on the horn. At this point you want only the boss - don't involve coworkers. Let boss know what's going on, and what your evidence is. Emanate calmness and rationality here! No one is helped by a sense of fear or excitement or, dare I say it, panic. Suggest that the boss may want to involve legal, public relations, HR, line of business stakeholders, other IT workers - but let the boss decide and make those calls. Let him know that you're going to take no action other than diagnostic and getting your notes in order while he's assembling the team.
Make preliminary decisions. Now that the team is together, you’re going to need to present the information you’ve gathered so far. Next your team have a couple of relatively simple decisions to make:

Who has ultimate authority? This is not the time to have everyone off playing hero in an uncoordinated response. You need a ‘fire chief’ – and the chief needs to know what everyone is doing.
Priorities. Which is more important – fixing damages, closing the hole, preserving evidence so you can prosecute the attacker? You’re aiming to get a ranked priority list here. A bit of advice: do what you can to make sure that blame is de-prioritized. Let all members of the team know that this is most definitely not the time to even think about finger pointing!
Who will work on what?
How often should the response team get together to assess status and make decisions?
Communication plan. IT workers will want to be able to concentrate – their effectiveness is vastly reduced if they have to answer a ringing phone every five minutes. Work out a quick plan to avoid that, and put one person in charge of disseminating information to whomever else needs it. Let him be the guy whose phone rings every 5 minutes.
Work plan. Whomever is in charge will need to marshal forces carefully. In an event like this, people tend to overwork themselves – that can lead to mistakes, or a skeleton crew available tomorrow when the next problem happens. Be sure to rotate people effectively.

Assess damage (and damage potential). Now, with the team assembled and preliminary decisions made, it’s time to reconsider the damage done and the potential for more damage. Your team may need to do more investigation at this point, or it may have enough information to procede to the next step.
Stop the spread. Consider whether you need to shut down servers, services, network links, etc. Maybe a firewall rule change is enough. Think about how to make the smallest (and least destructive) changes that have the greatest impact on slowing the growth of the problem.
Notification plan. You’re probably going to have to tell your users about an outage. But what about customers, business partners, law enforcement? Consider carefully the phrasing and timing of such notifications. This is a management task.
Remediate. OK, now that you understand the damage, and you’ve stopped the spread, it’s time to consider the cleanup plan. This is going to depend much on some of the decisions made at step #4 – do you need to preserve evidence for later legal or disciplinary action? Which systems need to be cleaned, secured, and brought back online first? What can wait?
Post-Mortem. The post mortem is often skipped, as exhausted people all take their much-deserved rest, and then return to all those tasks they deferred whilst working on the outage. But impress on your boss the need for spending a couple of hours critiquing the incident, running down loose ends, and so on. It’ll be worth it – though you may never be able to measure that worth. Because you’ll never know about the problems your post-mortem successfully prevents!

Even if you can't get the post-mortem going, this is the best time to hit up your boss for funding the protective measures you wish you'd had before. Could be a new backup system, better network partitioning via VLANs - whatever. The iron is hottest about a week after the event. Strike it!

To me, most of the above seems like just plain common sense. But, having been through more than a few SHTF scenarios, I am always amazed by the number of people who, in their zeal to fix the problem, take leave of common sense, or do something which looks sensible from their POV but proves wasteful or even counterproductive when you step back and take the wide angle view.

Let’s be careful out there!

MTBF: Not What You Thought

noreply@blogger.com (quux) — Fri, 23 Mar 2007 18:13:00 +0000

It has recently come to my attention that most of us in the trade have basically the wrong idea about MTBF. You know the term, right? Mean Time Between Failures - which is a figure we often look at; especially when we choose disks for our arrays. The problem? In the words of the great Inigo Montoya:
You keep using that word. I do not think it means what you think it means.

A little reading over at Wikipedia will soon dispell you of the idea that when a disk's spec-sheet says the MTBF is 50 years, that disk will actually last 50 years. Oh, hmm, you say you took a look at that Wikipedia page, and the math scared you? Yeah - me too. So let's ask Richard Elling (Sun.com) to cut through the fog with a real-world example:

MTBF is a summary metric, which hides many important details. For example, data collected for the years 1996-1998 in the US showed that the annual death rate for children aged 5-14 was 20.8 per 100,000 resident population. This shows an average failure rate of 0.0208% per year. Thus, the MTBF for children aged 5-14 in the US is approximately 4,807 years. Clearly, no human child could be expected to live 5,000 years. Similarly, if a vendor says that the disk MTBF is 1 Million hours (114 years), you cannot expect a disk to last that long.

Oh. Well, dang. I guess that disk ain't very likely to last 50 years!

My rule of thumb? Check the warrantee coverage. The manufacturer with the longest warrantee at the lowest price is the one most confident in the durability of their drives.

Adminfoo is baaaaaack!

noreply@blogger.com (quux) — Mon, 19 Mar 2007 15:29:00 +0000

OK, I can't begin to express the angst, frustration, and finally denial states that I went through over the problems with my former web host. So, I won't. But I will direct your attention to the incredible, hulking 107-page thread about this debacle, over on ars technica. It is not pretty - and if you run out of patience long before you finish reading those 107 pages, I don't blame you. I paid those jokers for lifetime service - and the joke was on me.

I have to admit a bit of embarrassment here. I've always preached the Gospel of Backups, and so it's bitterly ironic that I didn't have a backup of the old adminfoo site. Now, this really is something I checked into before signing up with my former host - and they assured me they kept backups in three separate places. So I let that lull me into a false sense of confidence. I should have been keeping my own backups - and I can assure you I will be doing so from here on out. They'll be a simple wget dump of the site - but at least they'll exist.

If you happen to be one of the adminfoo readers who has found your way back to the site after this long hiatus: welcome back! I can't help but feel that I owe you two apologies: one for having dropped out of site for so long, and another for breaking the links to any articles you might have bookmarked. Sorry about that. Really, really sorry about that!

And so, here we are again. Back from the grave, and now hosted on Blogger. I did spend a lot of time looking into other host providers, other platforms, and so on ... and I finally decided to just let Google (who owns Blogger) do all the work for me. Sure I lose a couple of features over the excellent Drupal platform I was using before, but this is gonna save me a lot of work. No longer will I worry about my host provider being down, or how to implement a new anti-spam system, or the overhead of a lot of features that I really didn't need. It has always been my intent to write often and insightfully about the craft of systems administration, and I now realize that I was putting far too much effort into presentation, and not enough into the writing.

Ah, yes, the writing. Over the years my propensity for writing has ebbed and flowed on a schedule all its own. But the dry spells always end, and right now I feel a new wellspring of creativity a-blooming (pardon the mixed metaphor). Coming up, I plan to continue in the old vein, with commentary on strategy, tactics, and tools. I'll also bring back much of the old content, as I've found it's not that hard to just cut-and-paste from my old data to this new copy of adminfoo. It's time-consuming, though, and I have to do some manual reformatting. So I'm bringing back the oldest posts first, and I'll be slowly working through all the old content.

But I have some new stuff in mind, too:

The 'So You Inherited A Network' series I planned out over a year ago, and never got around to writing.

Map of systems administration knowledge. This will be on wiki.adminfoo.net, which is right now just a twinkle in my eye. The SAKmap is not my idea, actually; it belongs to another fellow whom I hope to convince to write here as well. I'm not sure he's aware of that yet, so I'll keep his name to myself for the time being.

OS patch reports. I admin a few Windows servers, and a few Linux ones too. While it might seem boring, I think a lot of people will benefit from seeing a list of patches applied to running systems - and knowing that in at least one case, those patches did or did not blow up the systems. Where I run into troubles or gotchas, I'll let you know - but I expect (and hope!) that most of those entries will be boring, predictable: applied patches foo, bar, and baz. No problems!

A few other ideas, which I'm keeping to myself right now. Mainly because, well, I dunno if I'll ever get around to, er, doing them.

Don't cache 'negative' DNS lookups on Windows systems

noreply@blogger.com (quux) — Fri, 29 Oct 2004 02:29:00 +0000

This one is a little bit esoteric. Scenario:

You try to connect to somesystem.yourdomain.com and fail - the name cannot be looked up.
You discover that the DNS record is missing in your DNS server, and you fix it by adding the correct record.
... but you still can't connect to somesystem.yourdomain.com from your workstation!

What's happening here is that your system has cached a 'negative lookup'. Your local DNS cache basically doesn't think the DNS name exists - and it will go on thinking that until the cached entry expires.

Here is an example:

C:\Tools>ipconfig /displaydns

Windows IP Configuration
1.0.0.127.in-addr.arpa
----------------------------------------
Record Name . . . . . : 1.0.0.127.in-addr.arpa.
Record Type . . . . . : 12
Time To Live . . . . : 0
Data Length . . . . . : 4
Section . . . . . . . : Answer
PTR Record . . . . . : localhost

nosuchmachine.cojones.org
----------------------------------------
Name does not exist.

adminfoo.net
----------------------------------------
Record Name . . . . . : adminfoo.net
Record Type . . . . . : 1
Time To Live . . . . : 308
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 67.15.36.7

localhost
----------------------------------------
Record Name . . . . . : localhost
Record Type . . . . . : 1
Time To Live . . . . : 0
Data Length . . . . . : 4
Section . . . . . . . : Answer
A (Host) Record . . . : 127.0.0.1

Here we see that the machine nosuchmachine.cojones.org was looked up, and found to be nonexistent. Now, even if I go and create a DNS record for nosuchmachine, my host will not resolve that name until the 'negative result' entry is flushed from my cache. I can manually flush it with an ipconfig /flushdns command.

Or I could put the following registry entries into my system:

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters]
"NegativeCacheTime"=dword:00000000
"NetFailureCacheTime"=dword:00000000
"NegativeSOACacheTime"=dword:00000000

Essentially this will tell my system to never cache 'negative lookups'.

Find rogue DHCP servers!

noreply@blogger.com (quux) — Thu, 28 Oct 2004 01:27:00 +0000

Sorry about the lack of new content lately - life has been busy here at the adminfoo ranch. Here's a quick blurb for dhcploc, which can be handy for tracking down rogue DHCP servers.

Basically this commandline tool can run on a Windows host and send a DHCP request, then report all servers which answer. It won't actually claim the DHCP address, though. You can also leave it running for awhile and it will beep and add a new line of output anytime it sees a DHCP request or offer in the wire - all DHCP packets are broadcast so it doesn't need to make your NIC promiscuous. Output looks like this. Packet sniffing (with the right filter) using Ethereal or some such would accomplish the same goal, but dhcploc is a quick and easy tool you can have a remote user (or customer) run for you without a lot of hand-holding.

You can download dhcploc along with some other Windows diagnostic tools here.

*nix commandline for Windows (free!)

noreply@blogger.com (quux) — Tue, 21 Sep 2004 01:25:00 +0000

Maybe you already knew: MS is giving away Services for Unix 3.5 (SFU), which can give you a very functional *nix commandline on a Windows box. No more struggling with Cygwin, looking for windows ports of your favorite commandline tools, etc.

But did you know about the Interop Systems Tools Warehouse? Here you can pick up dozens of *nix tools which will run just fine within the SFU environment. Take a walk on the *nix side, baby!

Network Access Quarantine recipe

noreply@blogger.com (quux) — Tue, 31 Aug 2004 06:00:00 +0000

Awhile back, we noticed that MS now gives you a way to check for and update hotfixes, AV, and firewall software before connecting VPN clients. This is big stuff; the VPN is the entry point for quite a few malware breakouts. You control your LANs; isn't it time you controlled the VPN?

Security Focus' John Hassel has written a step-by-step guide to building a VPN quarantine control system. While this requires Win2003 server as your VPN endpoint, we think this is a great justification for upgrading at least one of your servers. This could be one of the three best steps you'll take all year in terms of minimizing malware outbreaks. (The other two? Service Pack 2, and keeping enterprise antivirus up to date.)

Rule the rack

noreply@blogger.com (quux) — Sun, 29 Aug 2004 05:54:00 +0000

Racking up a few servers are you? Worried about getting everything to fit right? This tape measure, graduated in inches and rack units, could be a big help. Fifteen bucks gets you one.

Registry Cleaners = placebos

noreply@blogger.com (quux) — Sun, 29 Aug 2004 00:33:00 +0000

UPDATE (2005/10/10): Mark Russinovich has a few things to say about registry cleaners. I've added a few comments at the end of the article in an attempt to interpret.

It's time for a little adminfoo debunking. I've noticed a bit more hype lately about various 'registry cleaner' utilities, and I have this to say about them: bunk!

First, a short explanation of what the registry is. If you remember the old days of Windows 3.1, you may recall the old *.ini files (unix people should be thinking of *.conf files). These were text files which held configuration data for a program - things like where program data might be found, or user preferences such as window size and placement, etc. As Windows evolved, the decision was made to centralize all of these *.ini files into one central repository. And so the registry was born. It is now several files, but essentially it is accessed like a single database. Registry keys hold values and data.

The basic registry cleaner parses the whole registry looking for keys which point to things (files or other registry entries) which are not there. Some cleaners claim to do more, "healing" broken entries (I would be especially wary of these!). Bottom line is, if you have an actual bad registry entry (as opposed to just an extra one), you already know it because something is throwing errors. But folks, it's been a long time since I've seen a system error which could be traced back to a bad registry entry.

Let's take a few registry cleaner vendor claims and examine them (claims in italic; response in normal text):

...your registry gets stuffed with corrupt and orphaned entries. Many invalid records remain in the registry long after the corresponding applications were removed. This slows down your computer and can cause errors in other installed programs. A large registry is no slower than a small one, since Windows does not have to read the registry sequentially until it hits the right key and value. The registry is mapped, acting more like a database than a large file, so the program can go right to the key it needs and retreive a value. Notice how no registry cleaner product ever advertises actual measurements of speed gains? Finally, the difference between a 'large' registry and a 'small' one might be all of 20 mb (or 60 mb in really extreme cases). On today's computers that's a negligible amount of space.

...and can cause errors in other installed programs. Programs keep registry entries within their own keys and rarely if ever reference keys written by other programs. Today's programming best practices include building an uninstall routine which removes these entries when the program is removed. While it's fair to say that many older programs don't do this, and even many modern programs don't do it as well as they should, it's also true that registry keys, values, and data left behind by uninstalled/abandoned programs are simply never referenced. They take up very little space on your system.

Advanced "Deep" registry clean. There are thousands of programs out there - many of which can write thousands of combinations of registry keys/values/data. Do the math and you quickly discover millions of possible combinations, few of which are publicly documented by the program's makers. The idea that some omnipotent registry cleaning tool is aware of what all these programs "should" have entered in the registry, and will properly remove or correct each individual entry, seems silly to me.

Problems with the Windows registry are a common cause of Windows crashes and error messages. Not in my experience, or the experience of any of the professionals I asked about this. Ask yourself: when was the last time you had a problem with Windows and you were sure that problem was traceable to a registry entry?

For example, a registry key pointing to a non-existing file is invalid and must be deleted. Statements like this appeal to the neat freak in all of us. But again, there's very little actual need to do this, since the entry will simply never be referenced, and the key or value takes up only a few bytes on your hard disk. It's like saying all unused files MUST be deleted from your system: sure they aren't doing anything anymore, but if they aren't hurting anything and you have gigabytes to spare, what's the point?

... there's a lot more in this vein. But really all registry cleaners make the same tired, unprovable claims over and over again. Bottom line: most users have no need for this class of utility, and should not be fiddling with the registry anyway. Make changes to software via the software's provided interface, and let the software and the system handle the registry! This is snake-oil, folks, and you shouldn't be buying it unless you've got money to burn.

Troubleshooting pros will occasionally make setting changes directly in the registry. But it's like taking the back off of your TV set: don't do it unless you know what you're doing. Even if you do know what you're doing, you should always back up the registry before making any changes. If you insist on using a registry cleaner placebo, you'd be very wise to backup your registry first.

In closing I will contradict myself just a bit. Some folks do have a legitimate need for registry cleaners. Typically these are programmers and QA staff developing new software. In their dev and test routines they may install and uninstall the same software hundreds of times, and each time they need to be sure to remove all artifacts of the prior install, including registry entries. In this case, a registry cleaner may be justified - though most of the folks I know will write a script which specifically strips out the registry entries used by the program they are testing.

That's my rant. I'd love to be proven wrong, so if you can find measurable evidence of a registry cleaner doing any real good in the world, hit the comment link!

UPDATE (2005/10/10): I read Mark Russinovich's recent comments on Registry Junk with great interest. As you may know, Mark is the leading light behind the supremely respected sysinternals.com, so he's like E.F. Hutton in my book. When he speaks, I listen!

When I first started reading his article, I thought: yay, Mark and I agree! But actually he presents a more nuanced picture, going deep into a special situation I hadn't considered before: uninstallers that leave bogus keys in HKEY_USER registry key/value sets. However I beleive the situation Mark describes is pretty rare (especially in on-the-job situations), and leads to actual problems a vanishingly small amount of the time. Mark ends up concluding that "Registry cleaners will continue to have a place in the anal-sysadmin’s tool chest". Hmm! I've been called anal before ... but even after taking Mark's comments to heart, I still conclude that Registry cleaners are uncalled for 98% of the time they are used. Your mileage may vary.

First Things First

noreply@blogger.com (quux) — Tue, 24 Aug 2004 00:30:00 +0000

For those of you wanting to get started implementing or learning about security, I highly recommend SANS' First Things First paper. Starting with the venerable OSI model and moving quickly through a few network device and protocol primers, various network security methodologies, antivirus, and finally various host-hardening guides, this is the nuts and bolts of security exposed in a purposeful way. I particularly applaud this guide for directing people right to the RFC's for various network protocols, rather than more 'explanations of explanations' type documentation which is becoming all too common these days.

Excellent reading for anyone who wants to become a better admin, whether focused on security or not.

-Thanks to the folks writing the ISC Handlers Diary, which has become part of my daily morning read.

Windows Startup Online and EventID.net

noreply@blogger.com (quux) — Sat, 21 Aug 2004 00:25:00 +0000

UPDATE 2005/08/22: Just over a year later, I have found Tasklist.org, another resource which maps programs and vendors to .exe names.

Yesterday I was troubleshooting some odd issues with my new laptop. Although unrelated, I noticed quite a few programs being started automatically when I boot my system up - most of them added by the laptop vendor. So I began investigating to see what they were. This isn't as easy as it should be - while MSCONFIG is a big help, you're still left wondering what 'TpKmapAp.exe' or 'SynTPEnh.exe' actually do, and whether or not you need them starting up every time you boot your computer.

And that's where windowsstartup.com's very handy database of startup programs comes in. It's actually meant as the backend database for their Startup Inspector program (which I haven't tried yet, though it looks nice and is free), but you can query it directly via the web. Excellent idea!

(Added 11/23/2004) For entries missed in the StartupInspector database, I recommend you try ProcessLibrary.com, a similar tool. Between these two, most startup programs should be identified.

I'll round out this post with a similar web database: EventID.net, an online database of things you might find in your event log. Query results will often provide more in-depth explanations of what the log entry means, and what you should do about it. Highly recommended for all you universal troubleshooters.

Transferring ownership of files

noreply@blogger.com (quux) — Fri, 20 Aug 2004 00:24:00 +0000

If you're a Windows admin, you've probably 'taken ownership' of files in the past. Did you know it's also possible to 'give ownership'? The normal GUI won't allow you to do it (unless your on a Win2003 server box), but this article shows the way for NT and Win2000 using subinacl.exe.

Manage your passwords!

noreply@blogger.com (quux) — Tue, 17 Aug 2004 18:41:00 +0000

Even if you work in a single-signon (SSO) environment, I'll bet you have at least 20 passwords. Probably more.

How are you keeping track of them all? Not the dreaded sticky note on the monitor I am sure - because you're a professional. But here's a funny thing: last Friday I visited a respected professional in his cube and watched him peek at a slip of paper from an unlocked drawer as he logged on to one of our lesser-used servers. He looked sort of sheepish doing it, but it's a real problem. You're supposed to create hard-to-crack passwords that are difficult to remember, and you're supposed to remember them without writing them down. Catch-22.

There are a lot of password managers out there. I use Bruce Schneier's Password Safe for these reasons:

I trust Bruce. He's a clear thinker and a hype basher.

It's small enough to fit easily on a floppy (remember those?) or a USB key.

No install routine, so I can store it on a network share and access from anywhere.

It's secure - and it's simple. That's the goal y'know: security made simple.

Of course you should choose whichever password manager you like - just about any password manager is more secure than a piece of paper in a drawer or a wallet, and if it's not comfortable you won't use it. Once you have and use a password manager, you will be more inclined to use really hard passwords - I usually generate random ones with Password Safe.

I keep two password files - one work related and one for my psersonal stuff. Every so often I dump the work related one to a floppy. I write the password on the floppy, seal it in an envelope, sign across the flap, and give it to my boss - in exchange for the old one. This means my replacement can get right to work after I have spontaneously combusted. Might also be handy if the server breaks - I won't have to wait for the full restore to get my passwords back!

I keep a copy of my 'not work' password file in our fireproof safe at home. But it'd be just as good to have a friend or relative hang on to it.

Linux: simple 'snapshot' backups with rsync

noreply@blogger.com (quux) — Mon, 16 Aug 2004 18:18:00 +0000

If adminfoo seems a little windows-centric, that's because, well, it is. It's not meant to be, but I don't have much choice in the matter, since I spend about half of my waking hours thinking about Windows. It's not a prejudice issue ... and to prove it (again!) I bring this note about backups on Linux.

One of the sexiest things about Netapp is those cool snapshot pseudo-backups. This article shows how to create a poor-man's functional snapshot on Linux, basically by the smart use of rsync, cp, and mv commands. It is well thought out, and finds an exciting strength in the *nix way of running a filesystem. As the author leads you through his excellent backup strategy for Linux, he also provides a thoughtful grounding on some Linux file management basics. An excellent return to first principles for gurus, and a wonderful and practical walk through for Linux newbs.

Highly recommended. (Oh, and if you'd like more *nix articles here, why not sign up and submit them?)

Antivirus: defense in depth

noreply@blogger.com (quux) — Thu, 12 Aug 2004 18:11:00 +0000

MS has released a 90-page Antivirus Defense-in-Depth Guide which is well worth a read. Although you'll already be familiar with many of the concepts outlined in the Guide, it brings everything together in a nice framework overview, and may well expose a few specific tips/tricks you hadn't been aware of. If nothing else, print out chapter 4 (Outbreak Control and Recovery) and keep it at your desk to use as a plan of attack when outbreaks do occur.

In related news, Jeffrey Lee Parson (pictured on the right) has pled guilty to releasing one strain of the MSBlast malware that was so troublesome last year, and will be sentenced in November.

Create an XP install CD with SP2 preinstalled

noreply@blogger.com (quux) — Sun, 08 Aug 2004 17:46:00 +0000

Service Pack 2 for Windows XP is out ... but you knew that already, right? It's been done to death at all the other sites, so we won't go nuts with a review of our own (though we do hope to post some interesting SP2 registry hacks soon). We will note that it's getting some pretty strong endorsments though.

So we'd just like to remind you that it's not hard to build a custom bootable XP install CD with SP2 preinstalled. It's called 'slipstreaming', in case you didn't already know, and it can save valuable time when building new systems. Here are a few links which show how to do it:

Bink.nu (instructions are for Win2k but work fine with XP and SP1 or SP2).

MSFN's 'definitive' unattended guide. Goes futher than the Bink.nu guide into hotfixes, Office updates, applications, device drivers, and more.

Autostreamer is a nice looking GUI which builds a slipstreamed disk for you.
...if none of these are to your liking, google a bit. There are many other guides out there.

RAID Info

noreply@blogger.com (quux) — Sat, 31 Jul 2004 17:33:00 +0000

When I saw this picture, I knew it was time to bring some mass storage links to the front page. So here we bring you some basic RAID information.

ArsTechica's The Skinny On RAID

Wikipedia's RAID entry

AC&NC presents their RAID tutorial

StorageReview has this well-rounded discussion of RAID (note the bit about caching, often forgotten in RAID performance discussions!)

These are the basics, and will at least clear up the confusion on which RAID level is which. The problem, however, is that deeper knowledge is hard to find. You'll see many pronouncements similar to 'RAID1 is faster than RAID5 because of increased mechanical overhead', but you'll find little actual test data proving these assumptions. It is important to note that mechanical performance is not the only determining factor in this day of cheap RAM caches. This dearth of empirical data seems to stem from the fact that all RAID controllers are different, and no one seems to be doing comprehensive tests in this area. We all have our little RAID prejudices, but few of us have good data to back up our claims.

If you know where to find good and recent performance data on a variety of RAID layouts and configurations, clue us in by adding a comment below!

The Egoless Admin / PacketAttack

noreply@blogger.com (quux) — Sat, 31 Jul 2004 15:09:00 +0000

Today we bring you a link to The Egoless Admin, a short but worthwhile read on dealing with difficult users. We really like point #5, but we think they are all good. Yesterday was sysadmin day, but every day is user day!

Since that was such a short blurb, we also recommend packetattack.com as an excellent education and resource site for the network administrators among us. Packetattack has several excellent tutorials, a comprehensive list of links to the best of Cisco's site, and a valuable security section, among other things. We're pretty sure Ann Coulter* wouldn't visit Packetattack!

--
*Actually we have no idea - this is just more gratuitous name-dropping from Google's list of popular searches. Is the hit counter going crazy yet?

GenControl - a better VNC

noreply@blogger.com (quux) — Fri, 30 Jul 2004 02:08:00 +0000

Many of you probably read the previous story about KVM-over-IP and wondered: why pay for such a thing? With Terminal Services, exportable X Windows, VNC and other remote control programs, what's the need? Of course the answer is that you most want remote control of a system when the fit has hit the shan: the system is hung, or rebooted and sitting at a BIOS prompt, etc. Without remote KVM capability, the system must stay down while you get dressed, find your keys, and drive to work. With remote KVM (and possibly remote power control, which I'll address at a later date), you simply connect from home and fix the problem.

But for those looking for a decent, non-intrusive and no-cost way of running Windows systems from afar, I present GenControl. This is a modified VNC with a unique twist: it doesn't have to be manually installed at the remote end. You pick a system to control, and GenControl reaches out over then net, installs its server component, then connects you to the console session of the remote system. This happens in under 60 seconds on most WANs. When you're done, GenControl reverses the process, silently de-installing itself. You must have administrative privs on the remote system for this to work. Thus GenControl has no builtin security; it relies on the target OS's security levels.

GC does suffer from the cursor lag found in many VNC versions; however this isn't difficult to get used to. Note that GC grabs the console session on the remote machine, getting around some issues where things act differently in a terminal services session than they would at the console. GC does alllow you to share keyboard/mouse with a person sitting at the console, making this a nice solution for helpdesk support scenarios.

KVM-over-IP becomes affordable

noreply@blogger.com (quux) — Fri, 30 Jul 2004 02:04:00 +0000

I like to think of server rooms as dark, chilly places, rarely visited by mankind. We go there to install systems, change tapes, and perform the occasional hardware upgrade/repair. But generally speaking, I think this should be a low traffic area. Less jostling leads to more uptime, in my opinion.

Remote management tools such as KVM-over-IP help keep down the traffic in your server room - and they make administration easier. It's just more comfortable to work from your own desk or home office than to spend hours standing at a keyboard in the loud/cold server room. Anytime you make work easier to do, you increase the likelihood that said work will actually get, y'know, done. The problem with KVM-over-IP has been its high price: management couldn't see spending the $10k prices this stuff was commanding just two years ago.

But the world has changed again: now you can get this Aten unit for $500, and there are several other single-port choices in the sub-$1000 range. Connect one of these to your existing KVM switch, and you just might have something that'll let you perform that 2am BIOS upgrade from home, in your jammies. The (pictured) AMI Megarac unit ($800) looks especially attractive to travelling sysadmin types - toss it in your toolbag and use it to connect your laptop directly to that problem headless server. No more lugging monitors from one end of the server room to another!

Has anyone out there used these low-price KVM-over-IP systems yet? Let us know your experiences!