adminfoo.net

One of the things I love about Windows is Performance Monitor a/k/a PerfMon. It's an amazing tool that goes far too often unused - and when it does get used, it is often misinterpreted. So today I'm going to take you on the nickel tour through PerfMon, and the ten counters most valuable to determining overall system health and activity.

To open PerfMon, just go to the Start Menu, choose Run and type perfmon.
Bottleneck analysis
The most common use of PerfMon is to answer the burning question: why is my system running slow?

With the five performance counters listed below, you can quickly get an overall impression of how healthy a system is - and where the problems are, if they exist. The idea here is to pick counters that will be at low or zero values when the system is healthy, and at high values when something is overloaded. A 'perfectly healthy' system would show all counters flatlined at zero. (Perfection is unattainable, so you'll probably never see all of these counters flatlined at zero in real life. The CPU will almost always have a few items in queue.)

• Processor utilization
• • System\Processor Queue Length - number of threads queued and waiting for time on the CPU. Divide this by the number of CPUs in the system. If the answer is less than 10, the system is most likely running well.
• Memory utilization
• • Memory\Pages Input/Sec - The best indicator of whether you are memory-bound, this counter shows the rate at which pages are read from disk to resolve hard page faults. In other words, the number of times the system was forced to retreive something from disk that should have been in RAM. Occasional spikes are fine, but this should generally flatline at zero.
• Disk Utilization
• • PhysicalDisk\Current Disk Queue Length\driveletter - this is probably the single most valuable counter to watch. It shows how many read or write requests are waiting to execute to the disk. For single disks, it should idle at 2-3 or lower, with occasional spikes being okay. For RAID arrays, divide by the number of active spindles in the array; again try for 2-3 or lower. Because a shortage of RAM will tend to beat on the disk, look closely at the Memory\Pages Input/Sec counter if disk queue lengths are high.
• Network Utilization
• • Network Interface\Output Queue Length\nic name - is the number of packets in queue waiting to be sent. If there is a sustained average of more than two packets in queue, you should be looking to resolve a network bottleneck.
  • Network Interface\Packets Received Errors\nic name - packet errors that kept the TCP/IP stack from delivering packets to higher layers. This value should stay low.

To highlight a particular counter's line on the graph, select that counter in the lower pane. Then click the lightbulb icon on the toolbar above the graph. This will make the line for that counter turn thick and white (or black on some systems - I never found out why this changes).

Pay close attention to the scale column! Perfmon attempts to automatically pick a scale that will magnify or reduce the counter enough to produce a meaningful line on the graph ... but it doesn't always get it right. As an example, Perfmon often chooses to multiply Disk Queue Length by 100. So, you might think the disk queue length is sustained at 10 (bad!) when in fact it's really at 1 (good). If you're not sure, highlight the counter in the lower pane, and watch the Last and Average values just below the graph. In the screenshot below, I modified all of the counters to a scale value of 1.0, then changed the graph's vertical axis to go from 0-10.
To change graph properties (like scale and vertical axis as discussed above), rightclick the graph and choose Properties. There are a number of things to customize here ... fiddle with it until you have a graph that looks good to you.
To get a more detailed explanation of any counter, rightclick anywhere in the perfmon graph and choose Add Counters. Select the counter and object that you are curious about, and click the Explain button.
This screenshot shows a very lightly-loaded XP system, with the Memory\Pages Input/Sec counter highlighted:




All we see here is the Proccessor Queue Length hovering between 1 and 4, and two short spikes of Pages Input/Sec. All other counters are flatlined at zero, which is easy to check by highlighting each of them and watching the values bar underneath the graph. This is a happy system - no problems here!
But if we saw any of the above counters averaging more than 2-4 for long periods of time (except Processor Queue Length: don't worry unless it's above 10 for long lengths of time), we'd be able to conclude that there was a problem with that subsystem. We could then drill down using more detailed counters to see exactly what was causing that subsystem to be overloaded. More detailed analysis is beyond the scope of this article, but if there's enough interest I could do a second article on that. Leave a comment if you're interested!
General activity counters
Well, the system is healthy - and that's good ... but how hard is it working? Is the processor workin' hard, or hardly workin'? How much RAM is in use, how many bytes are being written to or read from the disk or network? The following counters are a good overview of general activity of the system.

• Processor utilization
• • Processor\% Processor Time\_Total - just a handy idea of how 'loaded' the CPU is at any given time. Don't confuse 100% processor utilization with a slow system though - processor queue length, mentioned above, is much better at determining this.
• Memory utilization
• • Process\Working Set\_Total (or per specific process) - this basically shows how much memory is in the working set, or currently allocated RAM.
  • Memory\Available MBytes - amount of free RAM available to be used by new processes.
• Disk Utilization
• • PhysicalDisk\Bytes/sec\_Total (or per process) - shows the number of bytes per second being written to or read from the disk.
• Network Utilization
• • Network Interface\Bytes Total/Sec\nic name - Measures the number of bytes sent or received.

In the graph below, I added these five counters to my existing 'bottlenecks' graph, and changed the vertical axis to go from 0-100. I highlighted the Working Set\_Total counter, which is currently at about 123 megabytes for the system. Notice how it shows a thick line at the top of the graph - you could assume that it was pegged at 100, if you didn't read the values bar (123,052,03 divided by a million is approximately 123 megabytes).



And ... that's all for now. Hopefully this quick show-and-tell has given you enough information to use PerfMon more usefully in the future!

Sorry about the lack of new content lately - life has been busy here at the adminfoo ranch. Here's a quick blurb for dhcploc, which can be handy for tracking down rogue DHCP servers.

Basically this commandline tool can run on a Windows host and send a DHCP request, then report all servers which answer. It won't actually claim the DHCP address, though. You can also leave it running for awhile and it will beep and add a new line of output anytime it sees a DHCP request or offer in the wire - all DHCP packets are broadcast so it doesn't need to make your NIC promiscuous. Output looks like this. Packet sniffing (with the right filter) using Ethereal or some such would accomplish the same goal, but dhcploc is a quick and easy tool you can have a remote user (or customer) run for you without a lot of hand-holding.

You can download dhcploc along with some other Windows diagnostic tools here.

Maybe you already knew: MS is giving away Services for Unix 3.5 (SFU), which can give you a very functional *nix commandline on a Windows box. No more struggling with Cygwin, looking for windows ports of your favorite commandline tools, etc.

But did you know about the Interop Systems Tools Warehouse? Here you can pick up dozens of *nix tools which will run just fine within the SFU environment. Take a walk on the *nix side, baby!

Awhile back, we noticed that MS now gives you a way to check for and update hotfixes, AV, and firewall software before connecting VPN clients. This is big stuff; the VPN is the entry point for quite a few malware breakouts. You control your LANs; isn't it time you controlled the VPN?

Security Focus' John Hassel has written a step-by-step guide to building a VPN quarantine control system. While this requires Win2003 server as your VPN endpoint, we think this is a great justification for upgrading at least one of your servers. This could be one of the three best steps you'll take all year in terms of minimizing malware outbreaks. (The other two? Service Pack 2, and keeping enterprise antivirus up to date.)

• Part 1
• Part 2

Racking up a few servers are you? Worried about getting everything to fit right? This tape measure, graduated in inches and rack units, could be a big help. Fifteen bucks gets you one.

UPDATE 2005/08/22: Just over a year later, I have found Tasklist.org, another resource which maps programs and vendors to .exe names.

Yesterday I was troubleshooting some odd issues with my new laptop. Although unrelated, I noticed quite a few programs being started automatically when I boot my system up - most of them added by the laptop vendor. So I began investigating to see what they were. This isn't as easy as it should be - while MSCONFIG is a big help, you're still left wondering what 'TpKmapAp.exe' or 'SynTPEnh.exe' actually do, and whether or not you need them starting up every time you boot your computer.

And that's where windowsstartup.com's very handy database of startup programs comes in. It's actually meant as the backend database for their Startup Inspector program (which I haven't tried yet, though it looks nice and is free), but you can query it directly via the web. Excellent idea!

(Added 11/23/2004) For entries missed in the StartupInspector database, I recommend you try ProcessLibrary.com, a similar tool. Between these two, most startup programs should be identified.

I'll round out this post with a similar web database: EventID.net, an online database of things you might find in your event log. Query results will often provide more in-depth explanations of what the log entry means, and what you should do about it. Highly recommended for all you universal troubleshooters.

If you're a Windows admin, you've probably 'taken ownership' of files in the past. Did you know it's also possible to 'give ownership'? The normal GUI won't allow you to do it (unless your on a Win2003 server box), but this article shows the way for NT and Win2000 using subinacl.exe.

Even if you work in a single-signon (SSO) environment, I'll bet you have at least 20 passwords. Probably more.

How are you keeping track of them all? Not the dreaded sticky note on the monitor I am sure - because you're a professional. But here's a funny thing: last Friday I visited a respected professional in his cube and watched him peek at a slip of paper from an unlocked drawer as he logged on to one of our lesser-used servers. He looked sort of sheepish doing it, but it's a real problem. You're supposed to create hard-to-crack passwords that are difficult to remember, and you're supposed to remember them without writing them down. Catch-22.

There are a lot of out there. I use Bruce Schneier's Password Safe for these reasons:

• I trust Bruce. He's a clear thinker and a hype basher.
• 
  
• It's small enough to fit easily on a floppy (remember those?) or a USB key.
• 
  
• No install routine, so I can store it on a network share and access from anywhere.
• 
  
• It's secure - and it's simple. That's the goal y'know: security made simple.

Of course you should choose whichever password manager you like - just about any password manager is more secure than a piece of paper in a drawer or a wallet, and if it's not comfortable you won't use it. Once you have and use a password manager, you will be more inclined to use really hard passwords - I usually generate random ones with Password Safe.

I keep two password files - one work related and one for my psersonal stuff. Every so often I dump the work related one to a floppy. I write the password on the floppy, seal it in an envelope, sign across the flap, and give it to my boss - in exchange for the old one. This means my replacement can get right to work after I have spontaneously combusted. Might also be handy if the server breaks - I won't have to wait for the full restore to get my passwords back!

I keep a copy of my 'not work' password file in our fireproof safe at home. But it'd be just as good to have a friend or relative hang on to it.

MS tool for Windows 2003 remote access services (RRAS) allows remote clients (via VPN) to be quarantined until they prove certain computer settings have been made. Check it out. There is an additional component to allow ISA servers to create access rules based on these quarantine settings.

Disk full error. We've all seen it; and we've all spent hours finding old/unused files or ferretting out the worst offenders. Here are some visual utilities which can speed that process, and help you communicate the problem to your users and/or management.

• Sequoiaview: Freeware. Shows disk usage as a lot of subdivided squares or rectangles.
• SpaceMonger: Freeware. Similar to Sequoiaview, some find this visually easier to use.
• Scanner: Freeware. Uses a sort of 'concentric piecharts' metaphor. My favorite freeware.
• DiskData: $30. I use this most often. Produces management-friendly piecharts as well as bar charts and reports. (I take screenshots to of the pie graphs for my mgmt reports)
• Disk Space Inspector: $30-$250 depending on how you use it. I haven't tried this but it looks nice - especially the ability to save reports and graphics as html.

There are many more out there I am sure. Including more intelligent ones which will sniff out duplicate files, identify filetypes you may not want on your servers (mp3 comes to mind), and so on. Tell us what you're using!

Here are three free CD-based tools. The first two are more useful to Windows admins; the last one should please just about everyone. Just download and burn the ISO images.

Pnordahl's Offline Password and Registry Editor: One of the more common problems in the Windows world is the forgotten Administrator password. I used to use Winternals' ERD Commander to reset the password in these cases, but they have gone to a (much!) more expensive and restrictive licensing scheme. The free Pnordahl tool is much faster, though a little harder on the eyes. Note, I have found that this tool is much more reliable if you reset to a blank admin password (specify blank password with an asterisk (*) when prompted) - setting the password to an alphanumeric string doesn't always work. That's ok; just pull the network cord before reboot and set a complex admin password within Windows after blanking it with this tool.

Bart's PE Builder: This kit allows you to build a customized CD of your own. When you're done you'll have a bootable GUI 'mini-OS' with whatever tools you select - you can then use this as a repair environment for seriously sick systems. There's a growing body of plugins and enhancements available; check out the virus scanner plugins!

DBAN: There's only one reason to insert this CD into a system: you want to completely and securely nuke all data on all local writeable drives. Excellent for the final decommissioning of your systems.

(UPDATED 7/14/2004 - addition of susserver.com link)

If you have more than 10 Windows systems to admin and you haven't automated your patch management yet, what are you waiting for? SecurityFocus' Jonathan Hassell has written a nice article on SUS and a few tools that add to its capabilities. This is the easy/smart way to deploy updates, and it doesn't sleep like you and I must.

Article Part 1 - Article Part 2 - Article Part 3

Another great resource for SUS is susserver.com - the good folks there have documented some 'under the hood' registry tweaks which can be helpful for advanced tweaking. If there is any interest in the topic, I can add a few scripts I've written, as well as a colleague's impressions of a freeware SUS report tool which we use in production at my day job.

source: SecurityFocus