The Life of a Sysadmin

I'm not making much headway erasing flash, so I'm taking a bit of a break from that by turning my attention to the network interfaces.

There are five ethernet interfaces on the NWR04B, plus the wireless card. Only one of the ethernet interfaces comes up enabled in Linux, so I've been trying to track down how it's all initialized and how to change it. The problem is, once again, that I really don't know what I'm doing, and I'm having to teach myself as I go along.

For example: the driver and the datasheet for the CPU talk about the CPU port for the switch. What the hell is that? Originally I thought that might be a special ethernet interface -- you know, like the one that's enabled in Linux. But this info says no, it's essentially a logical interface that may or may not be connected to a particular ethernet interface.

Yeah, the info is for the ADM5120 switch (which in turn is based on a MIPS chip) "a/o/t the ADM5106 (which is ARM-based) I'm working with, but I think the principal should be the same. It certainly seems to match the sort of stuff I'm seeing in the driver code. I'm hopeful, too, that the configuration tool for the 5120 will be, broadly speaking, applicable to the 5106...they talk about ioctls in the driver being used for this sort of thing; not sure if they're in the 5106 driver (I suspect not), but the source code available for 5120-based routers might have enough info to let me cut-and-paste^Wport something over.

Update: So the config tool for the 5120 uses a couple ioctls, SIOCSMATRIX and SIOCGMATRIX, to control which interfaces are on which VLAN. SIOCSMATRIX is defined in the config tool's code as SIOCDEVPRIVATE, and that has to be defined by the driver. It wasn't there in the source code, but a quick search for SIOCSMATRIX turned up a few diffs against the main kernel tree for the 5120. Looks like the switch driver may be a bit more full-featured than what I've currently got for the 5106, and I think it may be more or less a simple cut-n-paste to get it working for the NWR04B. Here's hoping.

I got these 1U units 'cos I was running out of room, and I figured I'd test these out. Natch, I'm running into problems with them. First off, I've got one that has panicked on me a few times in the last week. There's never any indication in the logs of what's happened, of course. I thought it was heat, but it recurred after moving it around. Fortunately I had a monitor hooked up, and I saw the panic message. Nothing much that I can make sense of, except that it seems to be in the middle of dealing with an interrupt when it happens -- part of the message had scrolled off the screen. I've turned on console logging and hooked up a serial cable to another machine, so now I wait. I'm unsure if the second problem is on the same machine or its twin, but it's strange. When SSHing in, or using rsync to transfer big files between the two, I suddenly found the session gone and this error on the screen:

Disconnecting:  Corrupt MAC on input

A bunch of Googling turns up this thread on Google and this patch from Andrew Morton's kernel, which apparently just missed going into 2.6.11. I'll have to give that a try and see how it does. (I feel oddly proud of using a patch from Morton -- something about being that much closer to celebrity. Sometimes I have absolutely no clue how my mind works.) How things have changed, BTW...I had to check to see what the latest kernel was. Also, when I was trying to get these machines working, I had difficulty with the stock 2.6.8 kernel in Debian -- wouldn't recognize the hard drive. My response? Grab the 2.6.11 stock kernel from Debian, rather than compile my own. Strange days, indeed. One difference between the two: ACPI was enabled in the BIOS of one, and not the other. Weirdly, HT depends on ACPI...who knew? Also, the IRQ list is slightly differnet without them; you don't get IRQ set to Cascade if HT/ACPI is disabled.

In here attempting to patch the 35 or so Windows machines that we've got at work. So far, it looks like I should be able to do this remotely using SSH and Cygwin. That depends, of course, on having very fucking silent ways of running everything. So far this has worked for me, on XP/SP2 and 2K/SP4:

1. regsvr32: None of the bits I've seen from SANS mentions it, but there is a silent option. Do it like so: regsvr32 /s /u %windir%\wystem32\shimgvw.dll However: I cannot get %windir% to work with Cygwin. According to this it should work as %WINDIR%, but it doesn't for me. Two things do seem to work: either change directory to /cygdrive/c (Cygwin-specific location of the C: drive) and use an absolute path (winnt\\system32\\shimgvw.dll), or run CMD to get a DOS/Windows shell and use %windir%.
2. The Unofficial Patch: Use the options: /VERYSILENT /SUPPRESSMSGBOXES However, it has problems if you try applying it on a machine that already has had the patch -- remotely, execution will just hang. If you run it locally w/o those options, you'll get a message saying it's already been applied; I guess that case is not handled well when run silently. Oh, and when the patch is applied silently, it'll reboot the machine immediately and without warning.

Thanks to Cygwin, I've got SSH running on most Windows machines here; I should be able to come up with some way of doing this all in one step. I'll post whatever I can figure out. Update: Yep, a simple batch file does the trick: regsvr32 /s /u %windir%\system32\shimgvw.dll c:\cygwin\home\Administrator\wmffix_hexblog13.exe /VERYSILENT /SUPPRESSMSGBOXES chmod 755 both the batch file and the fix, and away we go. The machine passes the test made by the guy who wrote the unofficial patch, which is as close as I think I can come to being sure that it all works. Further Update: Four hours later, done...but I've finally got SSH set up on the few machines I had left, so that's what took up most of the time.

The Atmosphere Player for Acrobat and Adobe Reader is designed to enable use of Atmosphere environments within a PDF document enabling the user the ability to experience a rich variety of interactive content, including three-dimensional objects, directional sound, streaming audio and video, SWF animations, and physical behaviors.

In Visual Studio, a solution is the basic unit of project management.

-- "Working with Microsoft Visual Studio 2005", by Craig Skibo, Marc Young and Brian Johnson

Been a lot happening here that I haven't written down...time to correct that.

First off, work is BUSY. We have ten -- no, wait, twelve -- -- people starting this month. About 8 have started already, so that leaves four. Fortunately, one of them is a new sysadmin who will be helping me out. Thank whoever for small mercies.

The sheer number of people has been part of the reason I've been so busy; another has been the Windows patches this month. Three goddamn times I've been in this month patching machines: once with the unofficial WMF fix, once with the out-of-band official WMF fix, and once with the two regular patch Tuesday patches. I am sick and tired of Windows problems.

However, I have managed to cobble together Windflower, a small-and-so-far-stupid Perl wrapper around the Microsoft Security Baseline Analyzer. So far it will run MSBA on the target computer and come up with a list of fixes it would like to see applied. It'll run over SSH, which is a blessing; I envision this as a way of automagically applying Windows patches remotely without getting a copy of SUS and IIS. It's called Windflower because it's heavily influenced/inspired by Daisy and Ivy, two programs released by Virginia Tech. (It was originally gonna be called Sunflower, but it turns out VT has already released a program with that name...I had no idea 'til now.)

Why not stick with Ivy (which worked better for me than Daisy)? Ivy's great, but it needs a GUI and its UI is irritating (keeps stealing focus, new logs overwrite old logs, etc). I've long wanted something that can work over SSH, and this looks like it should be able to. Plus, Ivy was written in Winbatch, which I don't know and don't have a compiler for. Windflower is written in Perl, which I do know pretty well.

Version 0.1, in all its completely unfinished glory, is available here. GPL'd for open-source goodness!

There's also Amanda, which has been giving me grief. First the estimates were taking hours to finish, which meant that even if backups started at 9pm they wouldn't finish 'til noon the next day. This was fixed by upgrading to 2.4.5, which uses calcsize, quicker-but-slightly-more-inaccurate estimator of the Elder Gods.

Then I ran into another problem: estimate requests, including all the exclusions for each directory, were taking up more than 32KB -- so they were split up into more than one packet by the requesting process. Unfortunately, the receiving process still ignores all but the first packet. Patches, as they say, are welcomed; in the meantime, the workaround is to make the packets smaller. The easiest way to do that is to have one big list of exclusions, rather than specifying each item in that list for each backup. The problem with that is that leads to problems where you (say) want to exclude certain stuff for everyone, plus allow people to specify their own list of exclusions: only the first list gets accepted. My own special workaround, hereby released under the GPL, is:

for i in `ypcat passwd.byuid | awk -F":" '{print $1}'` ; do cat /path/to/onebiglist >> $i/.exclude_from_backup done

Arghhh.

One thing that has helped with work is Time Management for System Administrators, by Tom Limoncelli. I just got this last week, but it's already helped a lot. The sample chapter gives a good overview of The Cycle, the system that TL advocates. The book irritates me in a couple places -- the odd buzzword, and an illustrative anecdote about a friend who was late reviewing a chapter that, frankly, makes TL sound like a bit of an ass. But these are pretty minor complaints, and I recommend getting it.

One of the things he recommends is either a PDA or a PAA (personal analog assistant, aka DayTimer(tm)). I decided to hunt around Ebay for a PDA, thinking I would pick up a used Handspring or some such; instead, I got a Sharp Zaurus SL-5500. Woohoo! Should be arriving next Wednesday.

Finally, I managed to spend a couple hours last night hacking on the NWR04B. I got the driver for the ADM5120 switch compiled; however, it hung when it came time to initialize the switch. A liberal sprinkling of printks showed that the kernel was hung in register_netdev at the call to rtnl_lock. Just for fun, I tried taking that out, and the initialization continued...though other networking drivers complained about RTNL_ASSERT failing, and the ethernet interface didn't actually work, since it couldn't mount its home directory via NFS. Still, progress of a sort.

The Sharp Zaurus has arrived, and so far I'm pretty happy with it. It's a handheld, it runs Linux, it's nice. Handwriting recognition is pretty good with this -- better than I remember a Handspring (last handheld I tried, maybe two years ago) being. The keyboard is decent, but it doesn't take long before I can feel my thumbs start to seize up, and anyway there's not always a way to do something without tapping somewhere on the screen.

The software, however...hm. It came with the stock Sharp distro and apps, so I tried using that at first. They were pretty good -- calendar worked, to-do list worked -- but things were all separate from each other. A todo item might have a due date, but it wouldn't be added to the calendar. A calendar item can't include a link to contact information. All this means a lot of re-entering stuff, which is a pain in the ass.

But hey, beauty of open source, right? So I tried Opie, the QT-based OpenZaurus distro. I had some problems installing because I missed part of the instructions (when it says rename the file initrd.bin, don't rename it initrd), but once it came up things seemed good. You still can't have a todo item show up in your calendar, but there's a decent workaround: an app called "Today" that shows you what's coming up in your calendar and your todo list. Perfect!

Except for suspend: it doesn't always come back from the dead, at which point you need to flip a tiny switch on the back of the thing and reboot. (This is a known problem, and a kernel upgrade is supposed to make it better.)

So fine, beauty of open source, right? I trie GPE, the GTK-based OpenZaurus distro, and am even more annoyed with it. Suspend seems to work better, but the handwriting recognition software is different from stock/Opie, so I have to relearn strokes. Plus, there's no separate area for capital (as opposed to lower-case) letters, which I've come to find pretty damned handy. That's enough to push me back to Opie, and try the kernel upgrade.

Dear god, this took me a long time to download. I managed to finally track it down here, but I'm putting up a copy here as a backup.