The Life of a Sysadmin

It has been a busy ten days, no lie. My wife and I moved into our new place with only minor difficulties. Tuesday, though, I came down with flu and spent the next three days lying on the couch, dozing through CNN Headline News (if you're gonna rot your brain, you gotta do it right) and gobbling Tylenol by the handful. So much for my plan to spend the week assembling all the Ikea furniture in the world...

The computers made it through okay, except for the XBox (used as a MythTV frontend). First the hard drive crapped out -- all sorts of hard errors during fscking, followed by scary looking errors about how it couldn't find init. Fortunately I had a spare 80GB Seagate Barracuda, so I installed Xebian v1.1.4. Worked well, and in fact it fixed the display-offset-an-inch-to-the-left-and-up problem I'd been unable to fix before, so that was good.

And then last night, it started behaving weirdly. First, it wouldn't play a program in MythTV -- it just sat there beeping whenever it accessed the drive. I tried power cycling, but then it just sat and beeped at the Cromwell BIOS screen without going any further. I tried searching for beeping hard drives/XBoxes, but this was all I could find. This morning it's fine, so I suspect overheating -- it is a little more enclosed than it was at our new place. We'll see if it happens again.

The new place has imposed some network changes, too. Our last place was an apartment -- all one floor, so it wasn't hard to snake cables around to hook up the XBox, my wife's laptop and so on. Now, though, my computers and the cable modem are on the second floor, and the laptop/XBox are on the first. And while this house is only about four years old, it doesn't have built-in CAT5. :-(

I had brief fantasies of just poking holes in MY drywall (pride of house ownership picks weird times to pop into my head) and snaking a cable down to the TV (which is almost directly underneath me right now), but gave up. Then I ran a cable from the second floor to the first, thinking I could just run it along the edge of the walls and hide it nicely. It was worth trying, but it really didn't work and even I thought it just looked ugly. Only one way to go: wireless.

Since the NWR04B's on hold, I decided to pick up a couple Linksys WRT54GLs and run OpenWRT on them. They arrived on Thursday, and w/in 15 minutes I'd voided the warranty by installing White Russian on them. :-)

Man, OpenWRT is nice -- it's exactly what I've been hoping to achieve on the NWR. I'm still working on the configuration for these things, but the basic idea is to have one on the second floor, running as an AP that the laptop can connect to, and one on the first floor running in client mode. The XBox will be connected to the client, and Hunsacker (MythTV backend) will be connected to the AP. The laptop will connect to the internal network using OpenVPN; probably the MythTV boxen will use OpenVPN as well. The AP will be sitting inside my internal network, rather than outside, but will be firewalled by itself and my gateway to only allow OpenVPN and SSH connections in or out. It's a bit more complicated than I've set up for a home network before, but it's starting to come together in my head.

Of course, I could just run the AP as the firewall itself -- Lord knows the thing could do it just fine. But I just ordered OpenBSD 3.9 a couple weeks ago (plus sent 'em a nice donation), and it'd be a shame to waste that.

Version 0.3 of Windflower, the command-line-only, runnable-via-SSH Perl wrapper around the Microsoft Baseline Security Analyzer, is now ready for download. Improvements include:

• Verbose and talk-only modes
• Note re: MBSA initialization
• Allow specification of passive options for each patch, rather than counting on "/passive /quiet /norestart"
• Handle case where there are more than one set of items has been checked for patches (for example: both Office and Windows)
• Will tell you where patch can be downloaded, if not found on local repository
• Only reboot if patches actually applied (!)
• General cleanup of hacky Perl

It's not perfect -- it won't remove files after they're no longer needed, and the documentation needs to be better -- but it's pretty good.

Arghh. For weeks now, I've been trying to track down why a couple of XP laptops have had random print jobs drop to the floor. I finally got to the point last week where I could reliably duplicate the problem (print four emails from Outlook in quick succession; only three show up, no error on the printer), and today I spent six hours figuring out where the hell the problem was. (I didn't intend to spend that long, but the combination of vociferous complaints and sheer bull-headedness got to me.)

For no particularly good reason, the laptop in question is set to print to the local HP 4200 using IPP. When I looked at the traffic in Ethereal, I noticed that the failing job had a subtly different response to the print job submission from the printer, and at the end the TCP stream was only closed by the laptop -- the printer ACKed right away but did not FIN its end. Aha! Firmware bug!

The printer repair guy who's been working with me to try and fix this stopped by to take a look, and decided to call HP support. Their response: Don't Do That, Then. Apparently, IPP is a weird protocol to use for a LAN and I should really print to port 9100 like everyone else.

Okay, yes, this worked, and it was a stupid amount of time to spend on this problem. But it irritates me that they weren't interested in (what I think is) a firmware bug, and that I'll never probably never get to the bottom of what was going on. Although I'm pretty sure that the JetDirect card just uses an embedded ARM processor; I could just try looking at the firmware with a disassembler...:-)

In other news, something's going subtly wrong with the WRT54G; the bridging of OpenVPN's tap0 interface and the external ethernet interface has stopped working. The internal ethernet interface still works, and if you SSH in that way and run ifconfig vlan0 down ; ifconfig vlan0 up the external interface starts working again. I'm also having problems with the wireless interface. I suspect the bridging may be involved there, too, since it's bridged with the internal ethernet. However, I only have my wife's iBook to test with, so I can't be sure it's not a problem with that.

And my OpenBSD 3.9 CDs are in. Hurray! Time to finally get this firewall off my desktop machine.

I've been setting up OpenVPN on my wife's iBook, using 3.0-RC2 of Tunnelblick. It works well, but I did come across one bit of weirdness.

I'm using OpenVPN in bridging mode. The network looks like this:

iBook <-> WRT54G <-> Home Network <-> Firewall <-> Internet

When OpenVPN bridging is going on, the iBook appears to be sitting on the Home Network. During testing, I was able to ping the Firewall box and other boxes on the Home Network, but I was unable to connect to websites on the Internet, and pinging Internet hosts got me this error:

ping: sendto: No buffer space available
ping: wrote www.google.ca 64 chars, ret=-1

I've run into this problem before, but the last time it was because one end of the tunnel went down -- not the case now.

I tried looking at netstat -m but it all looked good:

98 mbufs in use:
        94 mbufs allocated to data
        3 mbufs allocated to socket names and addresses
        1 mbufs allocated to Appletalk data blocks
145/368 mbuf clusters in use
760 Kbytes allocated to network (41% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

...which pretty much matches what I've seen when I've had this message before in other situations. netstat -s was a little more interesting:

icmp:
        266 calls to icmp_error
        0 errors not generated 'cuz old message was icmp
        Output histogram:
                echo reply: 2
                destination unreachable: 266
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                echo reply: 865
                destination unreachable: 284
                routing redirect: 18
                echo: 2
        2 message responses generated
        ICMP address mask responses are disabled

"Destination unreachable"? WTF? I had a look at the interfaces, and saw three tap instances. Not only that: tap1 was the one being used by Tunnelblick, but tap0 was the default gateway. That can't be right, right? Right. Turned out there was another, older instance of OpenVPN running that should've been killed long ago. Kill that, kill Tunnelblick, restart Tunnelblick, and all is well.

I'm hoping I won't have this crop up again; it's not in my wife's nature to start looking for rogue tap interfaces screwing up the routing tables if her Internet connection goes down. :-)

One other problem I had with the WRT54G was bridging and setting the time. See, OpenVPN starts at boot on this thing, and it needs a tap interface. Since we're doing bridging, it needs to be bridged to the outside interface -- vlan1 on OpenWRT, the Linux firmware I'm using. In order to make that work, I get the firewall script to create the bridge right before it runs all the firewall rules. That happens right before OpenVPN starts, which happens right before OpenNTPD runs, which happens right before the boot scripts finish and we're open for business. Firewall, then OpenVPN, then OpenNTPD. Got it?

Setting the time is important because otherwise OpenVPN will complain that the iBook is connecting using an SSL certificate that's not valid yet, and refuses to connect. Well, no problem -- ntpd -s takes care of that, right? Wrong: every time I checked the date, the time was 1999. (Yeah, I could've tried to set the hardware clock to something more reasonable, but that's a stupid hack.)

ntpd -s was running but not setting the time. tcpdump on my NTP server showed that there were no NTP requests coming from the WRT after it booted. Yet I could kill ntpd, start it up again, and it would set the time right away. I tried ntpclient, but it behaved in exactly the same way.

In the end, I couldn't find out what was going on. I suspect it's a problem related to the bridge I set up -- ntpd binds to vlan1, then for some reason things stop working once the bridge is set up. I can't be sure without a serial port, though -- still haven't figured out logging on this thing -- so I used a slightly less awful hack: running ntpclient to set the time just before bridging starts, then running openntpd after as usual. It just dodges the problem, rather than fixing it, but it works.