The Life of a Sysadmin

Checked my email this morning and saw that backups of my wife's computer had timed out. Weird, I thought, but didn't look into it further. Then my wife comes out and says, "Hey, my computer's having a stroke.". Uh-oh.

So I have a look and it's constantly, randomly, power cycling. It will get to the Ubuntu splash page then shut off, then get halfway through the BIOS check and shut off, then get halfway through boot and shut off, then stay off for two minutes, then turn on again. WTF?

First thought is cooling, of course. But the power supply feels cool to the touch, and when I get to the BIOS temperature page it says the CPU is at 51C -- eminently reasonable. (Then it shut itself off.) Okay, flaky RAM? Wonky graphics card? Dying, though not from lack of cooling, power supply?

Then it makes it all the way to Ubuntu's login page. I switch to a console and start looking at logs. This thing has been rebooting all night -- as in log messages about how shutdown has been invoked. And then I check /var/log/acpid and I see lots and lots and lots of entries saying that event POWERBTN (or some such) had been receieved, so Ubuntu was executing /etc/acpi/powerbtn.sh and shutting down nicely. And then I saw a broadcast message from root saying that the system was going down for reboot NOW!

Tempted to just try booting w/o ACPI, but I think that would just mask the issue. Back to Google...

My wife's computer is working fine now that I've put it in my room. :-) Double plus good, since she wants to move to the iBook and I get a nice AMD 2600. Woohoo!

As for what was going on...dunno. Random shorting, maybe? It's awfully dusty in there, so maybe that would affect things too. 'Tany rate, weird.

Now to figure out how to add all her Evolution mailboxes to Apple Mail.

I have found the perfect calendar program and the perfect front end. Just dig this screenshot:

This is exactly what I want.

First release of oz2remind has been unleashed upon an unsuspecting world. Using Perl's XML::Simple, it converts the datebook.xml file from OpenZaurus/Opie's calendar app to Remind format.

On Friday morning a transformer blew up at my work's building and we lost maybe a third of the outlets. It wasn't as bad as it could've been, since we kept power to the servers.

In the middle of dealing with that, I was told that there were seven new machines that had to be on the network, like, now. A thought that B knew that when A talked about test machines, A meant machines that had working home directories. B didn't know that. A also thought B knew that the test machines were needed now. B didn't know that.

Friday afternoon we were told that power would be down Saturday from 8am to 1pm while they fixed it. I ran around letting people know and asking if anything needed to be turned on in a special order. (Some of our equipment is a little mysterious for the likes of me.)

Saturday morning I was in at 7am shutting down the network and unplugging everything I could. Saturday afternoon at 1pm I was back to turn things on, only the work wasn't done and no one could tell me when it would be back up. Did some small maintenance jobs but couldn't really tackle anything big w/o power to at least boot up the machines. Called people to let them know what was going on.

This morning I was in at 6.30 am and hallelujah, the power was back on. I did some other maintenance, then started bringing things back in earnest. It all went pretty well except for a fileserver with a bum disk. Arghh. Currently letting that freeze in hopes it'll start working again so I can then rsync all the data off to the big-ass fileserver, while waiting for cold meds to kick in.

After that, it's off to dinner for my father-in-law's birthday...I hope. It's 1.30pm and I'm still not sure I'm going to make it.

More fallout today from Saturday's power outage: two workstations that failed to boot up (BIOS checksum error for one of 'em, which is a new one for me), some NIS-related services that didn't get started properly (not sure what's going on there), and so on. Plus the return of the where-are-those-seven-machines? that didn't get done on Friday because of all of this.

But I did learn some stuff about Cfengine. For example, if you have something like:

my_url = ( http://www.example.com/foo/bar )

then you'd better precede it with:

split = ( "+" )

or some other character that isn't used. The colon is treated as a list separator by default, which means that later on, when you try and do something like:

shell::
    linux.need_some_file:
        "/bin/wget $(my_url)/baz"

what it'll actually do is this:

/bin/wget http/baz
/bin/wget //www.example.com/foo/bar/baz

'cos it's iterating over the two lists, see?

And SuSE's dhcp client, by default (I think), will change /etc/yp.conf without telling you, and then on exit put back the old version (saved conveniently at /etc/yp.conf.sv. It took me a long time to figure out that this was happening, and it pissed me off mightily. /etc/resolv.conf is filled with comments when the dhcp client modifies it -- hell, they even throw in the PID. So why not do that with yp.conf? At least you can turn it off by changing DHCLIENT_MODIFY_NIS_CONF in /etc/sysconfig/networking/dhcp.

I ran into The Neptune Project a while back; they're running sensors along the ocean floor off the west coast of North America, and then hooking 'em up via 100Mbit ethernet back to the lab. This lets you do continuous observation, rather than sending down a probe for an hour or two. Very neat stuff; they just got their first pictures from the bottom of Saanich Inlet.

Welp, now they're looking for a sysadmin. If the timing was a little better I'd throw in a resume...but having just bought a house w/my pregnant wife, the timing's a little off. :-)

1. Taking days off after lots of days working is fun. I am sitting at home, drinking good beer, and I don't have to go in tomorrow. (At least, not at this point in time.) Very, very cool.
2. Learning stuff is cool. I am booted into a Belanix live CD right now on my wife's former computer (she went for the iBook after random hardware problems). I don't know what's going on. I hardly know where to find things. Startup scripts on this thing are beyond my ken. ps and ifconfig take weird options. And I'm having the time of my life.

That is all.

As I haven't written about the NWR04B in a while, I thought I'd mention that it's because I haven't done anything with it in a while. Part of it has just been buying a house, getting ready to move, pregnant wife, and so on. But I've also just put it aside for a while, as I wasn't making much progress on either writing to flash or getting all the ethernet ports working. I may take it up again later this year, but I suspect that the new kid and my wandering mind means it'll be a while, if ever, before I return to it.

In other news: Just got a new Overland LoaderXpress at work yesterday, and it's...interesting. Very simple machine once you take the cover off: a plastic tape magazine in the middle, a robot arm along the left, a double-height Ultrium 960 drive from HP at the back on the right, a power supply in the middle on the right, and the control board along the right-hand edge. That's pretty much it. (I may take pictures, 'cos I'm just that big a geek.)

There were a couple little blemishes: the cover had half-fallen off the tape drive and was lying at an angle; I had to push it back on. And the two screws that were holding the tape drive in place were loose and had to be screwed back on. I realize this is a budget jukebox, but it's still $8000 list. Oh, and their sales guy doesn't return calls. Weird.

Once I got the cover back on and put it in the server room, it wasn't too hard to get it hooked up. I made the mistake of not attaching a terminator before hooking up the SCSI cable -- don't do that! And I had to recompile the kernel (the backup box runs FreeBSD) to add the ch device. But once I got that figured out, getting Amanda to see it was as simple as telling it the changer device (/dev/ch0) and the changer script (chg-chio). Perfect!

One slight hiccup: Ulrium 3 drives will read Ultrium 1 tapes (of which we have, oh, 50), but will only write Ultrium 2 and 3 tapes. I didn't find this out 'til after I placed the order...my bad. This'll change my backup plans a bit, but it shouldn't be a big problem.

Stupid Sendmail bug. Our firewall was running FreeBSD 4.8 with lots of patches, but this latest patch from FreeBSD didn't apply cleanly -- too many changes to Sendmail in the meantime, and I was just completely uncomfortable trying to patch it myself.

Fortunately, we had a spare rackmount box (shhh!) that I was able to do the upgrade on. I installed 4.8, rsynced it with the firewall box (/etc, /var and /usr, rebuilt 4.8 world, then checked out a copy of RELENG_4, rebuilt that version of world, and made sure everything still worked.

I set up the MAC addresses of the interfaces to be the same as the existing box, double-checked everything, then held my breath and swapped cables. Success! A total of 37 seconds of downtime.

Of course, there were a few things that went wrong, but the most serious was our IPv6 connectivity -- I borked the firewall moving it over, and had to dig through /etc/rc.firewall6 to figure out what I'd missed. Man, I hate the FreeBSD firewall scripts...they're a good starting place, but I always end up replacing them with a single-purpose, easy to maintain shell script -- none of this digging-through-the-script-trying-to-find-the-right-variables-to-edit nonsense.

Overall, I'm pretty happy...it went about as well as I could've hoped, especially for having done it in, like, 24 hours. <Peggy Hill>Ho-yeah!</Peggy Hill>