The Life of a Sysadmin

L'il Barley and the Wheat Germ.

That is all.

I swear, Sympa has the worst fucking documentation of any goddamned software I've ever used. I'd rather view /dev/random with Internet Explorer then try to figure out what the hell this software does.

Example: List creation. Since Sympa is mailing list software, this would seem to be pretty basic. It is if you're using the web management feature, which they document. But if you're not? Well, now, why would you do such a thing? Are you a Communist or something?

Here's what I know: there's a Perl script here that creates the Sympa config files for various lists as needed. Then these lists magically show up. This appears to be related to the task manager portion of Sympa, which seems to create lists for the new config files. Or maybe Sympa checks the config files when incoming email comes in, and that's how it works. Either way, it sure as hell isn't documented anywhere; not for the older version we have, nor for the newer version (which seems to be a nothing more than a wiki dump, with all its attendant lack of organization).

God, this is irritating.

Now that Clara's heading back to work, my schedule has changed a bit: I'm staying at home on Wednesdays to take care of Arlo, and then working from home on Saturdays to make up the time. I'm grateful to my boss for letting me do this, and I'm hopeful it will work out.

My first Wednesday (July 4th) went pretty darned well, really. Arlo ate, he played, he got vaccinated (Chicken pox; I had no idea they vaccinated for it), he napped and then he played some more. I didn't drop him, he didn't freak out and it was a great deal of fun.

As it happens I got to take care of him on Friday, too; my mother-in-law, who's going to be taking care of him two days a week, had a sudden trip to the emergency room. She's okay, but wasn't able to take care of him that day. (She was mad about it, too...) I called into work and let them know I wouldn't be in, then went in anyway just to make sure a few things were okay. I've got some karma built up and a fistful of sick days I rarely take, so all was well.

And then yesterday I worked from home. And man o man, did I get stuff done. Not quite as much as I wanted; I was hoping to use flar to duplicate a Solaris machine so I could test it, and ran into a bug that took a while to figure out. (If the patch I applied fixes the problem, I'll write it up here since there was only one other reference I could find.) But it was lovely to work for, like, four hours in a row on something and not be interrupted. Plus, there's the skipping of the 90-minute commute to enjoy.

My fondness for trivial patches continues. You may envy me.

I can't believe it's taken me three years to find this entry from Aaron Margosis' blog about how to run Windows or Internet Explorer as admin using runas. I've always cheated by running different .cpl files until I found something that'd launch IE, then forget what it was and curse myself for not writing it down. Anyhow, this allowed me to open up C:\Program Files\Mozilla Firefox for modification by selected users, which in turn allows FF's auto-update feature to work without me having to wake up.

Also: Can't believe I never knew about mutt's T command. Nice.

Which is very timely, as I'm trying to track down why nscd door access is taking so long: http://au.sun.com/news/onsun/2002-11/tech_tips.html

This DNS regression suite looks tres cool. I've just upgraded BIND at work to the latest version, so maybe this is the next thing to try.

I can get really, really focussed sometimes. Every now and then that happens with Nagios.

Yesterday I had some time to kill before I went home, so I looked over my tickets in RT. (I work in a small shop, so a lot of the time the tickets in RT are a way of adding things to my to-do list.) There was one that said to watch for changes in our web site's main page; I'd added that one after MySQL'd had problems one time — ran out of connections, I think — and Mambo had displayed a nice "Whoops! Can someone please tell the sysadmin?" page (a nice change from the usual cryptic error when there's no database connection). Someone did, but it would've been nice to be paged about it.

At home I use WebSec to keep track of some pages that don't change very often (worse luck…), and I thought of using that. It sends you the new web page with the different bits highlighted, which is a nice touch. But I wanted something tied in with Nagios, rather than another separate and special system.

So I started looking at the Nagios plugins I had, and I was surprised to find that check_http has a raft of different options, including the ability to check for regexes in the content. Sweet! I added a couple strings that'll almost certainly be there until The Next Big Redesign(tm), and done.

I started looking at the other plugins, and noticed check_hpjd. A few minutes later I was checking our printers for errors...just in time to notice a weird error that someone had emailed me about 30 seconds before. Nice!

This morning (I work from home on Saturdays in return for getting Wednesdays off to take care of Arlo) I was checking Cacti (which rocks even if they do call it a solution). /home/visitors with no free space? Wha'? Someone had run a job that'd managed to fill the whole damned partition.

Well, there's check_disk, but that's only for mounted disks — and I don't want the monitoring machine freezing if there's a problem with NFS. SNMP should do this, right? Right — the net-snmp project has the ability to throw errors if there's less than a certain amount of free space on a disk. For some reason I'd never set that up before, nor got Nagios to monitor for it. A few minutes later and check_snmp was looking for non-empty error messages:

$USER1$/check_snmp -H $HOSTADDRESS$ -o UCD-SNMP-MIB::dskErrorMsg.$ARG1$ -s ""

I looked ahead in snmpd.conf and noticed the process section. Well, hell! It's all very good to check that the web server is running, but what if there are too many Apache processes? Or too few of MySQL? Or no Postfix? Can't believe I never set this up before…

I've finally come up for breath. This wasn't what I planned on doing this morning, but I love it when a plan will come together next time.

Not the clearest title...what I mean is, I was writing a Bash script like this:

SOMEONE=foo
...
mv $SOMETHING ~$SOMEONE

only it kept failing with "~foo: No such file or directory". I had a look at the manual, but it wasn't terribly clear. In a nutshell, though, Bash wasn't doing the tilde expansion no matter what combination of braces and quotes tried.

Eventually, I came across a pirated copy of the ever-excellent Unix Power Tools, 3rd Ed. (mine's at work, or I'd've checked it a lot sooner), and it had a solution:

FINAL_RESTING_PLACE=$(/bin/bash -c "echo ~${SOMEONE}")

I'm sure there must be a better way of doing this, though...Woot, any suggestions?

I dreamt last night that I met Sammy Davis Junior and talked to him to release all of his songs under a Creative Commons license 20 years after his death.

"Hugh, I'm convinced," he said. "And I want to write a cheque for ten thousand dollars, too. Tell me which organization to give it too."

(I think I also piqued RMS's interest in the issue, too, but I can't remember that part.)

In other news, my old employers have been bought by another BC ISP. I can't say I'm surprised, but it's a shame the new company uses IIS for their webserver...yeah, I'm a Unix geek, all right.

And many thanks to tobutaz for providing a much better answer to the question "How do I get the home directory of a user whose name is in a variable?":

USER=jdoe
eval "USERHOME=~$USER"

Every now and then I'm reminded of eval and then forget it when it's useful. Thanks again, tobutaz!

Woot! The second edition of "The Practice of System and Network Administration" has finally started shipping! Just ordered my copy, along with Beautiful Code and Perl Best Practices. I love books, oh yes I do.