Carousel is a lie!

Woohoo! My first entry for the Sysadvent Calendar, on Development for Sysadmins, has been posted! Thanks to Jordan for tidying it up and adding integration testing, which I'd missed when writing the article. There will be another article from me coming up soon-ish on OCSNG and GLPI.

As of December 1st, Jordan and Matt were still accepting entries -- so head on over if you've got something to say.

Busy day:

• Wake up at 5am because the youngest son's teething and he's going to be up at 5.20am

• Clean up ZFS snapshots yet again; repeat "I must schedule this in cron" for the nth time

• Put in request to maintenance to look at server room humidifier; current block o' cold weather == 10% RH in there

• Find out why Mailman stopped working (probably permission problems on the logs), and how to monitor this (settle for web interface for now, since that wasn't working either; will probably need to make my peace with user accounts for Nagios on machines I'm monitoring)

• Figure out why Drupal is shitting cron.php files all over the place (still no idea)

• Fill out performance review for self

• Back up Windows 2003 server before tossing it over the fence to software installer

• Start writing article for SysAdvent at last on OCSNG/GLPI

• Struggle with cfengine tidy stanza that doesn't work; repeat "I must upgrade to cfengine 3 or puppet" for the nth time

Tonight, bed at 8.30pm. And there's no shame in that.

Irritating: chkconfig on RHEL/CentOS returns non-zero if a service isn't configured for a runlevel. IOW, you can do:

chkconfig --level 3 foo

and have 0 returned if it's on, 1 if it's not.

But not SuSE; nope, it just returns 0 whether or not it's enabled, or even if the service itself doesn't exist. Because, you know, grep doesn't get used enough.

I'm doing this because I'm trying to use cfengine 2 to manage services. This works well in CentOS, where you can add something like:

service_foo_on = (ReturnsZero("/sbin/chkconfig --level 3 foo"))

and it'll work. ("service_foo_on" is a bit of a misnomer, because I'm checking runlevels, not whether it's actually running.)

Update: Nope, I'm wrong. chkconfig --check does exactly what I want. Many thanks to yaloki on #openSUSE-server for the help.

Here we go:

• I've got a new article on lessons learned from moving up on Sysadvent
• And an older article I wrote on inventory management that I didn't mention
• Interesting post/pointer from Matt on rack-area networking, and another on predictions for 2010
• I'm firing up Cobbler at work and SQUEE! is it ever nice for ; setting up new VMs

And that's all for now.

At the risk of tempting fate, I just realized that my web server is five years old (and a bit). Happy birthday, Thornhill!

Hah! Thanks to Teridon, I can now edit Foswiki files from the command line:

rcs -l TextFileName.txt
ci -mnone -t-none -wusername -u TextFileName.txt

Sweet! Now to automate it in Emacs...

After coming back from LISA I've been wanting to try IPv6 at home; I've dabbled with it on and off for the last few years, but haven't made a serious go of it.

Originally I had a tunnel with SixXS.net, but:

1. I was having problems with uptime that, in the end, turned out to be my own silly firewall problems

2. The points system they use just seems needlessly complicated

3. Hurricane Electric was just 'way easier to set up, including getting a /64 right away

4. There are a number of complaints about SixXS.net arbitrarily (so it's claimed) closing accounts.

But enough gossip! Now you can visit http://ipv6.saintaardvarkthecarpeted.com. Soon as I get a chance I'll set it up so it doesn't require the separate hostname.

• Waiting while my wife furiously edits a blog entry, then finding she's posted it and getting to be the first to read it. And not just because she mentions me every now and then. Seriously, she rocks. Also? She got me the XKCD book for Xmas. Like I need more reasons to love her.

A nice thing about working at a university is that you get all this time off at Xmas, which is really nice; however, it's also the best possible time to do all the stuff you've been saving up. Last year my time was split between this job and my last; now, the time's all mine, baby.

Today will be my last of three days in a row where the machines have been all mine to play with^W^Wupgrade. I've been able to twiddle the firewall's NIC settings, upgrade CentOS using Cfengine, and set up a new LDAP server using Cobbler and CentOS Directory Server. I've tested our UPS' ATS, but discovered that NUT is different from APCUPSD in one important way: it doesn't easily allow you to say "shut down now, even though there's 95% battery left". I may have to leave testing of that for another day.

It hasn't all gone smoothly, but I've accomplished almost all the important things. This is a nice surprise; I'm always hesistant when I estimate how long something will take, because I feel like I have no way of knowing in advance (interruptions, unexpected obstacles...you know the drill). In this case, the time estimates for individual tasks were, in fact, 'way paranoid, but that gave me the buffer that I needed.

One example: after upgrading CentOS, two of our three servers attached to StorageTek 2500 disk arrays reported problems with the disks. Upon closer inspection, they were reporting problems with half of the LUNs that the array was presenting to them -- and they were reporting them in different ways. It had been a year or longer since I'd set them up, and my documentation was pretty damn slim, so it took me a while to figure it out. (Had to sleep on it, even.)

The servers have dual paths to the arrays. In Linux, the multipath drivers don't work so well with these, so we used the Sun drivers instead. But:

1. You have to rebuild the drivers after a kernel change.
2. This only showed up on two servers because the third server had not upgraded its kernel (or indeed, any of its packages). Why? cfservd had refused its connection because I had the MaxConnections parameter too low.
3. And of the two that did upgrade, the one machine we'd tested the Linux drivers on still had an old multipath.conf file in /etc, which even though the multipathd. service wasn't starting up was enough to get drivers loaded. This took a while to figure out because I'd completely forgotten how to tell which driver was in use.

I got it fixed in the end, and I expanded the documentation considerably. (49,000 words and counting in the wiki. Damn right I'm bragging!)

Putting off 'til next time, tempted though I am: reinstalling CentOS on the monitoring machine, which due to a mix of EPEL and Dag repos and operator error appears to be stuck in a corner, unable to upgrade without ripping out (say) Cacti. I moved the web server to a backup machine on Tuesday, and I'll be moving it back today; this is not the time to fiddle with the thing that's going to tell me I've moved everything back correctly.

(Incidentally, thanks to Matt for the rubber duck, who successfully talked me down off the roof when I was mulling this over. Man, that duck is so wise...)

Last day today. (Like, ever!) If I remember correctly I'm going to test the water leak detector...and I forget the rest; it's all in my daytimer and I'm too lazy to get up and look right now. Wish me luck.

And best of 2010 to all of you!

While trying to figure out why Nagios was suddenly unable to check up on our databases, I suddenly realized that the permissions on /dev/null were wrong: 0600 instead of 0666. What the hell? I've had this problem before, and I was in the middle of something, so I set them back and went on with my life. Then in happened again, not half an hour later. I was in the same shell, so I figured it had to have been a command I'd run that had inadvertantly done this.

Yep: don't run the MySQL client as root. Yes yes yes, it's bad anyway, I'll go to sysadmin hell, but this is an interesting bug. The environment variable MYSQL_HISTFILE is set to /dev/null for root...and when you exit the client, it sets the permissions for the history file to 0600. So, you know, don't do that then. (Still no fix committed, btw...)