The Life of a Sysadmin

I can't believe it...my youngest son, after nearly three weeks of being up four or five times each night, slept nearly all the way through without a break: he only woke up at 1am and 5:15am, which is close enough to my usual wakeup time as makes no difference. It was wonderful to have a bit of sleep.

This comes after staying up late (11pm!) on Sunday bottling the latest batch of beer, a Grapefruit Bitter recipe from the local homebrew shop. You know, it really does taste like grapefruit, and even this early I'm really looking forward to this beer.

My laptop has a broken hinge, dammit. I carry it around in my backpack without any padding, so I guess I'm lucky it's lasted this long. Fortunately the monitor still works and mostly stays upright. I've had a look at some directions on how to replace it; it looks fiddly, but spending $20 on a new set of hinges from eBay is a lot more attractive than spending $100. Of course, the other consideration is whether I can get three hours to work on it….But in the meantime, I've got it on the SkyTrain for the first time in a week; it's been hard to want to do anything but sleep lately.

Work is still busy:

• I'm trying to get tinyMCE and img_assist to work with Drupal

  1. tinyMCE is no problem, but the img_assist part wasn't working with it. Turns out you need to get dev versions of the img_assist and WYSIWIG modules, because the latest version of tinyMCE (which is required for Drupal 6) broke some parts of img_assist (which, in turn, was in the middle of a rewrite anyhow). Eventually, the admin ass't will be able to work on the website w/o having to know HTML which == major win.

• Contacting vendors to look at backup hardware. So far we're looking at the Dell ML6010 and the Sun SL500. They're both modular, which is nice; we've got (low) tens of TB now but that'll ramp up quickly. The SL500 seems to have some weird things; according to this post, it takes up to 30 minutes to boot (!) and you can't change its IP address without a visit from the service engineer (!!). Those posts are two years old, so perhaps things have changed.

• Trying to figure out what we want for backup software, too. I'm used to Bacula (which works well with the ML6010) and Amanda, but I've been working a little bit with Tivoli lately. One of the advantages of Tivoli is the ease of restoring it gives to the users…very nice. I'm reading Backup and Recovery again, trying to get a sense of what we want, and reviewing Preston's presentation at LISA06 called "Seriously, tape-only backup systems are dead". So what do we put in front of this thing? Not sure yet…

• Speaking of Tivoli, it's suddenly stopped working for us: it backed up filesystems on our Thumper just fine (though we had to point it at individual ZFS filesystems, rather than telling it to just go), then stopped; it hangs on files over a certain size (somewhere around 500kb or so) and just sits there, trying to renew the connection over and over again. I've been suspecting firewall problems, but I haven't changed anything and I can't see any logged blocked packets. Weird.

Update: turned out to be an MTU problem:

• The Thumper supports Jumbo frames
• Our switch supports Jumbo frames
• Our firewall's inside interface, a GigE from Broadcom, does not support Jumbo frames
• Our switch will silently drop jumbo frames when directed to an interface that does not support it

I had no idea there were GigE NICs that did not support Jumbo frames. Though maybe that's just the OpenBSD driver for it. Hm.

Attended a talk today about the upgrade of UBC's network from supporting VLANs to supporting VRFs. Complicated but neat. I'm hoping that the presentation slides and video will make it to the IT Services website; I'll post a link if it does.

Also interesting is this external review of the IT department at UBC. It touches on some things I've been peripherally involved or interested in (funding models, culture and management); I've only skimmed it so far, but it's fascinating to read something so straightforward.

"Phycicists are fun to be around. I was watching TV with one, and a commercial came on for OxyClean. The announcer's voice comes in, strong and deep, and says, What's the most powerful force in the universe? The guy I'm with starts pumping his fist and chanting, Strong nuclear force! Strong nuclear force! The announcer comes back and says, That's right, oxygen! Poor bastard looked like someone just shat in his ear."

(Conversation with a friend just now.)

Two things that didn't work:

• Trying to keep my LDAP nice and clean by having an alias for a posixAccount in another ou.

Explanation: there's ou=Smith and ou=Jones, both of which are under ou=People,dc=example,dc=org. Smith wants to offer Jones the use of a few of his machines, which means setting up accounts for Jones and a few of his folks (cn=Alice, cn=Bob, and cn=Charlie). Obviously, these should be in ou=Jones, right? But if Smith's machines, through the wonders of pam_ldap, are set to check ou=Smith, how do Jones' folks log in?

(Digression: actually, Smith's machines right now check under ou=People — not ou=Smith,ou=People. Smith is the first one to use LDAP, so I stuck with that. I was going to change that at some point anyway, and I thought this might be a good chance to do just that.)

I thought I could try adding an alias, under ou=Smith, that'd point to cn=Alice,ou=Jones. And if I told LDAP that it was a posixAccount as well, then I could look at the account details with id and getent. But the logs showed that it just didn't work:

pam_ldap: error trying to bind as user "uid=Alice,ou=Jones,ou=People,dc=example,dc=org" (Inappropriate authentication)

Couldn't track down the error quickly, so went to plan B: stick with the current setup (machines checking ou=People) and put 'em under ou=Jones. I can always add host restrictions later on.

• Rebooting an IBM eseries 336m server remotely.

Explanation: Smith had a bunch of these machines at another location before getting server room space at UBC (and new servers). My access to them previously was via SSH only — there was no console access at all (sigh). Now they're at UBC, and one of 'em's gonna be my monitoring machine/second LDAP server ("The new server room: now with redundancy!") But while it was simple to turn on console redirection and choose PXE booting from the comfort of my office, I ended up borking the kickstart process and having to come back here anyway to set up the install. There's the BMC, which apparently I can access via the serial console if I so choose, but I'm still trying to figure out what that'll get me — ie, I can't find a manual in 11 seconds, so I'm putting that off for now.

Oh, and my new (work) laptop is in. Yay! It's a Dell D630, and aside from it's obscene footprint compared to my (ailing) C400, it's great. Ubuntu (Hardy for compatibility with the desktops here) is on so far, and CentOS (server work) and OpenBSD (instant firewall) aren't far behind.

Good god, it's the downhill run to 40. Weird.

(Oh, and you know who else's birthday it is? Alex Jones, who's two years younger than me. That's also weird.)

Just upgraded my laptop to Debian Lenny with only minor hiccups (my own fault). Not only have I got the latest version of Iceweasel/Firefox without any GTK version nonsense, but I've got wicd working, including my Broadcom wireless and WPA2! (I never could figure out the settings to get encryption working with the various /etc/network/ files...) I'm happy…

Nicks' post on customizing your home was interesting. Over the last year or so, I've been slowly improving the way I do this. My results have been mixed, probably because of the way I use Mercurial.

So I've got a repo to keep my dotfiles. There's a truly awful script that will symlink the real files to the repo, and doesn't clobber the originals more than one time out of three. I clone to work, or to a laptop, and start customizing. Overall, I feel like this should work…but it's decidely awkward.

Let's take the case of bash init files. I've got mine divided into .bashrc and .bashrc_local. The latter, as you'd expect, is machine/situation-specific — ssh aliases, commands for work, etc. .bashrc sets various aliases and functions that are unlikely to change. Just before exporting all the environment variables, .bashrc_local is sourced, which gives me a chance to override anything.

.bashrc should be in the repo — no question about that. But .bashrc_local should be there too, since I may clone my repo at work (say) to another filesystem. Since Mercurial is distributed, there's no problem with this — except when it comes to merging things back home. Since I think about home as The One True Repo, I want to keep everything there. But usually I've run hg push ssh://home, which promptly clobbers .bashrc_local there (at least when I do an hg update. Or if I merge from home, I end up creating new heads in my repo, and a multi-headed repo can't be pushed. (I'm fuzzy on the details; usually when this happens I bang away at it randomly until merges happen, and swear until I'm blind.)

As outlined here, the difficulty is probably in the way I use Mercurial and the way I've become used to SVN's (and CVS's) idea of branches that look like directories (and are thus very, very visible and easy for me to think about). xyld says, "I'm fed up with having to do hg merge and not actually merge anything, but just to satisfy the Mercurial internals." That's pretty much how I'm starting to feel. There's the option of doing pull, rather than push, to cherrypick the changes I want, but it's still a bit awkward for me to think about.

I understand SVN; it fits well with my brain, which is not a developer's. I understand hg, and I like the idea of distributed repos for certain things. But xyld's comments about switching to git resonate with me, and I may start trying that out.

Last week was reading week here at UBC. Monday I was off sick. Tuesday we got an email from the folks at the building where we've got guest access to one of their server rooms: the cooling was being shut down from 7am on Wednesday to 3pm on Thursday, so we'd have to turn off our servers. We're guests, so it's not like we've got a lot of say in the matter.

Natch, Thursday 3pm came and went. We got an email at 3:45pm from a manager there, saying that unexpected problems had arisen; they were hoping to have things back up by the weekend. That night I pointed our website at a backup server; it was not serving my boss' big web app, as there was no way to make that tiny little box serve a nearly 1TB database.

Friday I obsessed over the ambient temperature on our firewall (which I'd left turned on); it hovered around 35C. Around 10am we were told that they were hoping to have it on later that day, but that another shutdown might need to be scheduled for the next week (this week). At noon we were told that things were looking hopeful, but they couldn't guarantee cooling over the weekend.

At 2pm I found a local A/C rental agency who told us they'd be out to look at the room on Monday. 4pm I emailed my contact at the other department, plus his manager, to ask for updates and whether any further shutdowns could be scheduled after we'd arranged for cooling.

Over the weekend I obsessed over the temperature some more; it had dropped to 21C and stayed there, but without feedback from the facilities people I was reluctant to trust it.

Monday (yesterday; wow, time flies) we were told that the cooling system should perform well; however, a part still needed to be replaced. It was on order and would be coming in late this week or early next, and would require a four-hour outage.

This morning the cooling guy visited (he was at a funeral yesterday, so fair enough) and said that, yep, we could get a nice portable unit in for around $400 for a week.

I'm not writing this down because I'm proud of how I handled this. I'm writing this down so that someone else can maybe learn the things I should've known:

• If the cooling is going to be down, arrange for backup. This can be cheap if it's a small room, and it's a hell of a lot nicer than being at other people's mercy.
• Outage times are estimates, and you should treat them as such.
• 4pm on a Friday afternoon is not the time to bring up questions that should have been raised on Tuesday.

I have a habit of thinking "There's not much that can be done about that." Actually, it goes even further than that; it doesn't occur to me sometimes to think about what can be done. I'm not sure if this is lack of confidence, or trying too hard to get along, or just sheer laziness, but I'm trying hard to stop doing that.

As mentioned on Undeadly.org and openbsd-misc, OpenBSD is asking for donations for BGP routers and a new CVS server. I've donated, since I wouldn't be able to do half my job without them; if you feel the same and can spare some money, I urge you to do the same.

After this entry about the difference between push and pull for Mercurial, and how that doesn't fit with the way I instinctively want to use a repository, it's interesting to read Ted Tso responding to a similar complaint from a git user. Tso explains the discrepancy well:

Part of the problem here is that for most git workflows, most people
don't actually use "git push". ....in most large projects, the number
of people [who] need to use the "scm push" command is a very small
percentage of the developer population, just as very few developers
have commit privileges...

Ah, but in a distributed SCM world, things are more
democratic....While this is true, the number of people who need to be
able to publish their own branch is small....

There is one exception to this, of course, and this is a developer
who wants to get started using git for a new project which he or she
is starting and is the author/maintainer, or someone who is
interested in converting their project to git.

The whole entry, plus the comments, are worth reading.