The Life of a Sysadmin

The last few weeks, I've been setting up a small (5 racks) server room with the purchases that $OTHER_JOB recently made: 10 Sun X4140s, 2 — wait, 4 — X4240s, and one Thumper.

It's occupied a lot of my time, and before I lose the impulse, or fall asleep on my feet (second kid up at 4:30am for the last week or so; simultaneous discovery that at 4:30am I have a hard time getting back to sleep), I want to put down the things I learned.

• Power, power, power! Count it forward, count it backward, count it per rack, count it per PDU, count it per UPS, count it per breaker, count it in amps and watts, count it for 120V and 220V, count it idle and at full utilization. And then measure.
• Have a spreadsheet with the machine specs, host name, MAC addresses (if known), purpose, IP address(es), switch and console ports, rack assignment, RAID arrangement, stripe size, and favourite beer. Have it ready for the installer beforehand. I had a lot of this, but not all.
• Figure out how the hell they or you are gonna work in your space. That goes double if it's small. Tables, outlets, extension cords, network cables, switch ports…
• Figure out disk layout and OS in advance. God, what an embarrassing oversight.
• We paid for installation, and brother it was worth it.
• Sun's ldap_cachemgr does not like being told to connect to an IP address (via an entry in ou=Profiles) via SSL, and have the CN be a hostname instead. This took me a while to figure out.

But...my first batch of homebrew beer has been bottled, and a second brew day is coming up on Saturday. And apparently I'm not the only sysadmin who brews...though I'm not nearly ready to do all-grain just yet.

I've been hlding off mentioning this 'til all my ducks were in a row, but at last it's settled. The job I've been working at part-time for the last six months will be my full-time job starting next Wednesday. w00t!

I've been spending my time at $job_1 making sure the documentation is complete, getting a spare workstation set up and ready to go, and dumping my brain into the sysadmin who will be helping fill in 'til a new person is hired (which might take a while).

I'm really excited about this. First off, I'll get my lunch hours back; I've been walking between the two offices (mornings at one, afternoons at the other, back to the first for the last half hour), and it'll be nice to have an hour to myself again. But the new job is exciting for me: nice big servers used for scientific computation, the chance to build an infrastructure from scratch, and some big projects. The people are friendly. The boss is nice. The place has funding for the next five years or so. It's all good. About the only thing missing is a rocket pack so I can cut down on this 90-minute commute.

And on top of all that, they're open to the idea of sending me to LISA this year. Now that would be nice…have to see if it works with the family, but I'm keeping my fingers crossed.

In other news:

• Though I'm starting to be a bit frustrated with its organization, the Postfix book has proved its worth again by helping me not only get SASL set up at $job_2, but understanding what it's doing. This is in contrast to the don't-touch-or-you'll-lose-a-finger approach I'd taken to getting this working at my current job.
• I'm growing increasingly comfortable with Fedora Directory Server, thanks in no small measure to being able to play with it in a couple of virtual machines. Man, it makes a difference to have that freedom. (And in other news, water continues to be wet.)
• Got my third batch of home-made beer (Grapefruit Bitter, from my local homebrew shop's recipe) into the fermenter on Saturday. My father-in-law is a retired millwright, and he made me an immersion chiller 'cos he's just that cool. Worked a treat; not so sure about the washed yeast. Here's hoping. My parents are coming for a visit in the summer; I figure my dad and I should be able to do a batch and bottle it before they leave, which'll give him a good souvenir.
• Enjoying the hell out of Anathem, which my wife got me for Xmas.
• Closer to migrating to Chronicle. The only thing I want is to not dump all generated HTML pages into one directory; shouldn't take too long to add. Wouldn't that be a lovely thing?

I can't believe it...my youngest son, after nearly three weeks of being up four or five times each night, slept nearly all the way through without a break: he only woke up at 1am and 5:15am, which is close enough to my usual wakeup time as makes no difference. It was wonderful to have a bit of sleep.

This comes after staying up late (11pm!) on Sunday bottling the latest batch of beer, a Grapefruit Bitter recipe from the local homebrew shop. You know, it really does taste like grapefruit, and even this early I'm really looking forward to this beer.

My laptop has a broken hinge, dammit. I carry it around in my backpack without any padding, so I guess I'm lucky it's lasted this long. Fortunately the monitor still works and mostly stays upright. I've had a look at some directions on how to replace it; it looks fiddly, but spending $20 on a new set of hinges from eBay is a lot more attractive than spending $100. Of course, the other consideration is whether I can get three hours to work on it….But in the meantime, I've got it on the SkyTrain for the first time in a week; it's been hard to want to do anything but sleep lately.

Work is still busy:

• I'm trying to get tinyMCE and img_assist to work with Drupal

  1. tinyMCE is no problem, but the img_assist part wasn't working with it. Turns out you need to get dev versions of the img_assist and WYSIWIG modules, because the latest version of tinyMCE (which is required for Drupal 6) broke some parts of img_assist (which, in turn, was in the middle of a rewrite anyhow). Eventually, the admin ass't will be able to work on the website w/o having to know HTML which == major win.

• Contacting vendors to look at backup hardware. So far we're looking at the Dell ML6010 and the Sun SL500. They're both modular, which is nice; we've got (low) tens of TB now but that'll ramp up quickly. The SL500 seems to have some weird things; according to this post, it takes up to 30 minutes to boot (!) and you can't change its IP address without a visit from the service engineer (!!). Those posts are two years old, so perhaps things have changed.

• Trying to figure out what we want for backup software, too. I'm used to Bacula (which works well with the ML6010) and Amanda, but I've been working a little bit with Tivoli lately. One of the advantages of Tivoli is the ease of restoring it gives to the users…very nice. I'm reading Backup and Recovery again, trying to get a sense of what we want, and reviewing Preston's presentation at LISA06 called "Seriously, tape-only backup systems are dead". So what do we put in front of this thing? Not sure yet…

• Speaking of Tivoli, it's suddenly stopped working for us: it backed up filesystems on our Thumper just fine (though we had to point it at individual ZFS filesystems, rather than telling it to just go), then stopped; it hangs on files over a certain size (somewhere around 500kb or so) and just sits there, trying to renew the connection over and over again. I've been suspecting firewall problems, but I haven't changed anything and I can't see any logged blocked packets. Weird.

Update: turned out to be an MTU problem:

• The Thumper supports Jumbo frames
• Our switch supports Jumbo frames
• Our firewall's inside interface, a GigE from Broadcom, does not support Jumbo frames
• Our switch will silently drop jumbo frames when directed to an interface that does not support it

I had no idea there were GigE NICs that did not support Jumbo frames. Though maybe that's just the OpenBSD driver for it. Hm.

Full day:

• Prepare new network map
• Take stand-in techie around server room and explain new network setup
• Check UPS; still not crashed
• New Sun 4240 server unable to get past POST after hooking up fibre cable yesterday to SL-500 library. Try various things, no luck. Fortunately installers coming back next week to finish the job.
• Over to server room w/boss to take pictures for website
• Get programmer familiar w/the server she'll be using, how to set up services, etc. Arguably my job, but a) she'll want to learn and b) I'm off on vacation next week.
• Unless of course the UPS folks need to schedule downtime to make it work. But then I'll just use it as an excuse to show my dad and kids around the server room.
• Gotta pick out an IPA recipe to brew with my dad. Leaning toward the Cream IPA from Radical Brewing. May need to get a cooler to use as a lauter tun, since I think it's around 13 pounds of grain — more than I can comfortably do in my paint bag strainer setup.
• Still got out to walk around at lunch time, which was nice; I have a bad habit of skipping that.

My in-laws got us a family membership at Science World for Xmas last year. Yesterday I got to take my 4 year-old (how should that be hyphenated?) son for the morning. It was his third trip and my second.

We headed right for the Eureka room, which is aimed at the young 'uns, and he ran around showing me everything. "Daddy, here's a big tube where you can shoot out parachutes! And this air gun shoots balls up into the water!" We found out that you could stuff three plastic balls into the air gun at once (poom poom poom!).

Oh, and when we got home he wanted to do an experiment. He got some pennies and put them in a jar with water, to leave them for a few days and see if they would dissolve. I had a maple syrup candy in my pocket (no idea where I got it), so I threw that in too. The candy has dissolved and made the water brown, so I'm curious to see what he makes of that.

Science World is just incredible. I long to go see the grownup stuff, but even the kid stuff is enormously fun and moderately educational (though that's not my son's priority right now) (dang kids). I grew up in small towns as a kid, so trips to museums like this were rare, enormous fun. (And I never did get to go to Science North...) It's amazing to me that this stuff is right here, only a half hour away by transit. I'm still a little shocked we don't go, like, every weekend.

In other news, I've got a starter going for a batch of beer next weekend. It's a Belgian yeast, harvested from my January batch. The yeast was washed following these instructions, and the starter took off in about 18 hours. It seems to be doing quite nicely; I'll probably stick it in the fridge on Wednesday or so and cold-crash it.

The ingredients are pretty much whatever I have around the house: the last of my Gambrinus ESB, some biscuit and wheat malt, a bit of roasted barley, and the hops are Centennial, Goldings and Mt Hood. My father-in-law would call this a "ministrone" -- Italian not just for that kind of soup, but for "dog's breakfast" or "big ol' mixup". (I kind of like the idea of an Italian sounding like he's from Missouri.)

Still looking for a name; suggestions on a postcard, please. Sponsorship options are available. :-)

After that's in the bag, it's time to head back to Dan's for a shopping trip. This time, I think it'll be a 50-lb bag of plain ol' pale malt, and I'll see what difference that makes.