The Life of a Sysadmin

Last year I bought a Galileoscope for $15. It's a cheap (though well-made) telescope that was meant to celebrate the 400th anniversary of Galileo's first astronomical observations. It was $15 -- so cheap!

Jupiter has been visible all this month out our bedroom window around 4:30am, and this morning I pointed the telescope at it and saw its moons and, I think, a band across the middle. If I had a tripod to hook it up to, I would have got an even better view...but even balanced on the window, it's amazing what you can see.

Work yesterday was interesting -- which is good, because it's been a bit of a slow month. A vendor bought me coffee, and it was actually an interesting conversation. I finally got an LDAP server migrated to a VM in preparation for re-installing the host it's on; this took a while because I refused to read my own instructions for how to set up replication (sigh). And that brought up other problems, like the fact that my check for jumbo frames being enabled wasn't actually complaining about non-jumbo frames...or that the OpenSuSE machines I've got didn't get their LDAP configuration from Cfengine the way I thought.

All stuff to solve tomorrow...I mean, today. (Dang getting up at 4am...)

At work, I've been playing with a tiny cluster: 3 Sun V20z servers, each with a 2.2GHz dual-core Opteron and 2GB of memory. It's nothing special at all, but it's been a good way of getting familiar wiht Rocks.

One thing that's bitten me a few times is the documentation. The 411 service is described only in the 4.x documents, but still appears to be a going concern in the 5.x series...indeed, that's how I got LDAP and autofs working. And to test the cluster, the HPC roll documentation says to use cluster-fork...yet running cluster-fork gives me the message "cluster-fork has been replaced with 'rocks run host'", which is documented in the base roll

My in-laws got us a family membership at Science World for Xmas last year. Yesterday I got to take my 4 year-old (how should that be hyphenated?) son for the morning. It was his third trip and my second.

We headed right for the Eureka room, which is aimed at the young 'uns, and he ran around showing me everything. "Daddy, here's a big tube where you can shoot out parachutes! And this air gun shoots balls up into the water!" We found out that you could stuff three plastic balls into the air gun at once (poom poom poom!).

Oh, and when we got home he wanted to do an experiment. He got some pennies and put them in a jar with water, to leave them for a few days and see if they would dissolve. I had a maple syrup candy in my pocket (no idea where I got it), so I threw that in too. The candy has dissolved and made the water brown, so I'm curious to see what he makes of that.

Science World is just incredible. I long to go see the grownup stuff, but even the kid stuff is enormously fun and moderately educational (though that's not my son's priority right now) (dang kids). I grew up in small towns as a kid, so trips to museums like this were rare, enormous fun. (And I never did get to go to Science North...) It's amazing to me that this stuff is right here, only a half hour away by transit. I'm still a little shocked we don't go, like, every weekend.

In other news, I've got a starter going for a batch of beer next weekend. It's a Belgian yeast, harvested from my January batch. The yeast was washed following these instructions, and the starter took off in about 18 hours. It seems to be doing quite nicely; I'll probably stick it in the fridge on Wednesday or so and cold-crash it.

The ingredients are pretty much whatever I have around the house: the last of my Gambrinus ESB, some biscuit and wheat malt, a bit of roasted barley, and the hops are Centennial, Goldings and Mt Hood. My father-in-law would call this a "ministrone" -- Italian not just for that kind of soup, but for "dog's breakfast" or "big ol' mixup". (I kind of like the idea of an Italian sounding like he's from Missouri.)

Still looking for a name; suggestions on a postcard, please. Sponsorship options are available. :-)

After that's in the bag, it's time to head back to Dan's for a shopping trip. This time, I think it'll be a 50-lb bag of plain ol' pale malt, and I'll see what difference that makes.

I'm trying to get Bacula to make a separate copy of monthly full backups that can be kept off-site. To do this, I'm experimenting with its "Copy" directive. I was hoping to get a complete set of tapes ready to keep offsite before I left, but it was taking much longer than anticipated (2 days to copy 2 tapes). So I cancelled the jobs, typed unmount at bconsole, and went home thinking Bacula would just grab the right tape from the autochanger when backups came.

What I should have typed was release. release lets Bacula grab whatever tape it needs. unmount leaves Bacula unwilling to do anything on its own, and it waits for the operator (ie, me) to do something.

Result: 3 weeks of no backups. Welcome back, chump.

There are a number of things I can do to make sure this doesn't happen again. There's a thread on the Bacula-users mailing list (came up in my absence, even) detailing how to make sure something's mounted. I can use release the way Kern intended. I can set up a separate check that goes to my cel phone directly, and not through Nagios. I can run a small backup job manually on Fridays just to make sure it's going to work. And on it goes.

I knew enough not to make changes as root on Friday before going on vacation. But now I know that includes backups.

I'm back from 3 weeks' vacation. My family and I drove (!) from New Westminster, BC to Brandon, MB to visit my brother and sister-in-law and my parents. We took our time driving out and back: 6 days out, 6 days back, and, um, 9? days there.

Originally our plan was to camp most of the way there and back, but after two days we realized that the kids were a tad young for that...between mosquitoes, not going to sleep on time, and long-ass days in the car, it made a lot more sense to stay in hotels. (Kids dug it a lot...every new Super-8/HoJo/Travellodge prompted a "Wow, TWO TABLES! This is the best room EVER!" from the oldest.)

Family was fun. The prairies were hot (who knew?). Hotels with swimming pools are to be celebrated. The Rockies were, as ever, amazing. Canmore has a good brewpub and is a very, very pretty town. Aylesbury, SK had a horse to look at, which made it a good place for a picnic with the kids. Regina is made entirely of ass. Saskatoon, home of The Dymunds, is as pretty as ever.

TThe Royal Tyrrell Museum was incredible, even in a visit abbreviated by two toddlers...first time I'd ever seen real dinosaur skeletons before ("and I saw a T Rex anna stegasaurus anna triceratops anna archaeopteryx anna...). I could have easily spent a week there. If you ever get the chance, visit by any means necessary.

Despite some last-minute panics before leaving, I only received one phone call while away, and that from a salesman. I managed to forget about work entirely, and I expect to have to be guided to my desk tomorrow. And that is the capstone of a good vacation.

Yesterday was my first day back after a week off sick...and it wasn't bad, I have to say. It's summer at a university, so that helps, and it was a sunny day like we haven't had in weeks, so that was even better. I'm going off on vacation in three weeks, so I'm focussing now on getting things ready. (I'm asymptotically approaching a full realization of how to plan work over long periods of time...)

There's one big project (LDAP + website + email on four VMs) that I need to hand over to the owners. I found out/confirmed my suspicions that the people maintaining it while I'm gone will be moderately technical, not administrative, so that simplifies things a lot.

There's documentation I need to improve for the folks who'll be helping while I'm gone, including network maps. Ugh...I hate making network maps; it reminds me of web design in its fiddliness (is too a word). But I will do it. (I wonder if using a tablet would help. Anyone have any experience w/that?)

Oh, and DRI...sigh. One of the groups at work uses a protein visualization program that wants to use DRI. Without it, it reverts to a lesser form of rendering that looks like a strobe light. In the X.org troubleshooting guide, they recommend a stanza that looks like this:

Section "DRI"
    Mode 0666
EndSection

I don't like those permissions just on principle. You can put in a "Group foo" directive, which changes the group ownership as you'd expect, and for some workstations that works...but for another does not, possibly because of different driver versions. This is what comes of not having fully automated installs.

And there's son #2 waking up again. He woke up at 5 but was persuaded to go back to sleep. Bad: I was up at 5am. Good: I had time to write this.

A week ago this afternoon, I started feeling bad: achy, kinda fevery, lethargic. I chalked it up to flu. But then I got these massive headaches. At night I had intense, repetitive, feverish dreams about loading some random .el file in Emacs (no, seriously), and would wake up to more headaches, drenched in sweat.

First doctor said it was allergies or viral; second doctor thought it was bacterial, and gave me a prescription for antibiotics. (I don't usually get second opinions, and I didn't ask for antibiotics, but I've never felt so awful for so long.)

On Wednesday, I stupidly decided to go into work to take care of a few things, and scared a few people with how I looked. I slowly shuffled back to the bus and came home.

On Thursday, I was supposed to go with my wife and kids to Penticton for a weekend family gathering...that was out. I think I'd have thrown myself out the door within a few blocks. They left, and I've been having a sickie bachelor weekend. I started watching "Battlefield Earth" as soon as they left; I lay there on the couch, letting John Travolta's cackle wash over me and laughing weakly.

I watched a lot of movies. I read the Internet. I ate frozen pizza. I upgraded my wife's laptop to the latest version of Ubuntu, and fixed a Flash bug that's been irritating her. I had vague doubts about my choice of career, but I'm putting that down to lack of sleep...haven't had a good night's worth in a week now.

I've been feeling slowly better after starting the antibiotics. And once I figured out I was taking entirely too much acetominophen (I think I was getting rebound headaches), I started feeling even better. For some reason I'm still getting woken up at 1:30am by headaches, but I'm hopeful that'll go.

The family's due back this afternoon. I'll be back at work tomorrow, and I'm glad: I'm pretty bored of sitting on my ass all the time. (I'd be happier to spend more time with my family, but we've got another, longer vacation coming up in OH CRAP three weeks.)

Okay, enough. Go listen to "Volunteers" by Megafaun. (That's why I'm having career doubts. Beard + acoustic music...sigh.)

I've been upgrading my wife's laptop to Lucid Lynx today, and I've finally had a chance to track down one little bug that's been annoying her. On Youtube, flash controls don't work on videos; she can't close ads, the volume/mute controls don't work, and so on. This is a 64-bit machine, but 32-bit flash (natch).

The solution, as detailed here (post 42!), is to edit /usr/lib/nspluginwrapper/i386/linux/npviewer and insert this before the last line:

export GDK_NATIVE_WINDOWS=1

Restart the browser, and now it all works!

Wow:

Since I have started to use Org-mode, I though it was missing
something to have appointment locations on a map. Of course, it's
easy to get a LOCATION property from an entry, and then browse-url
on Google Maps.

But it is too easy for me, so once again I said: challenge accepted! I
will bring Google Maps into Emacs!

Simply amazing.

Quote:

You come into medicine and science at a time of radical transition. You have met the older doctors and scientists who tell the pollsters that they wouldn't choose their profession if they were given the choice all over again. But you are the generation that was wise enough to ignore them: for what you are hearing is the pain of people experiencing an utter transformation of their world. Doctors and scientists are now being asked to accept a new understanding of what great medicine requires. It is not just the focus of an individual artisan-specialist, however skilled and caring. And it is not just the discovery of a new drug or operation, however effective it may seem in an isolated trial. Great medicine requires the innovation of entire packages care -- with medicines and technologies and clinicians designed to fit together seamlessly, monitored carefully, adjusted perpetually, and shown to produce ever better service and results for people at the lowest possible cost for society.

Very, very interesting reading, particularly as it seems to recap arguments about the role of system administration.