The Life of a Sysadmin

Yesterday was my first day back after a week off sick...and it wasn't bad, I have to say. It's summer at a university, so that helps, and it was a sunny day like we haven't had in weeks, so that was even better. I'm going off on vacation in three weeks, so I'm focussing now on getting things ready. (I'm asymptotically approaching a full realization of how to plan work over long periods of time...)

There's one big project (LDAP + website + email on four VMs) that I need to hand over to the owners. I found out/confirmed my suspicions that the people maintaining it while I'm gone will be moderately technical, not administrative, so that simplifies things a lot.

There's documentation I need to improve for the folks who'll be helping while I'm gone, including network maps. Ugh...I hate making network maps; it reminds me of web design in its fiddliness (is too a word). But I will do it. (I wonder if using a tablet would help. Anyone have any experience w/that?)

Oh, and DRI...sigh. One of the groups at work uses a protein visualization program that wants to use DRI. Without it, it reverts to a lesser form of rendering that looks like a strobe light. In the X.org troubleshooting guide, they recommend a stanza that looks like this:

Section "DRI"
    Mode 0666
EndSection

I don't like those permissions just on principle. You can put in a "Group foo" directive, which changes the group ownership as you'd expect, and for some workstations that works...but for another does not, possibly because of different driver versions. This is what comes of not having fully automated installs.

And there's son #2 waking up again. He woke up at 5 but was persuaded to go back to sleep. Bad: I was up at 5am. Good: I had time to write this.

A week ago this afternoon, I started feeling bad: achy, kinda fevery, lethargic. I chalked it up to flu. But then I got these massive headaches. At night I had intense, repetitive, feverish dreams about loading some random .el file in Emacs (no, seriously), and would wake up to more headaches, drenched in sweat.

First doctor said it was allergies or viral; second doctor thought it was bacterial, and gave me a prescription for antibiotics. (I don't usually get second opinions, and I didn't ask for antibiotics, but I've never felt so awful for so long.)

On Wednesday, I stupidly decided to go into work to take care of a few things, and scared a few people with how I looked. I slowly shuffled back to the bus and came home.

On Thursday, I was supposed to go with my wife and kids to Penticton for a weekend family gathering...that was out. I think I'd have thrown myself out the door within a few blocks. They left, and I've been having a sickie bachelor weekend. I started watching "Battlefield Earth" as soon as they left; I lay there on the couch, letting John Travolta's cackle wash over me and laughing weakly.

I watched a lot of movies. I read the Internet. I ate frozen pizza. I upgraded my wife's laptop to the latest version of Ubuntu, and fixed a Flash bug that's been irritating her. I had vague doubts about my choice of career, but I'm putting that down to lack of sleep...haven't had a good night's worth in a week now.

I've been feeling slowly better after starting the antibiotics. And once I figured out I was taking entirely too much acetominophen (I think I was getting rebound headaches), I started feeling even better. For some reason I'm still getting woken up at 1:30am by headaches, but I'm hopeful that'll go.

The family's due back this afternoon. I'll be back at work tomorrow, and I'm glad: I'm pretty bored of sitting on my ass all the time. (I'd be happier to spend more time with my family, but we've got another, longer vacation coming up in OH CRAP three weeks.)

Okay, enough. Go listen to "Volunteers" by Megafaun. (That's why I'm having career doubts. Beard + acoustic music...sigh.)

I've been upgrading my wife's laptop to Lucid Lynx today, and I've finally had a chance to track down one little bug that's been annoying her. On Youtube, flash controls don't work on videos; she can't close ads, the volume/mute controls don't work, and so on. This is a 64-bit machine, but 32-bit flash (natch).

The solution, as detailed here (post 42!), is to edit /usr/lib/nspluginwrapper/i386/linux/npviewer and insert this before the last line:

export GDK_NATIVE_WINDOWS=1

Restart the browser, and now it all works!

Wow:

Since I have started to use Org-mode, I though it was missing
something to have appointment locations on a map. Of course, it's
easy to get a LOCATION property from an entry, and then browse-url
on Google Maps.

But it is too easy for me, so once again I said: challenge accepted! I
will bring Google Maps into Emacs!

Simply amazing.

Quote:

You come into medicine and science at a time of radical transition. You have met the older doctors and scientists who tell the pollsters that they wouldn't choose their profession if they were given the choice all over again. But you are the generation that was wise enough to ignore them: for what you are hearing is the pain of people experiencing an utter transformation of their world. Doctors and scientists are now being asked to accept a new understanding of what great medicine requires. It is not just the focus of an individual artisan-specialist, however skilled and caring. And it is not just the discovery of a new drug or operation, however effective it may seem in an isolated trial. Great medicine requires the innovation of entire packages care -- with medicines and technologies and clinicians designed to fit together seamlessly, monitored carefully, adjusted perpetually, and shown to produce ever better service and results for people at the lowest possible cost for society.

Very, very interesting reading, particularly as it seems to recap arguments about the role of system administration.

There's a Linux/FreeBSD sysadmin position open at UBC with the Department of Earth and Ocean Sciences. Job listings are here; look for job #7867. (Silly web page doesn't allow direct links...) If you're in Vancouver, it looks pretty sweet.

At $WORK I've been setting up a server for a new project. It's going to be the foundation for a bunch of work that they're doing: bag o' passwords, code repository, website + wiki, email. I've been trying to set it up in such a way that a) it'll be a good foundation and b) it'll be easy for them to manage -- at least when it comes to adding/deleting/modifying users, email addresses and so on -- rather than having to come to me every time something needs changing.

I set up CentOS Directory server and populated it with the root and a few entries. Thus: cn=Me,ou=People,dc=example,dc=org. I've got authentication working, restricted to certain groups where necessary, and now it's time for me to think about how the non-propellerheads are going to manage accounts. That means a web front end, and I've been evaluating GOsa for the last week or so.

The good:

• It's nice-looking. That's not damning-by-faint-praise; whatever I give to people is going to need to look professional.

• It's got plugins for lots of different things: Samba settings, email settings (including easy vacation settings...nice), ACLs, FAI, Nagios (which, near as I can tell, is generated from FAI info), FTP quotas, you name it.

• It supports hooks in its config file: postcreate, postremove, postmodify and check. These allow you to (say) create a home directory when a user is created or other fun things.

• It includes an easy way to manage the LDAP server: export/import snapshots in CSV and LDIF, and snapshots of the tree that can be restored later. (Haven't tried that out yet.)

• Addressbook page included.

The bad:

• The documentation can best be described as sparse.

  • Some plugins depend on others, but this isn't shown anywhere. When I tried to remove the RPMs for some that I wouldn't need, I found that others I did need failed. If it should all be installed, I couldn't find that documented anywhere.

  • Allowing people to log in to GOsa and change certain details is important for me, but is documented poorly in the ACLs page.

  • But also, ACLs can be applied only to items GOsa recognizes as Departments (ie, objectClass=gosaDepartment and/or possibly objectclass=gosaAdministrativeUnit). And you can't tell GOsa that ou=People is one of these; it complains and says that's a reserved keyword.

  • The gosa.conf config file refers you to the manpage, but that doesn't appear to be included in the RPMs I downloaded. The wiki page on the file refers you to the raw man page in the source tree.

  • It looks like vacation settings depend on an LDAP-enabled vacation responder like Gnarwl; this is cool, but wasn't documented anywhere. The only place I found this mentioned was on this blog entry, which contained a lot of other info (like how to set Postfix up; yes, I shoulda RTFM but I didn't know that Postfix could grab so much info out of LDAP) that I didn't find on the GOsa website. In fact, that blog entry was probably the best source of info about GOsa configuration.

  • Adding email information for people requires a GOsa-specific LDAP entry for a mail server; again, this is best documented in that blog entry.

• GOsa is happiest when you point it at a new, empty LDAP tree and go from there. It appears to be much less happy with the idea of pointing it at an already-existing LDAP tree (though there is support, during installation, for converting accounts it has found in the tree).

  Taking the the ou=People ACL example above as a starting point, GOsa wants to create its own department entries in your root: ou=Accounting (say), with ou=Groups and ou=People under that. I have mixed feelings about this organization; it would work well for my current employer, but I'm not sure how well it'll fit for this side project.

• GOsa makes use of GOsa-specific LDAP schemas that you add to your server; this lets it label an object with (say) notes that users are allowed to change their own password. Perhaps not bad in itself -- I don't know enough to judge -- but it seems to mean that there are two sets of ACLs to manage: those in GOsa, and those in your LDAP server.

• GOsa's schema files are for OpenLDAP syntax (there's a proper name for that but I can't think of it right now). I'm using CentOS DS, which has its own. FInding out how to convert one to the other was hard enough; after that was done, though, CentOS DS still didn't like it, and I had to delete some additional entries. There doesn't appear to be a lot of support for CentOS/Fedora/RedHat/389 DS.

  (For the record, here's what I had to do:

  $ wget http://directory.fedoraproject.org/download/ol-schema-migrate.pl
  $ for i in $(grep 'schema/gosa' /usr/share/doc/gosa/slapd.conf-example | sed -e's/ldap/openldap/' | awk '{print $2}') ; do echo ../ol-schema-migrate.pl -b $i >>  98gosa.ldif; done
  

  And then remove references to gosaLoginRestriction and goFaxDivertNumber.)

• It might be better to think of GOsa as a complete system, of which the visible LDAP front-end is only one part: as described here, it's composed of Postfix, Courier for IMAP, Cyrus for authentication, and so on. That's not entirely fair; LDAP is LDAP, and I'm sure a lot of what seems app-specific could be generified (I spik Englitch good) by configuring Dovecot (say; I already use Dovecot instead of Courier) appropriately, or writing a plugin for GOsa.

Overall, I'm not sure how happy I am with GOsa. It's not doing what I want now, and it looks like the only way to make that happen is to wipe the tree and start from scratch. -- Well, no, not scratch: create a GOsa-visible department (ou=Something) and stick everything in there. Given the sparse documentation, it's probably silly for me to focus on that, but that's what sticks out at me right now.

This posting (PDF) came up on the VanLUG mailing list today; if you're looking for a Linux sysadmin job in Vancouver, it looks interesting.

For my own reference, here are a couple things I had to find out about Perl today:

• An overview of installing Perl modules with CPAN in your home directory.
• The local::lib module mentioned in that article; probably the best way of getting it going. Looks much easier than it was a few years ago.
• David Blank-Edelman's article for ;login: magazine about Perl and spawning (PDF)
• Parallel::Forkmanager is a simple Perl module for managing spawned processes.
• Proc::Queue is another module to manage forked processes.
• POE seems to be a much more sophisticated framework.

I've been setting up some new VMs for a separate project at work. I've realized that this is painful for two reasons: Bacula and Nagios.

Both are important...can't have a service without monitoring, and can't have a machine without backups. But each of these are configured by vast files; Bacula's is monolithic (the director's, anyhow, which is where you add new jobs) and Nagios' is legion. And they're hard to configure automagically with sed/awk/perl or cfengine; their stanzas span lines, and whitespace is important.

I've recently added a short script to my Nagios config; it regenerates a file that monitors all the Bacula jobs and makes sure they happen often enough. This is good...and I want more.

I found pynag, a Python module to parse and configure Nagios files. This is a start. I've had problems getting its head around my config files, because it didn't understand recursion in hostgroups (which I think is a recent feature of Nagios) or a hostname equal to "*". I've got the first working, and I'm banging my head against the second. The three books I got recently on Python should help (wow, IronPython looks nice).

There are a lot of example scripts with pynag. None do exactly what I want, but it looks like it should be possible to generate Nagios config files from some kind of list of hosts and services. This would be a big improvement.

But then there's Augeas, which does bi-directional parsing of config files. Have a look at the walk-through...it's pretty astounding. I realized that I've been looking for something like this for a long time: an easier way of managing all sorts of config files. Cfengine (v2 to be sure) just isn't cutting it anymore for me.

Now, the problem with Augeas for my present task is that there isn't anything in the current tree that does what I want, either. There is a commit for parsing nagios.cfg -- not sure if it's been released, or if it will parse everything in a Nagios config_dir. There's nothing for Bacula, either. This will mean a lot more work to get my ideal configuration management tool.

(On a side note, my wife said something to me the other day that was quite striking: I need tasks that can be divvied up into 45-minute chunks. That's how much free time I've got in the morning, bus rides to and from work, and the evening. Commute + kids != long blocks of free time.)

And I've got a congenital weakness for grand overarching syntheses of all existing knowledge...or at least big tasks like managing config files. So I'm trying to be aware of my brain.

...and there's son #2 waking up. Time to post.