The Life of a Sysadmin

The Life of a Sysadmin http://saintaardvarkthecarpeted.com/blog Carousel is a LIE! en Copyright 2006 Wed, 01 Oct 2008 15:31:54 -0700 Wed, 01 Oct 2008 15:31:54 -0700 aardvark@saintaardvarkthecarpeted.com Insomnia The good thing about being up at 3am is that, with a laptop, you can keep yourself entertained by whipping up a quick spreadsheet of the rack, switch and console server layout for the new server room. The bad thing is that you may not trip over Sun's handy-dandy power calculators (like for the <a href="http://www.sun.com/servers/x64/x4140/calc/">X4140</a> or the <a href="http://www.sun.com/servers/x64/x4440/calc/">X4440</a> until the next day, leaving you twelve hours to wonder blearily if you've blown your server room's power budget all in one go. http://saintaardvarkthecarpeted.com/blog/2008-10/insomnia.html That's a mighty big catchup I got goin' there Work…hell, life is busy these days. At work, our (only) tape drive failed a couple of weeks ago; <a href="http://www.bacula.org">Bacula</a> asked for a new tape, I put it in, and suddenly the "Drive Error" LED started blinking and the drive would not eject the tape. No combination of power cycling, paperclips or pleading would help. Fortunately, $UNIVERSITY_VENDOR had an external HP Ultrium 960 tape drive + 24 tapes in a local warehouse. Hurray for expedited shipping from <a href="http://www.richmond.ca">Richmond</a>! Not only that, the Ultrium 3 drive can still read/write our Ultrium 2 media. By this I mean that a) I'd forgotten that the <a href="http://www.lto-technology.com/">LTO standard</a> calls for R/W for the last generation, not R/O, and b) the few tests I've been able to do with reading random old backups and reading/writing random new backups seem to go just fine. <a href="mailto:comments+1222348453@saintaardvarkthecarpeted.com">Question for the peanut gallery</a>: Has anyone had an Ultrium tape written by one drive that couldn't be read by another? I've read about tapes not being readable by drives other than the one that wrote it, but haven't heard any accounts first-hand for modern stuff. <a href="mailto:comments+1222348453@saintaardvarkthecarpeted.com">Another question for the peanut gallery</a>: I ended up finding instructions from HP that showed how to take apart a tape drive and manually eject a stuck tape. I did it for the old Ultrium 2. (No, it wasn't an HP drive, but they're all made in Hungary…so how many companies can be making these things, really?) The question is, do I trust this thing or not? My instinct is "not as far as I can throw it", but the instructions didn't mention anything one way or the other. In other news, $NEW_ASSIGNMENT is looking to build a machine room in the basement of a building across the way, and I'm (natch) involved in that. Unfortunately, I've never been involved in one before. Fortunately, I got training on this when I went to LISA in 2006, and there's also <a href="http://amazon.com/o/ASIN/0321492668/tomontime-20">Limoncelli, Hogan and Chalup</a> to help out. (That link sends the author a few pennies, BTW; if you haven't bought it yet, get your boss to buy it for you.) As part of the movement of servers from one data centre across town to new, temporary space here (in advance of this new machine room), another chunk of $UNIVERSITY has volunteered to help out with backups by sucking data over the ether with Tivoli. Nice, neighbourly think of them to do! I met with the two sysadmins today and got a tour of their server room. (Not strictly necessary when arranging for backups, but was I gonna turn down the chance to tour a 1500-node cluster? No, I was not.) And oh, it was nice. Proper cable management…I just about cried. :-) Big racks full of blades, batteries, fibre everywhere, and a big-ass robotic Ultrium 2 tape cabinet. (I was surprised that it was 2, and not U3 or U4, but they pointed out that this had all been bought about four or five years ago…and like I've heard about other government-funded efforts, there's millions for capital and little for maintenance or upgrades.) They told me about assembling most of it from scratch — partly for the experience, partly because they weren't happy with the way the vendor was doing it ("learning as they went along" was how they described it). I urged them to think about presenting at <a href="http://www.usenix.org/events/lisa08/">LISA</a>, and was surprised that they hadn't heard of the conference or considered writing up their efforts. Similarly, I was arranging for MX service for the new place with the university IT department, and the guy I was speaking to mentioned using Postfix. That surprised me, as I'd been under the impression that they used Sendmail, and I said so. He said that they had, but they switched to Postfix a year ago and were quite happy with it: excellent performance as an MTA (I think he said millions of emails per day, which I think is higher than my entire career total :-) and much better <a href="http://en.wikipedia.org/wiki/Milter">Milter</a> performance than Sendmail. I told him he should make a presentation to the university sysadmin group, and he said he'd never considered it. Oh, and I've completely passed over the A/C leak in my main job's server room…or the buttload of new servers we're gonna be getting at the new job…or adding the Sieve plugin for Dovecot on a CentOS box…or OpenBSD on a Dell R300 (completely fine; the only thing I've got to figure out is how it'll handle the onboard RAID if a drive fails). I've just been busy busy busy: two work places, still a 90-minute commute by transit, and two kids, one of whom is about to wake up right now. Not that I'm complaining. Things are going great, and they're only getting better. Last note: I'm seriously considering moving to <a href="http://www.steve.org.uk/Software/chronicle/">Steve Kemp's Chronicle</a> engine. Chris Siebenmann's note about the <a href="http://utcc.utoronto.ca/~cks/space/blog/web/EntryAsFileAttraction">attraction of file-based systems for techies</a> is quite true, as is his note about it being hard to do well. I haven't done it well, and I don't think I've got the time to make it good. Chronicle looks damn nice, even if it does mean opening up comments via the web again…which might mean actually getting comments every now and then. Anyhow, another project for the pile. http://saintaardvarkthecarpeted.com/blog/2008-09/thats_a_mighty_big_catchup_i_got_goin_there.html Double-take That's <a href="http://www.flickr.com/photos/mtu1440/2778655707/sizes/l/">not quite</a> <a href="http://www.bigpointroad.com">my dad</a> at <a href="http://undeadly.org/cgi?action=article&sid=20080827114338">c2k8</a>, but damn if it wasn't enough to make me look twice. http://saintaardvarkthecarpeted.com/blog/2008-09/doubletake.html Well, that's a relief CentOS <a href="http://lists.centos.org/pipermail/centos-announce/2008-August/015195.html">not affected</a> by the Red Hat compromise. http://saintaardvarkthecarpeted.com/blog/2008-08/well_thats_a_relief.html A new tape without labels inside? <a href="http://www.imdb.com/title/tt0357413/quotes">That's bush. Bush league.</a> You hear me, Fuji? Look at me! I knew there was a reason to compulsively squirrel away every half-used set of tape labels. http://saintaardvarkthecarpeted.com/blog/2008-08/a_new_tape_without_labels_inside.html Fedora Directory Server So one of the things I need to set up at <tt>$JOB_2</tt> is some kind of unified bag o' passwords…which, since I hate NIS, pretty much means LDAP. This is the first chance I've had to set up an LDAP system from scratch, rather than either being afraid to try or being stuck with (and, sadly, contributing to the further divergence of) a mishmash of semi-borked LDAP servers. I've been trying out <a href="http://directory.fedoraproject.org">Fedora Directory Server</a> the last few days, and so far I'm pretty happy with it. It's nice to have the luxury of learning what the hell I'm doing before it all goes live, of screwing up a bunch of times on a non-production system. Likes: Welp, it's a lot like Sun's Directory Server…at least as far as the logging and console go, anyhow. Not surprising, given the heritage. You can automate installation by giving it a configuration file — something I didn't realize you could do with Sun's DS. Other likes: PHPLDAPAdmin is nice. The latest version has E-Z-Reed XML templates for things like account creation, meaning I can keep my ignorance of Javascript intact. (Hurray!) Minor irritants: there are a few. First off, there are no RPMs for CentOS 5 for the 1.1 series; you have to jump through some hoops to get the FC6 RPMs of 1.1 installed. I'd originally tried the 1.0 series on Debian, and hadn't realized that the 1.1 series does not include the org chart or E-Z-Account-Maker web app. (This is where y'all can go, "Muffin!") Third, I'm so far not able to get the automated installation working…can't figure out why. Not terribly important, since <tt>$JOB_2</tt> is small and likely to stay that way; a couple of servers is likely to be the max. But installation of this thing, just like with Sun DS, has lots of knobs that you can twiddle if you want, and part of the problem with the mishmash at <tt>$JOB_1</tt> is that no one ever standardized the settings — never wrote down the answers to the questions, or scripted it, or came up with a config file, or anything. And it's hellish if you want to add another install to the mix. Anyhow…so far it's cool. I've been playing with it on a machine at <tt>$JOB_2</tt> plus an installation of CentOS 5 on my laptop. Still to learn: SSL, replication, and (maybe) multi-master replication. (Incidentally, I'm surprised that there isn't a more recent version of O'Reilly's <a href="http://oreilly.com/catalog/9781565924918/">LDAP Administration</a> by <a href="http://plainjoe.org">Gerald Carter</a>. Yes, there's still OpenLDAP and I don't imagine it's changed very much (feel free to correct me), but something that included Fedora DS, and maybe (<a href="http://directorymanager.wordpress.com/2007/11/28/an-open-letter-to-the-opends-community-and-to-sun-microsystems/">maybe</a>) <a href="http://www.opends.org/">OpenDS</a> would be good. (And speaking of Sun gossip, I've been meaning to mention <a href="http://www.cuddletech.com/blog/pivot/entry.php?id=960">this</a> for a while…and now <a href="http://www.blastwave.org/dclarke/blog/?q=node/111">this</a>.) http://saintaardvarkthecarpeted.com/blog/2008-08/fedora_directory_server.html Defcon NOC Interesting article from <a href="http://blog.wired.com/27bstroke6/2008/08/a-first-ever-lo.html">Threat Level</a> about the Defcon NOC. Now there'd be an interesting job… http://saintaardvarkthecarpeted.com/blog/2008-08/defcon_noc.html A note on comments About a year ago, I started using a cobbled-together system of Bash and Perl scripts and Makefiles to put together this blog. One of the reasons was my general dislike for PHP; another was my desire to try living (at least in some small way) by <a href="http://saintaardvarkthecarpeted.com/wordpress/?p=255">Saint Aardvark's Axiom of Information Utility</a>, and try keeping this in plain text. (Another was a desire to use Emacs to write these damn things; I want the control that's thrown out when you start using a GUI to edit.) But one of the problems that faced me was how to deal with comments, and comment spam. Having a web form that allowed comments made commenting easy, but the downside was that it made spamming easy too. WP and others keep this down to a dull roar, but it's not perfect and I've had problems with false positives — people being unable to post comments because their IP address was on some blacklist, and the plugin had made no provision for whitelisting. I decided to lash together something that would use email. For me — a very small, low-traffic website, with a blog devoted to a rather obscure set of concerns and a tech-savvy audience (Hi Dad!) — this seemed like a good choice. Email spam, for me, has been pretty much solved by greylisting and SpamAssassin. (There's the problem of a ten — no, fourteen — year-old email address that I've been meaning to get changed for a while now, but that's another story; they don't seem to do greylisting, and SpamAssassin does catch most of it.) So taking comments by email seemed, you know, righteous, dude. The system for comments is pretty simple: every post gets an epoch timestamp embedded in it. (I think if you look in the HTML source, you can see it.) I use it for sorting the order of the posts, and I use it to generate email addresses for post-specific comments. The format is simple: <tt>comments+(seconds since the epoch)@saintaardvarkthecarpeted.com</tt>. The address is included in the post, though I haven't done much to make it obvious. (This blog, and I think this whole website, would make baby Jacob Nielson cry.) My thinking was that, even though I was publishing the addresses, it wouldn't matter: as I mentioned, spam for me has been mainly solved (insert disclaimers here). Between greylisting and SpamAssassin, I figured I pretty much wouldn't see any spam at all. Turns out there's another benefit: the addresses have been picked up by spam bot crawlers, but they're screwing up the scraping. From 24 days of mail logs, I see a crapload of attempts to deliver to the wrong address: <div class="listingblock"> <div class="content"> <pre><tt>$ perl -ne'/NOQUEUE/ && s{.*to=<(\S+?)>.*}{$1} && print "$_\n";' mail.log* | sort | uniq -c | sort -n [much snippage] 36 1181577610@saintaardvarkthecarpeted.com 36 1182947701@saintaardvarkthecarpeted.com 37 1181326150@saintaardvarkthecarpeted.com 37 1183667208@saintaardvarkthecarpeted.com 38 1182949918@saintaardvarkthecarpeted.com 40 1183349604@saintaardvarkthecarpeted.com</tt></pre> </div></div> There were more than 2500 of these messages turned away by greylisting. They've all stripped off everything up to the plus, not realizing (as I didn't until a few years ago) that a plus in an email is valid. In fact, the only attempts to deliver to legitimate comment addresses were two actual comments to my blog…which brings up a shortcoming: I never got that many comments with WordPress, but I sure got more than I do now. It's possible my writing has just gone 'way downhill, but I think it's more likely that this system just puts people off, or they're just unable to find it with my current (crappy) design. (One interesting problem: <a href="http://torturedpotato.com/cheeseblog">my wife</a> tried to comment once, using Lotus Notes at her workplace. It converted the plus sign into an underscore. Weird.) I still regard this setup for comments as an experiment. Its results are definitely mixed; no spam, but fewer comments as well. Given the tiresome mess that comes with the lack of an HTTP equivalent of greylisting, I'm inclined to keep doing it. Anyhow…that's my interesting research result for the day. You may now talk amongst yourselves. http://saintaardvarkthecarpeted.com/blog/2008-08/a_note_on_comments.html cfengine: Received signal 2 (SIGKILL) while doing pre-lock-state Ran into a problem today when adding this stanza to cfengine on a Debian Etch machine: <div class="listingblock"> <div class="content"> <pre><tt>editfiles: { /etc/aliases AppendIfNoSuchLine "root: sysadmin@pims.math.ca" DefineClasses "rebuild_aliases:restart_postfix" }</tt></pre> </div></div> The cfengine reference file I've got, which sez it's for version 2.2.1, says you can define multiple classes in DefineClasses (or DefineInGroup), as long as they're separated by commas, spaces or dots. (The version in Etch is 2.2.20.) However, when I ran cfagent, it just hung immediately after performing the edit, and gave this error when I ctrl-c'd it: <div class="listingblock"> <div class="content"> <pre><tt>cfengine: Received signal 2 (SIGKILL) while doing [pre-lock-state]</tt></pre> </div></div> Running cfengine with <tt>-d2</tt> showed endless repetitions of <tt>AddClassToHeap()</tt> at this point, so either there's something wrong with my syntax or there's a bug in cfengine. (I'm guessing the former.) Searching for <tt>pre-lock-state</tt> and cfengine only turned up cases where the clients were syncing with the master; thus this note. The fix was to just make it one class: <div class="listingblock"> <div class="content"> <pre><tt> DefineClasses "rebuild_aliases"</tt></pre> </div></div> Asking to restart Postfix was probably a bit of overkill anyhow… http://saintaardvarkthecarpeted.com/blog/2008-07/cfengine_received_signal_2_sigkill_while_doing_prelockstate.html Happy Sysadmin Day! <a href="http://www.sonador.com/arseely/">Andy</a> just pointed out to me that it's <a href="http://sysadminday.com/">Sysadmin Day</a>, which I'd totally forgot about. So here's to ya, everyone! Which means I'll have to save the dream I just had (one of the Rohirrim being held hostage by the housewives of the OC at one of their garden parties, and insisting, with murderous glint in their eyes, that he finish all of his cake before leaving despite the rest of the Rohirrim showing up to rescue him, and me trying desperately to resolve the standoff by telling Peter Gallagher that, really, there's still time to let him go) for another entry…/me shakes head. http://saintaardvarkthecarpeted.com/blog/2008-07/happy_sysadmin_day.html