the life of a sysadmin.

Deep thoughts

Tue Feb 26 20:30:23 PST 2008

I've been listening to the presentations from LISA07, and I have a few observations.

Trey Darley's presentation reminded me a lot of my last job, but much more intense: fast growth, no control, and no budget. The difference is that he had the experience and the chops to deal with it well. Also, if he can present at LISA, so can I.

Andrew Hume's presentation, "No Terabyte Left Behind", was interesting, by which I mean frightening. People mostly just trust that hardware does what it says it does/will do when it comes to storage. But that doesn't always work: he tells the story of a prof he worked with who checksummed all his files once a week. When a checksum changed — and it did about every 6 months — he'd retrieve it from backup. His rough guess for undetectable errors: 1 per 10 terabyte-years. And we're getting to the point where that's going to be significant very soon.

Tony Cass' presentation on grid computing for CERN was fascinating. This is the place I wanted to work (though as a particle physicist). UBC/TRIUMF is doing some work for this project as well, which makes me think I should jump over.

David Josephson's presentation was interesting, as much for the Q&A afterward as for his point. Which was? Glad you asked: that focussing on IP-based spam filtering (RBLs, greylisting) provides an incentive to spammers to hijack network prefixes via BGP attacks, and generally do nasty things to the Internet; please switch to content-based filtering post-haste. (To clarify, he was talking in particular about fast naive Bayesian classifiers, not SpamAssassin.) Since IP-based filtering treats IPs as valuable things — tokens that demonstrate your email is worth accepting — spammers steal IP addresses.

I'm not sure how much I buy his argument; he kept promising that the BGP attacks he described were only part of the problem, but he never seemed to get beyond that. But during the Q&A Brad Knowles got up and said (my summary) Content filtering doesn't scale, at least in his experience (as Senior Internet Mail Systems Administrator for AOL). At that point, another guy got up and said (again, my summary) that sort of thing is heard all the time, but with no data to back it up. The responder had co-authored a paper with Josephson that got Best Paper award at LISA '04, and they'd made damn sure to include a ton of footnotes. If their conclusions were wrong, people were free to challenge them; if Knowle's were wrong, they were unchallengeable because there was no data to back it up — it was all just story that got passed along and became myth.

Knowles' response was "I don't have time to write papers; I'm a technician, not an academic." Which is true, in lots of ways. And I don't mean any insult to Knowles; he's done things I will probably never match, we are all flooded with work, and so on. I'm one guy, working at a small shop, with none of his experience, or chops, or rep, or audience.

But there's a reason my .signature says "Because the plural of Anecdote is Myth": it's to remind me that unless you can back something up with facts, preferably written down and logged and repeatable, all you've got is a bunch of stories that become more and more True the more you repeat them.

It's obnoxious to sneer and say, "Cite, please"; it's worse to be ignorant.

Lots more listening to do. If you haven't downloaded them yet, you really should.

(permalink) (comments)

Spring

Fri Feb 22 15:43:23 PST 2008

At last: I'm finally coming to the end of working with the verdammnt web registration forms. We're going from our awful hack of a glued-together mess of Mambo and custom PHP, to something that'll mainly be Drupal with no custom code. Allegedly it's six weeks 'til launch date; the registration forms in use right now will limp along 'til they're no longer needed (end of the summer).

The registration form I'm working on now is not complicated in the absolute sense, but it's the most complicated one we've got. Last year I was afraid to touch the (old, legacy, ugly) code, and mostly just changed dates. This year I thought "fuck it" and rewrote nearly all of it, using the tools and skills I'd picked up in the meantime. (I'm still not a great programmer, understand, but I have improved some over last year.)

After a full day banging my head against it, I'm finally coming to the point where I'm pretty confident that the code will do what it's supposed to. And that's a relief. Therefore, in the stylee du Chromatic, I give thanks to:

In other news…just downloaded the second dev preview of Indiana, which I'd managed to not hear about at all (the preview releases, that is). I love university bandwidth; 640MB in about 1 minute. Sweet. I'll give it a try at home and see how it feels.

I've just finished reading the summaries of LISA '07 in the latest issue of ;login:. I feel…incredibly left out. I'm starting to think this profession might not be such a simple thing, you know, man? Sir? The presentations on autonomic computing have left me feeling a bit like a buggy whip maker with his nose to the grindstone.

And yes, it's a way off, and yes, small shops and generalists will probably be around for a while to come. But I'm not sure how much I want to keep being at a small shop. Which means learning the big stuff. Which, natch, is hard to do when you're trying to figure out how to properly test registration forms. Sigh.

But: I just stuck my head out a door at work and saw a chickadee. It chirped for a while, sitting on a tree near our building, then flew off. On a rare sunny day in Vancouver in Frebruary, after a week of messed-up sleep and feeling like I've been spinning my wheels, this is nice.

(permalink) (comments)

Fiendish Giggle

Fri Feb 8 10:10:54 PST 2008

New Dell 2950 server. 2 x quad-core Xeons, 2 x 6MB cache on each die, 16GB RAM, 6 x 300GB SAS 10K SCSI drives in a RAID-6 array using the PERC/6 controller.

/usr/src/linux-source-2.6.18# time make -j 9 bzImage
[snip]
Root device is (8, 3)
Boot sector 512 bytes.
Setup is 7295 bytes.
System is 1222 kB
Kernel: arch/i386/boot/bzImage is ready  (#1)

real    0m22.668s
user    2m20.425s
sys     0m14.537s

That's just insane.

(permalink) (comments)

Procmail bites me in the ass again

Wed Feb 6 13:49:27 PST 2008

Okay, so it isn't quite as bad as the time I threw 3,000 incoming messages for an ISP into my home directory. But I've just figured out that the reason a) $VENDOR didn't get back to me and b) it's been so quiet for the last few days is because all email was going to a file called X-Original-Sender because of one missing *. (In fact, that may also have been the cause of the first big error…)

God, I hate procmail sometimes.

(permalink) (comments)

My perfect notebook

Wed Feb 6 12:32:17 PST 2008

I agree completely with Chris Siebenmann's entry on the utility of keeping a notebook. I've done this almost as long as I've been working in IT, and it's saved my ass repeatedly. Also, the way I keep my journal — random notes at the front working toward the back, daily summary at the back working toward the front — means that it's fairly simple to search for my notes on a particular task, or explain to management just what I do with my time.

I love paper. I tried a PDA for a while; hated it, didn't trust it, and gave it up promptly. Scribbling with a pen is faster, more satisfying, and doesn't make me wait for something to reboot or awaken, or force me to learn a different way to scribble. At the best of times, it forces me to think a bit about what I'm doing or seeing, rather than just typing blindly at the problem. (What, you never do that?)

But while a paper notebook is wonderful, it's not perfect. Here's what would be perfect:

Let me paste screen captures right into my notebook. (I'm talking both screenshots and the log files from GNU screen.)

Let me paste sections of my .history file into my notebook complete with timestamps.

Let me cut-and-paste from my notebook to Emacs (or vi, you heathens), and vice-versa.

Let everything I write or paste be timestamped automagically.

Let everything I write or paste be sync'd automagically to some plain text-like format, suitable for grepping, munging, merging into a database, pushing to syslogd, or what have you.

(permalink) (comments)

Heh

Sun Feb 3 08:50:14 PST 2008

Matthew Garret's presentation on Suspend-to-Disk make fun reading.

Arlo's sick with flu or something; I was up 'til 1am last night rocking him to sleep. Haven't done that in a while…

Telling detail: I'm about to blow away Debian testing on my desktop machine and install Ubuntu's Gutsy Gibbon. Partly it's because I'm tired of installing 80MB worth of updates every two weeks, and partly it's because it'll make setting up the printer a breeze.

I'll probably leave half the drive aside for good ol' Debian stable, but Ubuntu'll stay there for experimenting and so my parents, on their next visit, will not have to bring out their 4-tonne laptop.

I'll be reinstalling Ubuntu on my laptop as well; due to a stupid error, I installed Dapper, not Gutsy. I tried updating in one fell swoop, and after three days of apt-get -f install I finally got things working…except for the boot artwork, and GDM doesn't start one time out of three. Interesting experiment, but I think I'll take a do-over.

I may even install it twice, so that I can try out The Depenguinator, which appears to be a lot easier than trying to figure out PXE booting for FreeBSD. Unlike OpenBSD, there's no readily apparent "official way" of doing it, and the handful of HOWTOs I've found have contradicted each other. At this point I'm just too lazy to keep trying and seeing what I'm doing wrong.

(permalink) (comments)

Wah

Deep thoughts

Spring

I never knew this existed

Fiendish Giggle

Procmail bites me in the ass again

My perfect notebook

Heh

Happy 2^5, everyone!