Gnuplot...how'd I live without it?

Thanks to the good folks who wrote this page, I was able to come up with some quick graphs showing how often our RealMedia files are used:

Gnuplot Example

All it took was five minutes mucking about ... cf OpenOffice.org, where I could not, for the life of me, figure out how to get a decent-looking output.

Tags:

Can a mouse crash?

Just had a repeat of the weird mouse-X disconnect I've encountered before. This time though, I'm running Debian Etch — so no more blaming the problem on SuSE (as I secretly always did :-).

One noticeable problem this time was that the middle button did not work, making click-to-paste impossible; I even ran xev and saw no events for middle-clicking. (This in addition to clicking being inconsistent, the client receiving the click being inconsistent, etc). Running cat /dev/input/mouse0 did not work. What did work was disconnecting the mouse (a USB 3-button optical jobbie), then plugging it back in. Sure, coulda been the mouse driver, or X, or something, but I wonder if the hardware itself — whatever little controller chip is in there — maybe got wedged. Interesting to think about…

Tags: hardware

Sick, sick, sick

Flu sucks. I've been down with it for the last two days. Today I'm feeling a bit better — a little bit, mostly with lots of drugs. We'll see if I make it in to $WORK today or not.

What else have I been doing? Let's see…

  • Our Roku SoundBridge M1001 has died after just over a year, which turns out to be a semi-common thing. Fortunately I've found a local electronic repair store that I'm hoping will fix things up. Other than that I've been fairly happy with it; it's definitely a lot handier with the SoundBridge Commander, though.
  • Started playing with Xen on my web server. I'm going to try setting things up more complicated than they are now, but hopefully more secure: one instance to serve files, with /home (where all the sites are) mounted read-only; one with a database server; one for mail; one for SSH access; and one with the bag of passwords (LDAP, probably). Yeah, it's hopelessly baroque for a home server with, what, 8 websites, but it's a work in progress and it's fun to play. :-)
  • Installed the Awesome window manager after reading about it for the nth time. Happy so far, but I'm still exploring.
  • Tried making bread that rises overnight. I got into the habit of making my own bread at the beginning of the year, and I really got used to it. But two kids means much less time, and half an hour to mix and knead the dough is getting hard to find. So the latest trick has been using the KitchenAid and its dough hook (I did it all by hand previously), and fiddling with the amount of yeast so that, in theory, it'll rise overnight and be ready to bake when I get up at 5:30am to go to work. This used to work, but now I'm having problems with it collapsing. That may be because I'm using instant yeast rather than the regular, active stuff. Updates will follow as events unfold.

And that's that. Time to put together some lunch and see about going to work.

Tags: geekdad

Today's lesson:

$TTL != refresh.

This lesson brought to you by the ISC, Cricket Liu and RFC 1035.

Tags: rtfrfc

Unattended...working at last

Looks like we're going to be getting a bunch more Windows desktops in the near future at $WORK, so I've been looking into Unattended again. I'm having much better luck this time than the last time I tried, a couple or three years ago.

I can't remember what went wrong then, but this time it took me a stupidly long time to figure out that an error message about a missing djgpp.env file means you forgot to unzip the support files under a directory called djgpp.

Another thing that tripped me up this time: unattended installations from OEM media are not allowed by default, even with a legit key from the sticker on the side of your new shiny box. This mailing list post pointed out the magic key, which seems to be working for mwe.

Now that I got those things sorted out, things are going much better.

Tags: windows installation

Back at work...

...after a month off, and almost no emergencies in my absence. Sweet!

Now if only I could catch up on sleep. I remember this from the first kid: you never know just how much you can accomplish on so little sleep.

Tags: work geekdad

Now we are four

Tags: geekdad

Possibly my last free Saturday afternoon for a while...

As Project U-14 draws to a close, I've been spending a wonderful couple of hours on the couch playing with my laptop while Arlo naps.

Here's what I've been doing:

  • Tracked down a bug that in gpsbabel that would make it crash if you gave it no filename to process. Turned out it was fixed late last year, and the new version will be in Lenny, but that was okay; the practice was good, it didn't take too long, and it turns out both the developers and I came to the same conclusion. I hope they know what they're doing, at least. :-) I'm unsure if I should file a bug in Debian or not; Lenny's coming soon, and I'm not sure of the usual practice in this case.
  • Actually began work on Project U-13; I'm trying out fai-cd, rather than GRML. GRML's nice, but it's a way non-stock Debian install by the time it's done, and I'd like to stick as much as possible to stock Debian (OS of the OElder Goeds). We'll see how it goes.
  • Read about how Dan Fucking Kaminsky managed to Rickrolled Facebook thanks to nasty DNS hijacking (similar to Verisign's Site Finder) by Earthlink and Comcast. No copy of the presentation up on his site, from what I can see, but I'm looking forward to it.
  • Eating chocolate chip cookies my wife made. Yum.

Tags: geekdad

Blastwave upgrade: heads up

Heads up for those of you using Blastwave and CUPS: after upgrading to the latest stable version, printing stopped working for me (and a few users :-). I eventually tracked it down to the movement of two files: suddenly

/opt/csw/lib/cups/filter/pstopxl
/opt/csw/lib/cups/filter/pstoraster

were moved to

/opt/csw/lib/cups/pstopxl
/opt/csw/lib/cups/pstoraster

resulting in many error messages like Unsupported format text/plain! and Hint: is ESP ghostscript installed?. Moving them both back into place and restarting CUPS fixed things just fine.

According to Bacula (yay Bacula!) both files were in the right directory as of last night, and Blastwave's file list for Ghostscript shows the new location for these two files. A bug has been filed.

Tags: solaris packagemanagement bug

Sigh ----

This is one of the few things that would make me consider moving to the US right now.

Tags: work

With apologies to Mark Twain

There are always timesinks at a job: the things that suck up all your spare time, that interrupt what you're doing and force you on to something else. They're urgent, or they're complicated, or they're obscure and you only ever touch them every six months. If you're really unlucky, they're all three. They drain the life from you; a good day turns shitty, and an already-shitty day becomes nigh-unbearable.

The website is one such timesink at my current job. It's a veritable Grand Canyon of different technologies, databases, and code. You can examine it and, like a geologist, date particular pages or code with great accuracy, judging by clues like composition, surroundings, indentation patterns ("Oooh, K&R crossed with…crack?"), and previous experience. When an Urgent Request for Web Changes comes in (and they're all urgent), figuring out how to do it means figuring out how that particular page was generated in the first place: static? dynamic? CMS? And then you have to figure how you can meddle with it: logging into Mambo, the CMS of the damned? If it's static: does the URL map nicely to the filesystem, or is there a hidden Apache Alias directive somewhere? Do you have permissions to open the file, or will it take sudo to chown it, or another nagging email to a coworker to please check their changes into RCS? And if, God help you, it's dynamic…but no; that mess of spaghetti should stay down. There's no sense bringing it up again simply for prurient purposes.

Sunray terminals are another timesink. When they work they work very well. I like the energy-saving aspects of it — both electrical and my own; one machine to manage is always better than 40. But when they don't work, it's a pain. Has a session become wedged? Is it GNOME's fault? Has Adobe Acrobat decided to eat up all the CPU again? If so, is that worse than the security holes that remain unfixed in the later version? Why is Solaris 10 randomly not sending RST packets when it receives a SYN on a port it's not listening on? (If anyone has any ideas, please let me know.) Has a cheap switch, installed because no one believed that an office meant for one might someday hold four, gone off its meds again?

These things make me throw up my hands and and curse my fortune. I have no one unfortunate enough to be my subordinate, so it's up to me to hack and slash through the possibilities until it's finished, or at least put off for another day.

But LDAP is worse.

When it works it works very, very well. Failover works, replication works, and an account created here zips there without a moment's thought. But when it fails, it's urgent and complicated and obscure all at once, and sometimes in degrees polynomial.

At last count we have four different master-master replicas, running three or possibly four different versions of Sun's Directory Server (under six different product names, no less). There are replication agreements spanning versions that aren't even supposed to tolerate each other's existence, using two different encryption protocols and NetBEUI. Two completely different "helpful" management tools vie for our attention, lacking only flash plugins to trigger seizures. Only one server can be poked or prodded with a command line tool. Diagnostics are by turns nonexistent or endearingly fickle.

To be fair, the vendor documentation is vast and makes fine kindling, though its promise to fully document error codes like error 457758854b: BER error 45775885b4 is best regarded as a bitter joke by a jaded software engineer who died alone, unloved and without stock options. (Our own documentation is marginally better: no carbon is released when it is destroyed.) Thus, keeping track of ACLs (say), and exactly which unholy wrath you will invite upon your head should you make a mistake when granting or revoking privileges to read a particular entry, means digging through half-remembered conversations, drunken Google searches, year-old notebooks and a quiet, solitary introspection normally reserved for contemplating your own impending doom.

On top of everything else, LDAP encompasses everything, or nearly so. Email routing, website privileges, database access, even TCP checksum computation: all are kept in, or depend on, or just like to hold hands with, LDAP. It's enough to make me wistful for the good old days of NIS.

In a few minutes I am going to go back to work and try to figure out why a new account has stopped, in mid-replication, halfway between $UNIVERSITY and $OTHER_UNIVERSITY. It will take me the rest of the afternoon. I will use words that my own son does not know I know. And I will come out of it shrunken, withered, beaten down and humble.

Tags: rant ldap

Saturday work

Yesterday I spent the day at work testing our installation of APCUPd and tidying up the goram rat's nest of network and electrical cables my predecessor left me.

APCUPSd worked with only a few hitches:

  1. I had one machine polling a UPS, and told it to shut down when there was 30% charge left. The other machines, which poll the master, were set to shut down 30 seconds after the power went out. They shut down, but that bumped up the charge reading on the battery because the load was that much lower. So I didn't get to test the automatic shutdown of the master.
  2. The other three machines were all set to shut down after 30 seconds; however, NFS cross-mounting made for problems with one of them. I'll need to stagger those three machines, whether they're looking at the charge or just shutting down n minutes after the power goes out.
  3. The Solaris 10 box shut down just fine, but when it restarted it did not let me log in — even in the console. Since Solaris 10's boot sequence is dead silent by default (thank you, Sun), it was hard to be sure what was happening. The last time I was patching this machine, reboots took 10 minutes; I gave 20 this time before giving up and going to single-user mode. The problem appears to be /etc/nologin, stuck there from the shutdown. This prevented a login prompt from coming up even in the console, without any sort of warning. Arghh.

As for the cleanup: satisfying. I'm no longer quite so ashamed of the server room.

Tags: hardware

The LugRadio presenters are little girls

Hah! Just listened to Season 5, Episode 13 of LugRadio, and they read my letter! It was about how, if you're working tech support, you usually have no idea whether or not the other person is actually clueful or not; thus, the inevitable "Is it plugged in?" questions which frustrate techies. Or, as Jono Bacon put it, you have no idea whether the customer is "chuffing up the bong pole". (And the giant round of Whiskey Tango Foxtrot? goes to…Jono Bacon! :-) So this is what fame feels like.

Incidentally, LugRadio Live USA looks like it's going to be freakin' sweet. I wish I could go, but Project U-14 is going to be in pre-release testing at that point...

Tags: geekdad

USENIX conference proceedings open

USENIX has done a wonderful thing: their conference proceedings are now open to the public, rather than requiring a USENIX membership.

This is very, very good. If you haven't gone through the list of presentations and papers from LISA, FAST, WOOT, or the USENIX conference itself, you really need to.

Come to that, if you haven't picked up a membership yet to USENIX and SAGE, you really need to. A dead-tree copy of ;login: magazine is the most interesting single publication I've found about computing in general, and system administration in particular. You owe it to yourself.

Tags: lisa reading

Wah ---

Airport Toy X-Ray Machine, c/o Saint Schneier. Maybe I'll get one for Arlo.

Tags: geekdad

Deep thoughts

I've been listening to the presentations from LISA07, and I have a few observations.

Trey Darley's presentation reminded me a lot of my last job, but much more intense: fast growth, no control, and no budget. The difference is that he had the experience and the chops to deal with it well. Also, if he can present at LISA, so can I.

Andrew Hume's presentation, "No Terabyte Left Behind", was interesting, by which I mean frightening. People mostly just trust that hardware does what it says it does/will do when it comes to storage. But that doesn't always work: he tells the story of a prof he worked with who checksummed all his files once a week. When a checksum changed — and it did about every 6 months — he'd retrieve it from backup. His rough guess for undetectable errors: 1 per 10 terabyte-years. And we're getting to the point where that's going to be significant very soon.

Tony Cass' presentation on grid computing for CERN was fascinating. This is the place I wanted to work (though as a particle physicist). UBC/TRIUMF is doing some work for this project as well, which makes me think I should jump over.

David Josephson's presentation was interesting, as much for the Q&A afterward as for his point. Which was? Glad you asked: that focussing on IP-based spam filtering (RBLs, greylisting) provides an incentive to spammers to hijack network prefixes via BGP attacks, and generally do nasty things to the Internet; please switch to content-based filtering post-haste. (To clarify, he was talking in particular about fast naive Bayesian classifiers, not SpamAssassin.) Since IP-based filtering treats IPs as valuable things — tokens that demonstrate your email is worth accepting — spammers steal IP addresses.

I'm not sure how much I buy his argument; he kept promising that the BGP attacks he described were only part of the problem, but he never seemed to get beyond that. But during the Q&A Brad Knowles got up and said (my summary) Content filtering doesn't scale, at least in his experience (as Senior Internet Mail Systems Administrator for AOL). At that point, another guy got up and said (again, my summary) that sort of thing is heard all the time, but with no data to back it up. The responder had co-authored a paper with Josephson that got Best Paper award at LISA '04, and they'd made damn sure to include a ton of footnotes. If their conclusions were wrong, people were free to challenge them; if Knowle's were wrong, they were unchallengeable because there was no data to back it up -- it was all just story that got passed along and became myth.

Knowles' response was "I don't have time to write papers; I'm a technician, not an academic." Which is true, in lots of ways. And I don't mean any insult to Knowles; he's done things I will probably never match, we are all flooded with work, and so on. I'm one guy, working at a small shop, with none of his experience, or chops, or rep, or audience.

But there's a reason my .signature says "Because the plural of Anecdote is Myth": it's to remind me that unless you can back something up with facts, preferably written down and logged and repeatable, all you've got is a bunch of stories that become more and more True the more you repeat them.

It's obnoxious to sneer and say, "Cite, please"; it's worse to be ignorant.

Lots more listening to do. If you haven't downloaded them yet, you really should.

Tags: lisa spam

Spring

At last: I'm finally coming to the end of working with the verdammnt web registration forms. We're going from our awful hack of a glued-together mess of Mambo and custom PHP, to something that'll mainly be Drupal with no custom code. Allegedly it's six weeks 'til launch date; the registration forms in use right now will limp along 'til they're no longer needed (end of the summer).

The registration form I'm working on now is not complicated in the absolute sense, but it's the most complicated one we've got. Last year I was afraid to touch the (old, legacy, ugly) code, and mostly just changed dates. This year I thought "fuck it" and rewrote nearly all of it, using the tools and skills I'd picked up in the meantime. (I'm still not a great programmer, understand, but I have improved some over last year.)

After a full day banging my head against it, I'm finally coming to the point where I'm pretty confident that the code will do what it's supposed to. And that's a relief. Therefore, in the stylee du Chromatic, I give thanks to:

In other news...just downloaded the second dev preview of Indiana, which I'd managed to not hear about at all (the preview releases, that is). I love university bandwidth; 640MB in about 1 minute. Sweet. I'll give it a try at home and see how it feels.

I've just finished reading the summaries of LISA '07 in the latest issue of ;login:. I feel…incredibly left out. I'm starting to think this profession might not be such a simple thing, you know, man? Sir? The presentations on autonomic computing have left me feeling a bit like a buggy whip maker with his nose to the grindstone.

And yes, it's a way off, and yes, small shops and generalists will probably be around for a while to come. But I'm not sure how much I want to keep being at a small shop. Which means learning the big stuff. Which, natch, is hard to do when you're trying to figure out how to properly test registration forms. Sigh.

But: I just stuck my head out a door at work and saw a chickadee. It chirped for a while, sitting on a tree near our building, then flew off. On a rare sunny day in Vancouver in Frebruary, after a week of messed-up sleep and feeling like I've been spinning my wheels, this is nice.

Tags: perl web programming solaris lisa

I never knew this existed

I'll be damned: a GPL'd Windows Eventlog-to-syslogd interface. Thanks to Will for the pointer!

Tags: windows

Fiendish Giggle

New Dell 2950 server. 2 x quad-core Xeons, 2 x 6MB cache on each die, 16GB RAM, 6 x 300GB SAS 10K SCSI drives in a RAID-6 array using the PERC/6 controller.

/usr/src/linux-source-2.6.18# time make -j 9 bzImage
[snip]
Root device is (8, 3)
Boot sector 512 bytes.
Setup is 7295 bytes.
System is 1222 kB
Kernel: arch/i386/boot/bzImage is ready  (#1)

real    0m22.668s
user    2m20.425s
sys     0m14.537s

That's just insane.

Tags: hardware linux dell

Procmail bites me in the ass \*again\*

Okay, so it isn't quite as bad as the time I threw 3,000 incoming messages for an ISP into my home directory. But I've just figured out that the reason a) $VENDOR didn't get back to me and b) it's been so quiet for the last few days is because all email was going to a file called X-Original-Sender because of one missing *. (In fact, that may also have been the cause of the first big error...)

God, I hate procmail sometimes.

Tags: screwup procmail rant