04 Jul 2008
Thanks to the good folks who wrote this page, I was able to come up with some quick graphs showing how often our RealMedia files are used:

All it took was five minutes mucking about ... cf OpenOffice.org, where I could not, for the life of me, figure out how to get a decent-looking output.
Tags:
02 Jul 2008
Just had a repeat of the weird mouse-X disconnect I've
encountered before. This time though, I'm running Debian Etch — so no
more blaming the problem on SuSE (as I secretly always did :-).
One noticeable problem this time was that the middle button did not
work, making click-to-paste impossible; I even ran xev
and saw no
events for middle-clicking. (This in addition to clicking being
inconsistent, the client receiving the click being inconsistent,
etc). Running cat /dev/input/mouse0
did not work. What did work
was disconnecting the mouse (a USB 3-button optical jobbie), then
plugging it back in. Sure, coulda been the mouse driver, or X, or
something, but I wonder if the hardware itself — whatever little
controller chip is in there — maybe got wedged. Interesting to think
about…
Tags:
hardware
19 Jun 2008
Flu sucks. I've been down with it for the last two days. Today I'm feeling a bit better — a little bit, mostly with lots of drugs. We'll see if I make it in to $WORK
today or not.
What else have I been doing? Let's see…
- Our Roku SoundBridge M1001 has died after just over a year, which turns out to be a semi-common thing. Fortunately I've found a local electronic repair store that I'm hoping will fix things up. Other than that I've been fairly happy with it; it's definitely a lot handier with the SoundBridge Commander, though.
- Started playing with Xen on my web server. I'm going to try setting things up more complicated than they are now, but hopefully more secure: one instance to serve files, with /home (where all the sites are) mounted read-only; one with a database server; one for mail; one for SSH access; and one with the bag of passwords (LDAP, probably). Yeah, it's hopelessly baroque for a home server with, what, 8 websites, but it's a work in progress and it's fun to play. :-)
- Installed the Awesome window manager after reading about it for the nth time. Happy so far, but I'm still exploring.
- Tried making bread that rises overnight. I got into the habit of making my own bread at the beginning of the year, and I really got used to it. But two kids means much less time, and half an hour to mix and knead the dough is getting hard to find. So the latest trick has been using the KitchenAid and its dough hook (I did it all by hand previously), and fiddling with the amount of yeast so that, in theory, it'll rise overnight and be ready to bake when I get up at 5:30am to go to work. This used to work, but now I'm having problems with it collapsing. That may be because I'm using instant yeast rather than the regular, active stuff. Updates will follow as events unfold.
And that's that. Time to put together some lunch and see about going to work.
Tags:
geekdad
10 Jun 2008
$TTL
!= refresh
.
This lesson brought to you by the ISC, Cricket Liu and RFC 1035.
Tags:
rtfrfc
05 Jun 2008
Looks like we're going to be getting a bunch more Windows desktops in
the near future at $WORK, so I've been looking into Unattended
again. I'm having much better luck this time than the last time I
tried, a couple or three years ago.
I can't remember what went wrong then, but this time it took me a
stupidly long time to figure out that an error message about a
missing djgpp.env
file means you forgot to unzip the support
files under a directory called djgpp
.
Another thing that tripped me up this time: unattended installations
from OEM media are not allowed by default, even with a legit key from
the sticker on the side of your new shiny box. This mailing list
post pointed out the magic key, which seems to be working for mwe.
Now that I got those things sorted out, things are going much better.
Tags:
windows
installation
26 May 2008
...after a month off, and almost no emergencies in my
absence. Sweet!
Now if only I could catch up on sleep. I remember this from the first
kid: you never know just how much you can accomplish on so little
sleep.
Tags:
work
geekdad
20 Apr 2008

Tags:
geekdad
19 Apr 2008
As Project U-14 draws to a close, I've been spending a wonderful couple of hours on the couch playing with my laptop while Arlo naps.
Here's what I've been doing:
- Tracked down a bug that in gpsbabel that would make it crash if you gave it no filename to process. Turned out it was fixed late last year, and the new version will be in Lenny, but that was okay; the practice was good, it didn't take too long, and it turns out both the developers and I came to the same conclusion. I hope they know what they're doing, at least. :-) I'm unsure if I should file a bug in Debian or not; Lenny's coming soon, and I'm not sure of the usual practice in this case.
- Actually began work on Project U-13; I'm trying out fai-cd, rather than GRML. GRML's nice, but it's a way non-stock Debian install by the time it's done, and I'd like to stick as much as possible to stock Debian (OS of the OElder Goeds). We'll see how it goes.
- Read about how Dan Fucking Kaminsky managed to Rickrolled Facebook thanks to nasty DNS hijacking (similar to Verisign's Site Finder) by Earthlink and Comcast. No copy of the presentation up on his site, from what I can see, but I'm looking forward to it.
- Eating chocolate chip cookies my wife made. Yum.
Tags:
geekdad
14 Apr 2008
Heads up for those of you using Blastwave and CUPS: after upgrading to the latest stable version, printing stopped working for me (and a few users :-). I eventually tracked it down to the movement of two files: suddenly
/opt/csw/lib/cups/filter/pstopxl
/opt/csw/lib/cups/filter/pstoraster
were moved to
/opt/csw/lib/cups/pstopxl
/opt/csw/lib/cups/pstoraster
resulting in many error messages like Unsupported format text/plain! and Hint: is ESP ghostscript installed?
. Moving them both back into place and restarting CUPS fixed things just fine.
According to Bacula (yay Bacula!) both files were in the right directory as of last night, and Blastwave's file list for Ghostscript shows the new location for these two files. A bug has been filed.
Tags:
solaris
packagemanagement
bug
09 Apr 2008
This is one of the few things that would make me consider moving to the US right now.
Tags:
work
04 Apr 2008
There are always timesinks at a job: the things that suck up all your
spare time, that interrupt what you're doing and force you on to
something else. They're urgent, or they're complicated, or they're
obscure and you only ever touch them every six months. If you're
really unlucky, they're all three. They drain the life from you; a
good day turns shitty, and an already-shitty day becomes
nigh-unbearable.
The website is one such timesink at my current job. It's a veritable
Grand Canyon of different technologies, databases, and code. You can
examine it and, like a geologist, date particular pages or code with
great accuracy, judging by clues like composition, surroundings,
indentation patterns ("Oooh, K&R crossed with…crack?"), and
previous experience. When an Urgent Request for Web Changes comes in
(and they're all urgent), figuring out how to do it means figuring out
how that particular page was generated in the first place: static?
dynamic? CMS? And then you have to figure how you can meddle with it:
logging into Mambo, the CMS of the damned? If it's static: does the
URL map nicely to the filesystem, or is there a hidden Apache Alias
directive somewhere? Do you have permissions to open the file, or will
it take sudo
to chown it, or another nagging email to a coworker to
please check their changes into RCS? And if, God help you, it's
dynamic…but no; that mess of spaghetti should stay down. There's no
sense bringing it up again simply for prurient purposes.
Sunray terminals are another timesink. When they work they work very
well. I like the energy-saving aspects of it — both electrical and my
own; one machine to manage is always better than 40. But when they
don't work, it's a pain. Has a session become wedged? Is it GNOME's
fault? Has Adobe Acrobat decided to eat up all the CPU again? If so,
is that worse than the security holes that remain unfixed in the later
version? Why is Solaris 10 randomly not sending RST packets when it
receives a SYN on a port it's not listening on? (If anyone has any
ideas, please let me know.) Has a cheap switch, installed because
no one believed that an office meant for one might someday hold four,
gone off its meds again?
These things make me throw up my hands and and curse my fortune. I
have no one unfortunate enough to be my subordinate, so it's up to me
to hack and slash through the possibilities until it's finished, or at
least put off for another day.
But LDAP is worse.
When it works it works very, very well. Failover works, replication
works, and an account created here zips there without a moment's
thought. But when it fails, it's urgent and complicated and
obscure all at once, and sometimes in degrees polynomial.
At last count we have four different master-master replicas, running
three or possibly four different versions of Sun's Directory Server
(under six different product names, no less). There are replication
agreements spanning versions that aren't even supposed to tolerate
each other's existence, using two different encryption protocols and
NetBEUI. Two completely different "helpful" management tools vie for
our attention, lacking only flash plugins to trigger seizures. Only
one server can be poked or prodded with a command line
tool. Diagnostics are by turns nonexistent or endearingly fickle.
To be fair, the vendor documentation is vast and makes fine kindling,
though its promise to fully document error codes like error
457758854b: BER error 45775885b4
is best regarded as a bitter joke by
a jaded software engineer who died alone, unloved and without stock
options. (Our own documentation is marginally better: no carbon is
released when it is destroyed.) Thus, keeping track of ACLs (say), and
exactly which unholy wrath you will invite upon your head should you
make a mistake when granting or revoking privileges to read a
particular entry, means digging through half-remembered conversations,
drunken Google searches, year-old notebooks and a quiet, solitary
introspection normally reserved for contemplating your own impending
doom.
On top of everything else, LDAP encompasses everything, or nearly
so. Email routing, website privileges, database access, even TCP
checksum computation: all are kept in, or depend on, or just like to
hold hands with, LDAP. It's enough to make me wistful for the good old
days of NIS.
In a few minutes I am going to go back to work and try to figure out
why a new account has stopped, in mid-replication, halfway between
$UNIVERSITY and $OTHER_UNIVERSITY. It will take me the rest of the
afternoon. I will use words that my own son does not know I know. And
I will come out of it shrunken, withered, beaten down and humble.
Tags:
rant
ldap
30 Mar 2008
Yesterday I spent the day at work testing our installation of
APCUPd and tidying up the goram rat's nest of network and
electrical cables my predecessor left me.
APCUPSd worked with only a few hitches:
- I had one machine polling a UPS, and told it to shut down when there was 30% charge left. The other machines, which poll the master, were set to shut down 30 seconds after the power went out. They shut down, but that bumped up the charge reading on the battery because the load was that much lower. So I didn't get to test the automatic shutdown of the master.
- The other three machines were all set to shut down after 30 seconds; however, NFS cross-mounting made for problems with one of them. I'll need to stagger those three machines, whether they're looking at the charge or just shutting down n minutes after the power goes out.
- The Solaris 10 box shut down just fine, but when it restarted it did not let me log in — even in the console. Since Solaris 10's boot sequence is dead silent by default (thank you, Sun), it was hard to be sure what was happening. The last time I was patching this machine, reboots took 10 minutes; I gave 20 this time before giving up and going to single-user mode. The problem appears to be
/etc/nologin
, stuck there from the shutdown. This prevented a login prompt from coming up even in the console, without any sort of warning. Arghh.
As for the cleanup: satisfying. I'm no longer quite so ashamed of the server room.
Tags:
hardware
17 Mar 2008
Hah! Just listened to Season 5, Episode 13 of LugRadio, and they
read my letter! It was about how, if you're working tech support, you
usually have no idea whether or not the other person is actually
clueful or not; thus, the inevitable "Is it plugged in?" questions
which frustrate techies. Or, as Jono Bacon put it, you have no idea
whether the customer is "chuffing up the bong pole". (And the giant
round of Whiskey Tango Foxtrot? goes to…Jono Bacon! :-) So this is
what fame feels like.
Incidentally, LugRadio Live USA looks like it's going to be
freakin' sweet. I wish I could go, but Project U-14 is going
to be in pre-release testing at that point...
Tags:
geekdad
13 Mar 2008
USENIX has done a wonderful thing: their conference proceedings
are now open to the public, rather than requiring a USENIX
membership.
This is very, very good. If you haven't gone through the list of
presentations and papers from LISA, FAST, WOOT, or the
USENIX conference itself, you really need to.
Come to that, if you haven't picked up a membership yet to USENIX and
SAGE, you really need to. A dead-tree copy of ;login:
magazine is the most interesting single publication I've found about
computing in general, and system administration in particular. You owe
it to yourself.
Tags:
lisa
reading
27 Feb 2008
Airport Toy X-Ray Machine, c/o Saint Schneier. Maybe I'll get one for Arlo.
Tags:
geekdad
26 Feb 2008
I've been listening to the presentations from LISA07, and I have
a few observations.
Trey Darley's presentation reminded me a lot of my last job, but
much more intense: fast growth, no control, and no budget. The
difference is that he had the experience and the chops to deal with it
well. Also, if he can present at LISA, so can I.
Andrew Hume's presentation, "No Terabyte Left Behind", was
interesting, by which I mean frightening. People mostly just trust
that hardware does what it says it does/will do when it comes to
storage. But that doesn't always work: he tells the story of a prof he
worked with who checksummed all his files once a week. When a checksum
changed — and it did about every 6 months — he'd retrieve it from
backup. His rough guess for undetectable errors: 1 per 10
terabyte-years. And we're getting to the point where that's going to
be significant very soon.
Tony Cass' presentation on grid computing for CERN was
fascinating. This is the place I wanted to work (though as a particle
physicist). UBC/TRIUMF is doing some work for this project as
well, which makes me think I should jump over.
David Josephson's presentation was interesting, as much for the
Q&A afterward as for his point. Which was? Glad you asked: that
focussing on IP-based spam filtering (RBLs, greylisting) provides an
incentive to spammers to hijack network prefixes via BGP attacks, and
generally do nasty things to the Internet; please switch to
content-based filtering post-haste. (To clarify, he was talking in
particular about fast naive Bayesian classifiers, not SpamAssassin.)
Since IP-based filtering treats IPs as valuable things — tokens that
demonstrate your email is worth accepting — spammers steal IP
addresses.
I'm not sure how much I buy his argument; he kept promising that the
BGP attacks he described were only part of the problem, but he never
seemed to get beyond that. But during the Q&A Brad Knowles
got up and said (my summary) Content filtering doesn't scale, at
least in his experience (as Senior Internet Mail Systems Administrator
for AOL). At that point, another guy got up and said (again, my
summary) that sort of thing is heard all the time, but with no data
to back it up. The responder had co-authored a paper with Josephson
that got Best Paper award at LISA '04, and they'd made damn sure to
include a ton of footnotes. If their conclusions were wrong, people
were free to challenge them; if Knowle's were wrong, they were
unchallengeable because there was no data to back it up -- it was all
just story that got passed along and became myth.
Knowles' response was "I don't have time to write papers; I'm a
technician, not an academic." Which is true, in lots of ways. And I
don't mean any insult to Knowles; he's done things I will probably
never match, we are all flooded with work, and so on. I'm one guy,
working at a small shop, with none of his experience, or chops, or
rep, or audience.
But there's a reason my .signature says "Because the plural of
Anecdote is Myth": it's to remind me that unless you can back
something up with facts, preferably written down and logged and
repeatable, all you've got is a bunch of stories that become more and
more True the more you repeat them.
It's obnoxious to sneer and say, "Cite, please"; it's worse to be
ignorant.
Lots more listening to do. If you haven't downloaded them yet, you
really should.
Tags:
lisa
spam
22 Feb 2008
At last: I'm finally coming to the end of working with the verdammnt
web registration forms. We're going from our awful hack of a
glued-together mess of Mambo and custom PHP, to something that'll
mainly be Drupal with no custom code. Allegedly it's six weeks 'til
launch date; the registration forms in use right now will limp along
'til they're no longer needed (end of the summer).
The registration form I'm working on now is not complicated in the
absolute sense, but it's the most complicated one we've got. Last
year I was afraid to touch the (old, legacy, ugly) code, and mostly
just changed dates. This year I thought "fuck it" and rewrote nearly
all of it, using the tools and skills I'd picked up in the
meantime. (I'm still not a great programmer, understand, but I
have improved some over last year.)
After a full day banging my head against it, I'm finally coming to the
point where I'm pretty confident that the code will do what it's
supposed to. And that's a relief. Therefore, in the stylee du
Chromatic, I give thanks to:
In other news...just downloaded the second dev preview of Indiana,
which I'd managed to not hear about at all (the preview releases, that
is). I love university bandwidth; 640MB in about 1 minute. Sweet. I'll
give it a try at home and see how it feels.
I've just finished reading the summaries of LISA '07 in the
latest issue of ;login:. I feel…incredibly left out. I'm starting
to think this profession might not be such a simple thing, you
know, man? Sir? The presentations on autonomic computing have left me
feeling a bit like a buggy whip maker with his nose to the grindstone.
And yes, it's a way off, and yes, small shops and generalists will
probably be around for a while to come. But I'm not sure how much I
want to keep being at a small shop. Which means learning the big
stuff. Which, natch, is hard to do when you're trying to figure out
how to properly test registration forms. Sigh.
But: I just stuck my head out a door at work and saw a chickadee. It
chirped for a while, sitting on a tree near our building, then flew
off. On a rare sunny day in Vancouver in Frebruary, after a week of
messed-up sleep and feeling like I've been spinning my wheels, this is
nice.
Tags:
perl
web
programming
solaris
lisa
08 Feb 2008
I'll be damned: a GPL'd Windows Eventlog-to-syslogd interface. Thanks to Will for the pointer!
Tags:
windows
08 Feb 2008
New Dell 2950 server. 2 x quad-core Xeons, 2 x 6MB cache on each die,
16GB RAM, 6 x 300GB SAS 10K SCSI drives in a RAID-6 array using the
PERC/6 controller.
/usr/src/linux-source-2.6.18# time make -j 9 bzImage
[snip]
Root device is (8, 3)
Boot sector 512 bytes.
Setup is 7295 bytes.
System is 1222 kB
Kernel: arch/i386/boot/bzImage is ready (#1)
real 0m22.668s
user 2m20.425s
sys 0m14.537s
That's just insane.
Tags:
hardware
linux
dell
06 Feb 2008
Okay, so it isn't quite as bad as the time I threw 3,000 incoming
messages for an ISP into my home directory. But I've just figured out
that the reason a) $VENDOR didn't get back to me and b) it's been so
quiet for the last few days is because all email was going to a file
called X-Original-Sender
because of one missing *
. (In fact, that
may also have been the cause of the first big error...)
God, I hate procmail sometimes.
Tags:
screwup
procmail
rant