As mentioned on Undeadly.org and openbsd-misc, OpenBSD is
asking for donations for BGP routers and a new CVS server. I've
donated, since I wouldn't be able to do half my job without them; if
you feel the same and can spare some money, I urge you to do the same.
I was >this< close to writing my own damn set of Perl scripts to
test a firewall, but I decided to search one last time. Good thing,
too: ftester looks pretty close to perfect.
I'm having trouble right now getting ftestd to work on an OpenBSD 4.3
system; this may be because I'm trying to get it to listen on an
interface that's part of a bridge. I'll have to look into this
further. But testing it out between my laptop and desktop works a
treat, whether my laptop is running OpenBSD or Linux 2.6. Sweet!
When I was at LISA, one of the sysadmins I met mentioned a firewall
unit testing script that a coworker of his had come up with. The idea
was to run your OpenBSD firewall in a QEMU instance, then try passing
traffic back and forth to make sure everything worked as
expected. I've been looking for that tool to be released, but haven't
seen it....or anything else like it either…
Until today, that is, when I stumbled on NetUnit. It's a
Java-based tool that tests basic network connectivity, using XML files
to specify tests. So far he's got tests for ICMP/port 7 (which I never
knew was the echo port), TCP ports, HTTP/HTTPS and MySQL. Not bad
at all, except for my lack of Java experience.
Of course, now I want to write my own tester using Perl and QEMU. Like
I've got time. But here's an idea for anyone who can use it: test your
firewall using three instances of QEMU (inside, outside and firewall),
and have the inside and outside hosts communicate using the serial
port. "I'm gonna send an echo request, did you see it?" "Yes, did you
see the reply?" It's a bit more feedback than simply noting the lack
of the expected reply.
And it's not at all like conversations that start out with, "I sent
you an email. Did you get it?"
Arlo's sick with flu or something; I was up 'til 1am last night
rocking him to sleep. Haven't done that in a while…
Telling detail: I'm about to blow away Debian testing on my desktop
machine and install Ubuntu's Gutsy Gibbon. Partly it's because I'm
tired of installing 80MB worth of updates every two weeks, and partly
it's because it'll make setting up the printer a breeze.
I'll probably leave half the drive aside for good ol' Debian stable,
but Ubuntu'll stay there for experimenting and so my parents, on their
next visit, will not have to bring out their 4-tonne laptop.
I'll be reinstalling Ubuntu on my laptop as well; due to a stupid
error, I installed Dapper, not Gutsy. I tried updating in one fell
swoop, and after three days of apt-get -f install I finally got
things working…except for the boot artwork, and GDM doesn't start one
time out of three. Interesting experiment, but I think I'll take a
I may even install it twice, so that I can try out The
Depenguinator, which appears to be a lot easier than trying to
figure out PXE booting for FreeBSD. Unlike OpenBSD, there's no readily
apparent "official way" of doing it, and the handful of HOWTOs I've
found have contradicted each other. At this point I'm just too lazy to
keep trying and seeing what I'm doing wrong.
My laptop hard drive started giving scary errors a couple days ago on
the way to work (I've got a 90-minute commute by public transit [uck]
so I fill the time by reading, listening to podcasts, or working
on Project U-13). Fortunately, working at a university means
that there are two computer stores on campus. I ran out at lunch,
picked up a 100GB drive, and had things back to normal by the next
Well, normal modulo one false start with Debian; I decided to try
encrypted filesystems just for fun. But then I suspended, came back
with a newere kernel, and it could not read the encrypted LVM group
Still lots of free space on this thing, and I'm thinking of installing
Ubuntu, FreeBSD and maybe NetBSD just for fun. Of course, I've got to
do it all via PXE since this thing doesn't have any CDROM drive, but
that just adds to the geek points.
Project U-13 is coming up on 0.0.3, btw; Andy suggested
adding Rackmonkey, which looks quite cool. There's no package for
it, so I'm having to do some rather ugly scripted installation…but I
can stand it for now. And I've got the barest skeleton of a cfengine
file in there too. Watch the skies!
Dude, my laptop screen just turned blue. I'd booted into OpenBSD (4.2) and was trying to figure out how to turn off the audible bell. I'd gone from X to a virtual console to see if the problem happened there (it did), then tried ctrl-alt-f5 to get back to X.
My laptop screen turned from black with white text to grey with grey text to light blue with dark blue text, over the course of a minute or so. I thought I'd suddenly borked the LCD screen, but when I rebooted to Debian it was all fine. Just tried switching to a console, then back to X (alsoin Debian), and that's fine too. Bizarre.
Just checked the logs in OpenBSD and found a series of entries like this:
Nov 1 16:47:17 laptop /bsd: agp_release_helper: mem 0 is bound
Nov 1 16:47:17 laptop /bsd: agp_release_helper: mem 1 is bound
Nov 1 16:47:17 laptop /bsd: agp_release_helper: mem 2 is bound
Nov 1 16:47:17 laptop /bsd: agp_release_helper: mem 3 is bound
Nov 1 16:47:17 laptop /bsd: agp_release_helper: mem 4 is bound
Nov 1 16:47:24 laptop /bsd: agp_release_helper: mem 5 is bound
Nov 1 16:47:24 laptop /bsd: agp_release_helper: mem 6 is bound
Nov 1 16:47:24 laptop /bsd: agp_release_helper: mem 7 is bound
Nov 1 16:47:24 laptop /bsd: agp_release_helper: mem 8 is bound
Nov 1 16:47:24 laptop /bsd: agp_release_helper: mem 9 is bound
Nov 1 16:47:31 laptop /bsd: agp_release_helper: mem 10 is bound
Nov 1 16:47:31 laptop /bsd: agp_release_helper: mem 11 is bound
Nov 1 16:47:31 laptop /bsd: agp_release_helper: mem 12 is bound
Nov 1 16:47:31 laptop /bsd: agp_release_helper: mem 13 is bound
Nov 1 16:47:31 laptop /bsd: agp_release_helper: mem 14 is bound
Nov 1 16:47:38 laptop /bsd: agp_release_helper: mem 15 is bound
Nov 1 16:47:38 laptop /bsd: agp_release_helper: mem 16 is bound
Nov 1 16:47:38 laptop /bsd: agp_release_helper: mem 17 is bound
Nov 1 16:47:38 laptop /bsd: agp_release_helper: mem 18 is bound
Nov 1 16:47:38 laptop /bsd: agp_release_helper: mem 19 is bound
Very weird. On the bus, so Googling that'll have to wait. Although I do have the code on that partition…here we go: says it's the AGPIOC_RELEASE ioctl for agp. Aha! Maybe I'll explain money laundering while I'm at it.
And btw, here's a memo for the world: if you're on the toilet, don't take a phone call. It's really not that important.
Update, October 15 2008: Still happening with OpenBSD 4.3. And for the record, this is a Dell C300 laptop.
I ordered the 4.2 CD set of OpenBSD at work, in another optimistic
step toward reorganizing the firewall there. In order to (ahem)
road-test it, I installed it on my new laptop (which, you'll
recall, is running Debian Stable) in a 5GB partition I'd left for
just this purpose.
Onboard wireless, like with Debian, did not work, and I didn't expect
it to; fuck you too, Broadcom. But my dad offered to send out a couple
of wireless cards he couldn't use, and I figured one of 'em would have
One was a Broadcom (op cit.), so that was out. The other, a DWL-650
(which appears to have umpty different versions over the years with
not one change in model number) looked promising: a Realtek chipset,
so should be good, right?
Well, it worked on OpenBSD -- but not in Linux. There's no driver in
the tree for it, and the outside project to make drivers for it
had its last official release in 2005. What's more, the CVS
version, for some reason, removes all of its source files when I
compile it, then complains that there are no files left to compile. To
be fair, I think this is because of a makefile included from
/lib/modules/2.6.22-2-686/build rather than the code itself.
What OpenBSD does not do on this laptop is suspend -- or more
accurately, come back from suspension. This works reasonably well
under Debian, which means that I still have one rose to give away to
The Next Laptop OS for Saint Aardvark.
The laptop I bought off eBay arrived at work on Wednesday...which is my
day at home with Arlo. Thursday I was off sick with flu. Yesterday I
was back at work and slashing open the box it came in, eager to see
what I'd got.
Well, I already knew: it's a Dell C400. 12" screen, 1.2GHz P3 (but
running at 800MHz with SpeedStep and all), 256MB RAM and a 30GB
drive. Not a whole lot of memory, and a bigger hard drive would
always be nice, but I can always upgrade. There's no CD drive in this
thing, and I hadn't plumped for the docking station, so I set up PXE
booting to install Debian. It was a trifle slow, but it worked!
(Especially the second time, after I'd accidentally overwritten Debian
trying to install OpenBSD on another partition. :-)
I'm surprised at how much Just Works in this thing: X.org (no
configuration needed, just start up XDM...mann, that's nice),
suspend-to-disk, ethernet (well, it's a 3c905; what do you
expect?). Even the battery, which I'd written off in advance, appears
to hold a decent charge -- about four hours so far. The one thing
that's dicy is the onboard wireless, a Dell 1370 from everybody's
favourite company. But again, I'd written that off in advance.
Next up: I've ordered the OpenBSD 4.2 CD set, so I'll be
installing that once it arrives. And Noah has shown the way to
longer battery life; I'm getting my 2.6.22 kernel now from
Backports. (Oh, the shame of not compiling my own kernel...)
On another note, I think someone had one too many Dilbert moments:
$ dig newcastle.edu.au mx
; <<>> DiG 8.3 <<>> newcastle.edu.au mx
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->> HEADER <<- opcode: QUERY, status: NOERROR, id: 2
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 4
;; QUERY SECTION:
;; newcastle.edu.au, type = MX, class = IN
;; ANSWER SECTION:
newcastle.edu.au. 11h59m12s IN MX 10 proactive.newcastle.edu.au.
newcastle.edu.au. 11h59m12s IN MX 10 synergy.newcastle.edu.au.
Just updated my resume for the first time since starting my
current job. It's nice to look back at what you've done and
realize that, hey, there's been a lot.
In other news, I finally gave in to lust the other day and bought a
Dell C400 on eBay. Nothing too special — 1.2GHz, 256MB, 30GB hard
drive — but I was mainly after the 12" screen, so that I'd be able to
(say) debug raw ethernet frames on my daily commute. About $280 when
all was said and done; the strong Canuckistan peso was part of the
incentive to buy now. Should be at the office in a week or so, and I
It amazed me to see how many off-lease laptops were available, and
just how cheap you could pick them up. A whilte back my boss got
a D420; with extra memory and a few other things, it came in at about
$1700 or so Canadian. But if you look around, there are plenty of
D400s and D410s around for less than $500 — even less than $400 if you
look hard. Add another $100 (say) for a working battery, and you're in
pretty good shape.
On Tuesday, I'm giving a short presentation on my work's subnet at
SNAG, the UBC System and Network Administrator's Group. I found
Bruce in OpenBSD's ports tree on my laptop; the documentation is
(ahem) thin, but it works. Wish me luck.
Now that Clara's heading back to work, my schedule has changed a
bit: I'm staying at home on Wednesdays to take care of Arlo, and then
working from home on Saturdays to make up the time. I'm grateful to my
boss for letting me do this, and I'm hopeful it will work out.
My first Wednesday (July 4th) went pretty darned well, really. Arlo
ate, he played, he got vaccinated (Chicken pox; I had no idea they
vaccinated for it), he napped and then he played some more. I didn't
drop him, he didn't freak out and it was a great deal of fun.
As it happens I got to take care of him on Friday, too; my
mother-in-law, who's going to be taking care of him two days a week,
had a sudden trip to the emergency room. She's okay, but wasn't able
to take care of him that day. (She was mad about it, too...) I called
into work and let them know I wouldn't be in, then went in anyway just
to make sure a few things were okay. I've got some karma built up and
a fistful of sick days I rarely take, so all was well.
And then yesterday I worked from home. And man o man, did I get stuff
done. Not quite as much as I wanted; I was hoping to use flar to
duplicate a Solaris machine so I could test it, and ran into a bug
that took a while to figure out. (If the patch I applied fixes the
problem, I'll write it up here since there was only one other
reference I could find.) But it was lovely to work for, like, four
hours in a row on something and not be interrupted. Plus, there's
the skipping of the 90-minute commute to enjoy.
For a while now I've been irritated with the behaviour of OpenRCS and
Emacs on my OpenBSD machines: every time I try to check out a file
kept in RCS, using C-x v v (vc-next-action), I still have to
toggle read-only status on the file. Then, when I try to check it in,
it asks if I want to steal the lock from myself, and never actually
checks it in.
Finally had some time to track this down, and this bug appears to
be the cause. I may have to play around with Emacs a bit to get it to
ignore the permissions, or I may just use the OpenBSD package for GNU
When trying to install OpenBSD to a Sun Ultra 1 workstation over the
network, I got the Fast Data Access MMU Miss error when running
boot net bsd.rd. Turned out I'd copied the wrong boot loader to the
TFTP directory; copying ofwboot.net over it fixed the problem.
Arghh. For weeks now, I've been trying to track down why a couple of
XP laptops have had random print jobs drop to the floor. I finally got
to the point last week where I could reliably duplicate the problem
(print four emails from Outlook in quick succession; only three show
up, no error on the printer), and today I spent six hours figuring out
where the hell the problem was. (I didn't intend to spend that long,
but the combination of vociferous complaints and sheer bull-headedness
got to me.)
For no particularly good reason, the laptop in question is set to
print to the local HP 4200 using IPP. When I looked at the traffic in
Ethereal, I noticed that the failing job had a subtly different
response to the print job submission from the printer, and at the
end the TCP stream was only closed by the laptop -- the printer ACKed
right away but did not FIN its end. Aha! Firmware bug!
The printer repair guy who's been working with me to try and fix this
stopped by to take a look, and decided to call HP support. Their
response: Don't Do That, Then. Apparently, IPP is a weird protocol to
use for a LAN and I should really print to port 9100 like everyone
Okay, yes, this worked, and it was a stupid amount of time to spend on
this problem. But it irritates me that they weren't interested in
(what I think is) a firmware bug, and that I'll never probably never
get to the bottom of what was going on. Although I'm pretty sure that
the JetDirect card just uses an embedded ARM processor; I could just
try looking at the firmware with a disassembler...:-)
In other news, something's going subtly wrong with the WRT54G; the
bridging of OpenVPN's tap0 interface and the external ethernet
interface has stopped working. The internal ethernet interface still
works, and if you SSH in that way and run ifconfig vlan0 down ;
ifconfig vlan0 up the external interface starts working again. I'm
also having problems with the wireless interface. I suspect the
bridging may be involved there, too, since it's bridged with the
internal ethernet. However, I only have my wife's iBook to test with,
so I can't be sure it's not a problem with that.
And my OpenBSD 3.9 CDs are in. Hurray! Time to finally get this
firewall off my desktop machine.
Top Tip: Filenames with a tilde in them can confuse Samba.
Case in point: last week a user was
having problems loading his profile: W2K kept choking and saying that
the file Local Data\Applications\foo\backup\~AvariciousMonkeys.c was
in use. Naturally, lsof on the Samba server turned up nothing, and I
couldn't see any obvious problem. On a hunch, I tried renaming the
file to AvariciousMonkeys.c~, and hey presto! goodness all
This week I'm trying to get FAI going in seriousness. I've worked on
it before, but now I've got three developers who want to switch to
Linux. The last thing I want is another series of one-offs, so I'm
taking the time to do it right. Now there's a CD version in beta, and
so far it's working well. Cf. the usual way of doing it, which is to
do PXE booting and grab everything off the network. I'm not opposed to
that, but one of the things I wanted out of FAI before was the ability
to do CD-based, kickstart-like Debian installs; looks like it's
finally going to work.
Looks like we're having a problem with a Maxtor PCI IDE controller and
the Intel mobo in our backup server. It's been mysteriously crashing
in the middle of the night w/no log messages. Some checking in the
BIOS turned up another problem: going to the hardware monitoring page
to look at the CPU temperature made the damn thing freeze. WTF? Sure
seems like the symptom we were seeing, and backups running at night
make big use of the Vinum array that uses drives attached to the IDE
adapter...long story short, taking out the card stopped the BIOS
freezing. It remains to be seen if it'll work for the random midnight
freezes, but it's good to have something to try. I'm hopeful that FreeBSD will be able to handle SATA drives attached
to this thing...we'll have to see.
Which brings me to the next bit: fleshing out plans for server
upgrades. As I mentioned, last week we had a power supply fail on our
Very Important Server, and I want to try and keep that from happening
again. Of course, adding umpty thousand dollars worth of hardware to
your budget four months before the end of fiscal doesn't really work
too well, so as much as possible I need to do this w/o new
hardware. Ha! But I'll give it a try.
First off is setting up OpenLDAP and importing Samba's information
into it. That'll be neat, since I've never worked w/LDAP
before. Second is to set up some BDCs using OpenLDAP to query the
master. (Or do they just suck over the whole database? Hm. Either
way.) Third is to set up some Linux machines. Why? Two reasons:
LinuxHA + DRBD
LinuxHA and DRBD seem fantastic, and there just doesn't seem to be
anything comparable on the FreeBSD side. As for the hardware...well,
my first impression of server hardware from IBM, HP and the like (no,
don't talk to me about Dell) is that I'm going to need a newer version
of FreeBSD than we currently use in order to run SATA drives. (I know
SCSI is the way to go, but I was quoted two thousand dollars for
two IBM 73GB 15k drives! I know: 15k, IBM, etc, but even halving that
means two -- two! -- 73GB drives for a thousand bucks, a/o/t two 200GB
drives for, what, four hundred. Heh.)
We're using an older version of the 4-series FreeBSD here. I've
already set up one server using a newer 4-series release, and it's a
pain: too many differences, one more thing to keep in mind when making
changes, and so on. I haven't worked with the 5-series yet, and I
don't want to start now...not entirely sure that it'd work for
us. Plus, we'll probably migrate to Linux anyway, so I don't mind
doing it for a server.
Anyhow! Get a Real Server and throw Linux on it. Hook it up to our
drive array and start migrating home directories to ReiserFS from
UFS/FreeBSD. Not trivial, but doable. Add more Linux servers as budget
Well, I did the right thing today -- twice. Damn right I'm
First off, it turns out that the FreeBSD Foundation has run into
a (good!) problem: its donations have been too big. In order to keep
its US charitable status, it needs to have two-thirds of its donations
be relatively small. Due to a couple of big donations, this ratio is
a little out of whack at the moment, and they need a bunch of
Welp, I've been administering FreeBSD systems for a living
for...well, I was gonna say four years, but it's more like two and a
half or three. I've been working on them for four, though; my rent
and food has been paid in large part because of the generosity of the
people who put together FreeBSD. A donation went off in short
Then I remembered that I've been meaning to join the Free Software
Foundation for a while now. The motivation is the same: I've been
paying my bills for a long time now (and enjoying myself immensely in
the process) because of the generosity of Free-as-in-Freedom
software people: Stallman, Torvalds, Wall, and a
zillionothers. I have a hard time imagining what I'd be
doing now without Free software; I suspect that, if I was lucky, I'd
be working as a grocery store manager right now. So: off to the FSF
website to sign up for an associate membership.
And what did I find but two, count 'em TWO cool things:
If you refer three people to the FSF for associate memberships, RMS
or Eben Moglen will record a message for you, suitable for voicemail,
Hallowe'en or impressing the ladies. I did a quick search on Google,
but couldn't find anyone with the link...damn shame. Better than a
free iPod, cooler than a CmdrTaco TiVo -- join the FSF and get
RMS to say "All Hail Liddy!"
The FSF is looking for a senior sysadmin. God, that'd be
cool. Decent enough pay (no, it's not the sort of job you take
because of the money, but it's nice to think about), all the Free
software you can handle, and an IBM Thinkpad to run it on. Of course,
I think I'd have some 'plainin' to do about the laptop I'm writing
this on...and, of course, it would mean living in the US. Frankly,
that scares the crap out of me these days. Goddamned PATRIOT Act...
In other news, work continues apace. We're losing two coop students
and gaining one, gaining another full-time person, and I'm still
trying to get my RAID array -- credit app is with the boss, and
after that's done the order'll finally go in.
Rough guess (wild hope) at this point is that it'll be in my hands in
mid-January, which won't be a moment too soon. There's a new Linux
server I'm setting up that I'm desperately hoping won't have problems
due to proprietary kernel modules in the software I'm installing. (I'm
just writing myself further and further out of that job, aren't I?)
And I'm wondering if the simplest way to get Nagios to make sure the
right machines are exporting the right filesystems is to check if amd
is mounting them correctly. (No matter whether the machine or amd
fails, something needs to be fixed.) Or maybe I just need to figure
out the right wrapper for showmount -e.)
On the spam front: good god, what a smoking hole Movable Type is
turning out to be. First there were the license changes, then the
commentspammers (who seem to be posting a lot more
aggressive to MT than to WordPress)...Of course, comment
spam affects all blogs, not just MT. Still, this whole idea of
rebuilding static pages every time the stars move seems to be causing
them a lot of trouble. (Yep, that last sentence was pure FUD. Or
bullshit.) And okay, no, I don't use MT, so what precisely is my beef?
As I'm not going to put up, I should shut up. I still have to upgrade
WP -- though according to this posting, there are still lots of
XSS issues left unfixed. I'm also upgrading PHP, and I should
probably use ApacheToolbox to do that automagically, rather than
periodically editing my own Makefile.
The release party for Where Are They Coming From? came off JUST
FINE, thank you. EVERYONE was there. Top Stars include Topo,
Phil Knight and Mos Def, fresh from the set of HHGTTG. Uh huh.
Further thoughts on the MySQL + GPhoto2 thing: gphoto2 does have
the ability to pipe to STDOUT, which I don't think I knew...maybe it
won't be as much work to insert directly into a database as I
thought. Might even be able to do it as a Perl script.
Finally: what a gorgeous day. It's downtown Vancouver on the back
steps of the Art Gallery, it's sunny (in December, too) and just cold
enough to make you go "brr". The skater kids are practicing their
synchronised jumping -- just in time for the Olympics, I'm sure. A
far-too-generous co-worker has handed out chocolate, another has
handed out home-made rum and brandy balls, and I'm taking off early
to go drinking with a third. Feeling pretty damned good right
Update: Too bad Topo's not so great -- fever of 102.8F, as of
a couple minutes ago. (Still haven't figured out what that is in
Celsius; bad Canuckistanian!) It's down a bit from earlier this
afternoon, though, so I'm thinking good things. And thesepages say to not worry if it's less than a couple days, so I'm
not worrying. Nope.
I got into work today and found that the mail server had just come up
after *half a fucking hour* of being down because of the insane
load placed on it by spam -- just spam -- coming in. The owner of the
company couldn't send email. I started setting up the new mail server.
And it was nice. I got to go away, away from the help desk, sit down
and figure out how to make it work. FreeBSD's vinum + Promise raid
controller == kernel panic (details later on). Finally got vinum
figured out -- I've only worked w/it once before -- and before I was
grabbed back to help desk had the disk setup about 80% done.
So some more details: there's 4 x 40GB maxtor IDE drives. (Yeah yeah
yeah SCSI.) We've got an onboard Promise controller chip; I'll put in
the mobo tomorrow and make this all seamless. First it turns out we've
got the Promise Lite (Less Filling!) BIOS, which means we can only
have one (1) array of two disks; the other two disks can be single
arrays on their own, which is useful in some alternate universe I'm
sure. So okay, try setting up one mirrored (Raid 1? 0? I can't keep
'em straight) array, and we'll use vinum to tie it together with the
other single drives...
Only as soon as I try using vinum to do _anything_ with the
Promise'd arrays, BANG: kernel panic. This is 4.6, not the latest
(4.7RC1 as I type), but still. Arghh. Doesn't matter whether vinum
tries raid 0, 1 or 5 -- just panics right away. If I had more time and
a box of my own to fool around with, I'd try [Michael
SlashdotJournal_25September2002-02 (Buy his book!) and
contribute something useful to the FreeBSD folk. Alas, it's not my box
or my time, and if I were to post this message to
freebsd-hackers-important-vinum-people tomorrow I'd (deservedly) get
laughed at so hard I'd feel it over the ether.
Anyway. Point is I can't get vinum to play nice w/the Promise'd chip
even as an IDE controller. The BIOS of the box allows you to turn the
Promise chip on, off, or to ATA/IDE; but even set to the latter, it
panics once vinum touches /dev/ar*. You have been warned.
So get vinum using the four drives on the first two IDE channels, and
that works fine once I learn the intricacies of disklabel (set type to
vinum, kids!) and vinum init (and that takes a long time w/3*35GB
partitions^H^H^H^H^H^H^H^H^subsooperplexen). 1 5m 5o 133t!
OT: One of my side notes was going to be about how I'm posting this
w/Lynx 'cos Mozilla won't let me use vi, editor of the Elder Gods, as
an editor. Then I realized I could have just fired up a shell and used
vi in there. Sigh. Rumours of my cleverness have been exaggerated.