The Life of a Sysadmin

Update time.

I got into work today and found that the mail server had just come up after *half a fucking hour* of being down because of the insane load placed on it by spam -- just spam -- coming in. The owner of the company couldn't send email. I started setting up the new mail server.

And it was nice. I got to go away, away from the help desk, sit down and figure out how to make it work. FreeBSD's vinum + Promise raid controller == kernel panic (details later on). Finally got vinum figured out -- I've only worked w/it once before -- and before I was grabbed back to help desk had the disk setup about 80% done.

So some more details: there's 4 x 40GB maxtor IDE drives. (Yeah yeah yeah SCSI.) We've got an onboard Promise controller chip; I'll put in the mobo tomorrow and make this all seamless. First it turns out we've got the Promise Lite (Less Filling!) BIOS, which means we can only have one (1) array of two disks; the other two disks can be single arrays on their own, which is useful in some alternate universe I'm sure. So okay, try setting up one mirrored (Raid 1? 0? I can't keep 'em straight) array, and we'll use vinum to tie it together with the other single drives...

Only as soon as I try using vinum to do _anything_ with the Promise'd arrays, BANG: kernel panic. This is 4.6, not the latest (4.7RC1 as I type), but still. Arghh. Doesn't matter whether vinum tries raid 0, 1 or 5 -- just panics right away. If I had more time and a box of my own to fool around with, I'd try [Michael Lucas'|http://www.oreillynet.com/pub/a/bsd/2002/03/21/Big_Scary_Daemons.html] [SlashdotJournal_25September2002-02]1 (Buy his book!) and contribute something useful to the FreeBSD folk. Alas, it's not my box or my time, and if I were to post this message to freebsd-hackers-important-vinum-people tomorrow I'd (deservedly) get laughed at so hard I'd feel it over the ether.

Anyway. Point is I can't get vinum to play nice w/the Promise'd chip even as an IDE controller. The BIOS of the box allows you to turn the Promise chip on, off, or to ATA/IDE; but even set to the latter, it panics once vinum touches /dev/ar*. You have been warned.

So get vinum using the four drives on the first two IDE channels, and that works fine once I learn the intricacies of disklabel (set type to vinum, kids!) and vinum init (and that takes a long time w/3*35GB partitions^H^H^H^H^H^H^H^H^subsooperplexen). 1 5m 5o 133t!

OT: One of my side notes was going to be about how I'm posting this w/Lynx 'cos Mozilla won't let me use vi, editor of the Elder Gods, as an editor. Then I realized I could have just fired up a shell and used vi in there. Sigh. Rumours of my cleverness have been exaggerated.

Original entry.

growisofs will overwrite a DVD+RW when run non-interactively iff you use the option "-use-the-force-luke". Heh. Read the source for details.

To make a bootable FreeBSD CD, it's not enough to simply dd of=iso and burn that. Do this instead:

    mkisofs -R -b boot/cdboot -no-emul-boot -o foo.iso /tree/to/copy

This hints brought to you by the letter J, the letter O, and the number pi.

Original entry

Well, I did the right thing today -- twice. Damn right I'm bragging.

First off, it turns out that the FreeBSD Foundation has run into a (good!) problem: its donations have been too big. In order to keep its US charitable status, it needs to have two-thirds of its donations be relatively small. Due to a couple of big donations, this ratio is a little out of whack at the moment, and they need a bunch of small donations.

Welp, I've been administering FreeBSD systems for a living for...well, I was gonna say four years, but it's more like two and a half or three. I've been working on them for four, though; my rent and food has been paid in large part because of the generosity of the people who put together FreeBSD. A donation went off in short order.

Then I remembered that I've been meaning to join the Free Software Foundation for a while now. The motivation is the same: I've been paying my bills for a long time now (and enjoying myself immensely in the process) because of the generosity of Free-as-in-Freedom software people: Stallman, Torvalds, Wall, and a zillion others. I have a hard time imagining what I'd be doing now without Free software; I suspect that, if I was lucky, I'd be working as a grocery store manager right now. So: off to the FSF website to sign up for an associate membership.

And what did I find but two, count 'em TWO cool things:

1. If you refer three people to the FSF for associate memberships, RMS or Eben Moglen will record a message for you, suitable for voicemail, Hallowe'en or impressing the ladies. I did a quick search on Google, but couldn't find anyone with the link...damn shame. Better than a free iPod, cooler than a CmdrTaco TiVo -- join the FSF and get RMS to say "All Hail Liddy!"

2. The FSF is looking for a senior sysadmin. God, that'd be cool. Decent enough pay (no, it's not the sort of job you take because of the money, but it's nice to think about), all the Free software you can handle, and an IBM Thinkpad to run it on. Of course, I think I'd have some 'plainin' to do about the laptop I'm writing this on...and, of course, it would mean living in the US. Frankly, that scares the crap out of me these days. Goddamned PATRIOT Act...

In other news, work continues apace. We're losing two coop students and gaining one, gaining another full-time person, and I'm still trying to get my RAID array -- credit app is with the boss, and after that's done the order'll finally go in.

Rough guess (wild hope) at this point is that it'll be in my hands in mid-January, which won't be a moment too soon. There's a new Linux server I'm setting up that I'm desperately hoping won't have problems due to proprietary kernel modules in the software I'm installing. (I'm just writing myself further and further out of that job, aren't I?)

And I'm wondering if the simplest way to get Nagios to make sure the right machines are exporting the right filesystems is to check if amd is mounting them correctly. (No matter whether the machine or amd fails, something needs to be fixed.) Or maybe I just need to figure out the right wrapper for showmount -e.)

On the spam front: good god, what a smoking hole Movable Type is turning out to be. First there were the license changes, then the comment spammers (who seem to be posting a lot more aggressive to MT than to WordPress)...Of course, comment spam affects all blogs, not just MT. Still, this whole idea of rebuilding static pages every time the stars move seems to be causing them a lot of trouble. (Yep, that last sentence was pure FUD. Or bullshit.) And okay, no, I don't use MT, so what precisely is my beef?

As I'm not going to put up, I should shut up. I still have to upgrade WP -- though according to this posting, there are still lots of XSS issues left unfixed. I'm also upgrading PHP, and I should probably use ApacheToolbox to do that automagically, rather than periodically editing my own Makefile.

The release party for Where Are They Coming From? came off JUST FINE, thank you. EVERYONE was there. Top Stars include Topo, Phil Knight and Mos Def, fresh from the set of HHGTTG. Uh huh.

Further thoughts on the MySQL + GPhoto2 thing: gphoto2 does have the ability to pipe to STDOUT, which I don't think I knew...maybe it won't be as much work to insert directly into a database as I thought. Might even be able to do it as a Perl script.

Finally: what a gorgeous day. It's downtown Vancouver on the back steps of the Art Gallery, it's sunny (in December, too) and just cold enough to make you go "brr". The skater kids are practicing their synchronised jumping -- just in time for the Olympics, I'm sure. A far-too-generous co-worker has handed out chocolate, another has handed out home-made rum and brandy balls, and I'm taking off early to go drinking with a third. Feeling pretty damned good right now.

Update: Too bad Topo's not so great -- fever of 102.8F, as of a couple minutes ago. (Still haven't figured out what that is in Celsius; bad Canuckistanian!) It's down a bit from earlier this afternoon, though, so I'm thinking good things. And these pages say to not worry if it's less than a couple days, so I'm not worrying. Nope.

(Note: this was actually written back in May.)

Top Tip: Filenames with a tilde in them can confuse Samba.

Case in point: last week a user was having problems loading his profile: W2K kept choking and saying that the file Local Data\Applications\foo\backup\~AvariciousMonkeys.c was in use. Naturally, lsof on the Samba server turned up nothing, and I couldn't see any obvious problem. On a hunch, I tried renaming the file to AvariciousMonkeys.c~, and hey presto! goodness all over.

This week I'm trying to get FAI going in seriousness. I've worked on it before, but now I've got three developers who want to switch to Linux. The last thing I want is another series of one-offs, so I'm taking the time to do it right. Now there's a CD version in beta, and so far it's working well. Cf. the usual way of doing it, which is to do PXE booting and grab everything off the network. I'm not opposed to that, but one of the things I wanted out of FAI before was the ability to do CD-based, kickstart-like Debian installs; looks like it's finally going to work.

Looks like we're having a problem with a Maxtor PCI IDE controller and the Intel mobo in our backup server. It's been mysteriously crashing in the middle of the night w/no log messages. Some checking in the BIOS turned up another problem: going to the hardware monitoring page to look at the CPU temperature made the damn thing freeze. WTF? Sure seems like the symptom we were seeing, and backups running at night make big use of the Vinum array that uses drives attached to the IDE adapter...long story short, taking out the card stopped the BIOS freezing. It remains to be seen if it'll work for the random midnight freezes, but it's good to have something to try. I'm hopeful that FreeBSD will be able to handle SATA drives attached to this thing...we'll have to see.

Which brings me to the next bit: fleshing out plans for server upgrades. As I mentioned, last week we had a power supply fail on our Very Important Server, and I want to try and keep that from happening again. Of course, adding umpty thousand dollars worth of hardware to your budget four months before the end of fiscal doesn't really work too well, so as much as possible I need to do this w/o new hardware. Ha! But I'll give it a try.

First off is setting up OpenLDAP and importing Samba's information into it. That'll be neat, since I've never worked w/LDAP before. Second is to set up some BDCs using OpenLDAP to query the master. (Or do they just suck over the whole database? Hm. Either way.) Third is to set up some Linux machines. Why? Two reasons:

• LinuxHA + DRBD
• Server Hardware

LinuxHA and DRBD seem fantastic, and there just doesn't seem to be anything comparable on the FreeBSD side. As for the hardware...well, my first impression of server hardware from IBM, HP and the like (no, don't talk to me about Dell) is that I'm going to need a newer version of FreeBSD than we currently use in order to run SATA drives. (I know SCSI is the way to go, but I was quoted two thousand dollars for two IBM 73GB 15k drives! I know: 15k, IBM, etc, but even halving that means two -- two! -- 73GB drives for a thousand bucks, a/o/t two 200GB drives for, what, four hundred. Heh.)

We're using an older version of the 4-series FreeBSD here. I've already set up one server using a newer 4-series release, and it's a pain: too many differences, one more thing to keep in mind when making changes, and so on. I haven't worked with the 5-series yet, and I don't want to start now...not entirely sure that it'd work for us. Plus, we'll probably migrate to Linux anyway, so I don't mind doing it for a server.

Anyhow! Get a Real Server and throw Linux on it. Hook it up to our drive array and start migrating home directories to ReiserFS from UFS/FreeBSD. Not trivial, but doable. Add more Linux servers as budget allows.

Arghh. For weeks now, I've been trying to track down why a couple of XP laptops have had random print jobs drop to the floor. I finally got to the point last week where I could reliably duplicate the problem (print four emails from Outlook in quick succession; only three show up, no error on the printer), and today I spent six hours figuring out where the hell the problem was. (I didn't intend to spend that long, but the combination of vociferous complaints and sheer bull-headedness got to me.)

For no particularly good reason, the laptop in question is set to print to the local HP 4200 using IPP. When I looked at the traffic in Ethereal, I noticed that the failing job had a subtly different response to the print job submission from the printer, and at the end the TCP stream was only closed by the laptop -- the printer ACKed right away but did not FIN its end. Aha! Firmware bug!

The printer repair guy who's been working with me to try and fix this stopped by to take a look, and decided to call HP support. Their response: Don't Do That, Then. Apparently, IPP is a weird protocol to use for a LAN and I should really print to port 9100 like everyone else.

Okay, yes, this worked, and it was a stupid amount of time to spend on this problem. But it irritates me that they weren't interested in (what I think is) a firmware bug, and that I'll never probably never get to the bottom of what was going on. Although I'm pretty sure that the JetDirect card just uses an embedded ARM processor; I could just try looking at the firmware with a disassembler...:-)

In other news, something's going subtly wrong with the WRT54G; the bridging of OpenVPN's tap0 interface and the external ethernet interface has stopped working. The internal ethernet interface still works, and if you SSH in that way and run ifconfig vlan0 down ; ifconfig vlan0 up the external interface starts working again. I'm also having problems with the wireless interface. I suspect the bridging may be involved there, too, since it's bridged with the internal ethernet. However, I only have my wife's iBook to test with, so I can't be sure it's not a problem with that.

And my OpenBSD 3.9 CDs are in. Hurray! Time to finally get this firewall off my desktop machine.

When trying to install OpenBSD to a Sun Ultra 1 workstation over the network, I got the Fast Data Access MMU Miss error when running boot net bsd.rd. Turned out I'd copied the wrong boot loader to the TFTP directory; copying ofwboot.net over it fixed the problem.

For a while now I've been irritated with the behaviour of OpenRCS and Emacs on my OpenBSD machines: every time I try to check out a file kept in RCS, using C-x v v (vc-next-action), I still have to toggle read-only status on the file. Then, when I try to check it in, it asks if I want to steal the lock from myself, and never actually checks it in.

Finally had some time to track this down, and this bug appears to be the cause. I may have to play around with Emacs a bit to get it to ignore the permissions, or I may just use the OpenBSD package for GNU rcs instead.

Now that Clara's heading back to work, my schedule has changed a bit: I'm staying at home on Wednesdays to take care of Arlo, and then working from home on Saturdays to make up the time. I'm grateful to my boss for letting me do this, and I'm hopeful it will work out.

My first Wednesday (July 4th) went pretty darned well, really. Arlo ate, he played, he got vaccinated (Chicken pox; I had no idea they vaccinated for it), he napped and then he played some more. I didn't drop him, he didn't freak out and it was a great deal of fun.

As it happens I got to take care of him on Friday, too; my mother-in-law, who's going to be taking care of him two days a week, had a sudden trip to the emergency room. She's okay, but wasn't able to take care of him that day. (She was mad about it, too...) I called into work and let them know I wouldn't be in, then went in anyway just to make sure a few things were okay. I've got some karma built up and a fistful of sick days I rarely take, so all was well.

And then yesterday I worked from home. And man o man, did I get stuff done. Not quite as much as I wanted; I was hoping to use flar to duplicate a Solaris machine so I could test it, and ran into a bug that took a while to figure out. (If the patch I applied fixes the problem, I'll write it up here since there was only one other reference I could find.) But it was lovely to work for, like, four hours in a row on something and not be interrupted. Plus, there's the skipping of the 90-minute commute to enjoy.

My fondness for trivial patches continues. You may envy me.

Came across a mention of BSDstats.org on the Dragonfly BSD Digest, and I've set it up on my home machine. There are a ton of FreeBSD machines, and only 64 OpenBSD clients reported…time to change that!

I'm reading the documentation for Bacula right now, and it's amazing. Clearly written, thorough and extensive — almost 800 pages long. I'm very impressed.

Just updated my resume for the first time since starting my current job. It's nice to look back at what you've done and realize that, hey, there's been a lot.

In other news, I finally gave in to lust the other day and bought a Dell C400 on eBay. Nothing too special — 1.2GHz, 256MB, 30GB hard drive — but I was mainly after the 12" screen, so that I'd be able to (say) debug raw ethernet frames on my daily commute. About $280 when all was said and done; the strong Canuckistan peso was part of the incentive to buy now. Should be at the office in a week or so, and I can't wait.

It amazed me to see how many off-lease laptops were available, and just how cheap you could pick them up. A whilte back my boss got a D420; with extra memory and a few other things, it came in at about $1700 or so Canadian. But if you look around, there are plenty of D400s and D410s around for less than $500 — even less than $400 if you look hard. Add another $100 (say) for a working battery, and you're in pretty good shape.

Virtualbox has made it to Debian testing — hurrah! Only it won't run (Open)?Solaris. Dang.

On Tuesday, I'm giving a short presentation on my work's subnet at SNAG, the UBC System and Network Administrator's Group. I found Bruce in OpenBSD's ports tree on my laptop; the documentation is (ahem) thin, but it works. Wish me luck.

And there's Arlo up. Time to go get him.

The laptop I bought off eBay arrived at work on Wednesday...which is my day at home with Arlo. Thursday I was off sick with flu. Yesterday I was back at work and slashing open the box it came in, eager to see what I'd got.

Well, I already knew: it's a Dell C400. 12" screen, 1.2GHz P3 (but running at 800MHz with SpeedStep and all), 256MB RAM and a 30GB drive. Not a whole lot of memory, and a bigger hard drive would always be nice, but I can always upgrade. There's no CD drive in this thing, and I hadn't plumped for the docking station, so I set up PXE booting to install Debian. It was a trifle slow, but it worked! (Especially the second time, after I'd accidentally overwritten Debian trying to install OpenBSD on another partition. :-)

I'm surprised at how much Just Works in this thing: X.org (no configuration needed, just start up XDM…man, that's nice), suspend-to-disk, ethernet (well, it's a 3c905; what do you expect?). Even the battery, which I'd written off in advance, appears to hold a decent charge — about four hours so far. The one thing that's dicy is the onboard wireless, a Dell 1370 from everybody's favourite company. But again, I'd written that off in advance.

Next up: I've ordered the OpenBSD 4.2 CD set, so I'll be installing that once it arrives. And Noah has shown the way to longer battery life; I'm getting my 2.6.22 kernel now from Backports. (Oh, the shame of not compiling my own kernel...)

On another note, I think someone had one too many Dilbert moments:

$ dig newcastle.edu.au mx

; <<>> DiG 8.3 <<>> newcastle.edu.au mx
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 4
;; QUERY SECTION:
;;      newcastle.edu.au, type = MX, class = IN

;; ANSWER SECTION:
newcastle.edu.au.       11h59m12s IN MX  10 proactive.newcastle.edu.au.
newcastle.edu.au.       11h59m12s IN MX  10 synergy.newcastle.edu.au.

Perhaps they got the names from /dev/bollocks.

I ordered the 4.2 CD set of OpenBSD at work, in another optimistic step toward reorganizing the firewall there. In order to (ahem) road-test it, I installed it on my new laptop (which, you'll recall, is running Debian Stable) in a 5GB partition I'd left for just this purpose.

Onboard wireless, like with Debian, did not work, and I didn't expect it to; fuck you too, Broadcom. But my dad offered to send out a couple of wireless cards he couldn't use, and I figured one of 'em would have to work.

One was a Broadcom (op cit.), so that was out. The other, a DWL-650 (which appears to have umpty different versions over the years with not one change in model number) looked promising: a Realtek chipset, so should be good, right?

Well, it worked on OpenBSD -- but not in Linux. There's no driver in the tree for it, and the outside project to make drivers for it had its last official release in 2005. What's more, the CVS version, for some reason, removes all of its source files when I compile it, then complains that there are no files left to compile. To be fair, I think this is because of a makefile included from /lib/modules/2.6.22-2-686/build rather than the code itself.

Update: Just read Tourrilhes' page on the RealTek driver, and learned something: there's a fork/resurrection of the project I'd looked at, and it appears to be relatively current. I'll have to take a look. SooperUpdate: the new project fixes the let's-delete-all-the-files problem. Score!

What OpenBSD does not do on this laptop is suspend -- or more accurately, come back from suspension. This works reasonably well under Debian, which means that I still have one rose to give away to The Next Laptop OS for Saint Aardvark.

Dude, my laptop screen just turned blue. I'd booted into OpenBSD (4.2) and was trying to figure out how to turn off the audible bell. I'd gone from X to a virtual console to see if the problem happened there (it did), then tried ctrl-alt-f5 to get back to X.

My laptop screen turned from black with white text to grey with grey text to light blue with dark blue text, over the course of a minute or so. I thought I'd suddenly borked the LCD screen, but when I rebooted to Debian it was all fine. Just tried switching to a console, then back to X (alsoin Debian), and that's fine too. Bizarre.

Just checked the logs in OpenBSD and found a series of entries like this:

Nov  1 16:47:17 laptop /bsd: agp_release_helper: mem 0 is bound
Nov  1 16:47:17 laptop /bsd: agp_release_helper: mem 1 is bound
Nov  1 16:47:17 laptop /bsd: agp_release_helper: mem 2 is bound
Nov  1 16:47:17 laptop /bsd: agp_release_helper: mem 3 is bound
Nov  1 16:47:17 laptop /bsd: agp_release_helper: mem 4 is bound
Nov  1 16:47:24 laptop /bsd: agp_release_helper: mem 5 is bound
Nov  1 16:47:24 laptop /bsd: agp_release_helper: mem 6 is bound
Nov  1 16:47:24 laptop /bsd: agp_release_helper: mem 7 is bound
Nov  1 16:47:24 laptop /bsd: agp_release_helper: mem 8 is bound
Nov  1 16:47:24 laptop /bsd: agp_release_helper: mem 9 is bound
Nov  1 16:47:31 laptop /bsd: agp_release_helper: mem 10 is bound
Nov  1 16:47:31 laptop /bsd: agp_release_helper: mem 11 is bound
Nov  1 16:47:31 laptop /bsd: agp_release_helper: mem 12 is bound
Nov  1 16:47:31 laptop /bsd: agp_release_helper: mem 13 is bound
Nov  1 16:47:31 laptop /bsd: agp_release_helper: mem 14 is bound
Nov  1 16:47:38 laptop /bsd: agp_release_helper: mem 15 is bound
Nov  1 16:47:38 laptop /bsd: agp_release_helper: mem 16 is bound
Nov  1 16:47:38 laptop /bsd: agp_release_helper: mem 17 is bound
Nov  1 16:47:38 laptop /bsd: agp_release_helper: mem 18 is bound
Nov  1 16:47:38 laptop /bsd: agp_release_helper: mem 19 is bound

Very weird. On the bus, so Googling that'll have to wait. Although I do have the code on that partition…here we go: says it's the AGPIOC_RELEASE ioctl for agp. Aha! Maybe I'll explain money laundering while I'm at it.

And btw, here's a memo for the world: if you're on the toilet, don't take a phone call. It's really not that important.

Update, October 15 2008: Still happening with OpenBSD 4.3. And for the record, this is a Dell C300 laptop.

My laptop hard drive started giving scary errors a couple days ago on the way to work (I've got a 90-minute commute by public transit [uck] so I fill the time by reading, listening to podcasts, or working on Project U-13). Fortunately, working at a university means that there are two computer stores on campus. I ran out at lunch, picked up a 100GB drive, and had things back to normal by the next morning.

Well, normal modulo one false start with Debian; I decided to try encrypted filesystems just for fun. But then I suspended, came back with a newere kernel, and it could not read the encrypted LVM group anymore. Whoops.

Still lots of free space on this thing, and I'm thinking of installing Ubuntu, FreeBSD and maybe NetBSD just for fun. Of course, I've got to do it all via PXE since this thing doesn't have any CDROM drive, but that just adds to the geek points.

Project U-13 is coming up on 0.0.3, btw; Andy suggested adding Rackmonkey, which looks quite cool. There's no package for it, so I'm having to do some rather ugly scripted installation…but I can stand it for now. And I've got the barest skeleton of a cfengine file in there too. Watch the skies!

Matthew Garret's presentation on Suspend-to-Disk make fun reading.

Arlo's sick with flu or something; I was up 'til 1am last night rocking him to sleep. Haven't done that in a while…

Telling detail: I'm about to blow away Debian testing on my desktop machine and install Ubuntu's Gutsy Gibbon. Partly it's because I'm tired of installing 80MB worth of updates every two weeks, and partly it's because it'll make setting up the printer a breeze.

I'll probably leave half the drive aside for good ol' Debian stable, but Ubuntu'll stay there for experimenting and so my parents, on their next visit, will not have to bring out their 4-tonne laptop.

I'll be reinstalling Ubuntu on my laptop as well; due to a stupid error, I installed Dapper, not Gutsy. I tried updating in one fell swoop, and after three days of apt-get -f install I finally got things working…except for the boot artwork, and GDM doesn't start one time out of three. Interesting experiment, but I think I'll take a do-over.

I may even install it twice, so that I can try out The Depenguinator, which appears to be a lot easier than trying to figure out PXE booting for FreeBSD. Unlike OpenBSD, there's no readily apparent "official way" of doing it, and the handful of HOWTOs I've found have contradicted each other. At this point I'm just too lazy to keep trying and seeing what I'm doing wrong.

That's not quite my dad at c2k8, but damn if it wasn't enough to make me look twice.

When I was at LISA, one of the sysadmins I met mentioned a firewall unit testing script that a coworker of his had come up with. The idea was to run your OpenBSD firewall in a QEMU instance, then try passing traffic back and forth to make sure everything worked as expected. I've been looking for that tool to be released, but haven't seen it....or anything else like it either…

Until today, that is, when I stumbled on NetUnit. It's a Java-based tool that tests basic network connectivity, using XML files to specify tests. So far he's got tests for ICMP/port 7 (which I never knew was the echo port), TCP ports, HTTP/HTTPS and MySQL. Not bad at all, except for my lack of Java experience.

Of course, now I want to write my own tester using Perl and QEMU. Like I've got time. But here's an idea for anyone who can use it: test your firewall using three instances of QEMU (inside, outside and firewall), and have the inside and outside hosts communicate using the serial port. "I'm gonna send an echo request, did you see it?" "Yes, did you see the reply?" It's a bit more feedback than simply noting the lack of the expected reply.

And it's not at all like conversations that start out with, "I sent you an email. Did you get it?"

I was >this< close to writing my own damn set of Perl scripts to test a firewall, but I decided to search one last time. Good thing, too: ftester looks pretty close to perfect.

I'm having trouble right now getting ftestd to work on an OpenBSD 4.3 system; this may be because I'm trying to get it to listen on an interface that's part of a bridge. I'll have to look into this further. But testing it out between my laptop and desktop works a treat, whether my laptop is running OpenBSD or Linux 2.6. Sweet!

As mentioned on Undeadly.org and openbsd-misc, OpenBSD is asking for donations for BGP routers and a new CVS server. I've donated, since I wouldn't be able to do half my job without them; if you feel the same and can spare some money, I urge you to do the same.