The Life of a Sysadmin

Spent some time this weekend trying to get wireless working. I've got the WRT54G on the second floor, and my wife's iBook on the first. The iBook will pick up the signal more or less fine, but if you put it to sleep and come back in an hour (say), it won't find the signal anymore. I suspect it's the iBook's fault, but I can't be sure since I haven't got another wireless notebook to check it with.

As I mentioned, the signal the iBook gets is decent, but it surprises me how much depends on the orientation of the antennae -- which, on the iBook, means what angle you've got the monitor at. I built a couple of these antennas, and that does seem to help a bit. Plus it's just fun making something with cardboard and tape and aluminum foil...feels like I'm in grade one again. :-)

On Friday I had to set up a new Windows workstation for the first time in a while, and I remembered these guys. They've done a metric buttload of work since I last checked in with them (Lord, a year now?) We use this program at work to automate software installs on Windows machines, and even though we had problems setting it up (mainly getting it to run as a service using Cygwin's cygrunsrv) it's saved us a ton of time getting new workstations ready. I think it's time I took another look at using it for ongoing maintenance, rather than just first installs.

Slowly getting my OpenBSD firewall put together. It'd be cool to use the WRT, of course, but then I wouldn't get to use the 3.9 CD set I just bought. It still amazes me that I can put together a firewall using pf and not lock myself out.

Marcin posted recently that he got Linux working on his own WRT1133 clone. Rather than bother uploading an image to flash, though, he used OpenOCD to write the image to memory using a JTAG cable. I hadn't heard of OpenOCD before, and this raises the possibility of getting Flash writes working from Linux by watching what the original bootloader does when it loads another image.

My father has started a blog. I was going to write, "Now if only my uncle would post again...", but he beat me to it.

Oh, and the favicon comes courtesy of Chris. Many thanks!

Finally, my friend ZenRender has just got a haircut after, like, at least eleven years of rampant hippiedom. (He did get it partly cut last year, but I call that chickening out.) Of course, he still looks like a damned Communist.

The second release of oz2remind is now available for your GPL'd pleasure. Now, it'll convert from Remind format to OpenZaurus format: it'll either parse your .reminders file directly, or (recommended) parse the output of remind -n -b1 -s. You can check out the Freshmeat page, or just go and grab it now.

Came across a weird problem on the firewall at work last week. It's running 4-STABLE, and was last updated about a month ago. It's got fxp0 for an outside interface, and em0 plus a bunch of vlan devices for inside interfaces.

When I added either of these two rules:

ipfw allow tcp from 192.168.16.34 to 192.168.19.33 1230,1236 keep-state via vlan19
ipfw allow tcp from 192.168.19.33 to 192.168.16.34 1230,1237 keep-state via vlan19

then suddenly DNS queries from inside our main LAN (192.168.0.0/24 on em0) to outside servers -- say, our main inside nameserver doing recursive queries for A records for Google -- stopped working: queries would pass through natd and go out with the source address changed, but the reply from the server would be accepted by the firewall box, rather than passed to natd and then back inside to the machine that'd made the query. Since the firewall box hadn't made the request, it would send back an ICMP port-unreachable packet to the outside nameserver. In other words:

1. 192.168.0.2 -> ns.google.com: www.google.com A?
2. 192.168.0.1 (firewall box) passes that to natd
3. natd changes packet to...
4. firewall outside IP -> ns.google.com: www.google.com A?
5. ns.google.com -> firewall outside IP: www.google.com A 1.2.3.4
6. firewall accepts that packet...
7. ...but realizes it doesn't have anything listening for a UDP packet from ns.google.com...
8. ...and rejects it:
9. firewall outside IP -> ns.google.com: ICMP port-unreachable

Took me most of the day to figure this out, because I found a separate problem and was convinced that these rules had nothing to do with it. And they don't, really -- wrong protocol, wrong interface, wrong addresses -- but remove the rules and everything's fine. Freakin' bizarre.

I spent a lot of time checking out state rules and such, and I'm pretty certain that's not it. At this point, I'm assuming that it's either a bug in ipfw (possibly related to this PR, or my upgrade from 4.8 to 4-STABLE did not go as cleanly as I thought. I'm going to try installing FreeBSD here and see if I can duplicate this...maybe get another one-character patch into FreeBSD. Woot!

I'm still trying to get Heartbeat all working on the two file servers at work. The bit that's getting me down is STONITH -- in particular, the apcmastersnmp plugin.

For some reason, it just will not send out the SNMP request saying "reboot that there outlet". It's not very specific about why, either. The weird thing (well, one of a few) is that running the stonith command will send the request (once you figure out the goddamned syntax for the config file...Christ on a crutch, the documentation is poor), but the hearbeat process itself, which just calls the library directly rather than using the stonith command, does not.

strace shows that heartbeat forks off a child to send the request. That child then goes about closing all its file descriptors, then trying to sendto(2) on a file descriptor (socket descriptor?) that's one of the bunch it just closed. We get EBADF, then it logs the failure.

(This is a little further than I was getting, BTW; it turns out to be essential to put the MIB file for the PDU into /usr/share/snmp/mibs. I didn't think about that, but it makes sense.)

So I've compiled a debug version of heartbeat (Debian rocks: DEB_BUILD_OPTIONS=debug,nostrip dpkg-buildpackage -rfakeroot and away you go), and it turns out to be snmp_synch_response that's failing. Of course, that's in the NET-SNMP library, so now I'm preparing to compile a debug version of that and see what's going on.

I'm of two minds on this. Failover would really be a good thing, and I can't do it w/o STONITH. And I hate like hell to just give up and say, "Oh, it's too hard for me." OTOH, this is just taking so damned long, and it is an older version (though it is the one in stable). I may take a look at the 2.0 series and see how that works...just hope I don't have to throw away all this work. <grumble / >

I've finally got Danconia up and running OpenBSD 3.9. It's now officially my firewall box, taking over duties from Rearden (Debian desktop machine). As always, the simplicity and featurefullosityness of pf just astounds me. A simple thing like not loading the rules if there's a syntax error is such a butt-saver, I'm amazed it hasn't been implemented in iptables or ipfw. (Of course, pf loads all the rules at once, rather than one at a time, so it's a different approach...but still.)

Next step is to get my IPv6 tunnel from HE up and running. I hadn't realized it, but OpenBSD does not use stf, the 6to4 IPv6 interface, because of security concerns. I'm gonna have to do some reading on this, I think. (Incidentally, why does this link say RFC 3694 is a "Threat Analysis of the Geopriv Protocol"?)

I've ordered a replacement power supply for the dying XBox I'm using for a MythTV frontend. It had been behaving badly for a while after the move, and then finally it just would not find the hard drive at all. The HD was fine -- I could plug it into another box and it'd work great (though in the process I had another hard drive actually catch fire -- 3" flame and all -- which was a pisser) -- and it could boot from a CD just fine. What's left? That's right, the power supply. Well, I hope so, anyhow. Inna meantime, I've set up the backend as a frontend; other than some occasional odd slowness deleting previously recorded shows, it's working fine.

Finally, as of last Friday [five years][8]. Since we're such hopeless romantics, I gave her a cupcake from Tim Horton's, and she gave me this fine dollar store sculpture:

We saw it a few weeks ago and it cracked me up. And then I read the label:

As I have, as instructed, planted the elucidation, I will be posting pictures as I witness the up long grass.

We are also less than one month from The Due Date. I am busy doing practical things like putting up smoke alarms, baking food to put in the freezer, and insisting that we pack the hospital bag now. Clara has shown amazing patience with my sudden neurotic compulsion to be A Responsible Adult^TM.

[8]: http://torturedpotato.com/cheeseblogmy wife and I have been married