The headers of doom

Had an interesting couple of problems at work this week.

First thing was a FreeBSD system where, suddenly, ipfw didn't work anymore. Only "suddenly"'s not exactly true: this happened after a kernel upgrade. And "didn't work" is a bit inaccurate too: it would list firewall rules -- it just couldn't add them. (Good thing this machine had "default accept" as its firewall policy...) So, like, WTF?

First I tried adding a very simple rule: /sbin/ipfw add 100 allow all from any to any Nope, didn't work: ipfw: getsockopt(IP_FW_ADD): Invalid argument I tried that rule on another machine to make sure my syntax was okay -- no problems. Well, what about the command itself? The MD5 checksum of /sbin/ipfw on both machines was the same. I considered briefly blaming the problems on 3133+ cR5><0rZ who'd found an MD5 collision in ipfw, but decided not to try that on my boss. (I did copy the command from the working machine to the stupid machine just to be sure...yep, same result. So much for superstition.)

Hey, wait a minute -- hadn't we patched the kernel on the stupid machine? Sure we had! So that means...well, I don't know what. I had a look at the patches (very simple stuff, so I was able to follow along), and couldn't see what might be causing the problem. I mean, yes they did change the firewall functionality, but...well, maybe we should chase that up, yes? Yes.

And here I fell down a rabbit hole: I started to wonder if maybe the fact that FreeBSD compiled modules for everything (sure seems that way) despite the functionality being included in your KERNCONF file maybe meant that said functionality might still actually reside in the modules -- that the kernel wasn't being statically compiled at all, or at least for this particular bit, but there were Secret! Invisible! Modules! that actually had the bit of code we were looking for. Oh sure, kldstat doesn't show them, but that just shows how tricky those damn FreeBSD kernel developers are, right? And yeah.

Since the stupid machine'd had its kernel copied over by hand -- ie, we did scp foo@bar:/kernel / (I KNOW, I KNOW) and rebooted, and didn't copy all those Secret! Invisible! Modules! over along with /kernel, why, sure we were gonna run into problems! Of course! It all makes sense now! It was the Freemasons all along!

Lemme tell you, I was yea close to copying over /modules/ipfw.ko and trying that (I did go so far as to try ldd /kernel (I KNOW, I KNOW), but it turns out that ldd actually tries to execute a file in order to figure out what libraries it uses, so it just gave me a smack for being such an idiot), but for some reason had another look at the patches we'd made. Okay, yep, that bit in here, that bit over there, and not one bloody file in /usr/src/sbin/ipfw/ipfw.c, so why the...

Oh. Header files.

1. We'd changed a header file
2. that was used in /sbin/ipfw's compilation
3. and I hadn't thought of that.

Well, crap. But hey, easy to test and easy to fix: patch the header file, recompile ipfw, and ha! Working! All I had to do was compose a suitably superior-sounding email about the dangers of passing /kernel files around willy-nilly, and all was well again.

Coming up next: Gentoo on a dual G5. Woohoo!