Had an interesting couple of problems at work this week.
First thing was a FreeBSD system where, suddenly, ipfw didn't work
anymore. Only "suddenly"'s not exactly true: this happened after a
kernel upgrade. And "didn't work" is a bit inaccurate too: it would
list firewall rules -- it just couldn't add them. (Good thing this
machine had "default accept" as its firewall policy...) So, like, WTF?
First I tried adding a very simple rule:
/sbin/ipfw add 100 allow
all from any to any Nope, didn't work:
getsockopt(IP_FW_ADD): Invalid argument I tried that rule on another
machine to make sure my syntax was okay -- no problems. Well, what
about the command itself? The MD5 checksum of /sbin/ipfw on both
machines was the same. I considered briefly blaming the problems on
3133+ cR5><0rZ who'd found an MD5 collision in ipfw, but decided not
to try that on my boss. (I did copy the command from the working
machine to the stupid machine just to be sure...yep, same result. So
much for superstition.)
Hey, wait a minute -- hadn't we patched the kernel on the stupid
machine? Sure we had! So that means...well, I don't know what. I had a
look at the patches (very simple stuff, so I was able to follow
along), and couldn't see what might be causing the problem. I mean,
yes they did change the firewall functionality, but...well, maybe we
should chase that up, yes? Yes.
And here I fell down a rabbit hole: I started to wonder if maybe the
fact that FreeBSD compiled modules for everything (sure seems that
way) despite the functionality being included in your KERNCONF file
maybe meant that said functionality might still actually reside in the
modules -- that the kernel wasn't being statically compiled at all, or
at least for this particular bit, but there were Secret! Invisible!
Modules! that actually had the bit of code we were looking for. Oh
sure, kldstat doesn't show them, but that just shows how tricky
those damn FreeBSD kernel developers are, right? And yeah.
Since the stupid machine'd had its kernel copied over by hand -- ie,
scp foo@bar:/kernel / (I KNOW, I KNOW) and rebooted,
and didn't copy all those Secret! Invisible! Modules! over along
with /kernel, why, sure we were gonna run into problems! Of course!
It all makes sense now! It was the Freemasons all along!
Lemme tell you, I was yea close to copying over
trying that (I did go so far as to try ldd /kernel
(I KNOW, I KNOW), but it turns out that ldd actually tries to
execute a file in order to figure out what libraries it uses, so
it just gave me a smack for being such an idiot), but for some reason
had another look at the patches we'd made. Okay, yep, that bit in
here, that bit over there, and not one bloody file in
/usr/src/sbin/ipfw/ipfw.c, so why the...
Oh. Header files.
- We'd changed a header file
- that was used in
- and I hadn't thought of that.
Well, crap. But hey, easy to test and easy to fix: patch the header
ipfw, and ha! Working! All I had to do was compose a
suitably superior-sounding email about the dangers of passing
/kernel files around willy-nilly, and all was well again.
Coming up next: Gentoo on a dual G5. Woohoo!