Mail server + 4am pages == fun

So update time on the not-so-new-anymore mail server.

SpamAssassin has been working out just ducky. I had the threshold set to 14, then 10, and I just lowered it to 9 yesterday. I'm keeping an eye on it as I go, because there are legitimate messages (mainly newsletters from Real Companies[tm]) that piss off SA -- "click here to unsubscribe", "you're getting this because", etc. -- and we need to whitelist 'em as we find 'em. Only, w/tens of thousands of messages being caught every day, that's a lot to look through...so it's taking a while.

As far as stats go, at threshold 10 we caught ~ 28k messages in 24 hours. In the 14 hours since I lowered it to 9, we've caught ~ 35k. Fuck me...

We've had one weird hardware problem. At 4am on Saturday morning I got a page (ugh) saying that the server was down. Tried pinging it, and yup, no response. I put our backup mail server on the front end and went back to sleep.

In the morning I went to check it out, and it seemed to be just frozen. Last log message sez:

xl0: watchdog timeout

WTF? Rebooted, saw a lot of "Stray IRQ" messages, and it seemed happy. Put it back on the front end, but let the backup server stay there too.

Dave the SysAdmin found this message on the FreeBSD mailing lists. It suggested that the problem might be because of a couple PCI slots sharing an IRQ; when the guy moved his network card to a slot that didn't share an IRQ, the problem went away. I checked the manual for the mobo (Gigabyte VR7XP), and it looks like the slot the card was in didn't share an IRQ. However, I took a few minutes, shut down the machine, and moved the card (3Com fill-in-the-blank-here) over a slot anyway.

While I was there, I checked out the BIOS and found something moderately interesting: APM was turned off, but in the options it had different IRQs it could wake upon. One of the four that were turned on was IRQ 7, which was the stray one that the box had been complaining about. I turned 'em all off. Bad me for not turning off all that in the first place.

It's held up fine after that last reboot, and now it's the only one on the front-end again. (Good thing, too; the backup mail server doesn't have SA installed, as it's also a webmail + web server.)

Original entry.