The Life of a Sysadmin

I promised a month ago (!) to put up the firewall rules I'd come up with for FreeBSD; here we are at last. With any luck this'll be useful for someone.

By way of background, I had the honeypot, a default server install of Redhat 6.2, sitting behind my main box running FreeBSD; the FreeBSD box had one external card (cable internet), one internal card for my LAN (yes, I was using my LAN while this was all going on...) and one internal card dedicated to the honeypot.

This is a fairly restrictive ruleset, but I didn't want to fuck up and risk letting Bad Packets (tm) out. As related earlier, I had to shut it all down about five minutes after I got cracked anyhow, so it was a moot point.

Oh, almost forgot: the funky part is not just the firewall rules below, but running the natd daemon w/the right options:

natd -t 10.0.1.1 -interface xl0

From man natd:

-t | -target_address address
Set the target address. When an incoming packet not associated
with any pre-existing link arrives at the host machine, it will be
sent to the specified address.

That made it a great deal easier to pass traffic initiated from the outside to the honepot. I'm sure there's a way to do this in Linux, but it's been long enough since I worked w/Linux firewall rules that I wouldn't know what that is.

Anyhow, w/o further ado:

#!/bin/sh

IPFW="/sbin/ipfw"
PRIVATE_LAN="10.0.0.0/24"
HONEYPOT_LAN="10.0.1.0/24"
HONEYPOT="10.0.1.1"
EXTERNAL_NIC="xl0"
INTERNAL_NIC="ep0"
HONEYPOT_NIC="ep2"
MY_HONEYPOT_IP="10.0.2.2"
PUBLIC_IP=`ifconfig $EXTERNAL_NIC | awk '/inet / {print $2}'`

$IPFW -f flush

$IPFW add 100 pass all from any to any via lo0
$IPFW add 200 deny all from any to 127.0.0.0/8
$IPFW add 300 deny ip from 127.0.0.0/8 to any

$IPFW add divert natd all from any to any via $EXTERNAL_NIC

$IPFW add check-state

$IPFW add skipto 40000 all from any to any via $INTERNAL_NIC

# Honeypot rules...be very careful!

# Logging.
$IPFW add allow log udp from $HONEYPOT to $MY_HONEYPOT_IP syslog
via $HONEYPOT_NIC

# Allow in from outside world. Remember, natd will be passing
  these
# packets on. Keep traffic from own network out.

$IPFW add allow log tcp from not $PRIVATE_LAN to $HONEYPOT via
$HONEYPOT_NIC
$IPFW add allow log tcp from not $PRIVATE_LAN to $HONEYPOT via
$HONEYPOT_NIC
$IPFW add allow log udp from not $PRIVATE_LAN to $HONEYPOT via
$HONEYPOT_NIC keep-state
$IPFW add allow icmp from not $PRIVATE_LAN to $HONEYPOT icmptypes
0,3,8,11,12,13,14 out via $HONEYPOT_NIC keep-state

# Allow replies from ftp, ssh, pop and web...put in mainly for ftp
  replies. NO SMTP!
$IPFW add allow log tcp from $HONEYPOT 20,21,22,110,80 to not
  $PRIVATE_LAN via $HONEYPOT_NIC

# Do we need this?

$IPFW add allow log tcp from any to $HONEYPOT in via $EXTERNAL_NIC
$IPFW add allow log udp from any to $HONEYPOT in via $EXTERNAL_NIC
$IPFW add allow log icmp from any to $HONEYPOT icmptypes
0,3,8,11,12,13,14 in via $EXTERNAL_NIC

# What we allow out: established, mail, ftp, web, domain, selected
  ICMP.

$IPFW add allow log tcp from $HONEYPOT to not $PRIVATE_LAN
established via $HONEYPOT_NIC
$IPFW add allow log tcp from $HONEYPOT to not $PRIVATE_LAN
20,21,22,25,80,110 via $HONEYPOT_NIC
$IPFW add allow log tcp from $HONEYPOT 20,21,22,110,80 to not
$PRIVATE_LAN via $HONEYPOT_NIC
$IPFW add allow log udp from $HONEYPOT to not $PRIVATE_LAN domain
via $HONEYPOT_NIC keep-state
$IPFW add allow log icmp from $HONEYPOT to not $PRIVATE_LAN
icmptypes 0,3,8,11,12,13,14 via $HONEYPOT_NIC

# Deny the rest

$IPFW add deny log all from $HONEYPOT to any

# Should this be in or via?

$IPFW add deny log all from any to any in recv $HONEYPOT_NIC
$IPFW add deny log all from any to any via $HONEYPOT_NIC

#add allow udp from any to any keep-state
$IPFW add 40000 allow udp from $PRIVATE_LAN to any keep-state
$IPFW add 40100 allow udp from $PUBLIC_IP to any keep-state

$IPFW add 40200 allow tcp from any to any established

$IPFW add 40300 allow tcp from any to $PUBLIC_IP 22,8000

$IPFW add 40400 allow all from any to any out via $EXTERNAL_NIC
$IPFW add 40500 allow all from any to any in via $INTERNAL_NIC
$IPFW add 40600 allow all from any to any out via $INTERNAL_NIC

#add allow all from any to ${PUBLIC_IP} 22, 8000 via ep0
$IPFW add 40700 allow all from localhost to 25 via lo0

$IPFW add 40700 allow icmp from any to any icmptypes 0,3,8,11,12,13,14

$IPFW add 40900 deny log all from any to any

Original entry.

ARGHHH...

I work at a small ISP, and among other things I help out the sysadmin w/the mail servers and the spam filtering we do (procmail-based, but we're trying to get SpamAssassin installed Real Soon Now). Yesterday I noticed that one of our front-end mail servers, which should also have been doing secondary DNS, was not doing secondary DNS. Turns out the /var partition was filled beyond capacity because of the fucking HUGE maillog generated because of the waves of spam we've been getting lately.

Admittedly, it's Not A Good Thing to have DNS + SMTP all on the same box; we've got a new mail server in [very hibby: half gig ram, 4x40GB drives, 2GHz or some-such P4] and we're trying to get it up, at which point the small box that crapped out can do just secondary DNS. But still.

And so but took a look at the queue, and took out 26,000 messages that were just bounces back to spamming mail servers refusing connections. 26k! We use both OR-something-or-other (ORDB died, I think...can't be bothered to look it up now, but we use whatever took up the mantle) and SpamCop's BL, and the fuckers still make it through. Arghhhhhhh.

As I said, it's a small ISP...which means that my official title is HelpDeskSuperHero (tm), which means that I get calls about all this, and have to talk people down off the ledge w/all the spam they're getting recently. It's the same talk every time, sometimes to people at the same office, and it's frustrating because a) I don't have the One Magic Wand they think I do, b) well, I might have the Magic Wand in a while, but not yet, and c) I keep having to explain why filtering out Naughty Words is really not the best idea.

Heh...not being terribly coherent right now; I'm usually better than this, but I'm still waking up and this all just makes me mad. Spammers fill up our mail queue, put the load on our mail servers through the roof, and anything I can do at the moment comes after the fact: it does nothing to prevent it, and very little to make things better. Sometimes for fun I tail /var/log/procmail.log and it's just insane: there is no possible way I can catch everything, or even react fast enough to catch what's coming in right now.

I realize none of this is News, but it's crazy to me how...how fucking crazy this is: we're spending our time, money and brainpower in what seems like a constantly losing race against a small number of people with the ability to ruin email. Spammers have power far out of proportion to their numbers, and it seems like it's enough to swamp us, and maybe lots of other people too.

On the subject of SpamAssassin: God it's good. I use it at home (FreeBSD + Procmail) and it fucking ROX. I dearly wish that there was a Windows version for Outlook Express, preferably free, that I could point our customers to, but nowt luck there: DeerSoft (kudos to them) have a version for Outlook, but not OExpress. Ah well.

Sigh. /me takes deep breath. Soon we will have SpamAssassin installed, I hope, and then everything will be better.

Original entry.

Update time.

I got into work today and found that the mail server had just come up after *half a fucking hour* of being down because of the insane load placed on it by spam -- just spam -- coming in. The owner of the company couldn't send email. I started setting up the new mail server.

And it was nice. I got to go away, away from the help desk, sit down and figure out how to make it work. FreeBSD's vinum + Promise raid controller == kernel panic (details later on). Finally got vinum figured out -- I've only worked w/it once before -- and before I was grabbed back to help desk had the disk setup about 80% done.

So some more details: there's 4 x 40GB maxtor IDE drives. (Yeah yeah yeah SCSI.) We've got an onboard Promise controller chip; I'll put in the mobo tomorrow and make this all seamless. First it turns out we've got the Promise Lite (Less Filling!) BIOS, which means we can only have one (1) array of two disks; the other two disks can be single arrays on their own, which is useful in some alternate universe I'm sure. So okay, try setting up one mirrored (Raid 1? 0? I can't keep 'em straight) array, and we'll use vinum to tie it together with the other single drives...

Only as soon as I try using vinum to do _anything_ with the Promise'd arrays, BANG: kernel panic. This is 4.6, not the latest (4.7RC1 as I type), but still. Arghh. Doesn't matter whether vinum tries raid 0, 1 or 5 -- just panics right away. If I had more time and a box of my own to fool around with, I'd try [Michael Lucas'|http://www.oreillynet.com/pub/a/bsd/2002/03/21/Big_Scary_Daemons.html] [SlashdotJournal_25September2002-02]1 (Buy his book!) and contribute something useful to the FreeBSD folk. Alas, it's not my box or my time, and if I were to post this message to freebsd-hackers-important-vinum-people tomorrow I'd (deservedly) get laughed at so hard I'd feel it over the ether.

Anyway. Point is I can't get vinum to play nice w/the Promise'd chip even as an IDE controller. The BIOS of the box allows you to turn the Promise chip on, off, or to ATA/IDE; but even set to the latter, it panics once vinum touches /dev/ar*. You have been warned.

So get vinum using the four drives on the first two IDE channels, and that works fine once I learn the intricacies of disklabel (set type to vinum, kids!) and vinum init (and that takes a long time w/3*35GB partitions^H^H^H^H^H^H^H^H^subsooperplexen). 1 5m 5o 133t!

OT: One of my side notes was going to be about how I'm posting this w/Lynx 'cos Mozilla won't let me use vi, editor of the Elder Gods, as an editor. Then I realized I could have just fired up a shell and used vi in there. Sigh. Rumours of my cleverness have been exaggerated.

Original entry.

So now the new mail server is up, though not accepting mail. I played around w/vinum last night and this morning from home (Let's hear it for OpenSSH!), and broke it -- but figured out what to do next.

The problem came in the way I'd laid out disk space w/the first install. Okay, so four drives; 128M for /, the rest for vinum. Only instead of setting up all the partitions on all the drives as I wanted them to be in the end (equal in size for raid5), I'd made a big /usr partition, then hoped I could juggle /usr over to a raid partition, re-disklabel the first drive to make the partitions the way I wanted them, add another subdisk/partition/slice each to the existing raid5 drives -- gah. No wonder it fell apart. The obvious solution was to make the /usr partition smaller (didn't need it as big as I'd originally set it), and go from there.

Got excited at this realization, so went into work an hour early (such a geek) and worked on it all day. Got it into the server room, up and running, even set up Sendmail for the first time (tested it for relaying, hands sweating), and got it all up and ready for tomorrow.

One weird thing was that I couldn't figure out how to get FreeBSD's port of net-snmpd (formerly ucd-snmpd) to work w/tcp wrappers and hosts.allow -- it would only allow connections with ALL : allow ; anything else would just refuse connections. Weird. I'll have to figure it out tomorrow.

Original entry.

So the new mail server is up and running. It looks, though, like I missed some fairly important things, and at least one critically important thing.

I was working on it from home last night around 8pm, and alla sudden it wasn't responding to ssh or pings. It came back up, and sure enough it had panicked and crashed. Fortunately the sysadmin (Hi Dave!) was there and was able to look at it and figure out what was wrong. My first thought was there were problems w/vinum and the promise controller again, but no: not enough file descriptors. Given that it's a moderate-to-fairly busy mail server that gets lots of spam, this was a pretty big fuckup.

Second, I'd set up vinum to make a bunch of separate raid5...um, partitions out of separate disk slices on each of four hard disks. Turns out you can divide one big vinum raid5 partition into separate slice-like entities (I'm still learning all this; forgive the imprecision of what I'm writing).

Third, I'd set MAXUSERS too low: 32, which seemed reasonable given that hardly anyone would be logging on to it. Of course, this setting controls lots of other resources, so I hadn't really thought that through. The SA set it to 0, which means FreeBSD will adjust it on the fly.

Fourth, he's got FreeBSD-stable (I think it's stable) built weekly on a box there, and I should've installed & mounted everything via NFS rather than installing from CDROM and putting everything on the disks.

All in all, I'm feeling a bunch humbler this morning. I did almost all of this on my own and thought I was doing pretty damned well, but I stil have an awful lot to learn -- plus, my first idea of what happened was completely wrong (not that I wouldn't have been able to figure it out eventually, probably, but that's not good). I'm starting to think I should set up some of the boxen I have at home here as a test lab -- I haven't really done much since setting up one as a honeypot -- and fuck around w/stuff like this: set up one as a mail server, set up the other one to hammer it w/a million messages, that sort of thing, and see where the bottleneck is. Plus NFS, plus all sorts of stuff. Plus buying Michael Lucas' excellent book and reading it cover to cover. Plus actually trying to understand Sendmail. Plus plus plus plus plus.

Sigh. On the other hand, the new mail server appears to be doing really well since Dave rebuilt the kernel last night, and the load on the back-end server (which customers use to send/receive mail) is dramatically lower -- only a half dozen big load spikes overnight, as opposed to one very half hour or less.

Original entry.