So I'm the Geek You Know for about six friends, and one of them needs a website. I volunteer my li'l server, no problem. But she wants something she can manage herself, and as she's not a geek that means something easy on the eyes and easy to use.
So I start looking at (can't use this phrase w/o gritting my teeth) content management systems, c/o Freshmeat and OpenSourceCMS.com (this site rocks). I've tried out two or three so far, and all have had the same results: my li'l server is dreadfully overworked.
It's a P200, 48MB of RAM, and it's fine at serving up static content: most of my site and my wife's site is just that, so it's not a problem most of the time. But start throwing some MySQL into the picture, and things slow down fast.
I'd settled, sort of, on Back-End as a likely contender; I liked its management pages, can't beat the street cred when CUPE uses it, there's an integrated gallery, and the installation went well. But when I tried it out...holy crap, it was slow: 10 seconds to throw up a page. Admittedly, better than the 30 seconds with some other packages, but still.
I figured it was time to move the database to the faster computer. I've got a Celeron, 450MHz, 384MB RAM, that I use for my desktop. Wasn't doing much besides 87 xterms and setiathome, so I figured out how to move it over there. Still slow. Well, decided to try some benchmarks. That's what separates us from the animals.
ab, against my own (static) site, shows 1000 requests being served, none dropped, concurrency 5, in 26 seconds. Against the Back-End demo site I set up, it timed out. I upped the timeout; same thing. In frustration I set concurrency to 500, set up iostat and top to watch on both the database and the web server (fancy!), and waited.
And waited.
After five minutes, the SSH session showing the stats on the P200 stopped. (Load on the Celeron and its disks barely registered, BTW.) I logged in via the KVM to see what was happening, and the answer is: not much. "eth0: card reports no resources", whole lotta processes being killed due to lack of memory, and fifteen minutes it is still chugging its way back to freedom. (Don't want to reboot and ruin that 60-day uptime...)
So: My questions to all of you are:
My thanks in advance for whatever help you can provide.
Interesting problem with The Inside Thing at work today. The Inside Thing runs headless -- no video card, no serial port (yet...sigh), connections only over IP. SSH to The Inside Thing has been no problem, and I never thought it would be.
One of the developers today loaded a debugging version of the FreeBSD kernel module he's working on, and found that it really slowed things down: a test script that would complete in a second, using the non-debugging module, would take a minute or more to run; in addition, the whole system would slow down to the point of near-unusability. WTF?
The debugging version does a lot of kernel printfs (I'm not a
developer, so forgive me for any imprecision in language
here). Logging is done to two places: /var/log/whatever, and over UDP
to The Outside Thing, which has its syslog daemon listening to port
grep syslog /etc/services. /var, on The Inside Thing, is just this
big (32 MB) memory filesystem, so that shouldn't be a problem. And the
network connection is gigabit ethernet, so that shouldn't be an issue.
I ran fstat while the program was running; it showed nothing unexpected: files open in /usr (where the program lives), the developer's home directory (NFS), /dev/insidething and /var/log/whatever. But run systat -vm, and hey, what's this: tons of interrupts on sio0.
This didn't work:
rm /dev/cuaa
mknod /dev/cuaa0 c 2 2
So on to less drastic measures.
I tried upping the serial port speed (we'd turned it on, but still haven't got a socket we can hook to yet) from 9600 to 115200 in /etc/ttys, and HUPping init; no change. (Incidentally, to get the serial port working on another FreeBSD machine over a null modem cable, I had to set it to 9600.3wire; strange. Or perhaps not.)
My boss came by at that point, and told me that the kernel printfs were not affected by stuff like getty and init; instead, there was a kernel option or possibly a sysctl that set that. Sure enough, look around and there's machdep.conspeed: 9600. Set it to 115200 and whee, look at things go! The debugging program ran in 30 seconds, which by this point seemed like a definite improvement.
I experimented a bit and found the highest machdep.conspeed could be set to something like 118900. Like before, this was better but by no means great. Then my boss came in again and announced a new sysctl MIB, greater than all the rest. This one was The Light, and the other one only came to announce The Light to the world:
kern.consmute
Set to 1, and all those kernel printfs still get logged to syslog, but never slow down The Inside Thing. I'm assuming that all this was trying to go out over the serial port after FreeBSD detected no video card...but I'm the first to admit that's probably a crack-addled dream.
In the grand tradition of ryanr's journal, let's see who figures out what's wrong with this Bourne shell script. First prizeis a Cadillac Eldorado. Second prize is a set of steak knives. Third prize is Cowboy Neal fires you.
Background: yesterday at work, my home-grown backup system choked when it tried to burn a 740MB ISO to CD. (Still waiting for a tape drive.) I decided to finally implement the long-delayed exclusion of certain files (.core, .o, etc.). I know, I know, should've put it in from the start, but this was the first time it became an issue.
Anyhow, I was testing to see if my changes worked, and they didn't: the files were not being excluded. What was doubly weird was that I could run, on its own, the command the script was running, and it would work fine: everything that should have been excluded was. I finally boiled it down to this script:
#!/bin/sh
TAR_EXCLUDE="--exclude='*.core' --exclude='*.a'
--exclude='*.o' --ignore-case --exclude='*cache*'"
# This command works:
/usr/bin/tar cvj --exclude='*.core' --exclude='*.a'
--exclude='*.o' --ignore-case --exclude='*cache*' -f
backup.tar /home/foo
# And this command doesn't:
/usr/bin/tar cvj $TAR_EXCLUDE -f backup.tar /home/foo
I tried putting echo behind each command, and I tried putting -x in the shebang; both showed the same output for both commands. (That make sense?)
What did I do wrong?
So I'm working on The Thing today, and it's decided that the automount daemon needs to be set up on The Inside Things. (The Inside Things are in a separate network, with The Outside Thing acting as a gateway between them and the rest of our internal network. And so everyone knows, The Thing is running FreeBSD.) It's not going to be left like this when The Thing is deployed, but it's handy for right now.
Only The Inside Things are on a separate network -- 10.0.0/24, as opposed to 192.168.0/24 for the rest of our network -- so ypbind isn't working. I'm not too familiar with NIS/NFS, so this is taking me a while to figure out.
Eventually I decide that I need to enable NIS for amd to work, and to do NIS I need to bind to the right server. Well, in the man page for ypbind I see the -S option: bind to a particular server. Should work, right?
So I boot The Inside Thing, and do these commands:
domainname thing ypbind ypset -h localhost -d thing 192.168.0.1
At the same time I'm running tcpdump on The Outside Thing to watch what happens, because these commands aren't working. And I see the weirdest thing: packets going to another, completely foreign IP address, port 111: RPC.
I scratch my head, try again: same thing. Reboot The Inside Thing, try again: same thing. The Inside Thing is running nothing more than NFS and SSH, I'm the only one on it, and still it keeps going to this IP.
I look up the IP address and it belongs to the Washington State Department of Transportation. WTF?
Try it on The Outside Thing -- unnecessary, since it's running amd quite happily, but I want to see what happens. Same thing.
I check the source code for ypset on the off chance that Theo de Raadt (he wrote it) put in some kind of trojan to...I don't know, ask for his driveway in Seattle to be plowed. Nothing -- but then, my rule of thumb has always been "If you're looking at source code, you're in over your head." (True for me, and if the source code is written in anything other than Perl or Bash. Still learning.)
I have no idea what the hell was going on. Anyone?
Flash! Just tried it at home on my FreeBSD gateway: same results. Jesus.
So I was trying to set up a diskless boot system on FreeBSD at work last week for The Thing. I'd found this article on booting FreeBSD with PXE, and it was nearly all I needed...except that this was about installation, and what I needed was a working system.
So I started fooling with it and trying to figure out how to add things like individual swap files and configuration information -- The Thing is going to have up to ten or so computers booting disklessly -- and then I came across a passing reference to /etc/rc.diskless1 and /etc/rc.diskless2. Bless their pants, the good folks at FreeBSD had already come up with a way of doing all this, and pretty much all I had to do was read those two scripts and /usr/share/examples/diskless/clone_root.
I got the hang of it pretty quickly and (mostly -- I have a habit of not following each and every instruction every single time, which is why Automation Rox) got it working...sort of. Something was going wrong and I couldn't figure out what it was.
See, what happens is you create, on the server, /disklessroot, which has a copy of almost everything the diskless unit is meant to use for a filesystem. /var is a memory filesystem (MFS) populated on boot with canonical files (see /etc/mtree), and /tmp is a symlink to /var/tmp. /etc, in turn, is filled with files specified in /conf on the server: you can set a default, then add stuff specific to particular hosts specified by IP address. And if a file called disklessremount is around in the right place, /etc will be filled first with the base files you'd expect, then overlaid with the stuff in the default section, then the host-specific stuff.
Only that last part wasn't happening. Somehow it'd get to the part where it was meant to copy the base files, then all these errors would pop up. I was convinced I was doing something wrong, but I couldn't figure out what. So, single-user then sh -x /etc/rc.diskless1 2>&1 | less. (God, I love the -x flag for sh/bash. Lovelovelove.) I print out the scripts so I can follow along. I follow along.
And I found a bug!
At one point it checks to see if the diskless_remount file is around; if it is, it creates the MFS, then proceeds to fill it. It uses eval a couple times in its checks, which always throws me; I've never been clear on what eval is for. But I muddled through it, then realized there needed to be a second slash on one line. Without it the MFS creation subroutine doesn't get the proper arguments.
Okay, no big deal; it's not like I found a ticking time bomb in the centre of the earth and had to alert everyone within 24 hours. But I was happy I was able to recognise it. I was all prepared to send in a PR when I found this one. Welp, there goes fame and fortune. :-)
In other news, it looks like FreeBSD does not like USB keyboards; we've got we're using for The Thing, where the PS/2 socket is inaccessible, but FreeBSD immediately panics and dumps registers when it finds it or when we plug it in. I'm going to see about maybe getting a core dump from it using Michael Lucas' excellent instructions.
So I'm at my new job (first job as a sysadmin...I'm so happy) and someone asks me if I can get an SSH server working on their machine. No big deal, right? I mean, everyone's running FreeBSD, right? Wrong: the computer this guy's talking about is running Windows 2000.
But I google, and fuck me if I don't come across instructions for running an OpenSSH server on Cygwin. Now don't get me wrong, I'd tried Cygwin before and had been very, very impressed w/the idea of running VI under W2K. But SSH? Cazart!
The instructions above were almost complete. But we've got a domain going (on Samba, natch), and the guy couldn't touch his files while logged in as Administrator. (I know, I know, shouldn't log in as Admin...but somehow it's hard to see that as being as bad as logging in as root...)
So some more google and came up with this. THe important bit is to change the GECOS field as suggested.
But: but but but: the other important bit is to start with a clean cygwin install. Follow the instructions here if need be. I banged my head against this long and painfully until I got it right.
On another note, I'm a Unix guy and I'm quite happy with that. But I've been thrust into W2K admin, and I find myself uncomforatably ignorant. One of the things I've always loved about Linux, FreeBSD et. al. is the sheer embarrassment of resources for the newbie. Where, o where are an equivalent humiliation of knowledge for Windows newbies? There has to be something out there for the technically proficient yet unfamiliar new guy. Bueller? Bueller?
So it looks like CNN has corrected the omission that I mentioned earlier. I guess it was just a mistake.
In other news, finally upgraded the link between Francisco (NFS server, FreeBSD) and Hardesty (NFS client, RedHat) to 100Mb/s cards. Aw yeah. Why did I leave it this long?
And I watched Logan's Run with my wife t'other night. To my surprise (she's not an SF fan) she liked it. And now I want to run around SkyTrain stations shouting, "Carousel is a LIE! You can LIVE!" Great stress reliever after a bad week at work.
Update March 21 2003: Found this Kuro5hin story on the transcript difference.
Cust. calls in having problems connecting: email works, but he can't go to websites. Check out his settings, everything looks find. So I give him an address to try typing in: http://207.102.64.2. "Type that into the address bar of your web browser, hit enter, and tell me what happens." Naturally, he can't dial in while on the phone w/me, so he hangs up and calls back a few minutes later.
"Didn't work," he said. "It said 'action cancelled'."
"'Action cancelled'?" I said. "That sounds weird. Internet Explorer said that?"
"No, Outlook Express. I don't use Internet Explorer to go to websites."
"I'm sorry?"
"I use Outlook Express."
"To go to web pages?"
"Yes."
Problem solved.
Message left on our voice mail:
"Hi, this is so-and-so. My email isn't working, hasn't been for about 12 hours or so, so something's obviously wrong at your end. I can still get in through your back door, but I don't want to do that. Give me a call, please."
I have someone on the phone with me right now complaining about the reception on their TV.
The gentleman went into great detail about the set (one year old Panasonic), what happens to the picture (solid horizontal line, thin; dancing white lines, also thin) and the variation over time (usually works for an hour in the evening, crappy other times). He's blaming it on the internet access built into his building (RJ45 ports in apartments, wireless access from the roof of one of the buildings), which we do helpdesk and provide connectivity for.
cf: http://www.cnn.com/2003/US/02/14/sprj.irq.un.transcript.1/index.html and
http://www.heraldsun.news.com.au/common/story_page/0,5478,5987651%255E663,00.html"
Notice the absence of this chunk of Blix' report in the CNN transcript:
I trust that the Iraqi side will put together a similar list of names of persons who participated in the unilateral destruction of other proscribed items, notably in the biological field.
The Iraqi side also informed us that the commission, which had been appointed in the wake of our finding 12 empty chemical weapons warheads, had had its mandate expanded to look for any still existing proscribed items. This was welcomed.
A second commission, we learnt, has now been appointed with the task of searching all over Iraq for more documents relevant to the elimination of proscribed items and programmes.
It is headed by the former Minister of Oil, General Amer Rashid, and is to have very extensive powers of search in industry, administration and even private houses.
The two commissions could be useful tools to come up with proscribed items to be destroyed and with new documentary evidence. They evidently need to work fast and effectively to convince us, and the world, that this is a serious effort.
The matter of private interviews was discussed at length during our meeting. The Iraqi side confirmed the commitment, which it made to us on 20 January, to encourage persons asked to accept such interviews, whether in or out of Iraq.
So far, we have only had interviews in Baghdad. A number of persons have declined to be interviewed, unless they were allowed to have an official present or were allowed to tape the interview.
Three persons that had previously refused interviews on UNMOVIC's terms, subsequently accepted such interviews just prior to our talks in Baghdad on 8 and 9 February. These interviews proved informative. No further interviews have since been accepted on our terms. I hope this will change. We feel that interviews conducted without any third party present and without tape recording would provide the greatest credibility.
At the recent meeting in Baghdad, as on several earlier occasions, my colleague Dr. ElBaradei and I have urged the Iraqi side to enact legislation implementing the UN prohibitions regarding weapons of mass destruction. In a letter just received two days ago, we were informed that this process was progressing well and this morning we had a message that legislation has now been adopted by the Iraqi National Assembly in an extraordinary session. This is a positive step.
Mr President, I should like to make some comments on the role of intelligence in connection with inspections in Iraq.
A credible inspection regime requires that Iraq provide full cooperation on "process" -- granting immediate access everywhere to inspectors -- and on substance, providing full declarations supported by relevant information and material.
However, with the closed society in Iraq of today and the history of inspections there, other sources of information, such as defectors and government intelligence agencies are required to aid the inspection process.
I remember how, in 1991, several inspections in Iraq, which were based on information received from a Government, helped to disclose important parts of the nuclear weapons programme.
It was realized that an international organization authorized to perform inspections anywhere on the ground could make good use of information obtained from governments with eyes in the sky, ears in the ether, access to defectors, and both eyes and ears on the market for weapons-related material. It was understood that the information residing in the intelligence services of governments could come to very active use in the international effort to prevent proliferation of weapons of mass destruction.
This remains true and we have by now a good deal of experience in the matter. International organizations need to analyse such information critically and especially benefit when it comes from more than one source.
The intelligence agencies, for their part, must protect their sources and methods.
Those who provide such information must know that it will be kept in strict confidence and be known to very few people. UNMOVIC has achieved good working relations with intelligence agencies and the amount of information provided has been gradually increasing. However, we must recognize that there are limitations and that misinterpretations can occur. Intelligence information has been useful for UNMOVIC.
In one case, it led us to a private home where documents mainly relating to laser enrichment of uranium were found. In other cases, intelligence has led to sites where no proscribed items were found. Even in such cases, however, inspection of these sites were useful in proving the absence of such items and in some cases the presence of other items -- conventional munitions. It showed that conventional arms are being moved around the country and that movements are not necessarily related to weapons of mass destruction.
The presentation of intelligence information by the US Secretary of State suggested that Iraq had prepared for inspections by cleaning up sites and removing evidence of proscribed weapons programmes. I would like to comment only on one case, which we are familiar with, namely, the trucks identified by analysts as being for chemical decontamination at a munitions depot. This was a declared site, and it was certainly one of the sites Iraq would have expected us to inspect.
This is a pretty big chunk. Without it, the dry humour of "Our reservation on this point does not detract from the appreciation of the briefing." just makes no sense -- there's no context to it.
I checked CNN's HTML, and it's not in there; there's no typo keeping it from view. Conspiracy theories are for the weak. Anyone got any other explanations?
Pages at 1.15am suck ass.
Writing this on RH8.0. And oh, the difference.
I started using Unix five years ago with Slackware when I bought my first computer (486, oh yeah) over the Internet (I had been thinking about Win9x but was worried about viruses); moved to Debian after reading CmdrTaco's raves; moved to FreeBSD after getting a job at an ISP that used (uses) FreeBSD pretty much exclusively; and now I've downloaded all five ISOs of RedHat 8.0, and I'm going to do my best to use it exclusively, at least at home.
I'm doing this because I'd like to take the RHCE exam. I've read about it, and it seems like a really good qualification -- I'm particularly taken w/the hands-on exam. As far as job qualifications go, I've got a fair amount of experience (enough to get me a junior position, if I had to look), but no certification; as I want a job as a sysadmin, this seems a bit of a lack. Becoming an RHCE seems the best way to fill that gap.
I must admit, I've forgotten what it's like not to have the packaging system do the thinking for you. One of the big reasons I moved to Debian was for the ease of installing new programs; I was sick to death of downloading a cool program, only to find that it depended on six separate libraries, each of which had four separate dependencies. It's such a thrill to just apt-get install foo or cd /usr/ports/devel/foo && make install distclean and then walk away. Trying to do that sort of thinking again is like...I don't know, forgetting how to walk and having to do the math by hand.
For example, I tried to install IceWM over the last couple of nights, and I couldn't get it to work. It depended on libdb3-1, but using RPMFind and FreshRPMs.net I was only able to find 3.3. Maybe not a show-stopper -- I didn't try forcing the installion and seeing -- but I didn't want to risk it; the current install is about the fourth in as many weeks (don't install Linux after !}fmt FreeBSD after Linux, kids!), and I didn't want to bother w/YARI. I gave up in the end, compiled from source (which, while surely part of The Linux Way doesn't seem to be part of The Reddat Way. Got it installed no problem, but then came the problem of how to start it up.
I went through a fairly default install of RH8.0, including selecting Gnome for a default environment (though installing KDE as well). That meant the default runlevel was 5, and so GDM started up. I found /etc/sysconfig/desktop, but setting DESKTOP to icewm or /usr/local/bin/icewm just didn't work. I gave up -- I was getting sick and tired of a) GNOME not working w/a home directory mounted over NFS (grr) and b) KDE trying to grab URLS whenever I highlighted something and c) both environments slowness (I've got a 450MHz celeron, 384MB ram, and d) both steal too many cycles for my liking) and lack of a terminal screen in easy and close and prominent proximity -- and set runlevel to 3. I rebooted, changed .xinitrc, typed startx and breathed a sign of relief. Cheating, sure, but I'd really like to have a working desktop before the year is out.
So now I get to learn about rpm. And hopefully I can put RHCE after my name (no, not really) within a year or so. That'd be nice.
So update time on the not-so-new-anymore mail server.
SpamAssassin has been working out just ducky. I had the threshold set to 14, then 10, and I just lowered it to 9 yesterday. I'm keeping an eye on it as I go, because there are legitimate messages (mainly newsletters from Real Companies[tm]) that piss off SA -- "click here to unsubscribe", "you're getting this because", etc. -- and we need to whitelist 'em as we find 'em. Only, w/tens of thousands of messages being caught every day, that's a lot to look through...so it's taking a while.
As far as stats go, at threshold 10 we caught ~ 28k messages in 24 hours. In the 14 hours since I lowered it to 9, we've caught ~ 35k. Fuck me...
We've had one weird hardware problem. At 4am on Saturday morning I got a page (ugh) saying that the server was down. Tried pinging it, and yup, no response. I put our backup mail server on the front end and went back to sleep.
In the morning I went to check it out, and it seemed to be just frozen. Last log message sez:
xl0: watchdog timeout
WTF? Rebooted, saw a lot of "Stray IRQ" messages, and it seemed happy. Put it back on the front end, but let the backup server stay there too.
Dave the SysAdmin found this message on the FreeBSD mailing lists. It suggested that the problem might be because of a couple PCI slots sharing an IRQ; when the guy moved his network card to a slot that didn't share an IRQ, the problem went away. I checked the manual for the mobo (Gigabyte VR7XP), and it looks like the slot the card was in didn't share an IRQ. However, I took a few minutes, shut down the machine, and moved the card (3Com fill-in-the-blank-here) over a slot anyway.
While I was there, I checked out the BIOS and found something moderately interesting: APM was turned off, but in the options it had different IRQs it could wake upon. One of the four that were turned on was IRQ 7, which was the stray one that the box had been complaining about. I turned 'em all off. Bad me for not turning off all that in the first place.
It's held up fine after that last reboot, and now it's the only one on the front-end again. (Good thing, too; the backup mail server doesn't have SA installed, as it's also a webmail + web server.)
So now the new mail server is up, though not accepting mail. I played around w/vinum last night and this morning from home (Let's hear it for OpenSSH!), and broke it -- but figured out what to do next.
The problem came in the way I'd laid out disk space w/the first install. Okay, so four drives; 128M for /, the rest for vinum. Only instead of setting up all the partitions on all the drives as I wanted them to be in the end (equal in size for raid5), I'd made a big /usr partition, then hoped I could juggle /usr over to a raid partition, re-disklabel the first drive to make the partitions the way I wanted them, add another subdisk/partition/slice each to the existing raid5 drives -- gah. No wonder it fell apart. The obvious solution was to make the /usr partition smaller (didn't need it as big as I'd originally set it), and go from there.
Got excited at this realization, so went into work an hour early (such a geek) and worked on it all day. Got it into the server room, up and running, even set up Sendmail for the first time (tested it for relaying, hands sweating), and got it all up and ready for tomorrow.
One weird thing was that I couldn't figure out how to get FreeBSD's port of net-snmpd (formerly ucd-snmpd) to work w/tcp wrappers and hosts.allow -- it would only allow connections with ALL : allow ; anything else would just refuse connections. Weird. I'll have to figure it out tomorrow.
ARGHHH...
I work at a small ISP, and among other things I help out the sysadmin w/the mail servers and the spam filtering we do (procmail-based, but we're trying to get SpamAssassin installed Real Soon Now). Yesterday I noticed that one of our front-end mail servers, which should also have been doing secondary DNS, was not doing secondary DNS. Turns out the /var partition was filled beyond capacity because of the fucking HUGE maillog generated because of the waves of spam we've been getting lately.
Admittedly, it's Not A Good Thing to have DNS + SMTP all on the same box; we've got a new mail server in [very hibby: half gig ram, 4x40GB drives, 2GHz or some-such P4] and we're trying to get it up, at which point the small box that crapped out can do just secondary DNS. But still.
And so but took a look at the queue, and took out 26,000 messages that were just bounces back to spamming mail servers refusing connections. 26k! We use both OR-something-or-other (ORDB died, I think...can't be bothered to look it up now, but we use whatever took up the mantle) and SpamCop's BL, and the fuckers still make it through. Arghhhhhhh.
As I said, it's a small ISP...which means that my official title is HelpDeskSuperHero (tm), which means that I get calls about all this, and have to talk people down off the ledge w/all the spam they're getting recently. It's the same talk every time, sometimes to people at the same office, and it's frustrating because a) I don't have the One Magic Wand they think I do, b) well, I might have the Magic Wand in a while, but not yet, and c) I keep having to explain why filtering out Naughty Words is really not the best idea.
Heh...not being terribly coherent right now; I'm usually better than this, but I'm still waking up and this all just makes me mad. Spammers fill up our mail queue, put the load on our mail servers through the roof, and anything I can do at the moment comes after the fact: it does nothing to prevent it, and very little to make things better. Sometimes for fun I tail /var/log/procmail.log and it's just insane: there is no possible way I can catch everything, or even react fast enough to catch what's coming in right now.
I realize none of this is News, but it's crazy to me how...how fucking crazy this is: we're spending our time, money and brainpower in what seems like a constantly losing race against a small number of people with the ability to ruin email. Spammers have power far out of proportion to their numbers, and it seems like it's enough to swamp us, and maybe lots of other people too.
On the subject of SpamAssassin: God it's good. I use it at home (FreeBSD + Procmail) and it fucking ROX. I dearly wish that there was a Windows version for Outlook Express, preferably free, that I could point our customers to, but nowt luck there: DeerSoft (kudos to them) have a version for Outlook, but not OExpress. Ah well.
Sigh. /me takes deep breath. Soon we will have SpamAssassin installed, I hope, and then everything will be better.
I promised a month ago (!) to put up the firewall rules I'd come up with for FreeBSD; here we are at last. With any luck this'll be useful for someone.
By way of background, I had the honeypot, a default server install of Redhat 6.2, sitting behind my main box running FreeBSD; the FreeBSD box had one external card (cable internet), one internal card for my LAN (yes, I was using my LAN while this was all going on...) and one internal card dedicated to the honeypot.
This is a fairly restrictive ruleset, but I didn't want to fuck up and risk letting Bad Packets (tm) out. As related earlier, I had to shut it all down about five minutes after I got cracked anyhow, so it was a moot point.
Oh, almost forgot: the funky part is not just the firewall rules below, but running the natd daemon w/the right options:
natd -t 10.0.1.1 -interface xl0
From man natd:
-t | -target_address address
Set the target address. When an incoming packet not associated
with any pre-existing link arrives at the host machine, it will be
sent to the specified address.
That made it a great deal easier to pass traffic initiated from the outside to the honepot. I'm sure there's a way to do this in Linux, but it's been long enough since I worked w/Linux firewall rules that I wouldn't know what that is.
Anyhow, w/o further ado:
#!/bin/sh
IPFW="/sbin/ipfw"
PRIVATE_LAN="10.0.0.0/24"
HONEYPOT_LAN="10.0.1.0/24"
HONEYPOT="10.0.1.1"
EXTERNAL_NIC="xl0"
INTERNAL_NIC="ep0"
HONEYPOT_NIC="ep2"
MY_HONEYPOT_IP="10.0.2.2"
PUBLIC_IP=`ifconfig $EXTERNAL_NIC | awk '/inet / {print $2}'`
$IPFW -f flush
$IPFW add 100 pass all from any to any via lo0
$IPFW add 200 deny all from any to 127.0.0.0/8
$IPFW add 300 deny ip from 127.0.0.0/8 to any
$IPFW add divert natd all from any to any via $EXTERNAL_NIC
$IPFW add check-state
$IPFW add skipto 40000 all from any to any via $INTERNAL_NIC
# Honeypot rules...be very careful!
# Logging.
$IPFW add allow log udp from $HONEYPOT to $MY_HONEYPOT_IP syslog
via $HONEYPOT_NIC
# Allow in from outside world. Remember, natd will be passing
these
# packets on. Keep traffic from own network out.
$IPFW add allow log tcp from not $PRIVATE_LAN to $HONEYPOT via
$HONEYPOT_NIC
$IPFW add allow log tcp from not $PRIVATE_LAN to $HONEYPOT via
$HONEYPOT_NIC
$IPFW add allow log udp from not $PRIVATE_LAN to $HONEYPOT via
$HONEYPOT_NIC keep-state
$IPFW add allow icmp from not $PRIVATE_LAN to $HONEYPOT icmptypes
0,3,8,11,12,13,14 out via $HONEYPOT_NIC keep-state
# Allow replies from ftp, ssh, pop and web...put in mainly for ftp
replies. NO SMTP!
$IPFW add allow log tcp from $HONEYPOT 20,21,22,110,80 to not
$PRIVATE_LAN via $HONEYPOT_NIC
# Do we need this?
$IPFW add allow log tcp from any to $HONEYPOT in via $EXTERNAL_NIC
$IPFW add allow log udp from any to $HONEYPOT in via $EXTERNAL_NIC
$IPFW add allow log icmp from any to $HONEYPOT icmptypes
0,3,8,11,12,13,14 in via $EXTERNAL_NIC
# What we allow out: established, mail, ftp, web, domain, selected
ICMP.
$IPFW add allow log tcp from $HONEYPOT to not $PRIVATE_LAN
established via $HONEYPOT_NIC
$IPFW add allow log tcp from $HONEYPOT to not $PRIVATE_LAN
20,21,22,25,80,110 via $HONEYPOT_NIC
$IPFW add allow log tcp from $HONEYPOT 20,21,22,110,80 to not
$PRIVATE_LAN via $HONEYPOT_NIC
$IPFW add allow log udp from $HONEYPOT to not $PRIVATE_LAN domain
via $HONEYPOT_NIC keep-state
$IPFW add allow log icmp from $HONEYPOT to not $PRIVATE_LAN
icmptypes 0,3,8,11,12,13,14 via $HONEYPOT_NIC
# Deny the rest
$IPFW add deny log all from $HONEYPOT to any
# Should this be in or via?
$IPFW add deny log all from any to any in recv $HONEYPOT_NIC
$IPFW add deny log all from any to any via $HONEYPOT_NIC
#add allow udp from any to any keep-state
$IPFW add 40000 allow udp from $PRIVATE_LAN to any keep-state
$IPFW add 40100 allow udp from $PUBLIC_IP to any keep-state
$IPFW add 40200 allow tcp from any to any established
$IPFW add 40300 allow tcp from any to $PUBLIC_IP 22,8000
$IPFW add 40400 allow all from any to any out via $EXTERNAL_NIC
$IPFW add 40500 allow all from any to any in via $INTERNAL_NIC
$IPFW add 40600 allow all from any to any out via $INTERNAL_NIC
#add allow all from any to ${PUBLIC_IP} 22, 8000 via ep0
$IPFW add 40700 allow all from localhost to 25 via lo0
$IPFW add 40700 allow icmp from any to any icmptypes 0,3,8,11,12,13,14
$IPFW add 40900 deny log all from any to any