The Life of a Sysadmin

While looking for an article on Windows XP's SP2 firewall doohickie, I came across this article. Check out these gems:

When you check Don’t allow exceptions, XP won't accept incoming connections for network services that appear on the exceptions list. This feature is handy when you suspect your machine is the target of malicious activity, as well as when you’re connected to the Internet using a public, possibly unsecure, connection.

When you suspect your machine is the target of malicious activity, as well as when you're connected to the Internet using a public, possibly unsecure, connection. Holy crap, did I miss the return of 1986 or something? It gets better:

This mode also is useful when a Trojan or worm attempts to propagate across a network. If detected early, you might be able to prevent a machine from becoming infected by disabling access to local shared resources and services. When the threat has passed, you permit XP to accept incoming requests for applications on the Exception list by clearing the Don’t allow exceptions check box.

Believe it or not, I'm speechless. Yep. Out of speech. No speech for me, thanks -- I'm full.

I decided this week to get Amanda working properly at home. I've got an old DDS3 tapedrive in Francisco, my FreeBSD firewall box, but all I've been doing so far is tarring to it once a week.

Setting up Amanda wasn't much of a problem, but I kept getting short write errors -- the damn thing was giving up and saying the tape was full after only about 3GB. I decided to run amtapetype, which takes about two hours per run with my hardware, in order to figure out exactly how much space I had. The first time, it said 2GB. WTF? The second time, the drive crapped out with errors about how a power reset had been detected. I decided to shut down Francisco and reseat the cables just in case. No problem, right?

Wrong! When I brought up Francisco again, it refused to boot -- lots of scary errors about how the hard drive couldn't be read, or found, and maybe the LIES about having a hard drive present should just stop now, huh? Francisco is old: it's an old P90 scrounged from an old job, stuck in this black case with non-working LEDs and a Punisher logo someone poked out in toothpick-sized holes on the front. No cooling fan, four ISA slots and three PCI, and I had to jiggle the BIOS so that it would boot from a 100MB partition at the beginning of an 80GB hard drive. Seems like as good a time as any to simply replace the damned thing...

...but first, a firewall. I tried booting it from an old laptop hard drive I had around, but that didn't work. I tried getting it to boot from a Slackware Live cd, but the whole concept of booting from a CD just made Francisco huddle in the corner in the fetal position.

Nothing else for it: it was time to do The Bad Thing. I grabbed one of the ethernet cards from Francisco, shut down Thornhill (P3, 500MHz, web and DNS server, Slackware and 2.6.7 kernel) and threw it in. A quick module recompile for tulip^Wvia-rhine and that was up; some judicious editing of the firewall set it up for NAT. Ph35r m3!

(Side note: Man, it's been far too long since I set up NAT on Linux; I still don't really understand what I've done. I've worked with FreeBSD for firewalls almost exclusively over the last four years, and I have some serious catching up to do.)

So now the question is: what do I do to replace Francisco? I know, finding a Pentium similar to Francisco is not that hard at all. But dammit, I'm tired of big, noisy boxes that are just waiting to die. I want something small, quiet, and reasonably new; I don't want to be fiddling with it, or worrying about it running out of memory (I tend to run far too much on a firewall, and 92MB of RAM just aggravates the problem).

It's complicated a bit by the recent heat-death of Hardesty, a 300MHz Celeron that had, 'til recently, been my desktop machine. I'd been hoping to replace or upgrade that, too; I've gotten quite used to a fast processor and lots of memory at work, and 15 seconds to render Slashdot's front page seems less like acceptable and more like a sign that civilization is in decline.

So...one option is a VIA Epia Cl6000. Dual ethernet, fanless goodness. That, and a case -- unless I decide to build my own Bubba can computer -- and some memory, and maybe a hard drive or maybe PXE booting. Whee! That'd make a pretty decent firewall and fileserver, no question.

But another option would be to let Thornhill keep doing the firewall thing, even though it's a webserver and should, like, rilly be outside the firewall, or at least in a DMZ. I could do something really funky like run Apache inside User-Mode Linux. Or maybe my own stuff, although I'm sure X would be a bear to get working.

A third option would be to keep using Francisco, but w/o a hard drive: let it PXE boot and do all the firewall stuff that way, totally stateless (well, hard drive-less). That could be interesting: almost no moving parts at that point. That would let me get a Mini-ITX something-or-other to use as a desktop machine. They're not the most powerful processors around, but when you can compile a kernel in 6 minutes, who the hell cares? Or maybe a Shuttle, so I could keep using my video card. Hm...

Well, enough of that for now; my cat needs chasing. And anyhow, King of the Hill season premiere tonight! @Woo!

I want my own WHOIS server! Completely made up stuff, of course, and back to its original purpose of white pages for th'Internet. Something like:

$ whois liddy
LIDDY: Dark God Of This Universe. The Black Goat
of the Woods with a Thousand Young. Your ass is grass, and He is the
fertilizer.
$ whois Tom Petty
TOM PETTY: Evil underling of Liddy, and His adopted son.

etc...Listen on port 43 and away you go. How cool would that be? Generate random responses (output of babble, perhaps?) to queries.

TODO: Find a Free WHOIS server post-haste.

This made me laugh.

The sumbitches are at it agin', mother. Comment spam is infecting both my blog and my wife's. So far a relatively small number of keywords -- poker, Texas, debt -- is sufficient to keep 'em away from where Google can see 'em. Well, that and OCD-like running of SELECT statements in MySQL. But the fuckers are gonna be the death of me, or at least blog comments. Although maybe some sort of SURBL plugin for URLs in the post...that'd be cool. Someone must have something like that already.

Not that I notice a whole lot of comments, anyhow, at least away from the Slashdot side of things...although I do notice that I've made it onto somebody's blogroll. How'd that happen?

In other news: I finally decided what to do about new computers: buy a new Shuttle Sk43G, Sempron processor, and make that my web server; then, make my current webserver (older Compaq P3-500 desktop machine) my desktop and firewall: lots of room for ethernet cards, tape drives and whatnot.

I agree, it's a little silly that the more powerful box becomes the horribly underutilized server, but such is life. If there was a comparably cheap shuttle that came with two onboard ethernet interfaces, I'd be buying that instead.

So dive right in, right? I got the new box home last night, assembled it and booted w/o problems. It took little effort to move the hard drive from the web server and put it in the new, tiny box; sure, I had to recompile the kernel (8 minutes! eat that, P90!) to get the right drivers in, but nothing big. Until, that is, it froze. Hard. And only a few minutes after booting. If I ran top and set it to update continuously, I could get to freeze within seconds.

Some fiddling with Grub (boot loader of the GODS, man) showed that the problem seemed to go away if I went with the original Slackware stock 2.4.20 kernel instead of the 2.6.7 kernel I'd last compiled. (I'm a packrat, and that includes keeping every kernel compiled on this damned thing, Just In Case, because You Never Know.) We've got one of these boxes at work with an Athlon XP and it works fine; admittedly, it's not doing much, but neither is my web server. (Ba-zing!)

God only knows what's going on there, but it didn't last: I left it on overnight to see if it'd keep going, and sure enough it froze again around 10pm. I put the HD back in the P3 and left it. I'm going to see Wilco tonight (Whoo! WilCO! WHOO!), so this'll take a back seat to some serious RAWK. Except I'll probably be speculating about crappy memory or badly applied heatsink paste the whole time. No. No, I won't. It's Wilco.

Actually, I'm thinking I may have to upgrade the BIOS in order to get it to work properly with the Sempron; originally it was detected as a 900MHz Athlon, and I had to tweak the bus speed and whatnot to get it to run at 1.5GHz. (Interestingly, this seemed to have no effect whatsoever on how quickly it would crash, compared to the difference the different kernel version made.) (God, that's an awful sentence. I'm sorry, everyone.)

Anyhow, there's probably lots wrong with the settings; I never really wanted to learn about memory spacings and CPU voltages and I don't know what-all.

In other other news, I mentioned that I moved last week, but I didn't mention that I came back to two, count 'em TWO dead computers. (Before you ask: Support contracts are for the weak, and I suspect I'm about to get very weak.) One was a Linux box whose hard drive gave up the ghost. Stupid IDE hard drives in a dusty, hot environment anyway! But the other was was an old Duron whose motherboard's capacitors yearned to be one with the cosmos (ie, they blew up real good). That was running Windows, so the whole let's-just-throw-the-hard-drive-into-another-box-and-see-if-it-boots thing was good for a very, very bitter laugh but little else.

Instead, I reinstalled not only Windows but Cygwin, too. That proved to be harder; we use Cygwin to compile very particular things that depend on version 2.2 of Python. Version 2.3 makes things cry. And no matter how much you tell the Cygwin installer that you don't want to upgrade Python, it goes ahead and does so anyway like some hyperactive sugar-fueled kid who's certain he knows how to fix things.

After far too much experimentation, I did what I should have done in the first place: I found an old archive of Cygwin, with the right version of Python, and I mirrored it. One gigantic, nine-hour long sucking sound later, and I had a local copy to point the Cygwin installer at. Thank god.

Finally, just got in the first 19" LCD monitor at work. This was, of course, two weeks after assuring someone that they were too expensive to get past the boss. My bad. I'm going to get a lot of mean looks, I think. But then, if I was a people person, why would I have become a sysadmin?

Recommendation of the Day: Vicious Battle Rap, by DJ Format and Abdominal. Bow down, baby.

SK43G, Sempron 2200. eth0: Via Rhine driver -- DLink 350TX? I'll have to look it up. eth1: RealTek 8139 onboard. ifconfig eth0 192.168.0.1 netmask 255.255.255.0 route add default gw 192.168.0.254 (log in as self) ssh 192.168.0.254 BAM -- freezes hard, and even the Magic SysRq key does nothing. Reboot... ifconfig eth1 192.168.0.1 netmask 255.255.255.0 route add default gw 192.168.0.254 (log in as self) ssh 192.168.0.254 Password: BAM! (the good BAM, this time) Yay! No BIOS upgrade required maybe! (UPDATE: Spelled out which one was eth1 [the onboard Realtek]. What a maroon!)

Here's a few more details on the problem with the new Shuttle. First, the card is a DLink DFE-530TX; the Shuttle is an SK43G. If the DLink is connected to my internal network switch, and from there to the gateway box, this sequence will make it freeze:

1. ifconfig eth1 192.168.0.1
2. route add default gw 192.168.0.254
3. ssh 192.168.23.254

Interestingly, if the network cable is unplugged, the problem doesn't show up...so it appears there's something about the response to the three-way handshake is what's causing the problems.

I managed to find some reports of wireless cards locking up hard with the VIA KM400 chipset, including cards from DLink. I tried setting all the IRQs to "Reserved" in the BIOS, and that didn't work; however, the card was grabbing IRQ 17, and the BIOS wouldn't let me reserve that one. I also tried upgrading the BIOS, and that didn't work either.

I'd love to pursue it further, but it's now officially the new webserver; I wanted to get it installed while I had a day to fool around with it and get everything working. So far there don't appear to be any problems.

And now, of course, I've got what used to be Thornhill as my desktop machine: P3 500MHz, 640Mb, and a new 160GB Seagate Barracuda. Once again, I'm going with Debian, God's own distro. Still gotta come up with a name for it.

I'm currently trying out KDE and Konqueror -- usually I use IceWM and Firefox, but I thought I'd give something fancier a try now that I've got a slightly hibbier machine. It's not bad so far, although having to set up all the keyboard shortcuts that come with Ice is a little annoying. We'll see how long it lasts.

I run my own web and mail server: thornhill.saintaardvarkthecarpeted.com. I host 6 or 7 domains right now for friends and family. I'm on Shaw Cable, a big-ass ISP up here in Canuckistan. Thornhill is listed as the MX for all the domains I host.

I recently gave up the SOHO package, which for $120/month gave me TV, internet access, the right to run servers and one (1) static IP address. Now I'm a renegade, hiding from the law and running my servers on addresses ladled out by DHCP. I run a client on Thornhill to update EveryDNS.net's records. (Good folks, by the way, and recommended.)

Today I tried SSHing to Thornhill, and it timed out. The websites were working, and I could ping the rest of the Internet ("ping 255.255.255.255"), so WTF?. I ran host thornhill.saintaardvarkthecarpeted.com, and got its old static IP address - the one that hasn't been in use since the end of October. I tried querying my ISPs nameservers directly using dig, and got the same result: both kept listing the old, static IP address for thornhill, but the correct address for www.saintaardvarkthecarpeted.com. Meanwhile, querying EveryDNS' nameservers, or any other nameservers I could think of, gave the correct, current, dynamic address. I queried many times but kept getting the same result.

No wonder mail seemed a little thin: no one on Shaw would be able to send us mail, and anything we sent to each other would also get lost, too (since we're both still using our ISP's mail server...still trying to get exim to work for me on Rearden, the new firewall box).

I thought about it, and decided it was worth trying to add another MX record. I added saintaardvarkthecarpeted.com, which has its own A record, and set the score/cost/preference to one less than Thornhill's. I figured that maybe Shaw's nameservers would at least check the MX when trying to bounce mail, or run the queue again, and see the updated record. I checked again with dig to make sure that Shaw's nameservers still had the correct IP address for saintaardvarkthecarpeted.com (yep), and again to see if it'd changed its mine about Thornhill (nope). Then I asked what they thought about MX records. Sure enough, two were listed.

Just for fun, I tried querying again about Thornhill's IP address, and fuck me if it hadn't suddenly changed to the new, dynamic, correct one! And not only that, but five minutes later all sorts of email from folks on Shaw started coming in.

Well, that was one nameserver down -- what about the other one? I queried it for Thornhill's IP, and it was the old one. I queried for the MX records -- both were listed. I repeated the query for Thornhill's IP, and bam -- just like that, it had been suddenly updated to the correct IP.

That's where things ended last night; my wife and I watched Coffee and Cigarettes (Bill Murray! Iggy and Tom!). But I set up a cron job to keep querying the nameservers about Thornhill's IP address. And you know what? 6AM it was fine. 7AM it was fine. But at 8AM I saw the same behaviour: get Thornhill's old IP address, query MX, get Thornhill's new IP address.

'Sfucked up, mang. The only thing I can think of is that maybe there's a crapload of DNS servers behind load-balancing, and I'm getting different ones at different times.

Simon Fraser University:

The core environment in support of research computing comprises
currently of a 200 processor compute cluster and 140 TB storage
facility, which also functions as the central storage facility for
the Western Canada Research Grid.

and NEPTUNE Canada:

As a member of the NEPTUNE Canada DMAS development team, the
successful candidate will be responsible for the installation,
implementation, administration, management, maintenance and
ongoing support of all operating systems, communications systems
and applications systems including hardware, software as well as
networking components necessary to develop the NEPTUNE Data
Management and Archiving System.

Sigh...someday...

Great story:

I expected that the contractor would be waiting for me with the cables finished when I got there. Nope. I found upon arrival that the electrical contractor doing the installation was not the same company that I’d been working with before. THAT company was a highly competent bunch, supplying trained workers capable of doing any task in a heavy industrial environment. High voltage cables were an everyday job for these folks. What was out there installing cables was small-town electrical contractor, apparently some sort of “brother-in-law” deal having been made. And while these guys might be fine installing the 277-volt feeders for a Burger King restaurant, they were over their heads in dealing with the 15,000-volt cable they were installing here.

Anyone who can tell me who this is (no fair if you saw the earlier entry) and the name of the song will totally get a link on the right-hand side of this blog. Fame and celebrity await!

Who knew? Do yourself a favour and go buy everything this man has written, and read it all. Then read it again. He's just that good.

So Gecko and I have been doing some interesting work with WordPress this weekend. My wife and I visited him and the lovely Arwen on Friday, drank too much wine, and when we woke up in the morning had a lovely Logitech USB headset, originally meant for a Sony PlayStation, sitting in our laps. We also had fuzzy memories of barked instructions to start "podcasting". Wha'?

First, we had to get the headset working. On my box (Debian Testing with a 2.6.9 kernel) it was a simple matter of getting ALSA modules compiled. Since I still hadn't got around to getting my sound card going after the big move, this had the pleasant bonus of being able to listen to music again.

When everything was done, I was able to run:

arecord -D plughw:Headset | oggenc - -o foo.ogg & sleep 60 killall arecord

and have a tasty OGG file at the end of it. Sweet!

But what about Ms Topo's computer? She's running RH9, and I had no interest in picking this weekend to migrate her to something newer. I knew (well, okay, I Googled and found out) that RH didn't do ALSA, so that left me with the fun of trying to bolt it on. I tried following these instructions, and it didn't work: whenever I tried to modprobe the new ALSA modules, I got lots of "unresolved symbol" errors. NFG.

Well, what about a new kernel? Could try upgrading to 2.6.9, right? Nope: RH uses initrd when booting, and I've never wrapped my head around that. But guess what? When booting back into 2.4, kudzu found and configured the USB headset automagically. Teach me to underestimate RH...

Okay, so that part solved. Next part was to figure out what the hell "podcasting" is. And for the love o' Linus, it's just a URI in an RSS 2.0 feed that points to a thing: an image, an MP3 file, whatever. They call it an enclosure, but it's just a fucking link! RSS is the new HTML. Somebody, somewhere, is going to figure out how to do TCP over RSS, and I won't know whether to laugh or cry.

(Hey! Google finds no pages with the phrase "TCP over RSS". You heard it here first, kids.)

But back to our story. So how the hell do you get an enclosure in your RSS 2.0 feed? Well, if you're using the ever-lovin' WordPress, you can either get the Alpha nightly releases, or you can make some judicous modifications to a few files. I backed up the originals, copied the others into place, made the right changes to the database, and baaaaaaaaaaaaaam!

Last step: oh yeah, an MP3. (Stupid patented file formats...) Quick look around found Audacity, and holy crap is that cool. The first time I started it, I got a little popup:

There was an error initializing the audio i/o layer. You will not be able to play or record audio. Error: Host error.

but turning off XMMS fixed that right up. I quickly recorded two tracks, exported the mess to MP3, put it up on the server, and hey-ho, let's go! Sir Gecko checked it out, and it worked on his iPod. What is it with Apple people, anyway?

Next step: Topo and Gecko do the ADD show. Watch for updates.

http://www.google.ca/search?hl=en&q=%22tcp+over+rss%22&btnG=Google+Search&meta=

As I mentioned, it's been a busy weekend for Gecko and I. With anything good and joyous on the Internet come spammers. Comment spam has been a minor irritant for a while -- nothing I couldn't handle by logging into MySQL directly and running DELETE statements with extreme prejudice -- but in the last few weeks it's gone off the hook. With dozens a day, it was time to start doing something automatically.

WordPress is pretty good this way -- you can set up your comments so that everything needs to be approved by the admin, or just stuff that matches certain words in the comment or URL fields. That worked for a while -- "poker", "debt" and "cialis" took care of most things. But it isn't a very sohphisticated filter, so I started looking around for something else.

I found Fahim Farook's WPBlacklist plugin, and it works pretty damned well. It imports a copy of Jay Allen's blacklist, then holds for approval anything that matches the HOLY CRAP two thousand three hundred forty five lines of regexes (a few) and domains (the bulk of the list). Plus, you can tell it to delete a comment and harvest information from it -- so it knows to watch out for that (domain, email address) in the future. All in all, I was pretty happy.

But then Gecko pointed out this elegant solution. My first name is not so obvious ("Saint? What kinda first name is that? Damn kids..."), so I put in my own simple question.

It's a brilliant idea, really: come up with a question with an answer that's obvious a) if you're at the site and b) are not a spammer's computer. Which makes me wonder what'll happen when/if AI gets a bit more common, or if spammers will start funding natural language parsing research...shudder.

In other comment spammer news, there's a really good article here about what one guy managed to find out about a comment spammer. Finally, turns out that what I was going to say was said a year ago:

...but just like everything else, the weblogging community seems intent on (a) thinking they're special and unique and nobody has ever had their problems before, and proceeding to (b) ignore all the work that has come before and reinventing the wheel. Now, certainly some adaptation of code and algorithms will be necessary. Existing tools probably can't be used as-is. Email spam fighting relies a lot on the structure of an email, the chain of headers that give away so much information to the trained eye, and none of that information is available in weblog spam. But I see from Jay's Comment Spam Clearinghouse that the latest and greatest tool available to us is a master list of domain names and a few regular expressions. No offense to Jay or all the people who have contributed to the list so far, but how quaint! I mean really. Savor this moment, folks. You can tell your children stories of how, back in the early days of weblogging, you could print out the entire spam blacklist on a single sheet of paper. Maybe with two or three columns and a smallish font, but still. Boy, those were the days.

Holy crap. I thought I was cynical. The entire article is highly recommended.

A quick Google turns up this entry on using SURBL to fight comment spam. More information here. A quick look at the WP-Blacklist plugin shows it shouldn't be that hard to add a quick DNS check...Hm. And the SURBL mailing list has discussed this too:

>The quick and easy answer, which may be wrong, is that they're >different folks, or at least different domains. > >Jeff C. > Oh please don't think that just yet!! Seriously. I'm working with some ninjas and the 6dos data and a new tool to let you look up this info! So far it ROCKS beyond belief! But more coming, and trying to keep data source anonymous of course. Also trying to tie in some other tools that other SURBL submitters have been asking for. Bottom line is that these guys ARE the same people. Data shows it.

Hm. Update, Nov. 30: Double hm

Dammit! Someone's already integrated SpamAssassin with Wordpress! Now I'll have to show my legs to get attention... Actually, something that could be useful here is a blog honeypot in order to figure out how effective different mechanisms are. That could be interesting...