The Life of a Sysadmin

SWAKS, a command-line utility for testing SMTP. Available as a Debian package. Sample output:

$ swaks
To: aardvark at thingy saintaardvarkthecarpeted communist
=== Trying thornhill.saintaardvarkthecarpeted.com:25...
=== Connected to thornhill.saintaardvarkthecarpeted.com.
< -  220 thornhill.saintaardvarkthecarpeted.com ESMTP Postfix All Hail Liddy!
 -> EHLO rearden.saintaardvarkthecarpeted.com
< -  250-thornhill.saintaardvarkthecarpeted.com
<-  250-PIPELINING
<-  250-SIZE 52428800
<-  250-ETRN
<-  250 8BITMIME
 -> MAIL FROM:<aardvark at="at" communist="communist" rearden="rearden" saintaardvarkthecarpeted="saintaardvarkthecarpeted" thingy="thingy">
< -  250 Ok
 -> RCPT TO:<aardvark at="at" communist="communist" saintaardvarkthecarpeted="saintaardvarkthecarpeted" thingy="thingy">
< -  250 Ok
 -> DATA
< -  354 End data with <cr><lf>.<cr><lf>
 -> Date: Sat, 11 Dec 2004 09:45:49 -0800
 -> To: aardvark at thingy saintaardvarkthecarpeted communist
 -> From: aardvark at thingy rearden saintaardvarkthecarpeted communist
 -> Subject: test Sat, 11 Dec 2004 09:45:49 -0800
 -> X-Mailer: swaks v20040404.1 jetmore.org/john/code/#swaks
 ->
 -> This is a test mailing
 ->
 -> .
< -  250 Ok: queued as 594AA2CF
 -> QUIT
< -  221 Bye
=== Connection closed by foreign host.

Wish I'd known about this a long time ago.

After a lot of consideration, and some reassurance from JWSmythe, I'm going with the Promise VTrak 15100 array for work. It has almost everything I want: serial ATA, dual SCSI adapters, and an ethernet interface. The downside is that Promise doesn't have an office in Canada, so there's the possibility that getting parts across the border could be a problem. However, there's a local company that'll do service, so that makes me feel better.

The other options just weren't as good: one was parallel ATA and had no ethernet interface. The other was the Fastora DAS-315, which certainly looked good -- but the local resellers couldn't be bothered to give me the time of day, let alone answer the questions I had. Best bit: when I asked for a copy of the service level agreement, the sales guy replied that he'd "have to see" if he could release it.

And at home, I've been running into problems with bridging, the 2.6.9 kernel and the 8139too driver. I thought I would enable bridging on Thornhill for some User-mode Linux fun, so I enabled it as a module, then rebuilt and reinstalled the modules. However, when I tried inserting it, I got unknown symbol: br_handle_frame_hook. Okay, what about rebuilding the kernel and including bridging within it? Tried that; when I booted, the kernel panicked as soon as it came time for the onboard 8139 interface to grab an address by DHCP.

It was similar to the earlier problems I had with the Shuttle, in that if I took out the ethernet cable everything was fine -- it was only when the response came in that the kernel panicked. And keep in mind this was without setting up a bridge at boot time, or anything like that. I had to go to the backup 2.6.7 kernel in order to calm things down.

I found this thread on LKML, and it seems to match pretty closely what I saw -- the stack trace matches what I saw; I wasn't able to see the whole message, because it would scroll off the screen. However, I'm reluctant to try this patch; I spent a whole evening rebooting (Sorry, Aaron) and trying different things before I finally confirmed that having bridging in the kernel was just a bad thing.

Interesting bit: I didn't realize that Linux does not have panic core dumping built into the kernel, as FreeBSD does; it's only available as a separate patch. Minus one for Linux.

Finally, it's the day after the office Xmas party, and what am I doing? Heading into work to unplug everything. The power is being shut off in our building (thirty-floor or so high-rise) while upgrades are done, so I'm shutting everything down and disconnecting it just in case. Tomorrow I go back in to reverse the process. Whee!

As I've mentioned before, I've set up Greylisting on my mail server. The basic principle is simple: if you haven't seen an IP and email address combo before, you give them a 450 ("Come back later") error. If they come back later, you let 'em in and whitelist 'em in the future. The theory is that spamming depends on volume, and a spammer bot won't try again. One thing I've been noticing, though, is that spammers are trying again -- but from different IP addresses, which means they still don't get past the Greylisting. How many IP addresses? Looking at my logs over the last week, here's what I see:

Number of connections from separate IPs	Number of occurrences
1	102
2	26
3	24
4	24
5	15
9 (!)	1
Total:	190

This means that more than half try once, then give up -- but more than 46% try again. It's only because they're trying from different IP addresses that Greylisting still works. What happens when someone decides to make their bot try again from the same proxy? BTW, all this reminds me that, while it's okay doing this with awk and sort, I still need to get msyslog working...this'd be a whole lot easier in SQL.

1. When compiling a Linux kernel, you need to run make config (with your saved .config file, of course!) after running make mrproper.
2. NFSv3 support is not included in the kernel just because you compiled in NFS support.
3. In the 2.4.28 kernel, at least, serial ATA drives are available at /dev/sd[abcd], not /dev/hd[efgh] like in Knoppix...at least, when using the libata interface.

tcpdrop looks 'way cool. More and more reasons to make my next server run OpenBSD.

Well, I did the right thing today -- twice. Damn right I'm bragging.

First off, it turns out that the FreeBSD Foundation has run into a (good!) problem: its donations have been too big. In order to keep its US charitable status, it needs to have two-thirds of its donations be relatively small. Due to a couple of big donations, this ratio is a little out of whack at the moment, and they need a bunch of small donations.

Welp, I've been administering FreeBSD systems for a living for...well, I was gonna say four years, but it's more like two and a half or three. I've been working on them for four, though; my rent and food has been paid in large part because of the generosity of the people who put together FreeBSD. A donation went off in short order.

Then I remembered that I've been meaning to join the Free Software Foundation for a while now. The motivation is the same: I've been paying my bills for a long time now (and enjoying myself immensely in the process) because of the generosity of Free-as-in-Freedom software people: Stallman, Torvalds, Wall, and a zillion others. I have a hard time imagining what I'd be doing now without Free software; I suspect that, if I was lucky, I'd be working as a grocery store manager right now. So: off to the FSF website to sign up for an associate membership.

And what did I find but two, count 'em TWO cool things:

1. If you refer three people to the FSF for associate memberships, RMS or Eben Moglen will record a message for you, suitable for voicemail, Hallowe'en or impressing the ladies. I did a quick search on Google, but couldn't find anyone with the link...damn shame. Better than a free iPod, cooler than a CmdrTaco TiVo -- join the FSF and get RMS to say "All Hail Liddy!"

2. The FSF is looking for a senior sysadmin. God, that'd be cool. Decent enough pay (no, it's not the sort of job you take because of the money, but it's nice to think about), all the Free software you can handle, and an IBM Thinkpad to run it on. Of course, I think I'd have some 'plainin' to do about the laptop I'm writing this on...and, of course, it would mean living in the US. Frankly, that scares the crap out of me these days. Goddamned PATRIOT Act...

In other news, work continues apace. We're losing two coop students and gaining one, gaining another full-time person, and I'm still trying to get my RAID array -- credit app is with the boss, and after that's done the order'll finally go in.

Rough guess (wild hope) at this point is that it'll be in my hands in mid-January, which won't be a moment too soon. There's a new Linux server I'm setting up that I'm desperately hoping won't have problems due to proprietary kernel modules in the software I'm installing. (I'm just writing myself further and further out of that job, aren't I?)

And I'm wondering if the simplest way to get Nagios to make sure the right machines are exporting the right filesystems is to check if amd is mounting them correctly. (No matter whether the machine or amd fails, something needs to be fixed.) Or maybe I just need to figure out the right wrapper for showmount -e.)

On the spam front: good god, what a smoking hole Movable Type is turning out to be. First there were the license changes, then the comment spammers (who seem to be posting a lot more aggressive to MT than to WordPress)...Of course, comment spam affects all blogs, not just MT. Still, this whole idea of rebuilding static pages every time the stars move seems to be causing them a lot of trouble. (Yep, that last sentence was pure FUD. Or bullshit.) And okay, no, I don't use MT, so what precisely is my beef?

As I'm not going to put up, I should shut up. I still have to upgrade WP -- though according to this posting, there are still lots of XSS issues left unfixed. I'm also upgrading PHP, and I should probably use ApacheToolbox to do that automagically, rather than periodically editing my own Makefile.

The release party for Where Are They Coming From? came off JUST FINE, thank you. EVERYONE was there. Top Stars include Topo, Phil Knight and Mos Def, fresh from the set of HHGTTG. Uh huh.

Further thoughts on the MySQL + GPhoto2 thing: gphoto2 does have the ability to pipe to STDOUT, which I don't think I knew...maybe it won't be as much work to insert directly into a database as I thought. Might even be able to do it as a Perl script.

Finally: what a gorgeous day. It's downtown Vancouver on the back steps of the Art Gallery, it's sunny (in December, too) and just cold enough to make you go "brr". The skater kids are practicing their synchronised jumping -- just in time for the Olympics, I'm sure. A far-too-generous co-worker has handed out chocolate, another has handed out home-made rum and brandy balls, and I'm taking off early to go drinking with a third. Feeling pretty damned good right now.

Update: Too bad Topo's not so great -- fever of 102.8F, as of a couple minutes ago. (Still haven't figured out what that is in Celsius; bad Canuckistanian!) It's down a bit from earlier this afternoon, though, so I'm thinking good things. And these pages say to not worry if it's less than a couple days, so I'm not worrying. Nope.

Thanks to these two posts, I've finally managed to turn off WP's stupid, borked, let's-throw-in-a-<BR>-tag-every-time-a-line-ends-in-the-editing-box behaviour. Since I use Mozex, Firefox plugin of the gods, this was seriously pissing me off. I agree with OtherMichael: this behaviour is a bug, and should be option-controllable.

Yes, it's trouble in paradise time:

• WTF does Vim try to connect to X?
• WTF does grip come with gconf?
• WTF does gconf refuse to give up information?

ARGHHH.

Holy crap:

IP addresses are easy to fake as well. The design principles of TCP/IP allows the sender of a packet to specify its IP address. The message will still be routed to its destination using the fake origin address. Return packets would be mis-routed, however, because TCP/IP would send responses to the true location of the IP address rather than where it actually came from. This means that IP spoofing is ineffective in situations where you need to interact with a remote server, but very effective in a one-way conversation. I can't retrieve a Web page using a spoofed IP address because I need to make the request and then have the server send me the page. But I can send requests all day long if I don't care about the response.

I thought this was just a slight muddying of the waters. But no. The VERY NEXT PARAGAPH:

Posting a comment (or TrackBack) doesn't require interaction. I can send a comment in a POST or GET message and not worry about the response if I don't care about receiving acknowledgment that it was successful.

...what, has Apache moved to UDP all of a sudden? Sweet Zombie Jesus! (And don't talk to me about guessing SYN numbers; that is not what this idiot is talking about.) (Although to give him his due, he is talking about this in an article explaining why blocking IP addresses from blogs won't work, and he comes up with a great summary: "This [approach] is fundamentally flawed because it assumes IP addresses are both unique and hard to come by.") (But oh, this is a very painful case of bending over backwards to be fair.) And then:

Now spammers have turned their attention to weblogs and comment forms. In order to increase search engine rankings you are posting advertisements to our Web pages. What you failed to understand is that bloggers are smarter, better connected, and more technologically savvy than the average email user. We control the medium that you are now attempting to exploit. You've picked a fight with us and it's a fight you cannot win. Bloggers will track you down and notify your hosting providers about your activities. We will tell your ISPs what you are using their connections for. We will let the makers of the products you are advertising know of your despicable sales methods. We will hit you where it hurts by attacking your source of income. You can move to a new host, find a new ISP, or sign up for a different affiliate plan. The end result will be the same. Each time you rise out of the muck we will strike you down and send you back to the hole you crawled out of.

Do you smell that? That is the sound of sweet, virgin superiority, fresh and and naive and unmingled. This is from Dive Into Mark. I quoted it before, but here's a bit more context:

If you want to be an anti-spam advocate, if you want to write software or maintain a list or provide a service that identifies spam or blocks spam or targets spam in any way, you will be attacked. You will be attacked by professionals who have more money than you, more resources than you, better programmers than you, and no scruples at all. They want to make money, this is how they have decided to make money, they really can make a lot of money, and you're getting in their way. This is old hat to anyone whos been involved in anti-spam efforts in other domains (Usenet and email spring to mind), but just like everything else, the weblogging community seems intent on (a) thinking they're special and unique and nobody has ever had their problems before, and proceeding to (b) ignore all the work that has come before and reinventing the wheel. [....]Someone challenged me, Well, how am I supposed to continue hosting these low-barrier discussions? I'm sorry, but I don't know. To quote Bruce Schneier, "I feel rather like the physicist who just explained relativity to a group of would-be interstellar travelers, only to be asked, 'How do you expect us to get to the stars, then?' I'm sorry, but I don't know that, either." The low barrier is exactly the problem here. We got away with it (please, come post random links on my site which is well indexed, poorly managed, and open to unlimited anonymous contributions!) because we were collectively very young and naive and thought no one could hurt us. Now it's like were turning 30 and being told we need to go on a diet and asking, "Well when can I go back to my old eating habits?" Um, you can't. Your old eating habits don't work anymore. Weblogging is growing up. Oh wait, you thought that would be a good thing? You must still be young.

It is still worth reading every single depressing and true sentence in there, if only to keep yourself from being drowned in bullshit, nonsense and fairy tales.

I don't like the WPBlacklist plugin as much as I used to. Reasons why:

1. Stupid insistance on banning IPs. There are so many zombie PCs out there acting as open proxies that this is a waste of time. I've watched the traffic, and comments don't come from the same IP twice.
2. Stupid insistance on looking at email addresses. When it's so easy to make up an email address, why bother? Tracking 1600 variations on byob@[some number].com is just filling up the tables.
3. Problems tonight with Topo's blog.

Topo mentioned tonight that not only had it been a while since a new comment was posted on her blog, but a test comment posted tonight never showed up. Part of the problem turned out to be a stupid PHP syntax error I'd introduced; I'd been editing one of the files in an attempt to force WPBlacklist stop emailing re: deleted posts. (Yes, we'd turned off all possible email-me-please settings, and it still kept filling up her inbox.)

And then somehow, our IP address and my URL got put into the blacklist tables. There's no note of when an entry is added to the blacklist (FIXME!), so I can't tell when it was added -- but every test comment I made was getting caught by this. Finally, there were at least two blank entries in the tables, and I'm afraid they might have been destroying everything in sight, too.

After a little bit of browsing around, it turns out the let's-insert-a-blank-line problem has been addressed in the 2.8 version of WPBlacklist, available from the new download page. I'll give that a try. Also, it'd be nice to clear up the license -- no mention of how WPBlacklist is available, and if I'm going to work on this (and hopefully improve it), I want to make sure I can distribute any changes. I'll post a question on the forum and see what happens.

Got a bad feeling in the pit of my stomach this morning when I came back to work. I'd deliberately stayed away from the usual non-Slashdot news sources (Internet Storm Center, Bugtraq, Full Disclosure), so there was a lot of catching up to do. Let's see: eighty-four new remote holes in Windows -- always fun -- and it turns out the phpBB worm is no longer a phpBB worm but a PHP worm. Jesus Christ.

I checked the logs on my home server, and sure enough there were tons of the little bastards hitting me. (The server at work was completely clean.) It looked like there was nothing there, but I couldn't be sure without more time spent on it than a few minutes' grepping -- which meant leaving it 'til I got home tonight. (Update: looks like I was fine. I tried the URLs in the logs, and none of them tried to fetch anything. Dodged a bullet there.)

OpenBSD has the right idea when it chroots Apache, but there's also the matter of initiating connections out. And yes, I'm guilty of this: Thornhill + port 80 + tcp syn should be firewalled off, but was not. Changed now, of course. Still, it would be nice to have Thornhill not be locked down entirely. Why not let me initiate a connection out, but prevent Apache from doing the same?

This gets back to What's Wrong With Unix?, and I still say a good part of it is the lack of fine-grained permissions on both ports and files. (That, and my inability to type a good post when I'm in a hurry...God, that was incoherent.) The sheer idiocy of continuing to insist on root permissions to open a port under 1024 is just ridiculous. Why do we do this? In a world of Unix on the desktop, where anyone can get root, what does this mean anymore? Nothing at all: it's a totem, a fetish, and the Unix equivalent of knocking on wood for luck.

Worse, by insisting that you need to be root to open port 80, you invite all sorts of security problems. Better hope you drop privileges effectively; better hope no one figures out a way to extract r00t from any lingering privileges; better hope you didn't make one single mistake, or you'll get 0wned. Serving web pages, answering DNS queries or answering QOTD requests (ports 80, 53 and 17, respectively) do not require root permissions. (This is quite a different question from whether or not J. Random User should be able to modify web pages, zone files, or the QOTD database.) qmail, Postfix and others have shown that delivering mail doesn't need root, either. (Other applications can be taken on a port-by-port basis; the full extent of my hand-waving is left as an exercise to the reader.)

So why is there no way to let UID www send a syn+ack, but not a syn? Or to let some range of UIDs do both? Why, Lord, can't I change ownership, groups and permissions on /proc/net/ipv4/tcp/port/80 so that UID www can open this port and nothing else? How long, O Lord, how long?

There is a patch I came across today that supposedly offers this sort of thing, but again: it SHOULD NOT be an option; it SHOULD NOT be a patch; it SHOULD be built-in and used, just like we use UIDs to restrict privileges now. (The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" are to be interpreted as described in RFC 2119.)

Ahem. In other news: At Staples today I picked up a Network Everywhere BWR04B 802.11b wireless router. --I'm sorry, "Network Everywhere"? Looks like Cisco/Linksys in disguise. But it was 18 Soviet Canuckistan pesos! Boxing Day special! How could I possibly resist? Better yet, it turns out that the damn thing can run Linux. It's got 8MB of RAM, 2MB of flash memory, and something like a 60MHz ARM CPU.

The folks over at the Hardware Recycling Initiative are working on getting this and other broadband router boards running Linux. Sweet! Now to figure out how the hell to get it to work on this thing...I can identify a soldering iron six times out of ten, but that's about it.