The Life of a Sysadmin

I found a link on Gecko's blog to Project Honeypot. Turns out it's a project to watch for, and attempt to track, spammer-run robots that scrape pages for email. I was intrigued, but a little put off by the terms of use. I did a big more digging around, and found I wasn't the only person who thought that way. However, there were some strong rebuttals from the SpamCop forums, discussion on SURBL mailing list, and from one of the principals (who also replied here).

Reassured, I signed up. It's still in the early stages, so there hasn't been a lot of spam received yet (350-odd pieces, according to the stats page on the site). Still, I'm hopeful it'll be a Good Thing.

Another approach: a Java SMTP honeypot. Huh.

Okay, so as I mentioned I'm trying to get a Subversion repository working in a way that a) keeps the repository safely on an NFS-exported, mirrored set of drives, and b) does not require YAFPF. Today I've been banging my head against Apache2 + mod_auth_pam. The problem is that while passwords are successfully checked (hurray! one less FPF!), group membership is not. this does not work:

AuthPAM_Enabled on
AuthPAM_FallThrough on
AuthGROUP_Enabled on
AuthGROUP_FallThrough on
AuthType Basic
AuthGroupFile /etc/group
AuthName "secure area"
Require group subversion

(For one brief, spastic moment I thought Satisfy any was the missing magic. Then I tried it without typing in a password. Sigh.) We're using FreeBSD and NIS; from what I've been able to find so far, that might be problematic. OTOH, I might have the entirely wrong idea about PAM and its ability to check group membership.

UPDATE: Logical as it seems, AuthGroupFile has no place in the modern kitchen. Removing that directive allowed everything to work. Whee!

"I say we take off and nuke the entire site from orbit. It's the only way to be sure."

Saturday afternoon my home web server got cracked. I found out because Google started refusing my searches, asking me to fill out a CAPTCHA form (incidentally, I hate the word CAPTCHA, and even typing it gives me hives) to prove I was human. What the hell?

So I checked on the server, which is also our firewall, which isn't good but frankly I was tired of maintaining a complex network at home, and sure enough there was some perl script running as user www-data (which Debian uses to run the webserver), sending off tons of Google queries and taking commands on IRC the way I keep hearing nobody does anymore. Crap.

Fortunately I've been running Bacula for a while now, backing up to an external hard drive, and so I figured that even though it probably would go away when I rebooted, I'd Do The Right Thing(tm) and rebuild from scratch.

This had to wait 'til the evening, so I shut down the webserver, ran backups a bunch more times, got more info, and moved the machine (a tiny li'l Shuttle box) from my youngest son's bedroom (apparently the only room in the house w/a phone outlet not covered by an ADSL filter) to our bedroom upstairs, running the network cable up the stairs.

In the end, it all went pretty smoothly. I was able to get all my packages back and restore from backup; the only thing I messed up was getting the ownership wrong on my restored crontab. (Debian uses a pool of UIDs for daemons, so you're not guaranteed to get the same UIDs if you reinstall.)

As a bandaid, I've firewalled off www-data from initiating connections out. I should have done this long before. Now I'm starting to think about the next step -- Xen, maybe, or SELinux. (I did briefly consider other distros, or even a BSD: CentOS for SELinux, FreeBSD for pf and jails. But I decided that one problem at a time was quite enough, thanks.)

• This Nagios check looks for extra Wordpress admins.$ARG1$ should look include "-w min:max -c min:max", giving the acceptable ranges; in my case, I know I should have 3 and only 3 admins, so I have "-w 3:3 -c 3:3".

  define command{
  command_name check_wp_admins
  command_line $USER1$/check_mysql_query -q 'SELECT COUNT(wp_users.user_login) AS "Admins"
                         FROM wp_users, wp_usermeta
                         WHERE wp_usermeta.meta_value LIKE "%administrator%" AND
                         wp_usermeta.user_id=wp_users.ID' -H $HOSTADDRESS$ $ARG1$
  }
  

• This one looks for nasty Wordpress posts. Note the dependency on MySQL's regex command. In my case, I know that I do not have any posts with these words, so in $ARG1$ I have "-w 0:0 -c 0:0".

  define command{
  command_name check_wp_nasty_posts
  command_line $USER1$/check_mysql_query -q 'SELECT COUNT(*)
                         FROM wp_posts
                         WHERE post_content REGEXP "iframe|noscript|display"' -H $HOSTADDRESS$ $ARG1$
  }
  

• And a Python script that picks out a random client, job and file from Bacula's database and tries to retrieve it. It's not ideal -- checking for sanity is left as a DARPA Grand Challenge -- but at least it's one way of exercising backups. I anticipate running this often. If anyone's interested, let me know.

The more I work with Python, the more I don't just like it but admire it.

Ugh...not much more right now. I've got a blocked eustachian tube that I'm self-medicating with a Python script^W^Wcold medicine, and the acetominiphen in it is making me hazy.

Holy crap. PDF supports multimedia, javascript, and launching arbitrary programs. Haven't checked the standard (warning: PDF) yet to see if they support the evil bit.