DNS and Emacs:
The last few weeks, I've been setting up a small (5 racks) server room with the purchases that $OTHER_JOB recently made: 10 Sun X4140s, 2 — wait, 4 — X4240s, and one Thumper.
It's occupied a lot of my time, and before I lose the impulse, or fall asleep on my feet (second kid up at 4:30am for the last week or so; simultaneous discovery that at 4:30am I have a hard time getting back to sleep), I want to put down the things I learned.
ldap_cachemgr does not like being told to connect to an IP address (via an entry in ou=Profiles) via SSL, and have the CN be a hostname instead. This took me a while to figure out.But...my first batch of homebrew beer has been bottled, and a second brew day is coming up on Saturday. And apparently I'm not the only sysadmin who brews...though I'm not nearly ready to do all-grain just yet.
You can configure OpenSSH's ~/.ssh/authorized_keys file to
restrict the commands that key is allowed to run via SSH...thus, say,
restricting a particular key to running rsync or dump. You can also
restrict it to connections only from certain hosts; as the manual
points out, this means that "name servers and/or routers would have to
be compromised in addition to just the key."
Tags: toptip
Matt asked how Amanda worked for people, and whether they'd recommend anything else. I tried to leave a comment, but Blogger's CAPTCHA (god, I hate that acronym) never seems to work for me. So here goes. (Irony of a man w/an email-based comment system complaining about someone else's left as exercise f/t reader.)
Amanda: Nice, but: At my last job (2.5 years ago now), we started running into problems when backing up a 1TB RAID5 array...simple Promise disk array, nothing special or terribly fast. Amanda would take hours to do an estimate of the backups…which, since Amanda tries to pack tapes as full as it can, it does all the time. This got to be a huge pain, and we didn't find a solution to this problem before I left. (We were using GNU tar for Amanda; not sure if that had anything to do with it, and I can't remember what the alternatives were…maybe dump? Dunno.) Not sure what the current state is.
Bacula: +1 on the nice. Very, very good at my current job; absolutely no problems with it at all. And the documentation is enough to cry for, it's so complete and wonderful and thorough and accurate and well done. Clients for Unix, Windows, and Mac. Total filesystmes here are…uh…less than 1TB, definitely, although it's creeping up there. So the smaller size may have something to do with it.
Tags: backups
I was >this< close to writing my own damn set of Perl scripts to test a firewall, but I decided to search one last time. Good thing, too: ftester looks pretty close to perfect.
I'm having trouble right now getting ftestd to work on an OpenBSD 4.3 system; this may be because I'm trying to get it to listen on an interface that's part of a bridge. I'll have to look into this further. But testing it out between my laptop and desktop works a treat, whether my laptop is running OpenBSD or Linux 2.6. Sweet!
Tags: bsd
When I was at LISA, one of the sysadmins I met mentioned a firewall unit testing script that a coworker of his had come up with. The idea was to run your OpenBSD firewall in a QEMU instance, then try passing traffic back and forth to make sure everything worked as expected. I've been looking for that tool to be released, but haven't seen it....or anything else like it either…
Until today, that is, when I stumbled on NetUnit. It's a Java-based tool that tests basic network connectivity, using XML files to specify tests. So far he's got tests for ICMP/port 7 (which I never knew was the echo port), TCP ports, HTTP/HTTPS and MySQL. Not bad at all, except for my lack of Java experience.
Of course, now I want to write my own tester using Perl and QEMU. Like I've got time. But here's an idea for anyone who can use it: test your firewall using three instances of QEMU (inside, outside and firewall), and have the inside and outside hosts communicate using the serial port. "I'm gonna send an echo request, did you see it?" "Yes, did you see the reply?" It's a bit more feedback than simply noting the lack of the expected reply.
And it's not at all like conversations that start out with, "I sent you an email. Did you get it?"
Tags: networking bsd
Seen while applying software updates to a new Mac at $WORK:
The Aluminum Keyboard Firmware Update will update the keyboard firmware on your aluminum Apple Keyboard. Important: Do not interupt the update, your keyboard will not function while it is being updated.
I guess a mouse crashing is not entirely out of the question...
Tags: hardware
The good thing about being up at 3am is that, with a laptop, you can keep yourself entertained by whipping up a quick spreadsheet of the rack, switch and console server layout for the new server room.
The bad thing is that you may not trip over Sun's handy-dandy power calculators (like for the X4140 or the X4440 until the next day, leaving you twelve hours to wonder blearily if you've blown your server room's power budget all in one go.
Tags: hardware
Work...hell, life is busy these days.
At work, our (only) tape drive failed a couple of weeks ago; Bacula asked for a new tape, I put it in, and suddenly the "Drive Error" LED started blinking and the drive would not eject the tape. No combination of power cycling, paperclips or pleading would help. Fortunately, $UNIVERSITY_VENDOR had an external HP Ultrium 960 tape drive + 24 tapes in a local warehouse. Hurray for expedited shipping from Richmond!
Not only that, the Ultrium 3 drive can still read/write our Ultrium 2 media. By this I mean that a) I'd forgotten that the LTO standard calls for R/W for the last generation, not R/O, and b) the few tests I've been able to do with reading random old backups and reading/writing random new backups seem to go just fine.
Question for the peanut gallery: Has anyone had an Ultrium tape written by one drive that couldn't be read by another? I've read about tapes not being readable by drives other than the one that wrote it, but haven't heard any accounts first-hand for modern stuff.
Another question for the peanut gallery: I ended up finding instructions from HP that showed how to take apart a tape drive and manually eject a stuck tape. I did it for the old Ultrium 2. (No, it wasn't an HP drive, but they're all made in Hungary...so how many companies can be making these things, really?) The question is, do I trust this thing or not? My instinct is "not as far as I can throw it", but the instructions didn't mention anything one way or the other.
In other news, $NEW_ASSIGNMENT is looking to build a machine room in the basement of a building across the way, and I'm (natch) involved in that. Unfortunately, I've never been involved in one before. Fortunately, I got training on this when I went to LISA in 2006, and there's also Limoncelli, Hogan and Chalup to help out. (That link sends the author a few pennies, BTW; if you haven't bought it yet, get your boss to buy it for you.)
As part of the movement of servers from one data centre across town to new, temporary space here (in advance of this new machine room), another chunk of $UNIVERSITY has volunteered to help out with backups by sucking data over the ether with Tivoli. Nice, neighbourly think of them to do!
I met with the two sysadmins today and got a tour of their server room. (Not strictly necessary when arranging for backups, but was I gonna turn down the chance to tour a 1500-node cluster? No, I was not.) And oh, it was nice. Proper cable management...I just about cried. :-) Big racks full of blades, batteries, fibre everywhere, and a big-ass robotic Ultrium 2 tape cabinet. (I was surprised that it was 2, and not U3 or U4, but they pointed out that this had all been bought about four or five years ago…and like I've heard about other government-funded efforts, there's millions for capital and little for maintenance or upgrades.)
They told me about assembling most of it from scratch...partly for the experience, partly because they weren't happy with the way the vendor was doing it ("learning as they went along" was how they described it). I urged them to think about presenting at LISA, and was surprised that they hadn't heard of the conference or considered writing up their efforts.
Similarly, I was arranging for MX service for the new place with the university IT department, and the guy I was speaking to mentioned using Postfix. That surprised me, as I'd been under the impression that they used Sendmail, and I said so. He said that they had, but they switched to Postfix a year ago and were quite happy with it: excellent performance as an MTA (I think he said millions of emails per day, which I think is higher than my entire career total :-) and much better Milter performance than Sendmail. I told him he should make a presentation to the university sysadmin group, and he said he'd never considered it.
Oh, and I've completely passed over the A/C leak in my main job's server room…or the buttload of new servers we're gonna be getting at the new job…or adding the Sieve plugin for Dovecot on a CentOS box...or OpenBSD on a Dell R300 (completely fine; the only thing I've got to figure out is how it'll handle the onboard RAID if a drive fails). I've just been busy busy busy: two work places, still a 90-minute commute by transit, and two kids, one of whom is about to wake up right now.
Not that I'm complaining. Things are going great, and they're only getting better.
Last note: I'm seriously considering moving to Steve Kemp's Chronicle engine. Chris Siebenmann's note about the attraction of file-based systems for techies is quite true, as is his note about it being hard to do well. I haven't done it well, and I don't think I've got the time to make it good. Chronicle looks damn nice, even if it does mean opening up comments via the web again…which might mean actually getting comments every now and then. Anyhow, another project for the pile.
That's not quite my dad at c2k8, but damn if it wasn't enough to make me look twice.
That's bush. Bush league. You hear me, Fuji? Look at me!
I knew there was a reason to compulsively squirrel away every half-used set of tape labels.
Tags: rant
So one of the things I need to set up at $JOB_2 is some kind of
unified bag o' passwords…which, since I hate NIS, pretty much means
LDAP. This is the first chance I've had to set up an LDAP system from
scratch, rather than either being afraid to try or being stuck with
(and, sadly, contributing to the further divergence of) a mishmash of
semi-borked LDAP servers.
I've been trying out Fedora Directory Server the last few days, and so far I'm pretty happy with it. It's nice to have the luxury of learning what the hell I'm doing before it all goes live, of screwing up a bunch of times on a non-production system.
Likes: Welp, it's a lot like Sun's Directory Server…at least as far as the logging and console go, anyhow. Not surprising, given the heritage. You can automate installation by giving it a configuration file — something I didn't realize you could do with Sun's DS.
Other likes: PHPLDAPAdmin is nice. The latest version has E-Z-Reed XML templates for things like account creation, meaning I can keep my ignorance of Javascript intact. (Hurray!)
Minor irritants: there are a few. First off, there are no RPMs for CentOS 5 for the 1.1 series; you have to jump through some hoops to get the FC6 RPMs of 1.1 installed. I'd originally tried the 1.0 series on Debian, and hadn't realized that the 1.1 series does not include the org chart or E-Z-Account-Maker web app. (This is where y'all can go, "Muffin!")
Third, I'm so far not able to get the automated installation
working…can't figure out why. Not terribly important, since $JOB_2
is small and likely to stay that way; a couple of servers is likely to
be the max. But installation of this thing, just like with Sun DS, has
lots of knobs that you can twiddle if you want, and part of the
problem with the mishmash at $JOB_1 is that no one ever standardized
the settings — never wrote down the answers to the questions, or
scripted it, or came up with a config file, or anything. And it's
hellish if you want to add another install to the mix.
Anyhow...so far it's cool. I've been playing with it on a machine at
$JOB_2 plus an installation of CentOS 5 on my laptop. Still to
learn: SSL, replication, and (maybe) multi-master replication.
(Incidentally, I'm surprised that there isn't a more recent version of O'Reilly's LDAP Administration by Gerald Carter. Yes, there's still OpenLDAP and I don't imagine it's changed very much (feel free to correct me), but something that included Fedora DS, and maybe (maybe) OpenDS would be good.
(And speaking of Sun gossip, I've been meaning to mention this for a while…and now this.)
Tags: ldap
Interesting article from Threat Level about the Defcon NOC. Now there'd be an interesting job...
Tags: networking
About a year ago, I started using a cobbled-together system of Bash and Perl scripts and Makefiles to put together this blog. One of the reasons was my general dislike for PHP; another was my desire to try living (at least in some small way) by Saint Aardvark's Axiom of Information Utility, and try keeping this in plain text. (Another was a desire to use Emacs to write these damn things; I want the control that's thrown out when you start using a GUI to edit.)
But one of the problems that faced me was how to deal with comments, and comment spam. Having a web form that allowed comments made commenting easy, but the downside was that it made spamming easy too. WP and others keep this down to a dull roar, but it's not perfect and I've had problems with false positives — people being unable to post comments because their IP address was on some blacklist, and the plugin had made no provision for whitelisting.
I decided to lash together something that would use email. For me — a very small, low-traffic website, with a blog devoted to a rather obscure set of concerns and a tech-savvy audience (Hi Dad!) — this seemed like a good choice. Email spam, for me, has been pretty much solved by greylisting and SpamAssassin. (There's the problem of a ten — no, fourteen — year-old email address that I've been meaning to get changed for a while now, but that's another story; they don't seem to do greylisting, and SpamAssassin does catch most of it.) So taking comments by email seemed, you know, righteous, dude.
The system for comments is pretty simple: every post gets an epoch
timestamp embedded in it. (I think if you look in the HTML source, you
can see it.) I use it for sorting the order of the posts, and I use it
to generate email addresses for post-specific comments. The format is
simple: comments+(seconds since the
epoch)@saintaardvarkthecarpeted.com. The address is included in the
post, though I haven't done much to make it obvious. (This blog, and I
think this whole website, would make baby Jacob Nielson cry.)
My thinking was that, even though I was publishing the addresses, it wouldn't matter: as I mentioned, spam for me has been mainly solved (insert disclaimers here). Between greylisting and SpamAssassin, I figured I pretty much wouldn't see any spam at all.
Turns out there's another benefit: the addresses have been picked up by spam bot crawlers, but they're screwing up the scraping. From 24 days of mail logs, I see a crapload of attempts to deliver to the wrong address:
$ perl -ne'/NOQUEUE/ && s{.*to=<(\S+?)>.*}{$1} && print "$_\n";' mail.log* | sort | uniq -c | sort -n
[much snippage]
```
36 1181577610@saintaardvarkthecarpeted.com
36 1182947701@saintaardvarkthecarpeted.com
37 1181326150@saintaardvarkthecarpeted.com
37 1183667208@saintaardvarkthecarpeted.com
38 1182949918@saintaardvarkthecarpeted.com
40 1183349604@saintaardvarkthecarpeted.comThere were more than 2500 of these messages turned away by greylisting. They've all stripped off everything up to the plus, not realizing (as I didn't until a few years ago) that a plus in an email is valid.
In fact, the only attempts to deliver to legitimate comment addresses were two actual comments to my blog…which brings up a shortcoming: I never got that many comments with WordPress, but I sure got more than I do now. It's possible my writing has just gone 'way downhill, but I think it's more likely that this system just puts people off, or they're just unable to find it with my current (crappy) design.
(One interesting problem: my wife tried to comment once, using Lotus Notes at her workplace. It converted the plus sign into an underscore. Weird.)
I still regard this setup for comments as an experiment. Its results are definitely mixed; no spam, but fewer comments as well. Given the tiresome mess that comes with the lack of an HTTP equivalent of greylisting, I'm inclined to keep doing it.
Anyhow...that's my interesting research result for the day. You may now talk amongst yourselves.
Ran into a problem today when adding this stanza to cfengine on a Debian Etch machine:
editfiles:
```
{ /etc/aliases
AppendIfNoSuchLine "root: sysadmin@pims.math.ca"
DefineClasses "rebuild_aliases:restart_postfix"
}The cfengine reference file I've got, which sez it's for version 2.2.1, says you can define multiple classes in DefineClasses (or DefineInGroup), as long as they're separated by commas, spaces or dots. (The version in Etch is 2.2.20.)
However, when I ran cfagent, it just hung immediately after performing the edit, and gave this error when I ctrl-c'd it:
cfengine: Received signal 2 (SIGKILL) while doing [pre-lock-state]
Running cfengine with -d2 showed endless repetitions of AddClassToHeap() at this point, so either there's something wrong with my syntax or there's a bug in cfengine. (I'm guessing the former.) Searching for pre-lock-state and cfengine only turned up cases where the clients were syncing with the master; thus this note.
The fix was to just make it one class:
DefineClasses "rebuild_aliases"
Asking to restart Postfix was probably a bit of overkill anyhow...
Tags: cfengine
title: Happy Sysadmin Day! date: Fri Jul 25 05:40:54 PDT 2008
Andy just pointed out to me that it's Sysadmin Day, which I'd totally forgot about. So here's to ya, everyone!
Which means I'll have to save the dream I just had (one of the Rohirrim being held hostage by the housewives of the OC at one of their garden parties, and insisting, with murderous glint in their eyes, that he finish all of his cake before leaving despite the rest of the Rohirrim showing up to rescue him, and me trying desperately to resolve the standoff by telling Peter Gallagher that, really, there's still time to let him go) for another entry…/me shakes head.
Tags:
Tags: bug
winerror.h describes this as a General access denied error. In the end, it turned out that when the account was created, the "user cannot change password" option was checked. Hope that'll help someone else's google-fu…dig +short porttest.dns-oarc.net TXT
and watch the skies.
Tags: toptip
How to quiet noisy cron entries that send far too much to STDERR:
exec 3>&1 ; /path/to/script 2>&1 >&3 3>&- | egrep -v 'useless|junk' ; exec 3>&-
I've been very busy of late, but the biggest news is that I've started a 3-month temporary part-time assignment here. It's a neat place, and feels a lot like a software startup. Even though it's a small group, they've got certain hardware requirements that are a lot bigger than what I've worked with before; it'll be interesting, to say the least.