Home again, home again, Jiggity-jog

I'm back from 3 weeks' vacation. My family and I drove (!) from New Westminster, BC to Brandon, MB to visit my brother and sister-in-law and my parents. We took our time driving out and back: 6 days out, 6 days back, and, um, 9? days there.

Originally our plan was to camp most of the way there and back, but after two days we realized that the kids were a tad young for that...between mosquitoes, not going to sleep on time, and long-ass days in the car, it made a lot more sense to stay in hotels. (Kids dug it a lot...every new Super-8/HoJo/Travellodge prompted a "Wow, TWO TABLES! This is the best room EVER!" from the oldest.)

Family was fun. The prairies were hot (who knew?). Hotels with swimming pools are to be celebrated. The Rockies were, as ever, amazing. Canmore has a good brewpub and is a very, very pretty town. Aylesbury, SK had a horse to look at, which made it a good place for a picnic with the kids. Regina is made entirely of ass. Saskatoon, home of The Dymunds, is as pretty as ever.

TThe Royal Tyrrell Museum was incredible, even in a visit abbreviated by two toddlers...first time I'd ever seen real dinosaur skeletons before ("and I saw a T Rex anna stegasaurus anna triceratops anna archaeopteryx anna...). I could have easily spent a week there. If you ever get the chance, visit by any means necessary.

Despite some last-minute panics before leaving, I only received one phone call while away, and that from a salesman. I managed to forget about work entirely, and I expect to have to be guided to my desk tomorrow. And that is the capstone of a good vacation.

Tags: geekdad

Randomized Title Words

Yesterday was my first day back after a week off sick...and it wasn't bad, I have to say. It's summer at a university, so that helps, and it was a sunny day like we haven't had in weeks, so that was even better. I'm going off on vacation in three weeks, so I'm focussing now on getting things ready. (I'm asymptotically approaching a full realization of how to plan work over long periods of time...)

There's one big project (LDAP + website + email on four VMs) that I need to hand over to the owners. I found out/confirmed my suspicions that the people maintaining it while I'm gone will be moderately technical, not administrative, so that simplifies things a lot.

There's documentation I need to improve for the folks who'll be helping while I'm gone, including network maps. Ugh...I hate making network maps; it reminds me of web design in its fiddliness (is too a word). But I will do it. (I wonder if using a tablet would help. Anyone have any experience w/that?)

Oh, and DRI...sigh. One of the groups at work uses a protein visualization program that wants to use DRI. Without it, it reverts to a lesser form of rendering that looks like a strobe light. In the X.org troubleshooting guide, they recommend a stanza that looks like this:

Section "DRI"
    Mode 0666
EndSection

I don't like those permissions just on principle. You can put in a "Group foo" directive, which changes the group ownership as you'd expect, and for some workstations that works...but for another does not, possibly because of different driver versions. This is what comes of not having fully automated installs.

And there's son #2 waking up again. He woke up at 5 but was persuaded to go back to sleep. Bad: I was up at 5am. Good: I had time to write this.

Tags:

Working Group on Sinus Decolonization

A week ago this afternoon, I started feeling bad: achy, kinda fevery, lethargic. I chalked it up to flu. But then I got these massive headaches. At night I had intense, repetitive, feverish dreams about loading some random .el file in Emacs (no, seriously), and would wake up to more headaches, drenched in sweat.

First doctor said it was allergies or viral; second doctor thought it was bacterial, and gave me a prescription for antibiotics. (I don't usually get second opinions, and I didn't ask for antibiotics, but I've never felt so awful for so long.)

On Wednesday, I stupidly decided to go into work to take care of a few things, and scared a few people with how I looked. I slowly shuffled back to the bus and came home.

On Thursday, I was supposed to go with my wife and kids to Penticton for a weekend family gathering...that was out. I think I'd have thrown myself out the door within a few blocks. They left, and I've been having a sickie bachelor weekend. I started watching "Battlefield Earth" as soon as they left; I lay there on the couch, letting John Travolta's cackle wash over me and laughing weakly.

I watched a lot of movies. I read the Internet. I ate frozen pizza. I upgraded my wife's laptop to the latest version of Ubuntu, and fixed a Flash bug that's been irritating her. I had vague doubts about my choice of career, but I'm putting that down to lack of sleep...haven't had a good night's worth in a week now.

I've been feeling slowly better after starting the antibiotics. And once I figured out I was taking entirely too much acetominophen (I think I was getting rebound headaches), I started feeling even better. For some reason I'm still getting woken up at 1:30am by headaches, but I'm hopeful that'll go.

The family's due back this afternoon. I'll be back at work tomorrow, and I'm glad: I'm pretty bored of sitting on my ass all the time. (I'd be happier to spend more time with my family, but we've got another, longer vacation coming up in OH CRAP three weeks.)

Okay, enough. Go listen to "Volunteers" by Megafaun. (That's why I'm having career doubts. Beard + acoustic music...sigh.)

Tags:

Regaining flash control functionality on Youtube in Ubuntu

I've been upgrading my wife's laptop to Lucid Lynx today, and I've finally had a chance to track down one little bug that's been annoying her. On Youtube, flash controls don't work on videos; she can't close ads, the volume/mute controls don't work, and so on. This is a 64-bit machine, but 32-bit flash (natch).

The solution, as detailed here (post 42!), is to edit /usr/lib/nspluginwrapper/i386/linux/npviewer and insert this before the last line:

export GDK_NATIVE_WINDOWS=1

Restart the browser, and now it all works!

Tags:

You magnificent bastard.

Wow:

Since I have started to use Org-mode, I though it was missing
something to have appointment locations on a map. Of course, it's
easy to get a LOCATION property from an entry, and then browse-url
on Google Maps.

But it is too easy for me, so once again I said: challenge accepted! I
will bring Google Maps into Emacs!

Google Maps in Emacs

Simply amazing.

Tags: Emacs

Medicine and system administration

Quote:

You come into medicine and science at a time of radical transition. You have met the older doctors and scientists who tell the pollsters that they wouldn't choose their profession if they were given the choice all over again. But you are the generation that was wise enough to ignore them: for what you are hearing is the pain of people experiencing an utter transformation of their world. Doctors and scientists are now being asked to accept a new understanding of what great medicine requires. It is not just the focus of an individual artisan-specialist, however skilled and caring. And it is not just the discovery of a new drug or operation, however effective it may seem in an isolated trial. Great medicine requires the innovation of entire packages care -- with medicines and technologies and clinicians designed to fit together seamlessly, monitored carefully, adjusted perpetually, and shown to produce ever better service and results for people at the lowest possible cost for society.

Very, very interesting reading, particularly as it seems to recap arguments about the role of system administration.

Tags:

Linux/FreeBSD sysadmin opening at UBC Vancouver

There's a Linux/FreeBSD sysadmin position open at UBC with the Department of Earth and Ocean Sciences. Job listings are here; look for job #7867. (Silly web page doesn't allow direct links...) If you're in Vancouver, it looks pretty sweet.

Tags:

Some thoughts on GOsa

At $WORK I've been setting up a server for a new project. It's going to be the foundation for a bunch of work that they're doing: bag o' passwords, code repository, website + wiki, email. I've been trying to set it up in such a way that a) it'll be a good foundation and b) it'll be easy for them to manage -- at least when it comes to adding/deleting/modifying users, email addresses and so on -- rather than having to come to me every time something needs changing.

I set up CentOS Directory server and populated it with the root and a few entries. Thus: cn=Me,ou=People,dc=example,dc=org. I've got authentication working, restricted to certain groups where necessary, and now it's time for me to think about how the non-propellerheads are going to manage accounts. That means a web front end, and I've been evaluating GOsa for the last week or so.

The good:

  • It's nice-looking. That's not damning-by-faint-praise; whatever I give to people is going to need to look professional.

  • It's got plugins for lots of different things: Samba settings, email settings (including easy vacation settings...nice), ACLs, FAI, Nagios (which, near as I can tell, is generated from FAI info), FTP quotas, you name it.

  • It supports hooks in its config file: postcreate, postremove, postmodify and check. These allow you to (say) create a home directory when a user is created or other fun things.

  • It includes an easy way to manage the LDAP server: export/import snapshots in CSV and LDIF, and snapshots of the tree that can be restored later. (Haven't tried that out yet.)

  • Addressbook page included.

The bad:

  • The documentation can best be described as sparse.
* Some plugins depend on others, but this isn't shown anywhere.
  When I tried to remove the RPMs for some that I wouldn't need, I
  found that others I _did_ need failed.  If it should _all_ be
  installed, I couldn't find that documented anywhere.

* Allowing people to log in to GOsa and change certain details is
  important for me, but is documented poorly in the [ACLs
  page][2].

* But also, ACLs can be applied only to items GOsa recognizes as
  Departments (ie, `objectClass=gosaDepartment` and/or possibly
  `objectclass=gosaAdministrativeUnit`).  And you can't tell GOsa
  that `ou=People` is one of these; it complains and says that's a
  reserved keyword.

* The `gosa.conf` config file refers you to the manpage, but that
  doesn't appear to be included in the RPMs I downloaded.  The
  [wiki page][6] on the file refers you to the [raw man page in
  the source tree][7].

* It looks like vacation settings depend on an LDAP-enabled
  vacation responder like [Gnarwl][3]; this is cool, but wasn't
  documented anywhere.  The only place I found this mentioned was
  on [this blog entry][4], which contained a lot of other info
  (like how to set Postfix up; yes, I shoulda RTFM but I didn't
  know that Postfix could grab so much info out of LDAP) that I
  didn't find on the GOsa website.  In fact, that blog entry was
  probably the best source of info about GOsa configuration.

* Adding email information for people requires a GOsa-specific
  LDAP entry for a mail server; again, this is best documented in
  [that blog entry][4].

  • GOsa is happiest when you point it at a new, empty LDAP tree and go from there. It appears to be much less happy with the idea of pointing it at an already-existing LDAP tree (though there is support, during installation, for converting accounts it has found in the tree).

Taking the the ou=People ACL example above as a starting point, GOsa wants to create its own department entries in your root: ou=Accounting (say), with ou=Groups and ou=People under that. I have mixed feelings about this organization; it would work well for my current employer, but I'm not sure how well it'll fit for this side project.

  • GOsa makes use of GOsa-specific LDAP schemas that you add to your server; this lets it label an object with (say) notes that users are allowed to change their own password. Perhaps not bad in itself -- I don't know enough to judge -- but it seems to mean that there are two sets of ACLs to manage: those in GOsa, and those in your LDAP server.

  • GOsa's schema files are for OpenLDAP syntax (there's a proper name for that but I can't think of it right now). I'm using CentOS DS, which has its own. FInding out how to convert one to the other was hard enough; after that was done, though, CentOS DS still didn't like it, and I had to delete some additional entries. There doesn't appear to be a lot of support for CentOS/Fedora/RedHat/389 DS.

(For the record, here's what I had to do:

    $ wget http://directory.fedoraproject.org/download/ol-schema-migrate.pl
    $ for i in $(grep 'schema/gosa' /usr/share/doc/gosa/slapd.conf-example | sed -e's/ldap/openldap/' | awk '{print $2}') ; do echo ../ol-schema-migrate.pl -b $i >>  98gosa.ldif; done

And then remove references to gosaLoginRestriction and goFaxDivertNumber.)

  • It might be better to think of GOsa as a complete system, of which the visible LDAP front-end is only one part: as described here, it's composed of Postfix, Courier for IMAP, Cyrus for authentication, and so on. That's not entirely fair; LDAP is LDAP, and I'm sure a lot of what seems app-specific could be generified (I spik Englitch good) by configuring Dovecot (say; I already use Dovecot instead of Courier) appropriately, or writing a plugin for GOsa.

Overall, I'm not sure how happy I am with GOsa. It's not doing what I want now, and it looks like the only way to make that happen is to wipe the tree and start from scratch. -- Well, no, not scratch: create a GOsa-visible department (ou=Something) and stick everything in there. Given the sparse documentation, it's probably silly for me to focus on that, but that's what sticks out at me right now.

Tags: ldap

Linux sysadmin job opening in Vancouver

This posting (PDF) came up on the VanLUG mailing list today; if you're looking for a Linux sysadmin job in Vancouver, it looks interesting.

Tags:

Perl multitasking, CPAN at home

For my own reference, here are a couple things I had to find out about Perl today:

  • An overview of installing Perl modules with CPAN in your home directory.
  • The local::lib module mentioned in that article; probably the best way of getting it going. Looks much easier than it was a few years ago.
  • David Blank-Edelman's article for ;login: magazine about Perl and spawning (PDF)
  • Parallel::Forkmanager is a simple Perl module for managing spawned processes.
  • Proc::Queue is another module to manage forked processes.
  • POE seems to be a much more sophisticated framework.

Tags: perl programming

Config file parsing

I've been setting up some new VMs for a separate project at work. I've realized that this is painful for two reasons: Bacula and Nagios.

Both are important...can't have a service without monitoring, and can't have a machine without backups. But each of these are configured by vast files; Bacula's is monolithic (the director's, anyhow, which is where you add new jobs) and Nagios' is legion. And they're hard to configure automagically with sed/awk/perl or cfengine; their stanzas span lines, and whitespace is important.

I've recently added a short script to my Nagios config; it regenerates a file that monitors all the Bacula jobs and makes sure they happen often enough. This is good...and I want more.

I found pynag, a Python module to parse and configure Nagios files. This is a start. I've had problems getting its head around my config files, because it didn't understand recursion in hostgroups (which I think is a recent feature of Nagios) or a hostname equal to "*". I've got the first working, and I'm banging my head against the second. The three books I got recently on Python should help (wow, IronPython looks nice).

There are a lot of example scripts with pynag. None do exactly what I want, but it looks like it should be possible to generate Nagios config files from some kind of list of hosts and services. This would be a big improvement.

But then there's Augeas, which does bi-directional parsing of config files. Have a look at the walk-through...it's pretty astounding. I realized that I've been looking for something like this for a long time: an easier way of managing all sorts of config files. Cfengine (v2 to be sure) just isn't cutting it anymore for me.

Now, the problem with Augeas for my present task is that there isn't anything in the current tree that does what I want, either. There is a commit for parsing nagios.cfg -- not sure if it's been released, or if it will parse everything in a Nagios config_dir. There's nothing for Bacula, either. This will mean a lot more work to get my ideal configuration management tool.

(On a side note, my wife said something to me the other day that was quite striking: I need tasks that can be divvied up into 45-minute chunks. That's how much free time I've got in the morning, bus rides to and from work, and the evening. Commute + kids != long blocks of free time.)

And I've got a congenital weakness for grand overarching syntheses of all existing knowledge...or at least big tasks like managing config files. So I'm trying to be aware of my brain.

...and there's son #2 waking up. Time to post.

Tags: python backups monitoring

Sale on O'Reilly ebooks today

O'Reilly is having a one-day sale on ebooks: $10 each. I've just picked up "Python for Unix and Linux System Administration", "Python Cookbook, Second Edition" and "Python in a Nutshell, Second Edition". (Detect a pattern?) They're DRM-free, the way Cory Doctorow intended. Downloading is a little slow right now, but I imagine that's just because of increased sales.

I'm not a shill for O'Reilly, just a satisfied customer. (Mostly. I wish OnLAMP.com was getting a bit more love.) (But then, I'm not paying for OnLAMP, am I?)

Tags:

Time well spent

Just spent two hours staring at this Python regex:

method_re = re.compile('(?P<urlcmd>\s*url\s.*)|(?P<nfscmd>\s*nfs\s.*)')

and trying to figure out why it wouldn't match this line:

install url --url http://cobbler.example.org/cobbler/ks_mirror/CentOS-5.4/os/x86_64

After much research, I can report two things:

  • In python, the match() method of a regular expression object matches from the beginning of the line. If you think in Perl, like I do, it's like if they prepended a ^ to the beginning of the regex.

  • (?P<urlcmd> puts the match in the variable urlcmd. That wasn't a problem, I just think it's neat (though obscure).

  • A kickstart file, which the line is taken from, is meant to have the install line be separate from the method (in this case, url). I did not know that.

And now to read more of Dr. Zhivago.

Update: by way of explanation, I was trying to use koan and Cobbler to install a new qemu instance like so:

/usr/bin/koan --nogfx --server=cobbler.example.com --virt --system=virtserver-01.example.com

and was getting two errors:

warning: kickstart found but no install_tree found
Cannot find install source in kickstart file, aborting.

The problem turned out to be that my kickstart file had this line:

install url --url http://cobbler.example.org/cobbler/ks_mirror/CentOS-5.4/os/x86_64

which should have been two lines:

install
url --url http://cobbler.example.org/cobbler/ks_mirror/CentOS-5.4/os/x86_64

Tags: python debugging

Michael W. Lucas has another book coming out!

Michael W. Lucas, of Absolute FreeBSD fame (among many others), has a book coming out on Network Flow Analysis. Sweet!

/me hurries off to pre-order...

Tags: network books debugging wow

Hopping

Been busy lately:

  • 3 new workstations with OpenSuSE. Can't figure out the autoinstall, so it's checklist time, baby.

  • Software upgrade for a fairly important server + 3 slave nodes. Natch, after rebooting one of the ILOMs for the servers just...went away. Can't ping it from the network. Works fine with an interactive ilom shell from Linux. Sometimes I really hate Dell software.

  • Got a call from the reseller for a major hardware vendor who just got taken over by a major database vendor; said db vendor has just turned off educational discounts we'd spent THREE MONTHS negotiating/waiting to have approved. I am unimpressed. Strongly tempted to call up random hardware vendors and throw money at them 'til they give us stuff.

  • Finally got leak detection working in the server room. Stupidly long time, it took.

  • Working on a "Lessons Learned" presentation for LISA that'll include mention of the leak detection (among other things). Not sure how it'll be received, but I figure it's their job to tell me it sucks, not mine.

  • New term coming, so about six new people coming. But at least I know about them in advance.

  • And this...and this...just amuse me. (Warning: Flash eats babies and sells them to Chinese hackers.)

  • Taxes.

But hey! Turns out we live in a constitutional democracy after all. There was some debate about this at 24 Sussex Drive, I understand. Score one for the good guys.

Tags: work dell politics hardware

My wife rocks

Wow: my wife just bought me a ticket to Hoppapalooza. I am so lucky.

Tags:

Where'd SNMP go?

Six books from O'Reilly on Windows Server 2003 (Cookbook, Security Cookbook, In A Nutshell, Netowrk Administration, Securing, and Learning) and there was ONE mention of SNMP in the indexes. What the hell?

Tags: windows

Putting a word in

Just got off the phone w/a Sun rep who called up to see how I was doing, did I need any coasters, etc. I took the opportunity to put a bug in his ear about Solaris.

If Oracle removes the entitlement to run Solaris on non-Sun hardware, then what the hell do I have to play with? I've got a bunch of Sun hardware, but only one machine running Solaris -- and that's in production, holding home directories on ZFS; I'm not playing with that.

OpenSolaris folks are asking for answers and not getting any. And saying "Go run OpenSolaris" ignores the problem of figuring out what's in Solaris proper, what's going to be there RSN, and what's two or more releases out.

If Solaris disappears, then I'm not going to figure out how it's better; that's just how:

  • my brain (I teach myself)
  • my budget (Dell hardware irritates me but it's easily half the price)
  • and my career (Open Source/Free Software FTW)

all work.

I like Solaris for precisely two things: ZFS and DTrace. Solaris has more, I know, but those are the things that matter to me. In all other respects, for me and my situation, Linux or the BSDs are good enough or better. And oh: FreeBSD has DTrace; DragonFly BSD has HAMMER; Linux has *@#%$)%! packaging.

No good ending for this, so we'll just call it quits.

Tags: solaris opensolaris dell

LDAP accounts

Trying to figure out how to add a bunch (well, 6) of LDAP user accounts. So far:

  • ldapuseradd: neat; wish there was a dry-run mode; wish the hooks supported arbitrary scripts

  • Python: neat; good to learn Python; have to avoid going down the rabbit hole of classes (Python made OOP clear to me in a way it hasn't before, and now everything looks like a nail)

  • phpLDAPadmin: arghh; nooooo; can't script

Realistically this won't happen very often, so perhaps I should just go ahead and use the damn browser to do this. But it kinda hurts me a little inside when I do.

Tags: ldap

A note on tracking Postfix messages through the logs

Recently I had to track messages to make sure they were being delivered, and this turned out to be much harder than I thought. Some of this is due to the way we have our mail set up, and some of it is just due to confusion (mine) about how Postfix logs things. So in the spirit of Chris Siebenmann, here's my explanation for my future self of how this works.

First, an explanation of the flow of messages at my workplace, whether sent or received from a local network or a remote network:

  • Postfix receives the message and passes to SpamAssassin
  • SpamAssassin scans the message, then requeues it with Postfix
  • Postfix receives the scanned message and delivers it

Here's a log with an example message. It was sent from pepperoni@pepperoni.pizza to otheruser@example.net, which we forward off to ilikesandwiches@club.sandwich. (Rememver the days when you could be assured that "club.sandwich" was a totally bogus TLD?)

   Apr  9 13:25:13 liddy postfix/smtpd[12203]: 4DC3914C086: client=mail.example.com[192.168.0.2]
   Apr  9 13:25:13 liddy postfix/cleanup[12206]: 4DC3914C086: message-id=<deadbeef@example.com>
   Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: from=<pepperoni@pepperoni.pizza>, size=2019, nrcpt=1 (queue active)
   Apr  9 13:25:13 liddy spamd[5110]: spamd: processing message <deadbeef@example.com> for nobody:99
   Apr  9 13:25:13 liddy spamd[5110]: spamd: result: . 0 - scantime=0.2,size=1981,user=nobody,uid=99,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=42925,mid=<deadbeef@example.com>,autolearn=failed
   Apr  9 13:25:13 liddy postfix/pickup[11886]: 85E1F14C089: uid=99 from=<pepperoni@pepperoni.pizza>
   Apr  9 13:25:13 liddy postfix/cleanup[12206]: 85E1F14C089: message-id=<deadbeef@example.com>
   Apr  9 13:25:13 liddy postfix/pipe[12207]: 4DC3914C086: to=<otheruser@club.sandwich>, orig_to=<ilikesandwiches@example.org>, relay=spamassassin, delay=0.25, delays=0.05/0/0/0.2, dsn=2.0.0, status=sent (delivered via spamassassin service)
   Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: removed
   Apr  9 13:25:13 liddy postfix/qmgr[995]: 85E1F14C089: from=<pepperoni@pepperoni.pizza>, size=2323, nrcpt=1 (queue active)
   Apr  9 13:25:43 liddy postfix/smtp[12212]: 85E1F14C089: to=<otheruser@club.sandwich>, relay=mail-relay.isp.tld[10.0.0.1]:25, delay=30, delays=0.01/0.01/30/0.35, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 92585180F0)
   Apr  9 13:25:43 liddy postfix/qmgr[995]: 85E1F14C089: removed

My confusion stemmed from thinking of the Postfix queue ID as the message ID. THIS IS WRONG. The message ID does not change. The queue ID changes if the message goes through the queue again. And, the way we have things set up, it goes through the queue twice: once when received, then again when SpamAssassin is done with it.

(Incidentally, I had wondered why SpamAssassin did not include the Postfix ID. This makes perfect sense when you realize it's a queue ID, and SA is not part of the Postfix queue.)

The other thing that makes this difficult is that the entries for a particular message are:

  • spread over many lines
  • only one of which will have the (unchanging) message ID
  • not necessarily adjacent

Thus, the process for tracking down what's happened to a particular mail goes like this:

Look for the bit you have (who the message is from, let's say):

$ grep pepperoni@pepperoni.pizza /var/log/maillog
Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: from=<pepperoni@pepperoni.pizza>, size=2019, nrcpt=1 (queue active)
Apr  9 13:25:13 liddy postfix/pickup[11886]: 85E1F14C089: uid=99 from=<pepperoni@pepperoni.pizza>
Apr  9 13:25:13 liddy postfix/qmgr[995]: 85E1F14C089: from=<pepperoni@pepperoni.pizza>, size=2323, nrcpt=1 (queue active)

Note the two queue IDs and search for those:

$ grep -E '85E1F14C089|4DC3914C086' /var/log/maillog.1
Apr  9 13:25:13 liddy postfix/smtpd[12203]: 4DC3914C086: client=mail.example.com[192.168.0.2]
Apr  9 13:25:13 liddy postfix/cleanup[12206]: 4DC3914C086: message-id=<deadbeef@example.com>
Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: from=<pepperoni@pepperoni.pizza>, size=2019, nrcpt=1 (queue active)
Apr  9 13:25:13 liddy postfix/pickup[11886]: 85E1F14C089: uid=99 from=<pepperoni@pepperoni.pizza>
Apr  9 13:25:13 liddy postfix/cleanup[12206]: 85E1F14C089: message-id=<deadbeef@example.com>
Apr  9 13:25:13 liddy postfix/pipe[12207]: 4DC3914C086: to=<otheruser@club.sandwich>, orig_to=<ilikesandwiches@example.org>, relay=spamassassin, delay=0.25, delays=0.05/0/0/0.2, dsn=2.0.0, status=sent (delivered via spamassassin service)
Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: removed
Apr  9 13:25:13 liddy postfix/qmgr[995]: 85E1F14C089: from=<pepperoni@pepperoni.pizza>, size=2323, nrcpt=1 (queue active)
Apr  9 13:25:43 liddy postfix/smtp[12212]: 85E1F14C089: to=<otheruser@club.sandwich>, relay=mail-relay.isp.tld[10.0.0.1]:25, delay=30, delays=0.01/0.01/30/0.35, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 92585180F0)
Apr  9 13:25:43 liddy postfix/qmgr[995]: 85E1F14C089: removed

Note the message ID and add it to the mix to get what SpamAssassin thought of it:

$ grep -E '85E1F14C089|4DC3914C086|deadbeef@example.com' /var/log/maillog.1
Apr  9 13:25:13 liddy postfix/smtpd[12203]: 4DC3914C086: client=mail.example.com[209.85.160.41]
Apr  9 13:25:13 liddy postfix/cleanup[12206]: 4DC3914C086: message-id=<deadbeef@example.com>
Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: from=<pepperoni@pepperoni.pizza>, size=2019, nrcpt=1 (queue active)
Apr  9 13:25:13 liddy spamd[5110]: spamd: processing message <deadbeef@example.com> for nobody:99
Apr  9 13:25:13 liddy spamd[5110]: spamd: result: . 0 - scantime=0.2,size=1981,user=nobody,uid=99,required_score=5.0,rhost=localhost.localdomain,raddr=127.0.0.1,rport=42925,mid=<deadbeef@example.com>,autolearn=failed
Apr  9 13:25:13 liddy postfix/pickup[11886]: 85E1F14C089: uid=99 from=<pepperoni@pepperoni.pizza>
Apr  9 13:25:13 liddy postfix/cleanup[12206]: 85E1F14C089: message-id=<deadbeef@example.com>
Apr  9 13:25:13 liddy postfix/pipe[12207]: 4DC3914C086: to=<otheruser@club.sandwich>, orig_to=<ilikesandwiches@example.org>, relay=spamassassin, delay=0.25, delays=0.05/0/0/0.2, dsn=2.0.0, status=sent (delivered via spamassassin service)
Apr  9 13:25:13 liddy postfix/qmgr[995]: 4DC3914C086: removed
Apr  9 13:25:13 liddy postfix/qmgr[995]: 85E1F14C089: from=<pepperoni@pepperoni.pizza>, size=2323, nrcpt=1 (queue active)
Apr  9 13:25:43 liddy postfix/smtp[12212]: 85E1F14C089: to=<otheruser@club.sandwich>, relay=mail-relay.isp.tld[137.82.224.74]:25, delay=30, delays=0.01/0.01/30/0.35, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 92585180F0)
Apr  9 13:25:43 liddy postfix/qmgr[995]: 85E1F14C089: removed

And now you know...the rest of the story.

Tags: postfix