My perfect notebook

I agree completely with Chris Siebenmann's entry on the utility of keeping a notebook. I've done this almost as long as I've been working in IT, and it's saved my ass repeatedly. Also, the way I keep my journal — random notes at the front working toward the back, daily summary at the back working toward the front — means that it's fairly simple to search for my notes on a particular task, or explain to management just what I do with my time.

I love paper. I tried a PDA for a while; hated it, didn't trust it, and gave it up promptly. Scribbling with a pen is faster, more satisfying, and doesn't make me wait for something to reboot or awaken, or force me to learn a different way to scribble. At the best of times, it forces me to think a bit about what I'm doing or seeing, rather than just typing blindly at the problem. (What, you never do that?)

But while a paper notebook is wonderful, it's not perfect. Here's what would be perfect:

Let me paste screen captures right into my notebook. (I'm talking both screenshots and the log files from GNU screen.)

Let me paste sections of my .history file into my notebook complete with timestamps.

Let me cut-and-paste from my notebook to Emacs (or vi, you heathens), and vice-versa.

Let everything I write or paste be timestamped automagically.

Let everything I write or paste be sync'd automagically to some plain text-like format, suitable for grepping, munging, merging into a database, pushing to syslogd, or what have you.

Tags: hardware

Heh ---

Matthew Garret's presentation on Suspend-to-Disk make fun reading.

Arlo's sick with flu or something; I was up 'til 1am last night rocking him to sleep. Haven't done that in a while…

Telling detail: I'm about to blow away Debian testing on my desktop machine and install Ubuntu's Gutsy Gibbon. Partly it's because I'm tired of installing 80MB worth of updates every two weeks, and partly it's because it'll make setting up the printer a breeze.

I'll probably leave half the drive aside for good ol' Debian stable, but Ubuntu'll stay there for experimenting and so my parents, on their next visit, will not have to bring out their 4-tonne laptop.

I'll be reinstalling Ubuntu on my laptop as well; due to a stupid error, I installed Dapper, not Gutsy. I tried updating in one fell swoop, and after three days of apt-get -f install I finally got things working…except for the boot artwork, and GDM doesn't start one time out of three. Interesting experiment, but I think I'll take a do-over.

I may even install it twice, so that I can try out The Depenguinator, which appears to be a lot easier than trying to figure out PXE booting for FreeBSD. Unlike OpenBSD, there's no readily apparent "official way" of doing it, and the handful of HOWTOs I've found have contradicted each other. At this point I'm just too lazy to keep trying and seeing what I'm doing wrong.

Tags: geekdad hardware linux bsd

Happy_25_everyone

title: Happy 2^5, everyone! date: Fri Feb 1 19:47:18 PST 2008

Next power of 2 day: March 4th (2^6).

Tags:

New toy

My workplace just got me a new cel phone: the Sony Ericsson W200a Sony Walkman Phone. The provider is Rogers; minus two points for not letting me make an MP3 into a ring tone, but plus three for letting MidpSSH work. It was a lark to be able to check mail on my firewall box; Mutt was surprisingly useful. No idea how much data costs on the plan I've got, and I don't plan on actually SSHing around very much, if at all…but still, fun. And, as mentioned elsewhere, kudos for including a USB cable and making it show up as an ordinary mass storage device.

Tags: hardware

Project U-13, 0.0.3

Version 0.0.3 of Project U-13, a distro for sysadmins, has been released!

The main change is the addition of RackMonkey, which its website describes as "a web-based tool for managing racks of equipment such as web servers, video encoders, routers and storage devices", at the suggestion of Andy Seely. Also, Lynx has been installed, and there's also the skeletal beginnings of a Cfengine config file.

The ISO has been signed with my public key. Share and enjoy, and comments on a postcard, please.

Tags: projectu13 cfengine

One_day_without_interruptions

title: One day without interruptions date: Wed Jan 23 16:24:35 PST 2008

It was everything I thought it would be. APCUPSd set up, new Postfix map in place for verdammt Sympa lists (replacing the old regexp-based one that allowed far too much backscatter), and a new (though very minimal) offsite Nagios installation. Beautiful.

Tags:

LOLCODE

Yes, I love LOLcats with a love that is fierce. (Though the comments all written in LOLcat just strike me as unneccessary. I know, but that's where the line is for me.)

But LOLCODE just makes me laugh and laugh and laugh:

HAI
CAN HAS STDIO?
PLZ OPEN FILE "LOLCATS.TXT"?

    AWSUM THX
        VISIBLE FILE
    O NOES
        INVISIBLE "ERROR!"

KTHXBYE

I may have to ask for the t-shirt for my birthday. Or maybe I'll just print out the syntax for the wall of my office.

Tags: funny

The Weekend

Tuesday, January 15: Notify users that there will be a brief interruption in our Internet access due to $UNIVERSITY network dep't cutover of our connection from old Bay switches to new Cisco switches. The cutover will be on Friday at 6:30am; the network dep't has said an hour, but it's expected to only be about 20 minutes.

Friday, January 18, 8:30am: Get into work to find that our Internet connection is down. I didn't get notified because the Nagios box can't send email to my cel phone if it can't get access to the Internet. Call network help desk and ask if there were problems; they say no, and everyone else is working just fine. I go to our server room and start trying to figure out what's wrong; can't find a thing. Call help desk back, who say they're going to escalate it.

10am: Get call back from the team that did the cutover. They tell me everything looks fine at their end; as we're the Nth connection to be cut over, it's not like they haven't had practice with it. I debug things with them some more, and we still can't find anything wrong: their settings are correct, mine haven't changed and yet I can't ping our gateway. (The firewall is an OpenBSD box with two interfaces, set up as a transparent bridging firewall.) As the firewall box is an older desktop that had been pressed into service long ago, I decide it'd be worth taking the new, currently spare (YOU NEVER HEARD ME SAY THAT) desktop machine and trying that.

Noon: Realize I have no spare ethernet cards (wha'?). Find two Intel Pro 100s at the second store I go to. Install OpenBSD 4.2 (yay for ordering the CD!), copy over config files, and put it into place. No luck. Still can't ping gateway. While working on the firewall, I notice something weird: I've accidentally set up a bridge with only one interface, while my laptop sits behind pinging the gateway (fruitlessly) ten times a second. (I got desperate.) When I add the second interface, the connection works — but only for 0.3 seconds. The behaviour is repeatable.

3pm: Right after that, the network people show up to see how things are going. I tell them the results (nothing except for 0.3 seconds) and they're mystified. We decide to back out the change from the morning and debug it next week. Things work again instantly. As the new firewall works, I leave it in place.

7.02pm: The connection goes down again. I don't get notified.

Saturday January 19, Noon: I get a call from the boss, who tells me that a meeting at the offices isn't going well because they have no Internet access. Call and verify that, yep, that's the case, and I can't ping there from home. Drive into work.

1.30pm: Arrive and start debugging. Again, nothing wrong that I can see but I can't ping our gateway or see its MAC address. Call help desk who say they have no record of problems. They'll put in a trouble ticket, but would like me to double-check before they escalate it. That's fine — I didn't wait long before calling them — so I do.

2pm: I get a call from the head of the network team that did the cutover; he'd seen the ticket and is calling to see what's going on. He and I debug further for 90 minutes. We try hooking up my laptop to the port the firewall is usually connected to, but that doesn't work; he can see my laptop's MAC address, but I can't see his.

4pm: He calls The Big Kahuna, who calls me and starts debugging further while his osso bucco cooks. We still can't get anywhere. I try putting my laptop on another port in another room, hoping that net access will work from there and maybe I can just string a cable across. It doesn't.

6pm: We call it a night; he and the other guy are going to come in tomorrow to track it down. I call nine bosses and one sysadmin to keep them filled in.

6.30pm: Drive home.

Sunday, January 20, 10.30am: We all show up and start working. We still can't find anything wrong. The boss calls to ask me to set up a meeting with the network department for tomorrow; I tell him I will after we finish fixing the problem.

11.30am: The network team lead gets desperate enough to suggest rebooting the switch stack. It works. We all slap our heads in disgust. Turns out that a broadcast storm on Friday evening triggered a logical failure in the switch we were connected to, resulting in the firewall's port alone being turned off.

Noon: The boss shows up to see how things are going. He talks with the network lead while I'm on the phone with The Big Kahuna; we've decided to try moving to the Cisco switches and make that work while everyone's here.

12.30pm: The Big Kahuna tells me that the problem is the Spanning Tree Protocol packets coming from my firewall box; the Cisco switch doesn't like that and shuts down the switch. I go through man pages until I find the blocknonip option for brconfig. 30 seconds later, everything is working. Apparently, I'm the only one they've come across who's running a transparent bridging firewall, so this is the first time they've seen this problem.

1pm: Debrief the boss. Notify other bosses, sysadmins and users that everything is back up again, then do some last-minute maintenance.

2pm: Drive home.

One thing: the usual configuration for other departments (that don't run their own firewall) is to have two Cisco switches running HSRP; they act as redundant gateways/firewalls that fail over automagically. The Big Kahuna mentions in passing that this doesn't work with OpenBSD bridging firewalls. (Our configuration had been simplified to one switch only on Friday as part of debugging the first problem; I mention this in case this is helpful to someone. I don't understand why this might be the case, so I'm going to ask him about this tomorrow.)

Tags: warstory networking

Coming up

My laptop hard drive started giving scary errors a couple days ago on the way to work (I've got a 90-minute commute by public transit [uck] so I fill the time by reading, listening to podcasts, or working on Project U-13). Fortunately, working at a university means that there are two computer stores on campus. I ran out at lunch, picked up a 100GB drive, and had things back to normal by the next morning.

Well, normal modulo one false start with Debian; I decided to try encrypted filesystems just for fun. But then I suspended, came back with a newere kernel, and it could not read the encrypted LVM group anymore. Whoops.

Still lots of free space on this thing, and I'm thinking of installing Ubuntu, FreeBSD and maybe NetBSD just for fun. Of course, I've got to do it all via PXE since this thing doesn't have any CDROM drive, but that just adds to the geek points.

Project U-13 is coming up on 0.0.3, btw; Andy suggested adding Rackmonkey, which looks quite cool. There's no package for it, so I'm having to do some rather ugly scripted installation…but I can stand it for now. And I've got the barest skeleton of a cfengine file in there too. Watch the skies!

Tags: hardware bsd cfengine projectu13

Merry Xmas!

Tags: geekdad

Stay on target...

Holy crap, it's been a while since I last wrote here. Mainly that's because I've been working on web stuff at work and have felt very little like a sysadmin of late. Thankfully we've got a webmaster hired, and to some extent the work'll be shifted to him in the new year. Of course, that still leaves the redesign of the website and its back end…that's not done 'til it's done.

This week, though, has been slow, and I've been catching up a little on sysadmin work. Part of it was setting up a devel server for the webmaster, and detailing what I was doing in Cfengine as I went along. It was gratifying to get LDAP working (I haven't done that on a Linux machine before; shame on me), and irritating when I realized that I couldn't mount the home directories from the server because I hadn't restarted nscd on the server.

The last two days were spent trying to get encrypted Bacula working between here and $other_university. This was an enormous pain in the ass for two reasons:

1. The Right Way (tm) of doing it is by using TLS, which is what the kids are calling SSL these days, and I have never fully grokked SSL, or the openssl command. I know that there's encryption going on; I know that there are certificates signed by CAs; I know that there's a lot of negotiating of different options. But start throwing in x509 versus PEM, Diffie-Helman parameters and the single most cryptic set of error messages I've ever come across, and I just feel thick. I was reduced to looking at tcpdump output of the negotiation to figure out what was going on, and I couldn't; the Bacula FD client complained that the Bacula Director wasn't producing a certificate, and that was all I knew. The otherwise incredibly excellent docs from Bacula were a trifle thin on all of this, and I couldn't find out much about my situation (going the self-CA route).

2. So okay, fuckit, right? That's why God invented OpenSSH. So whee, start tunnelling port 9102 over SSH so the Director can contact the FD at $other_university, and 9103 back so the FD can contact the Storage Daemon. Only it turns out (my bad for not knowing this before) that not only does the client want to contact the SD, so does the director. Thus, my plan to tunnel to the firewall at the other end and tell the client that it could find the Storage Daemon there didn't work, 'cos the director wanted to contact it there too. (I did briefly try allowing the director to contact the tunnel at the other end: so even though the Storage was working on the same machine as the director, for that one job the Director's connection to it was going to the remote end and getting tunnelled back over SSH. But:

   1. that's horrible, and
   2. I was afraid that when it came time to restore, the Director would figure that it had to contact the Storage Daemon remotely again, complicating an already complicated setup.)

And why was I trying to connect to the remote firewall via SSH, rather than the client I'm trying to back up itself? Because that client is a Solaris machine authenticating against LDAP, and that turns out to bork key-based logins over SSH. What a crock.

Oh well. I did add three other machines here to Bacula this week, so that's good.

Project U-13 is coming along. I'm pretty close to a 0.0.2 release (woot), which should have the following working:

• Cacti
• Nagios
• Request Tracker 3.6
• Cfengine

And by "working" I mean "installed". But I've got a decent setup on my laptop for building and testing it, which means I get up to a couple hours a day to work on it (New Westminster -> UBC == long). Thanks to Andy, he of the amazing speaking skills, for kicking my ass into action.

I'm learning a bit more about Mercurial in the process. After coming from CVS and Subversion, it seems really weird to me that the usual way of branching is "Go ahead, clone another repo! We're Mercurial! We don't care! Repos for everyone!" But if you figure on distributed development — something Linux-y than a controlled work environment — then it makes sense. Not that I think I'll have lots of people working on this thing, but it makes sense that if someone were to take this for their own ends, they wouldn't want to bother copying all the branches…just the one(s) they're interested in.

Last word to my son:

Q: What does a Camel say, Arlo? A: Purhl!

Tags: cfengine projectu13

New disk

At $other_university today adding a new hard drive to our server here: 300GB, instead of 30GB. The users will be very happy. And what with the snow coming down, I'll be very happy if transit keeps running 'til I'm all done.

And now a story about how sometimes it's not all Sympa's fault…

As part of the premptive strike against the mail server's impending failure, I upgraded Sympa (shudder) on $big_server using pkg-src. After copying the list config files over, and pointing it at a separate database, I tried telling it to update its list of subscribers. When I compared a few lists to the already-existing ones, though, they were short random numbers of subscribers. One that I used as a test case, for example, was short about 100 subscribers. That was a concern to me.

The lists that were short all seemed to be ones that grabbed information from the LDAP server, so I tried looking at the queries that Sympa made. The query itself was pretty simple:

(&(objectclass=inetLocalMailRecipient)(gidNumber=10000))

with Sympa being told that mailLocalAddress was the important bit. Should be simple to compare the results from this server and the one where things work…But I lost a good hour of my life, and possibly a couple years of life expectancy, when I became convinced that, somehow, replication to that server was failing, and big chunks of information (like mailLocalAddress) were being lost. Finally, though, I figured out that I'd stupidly been querying the two servers using different credentials. Unfortunately, I figured that out on the way home Friday night.

(Obviously I'd forgotten Aeileen Frisch's rules about system administration:

1. It's a permission problem.
2. If it's not a permission problem, it's a DNS problem.)

Monday, though, was a fresh start and a whole new day. I started by looking again at the queries Sympa made. On $old_server, we'd get about 210 results, with mailLocalAddress in each one of 'em. But on $big_server, we'd get about 210 results, with mailLocalAddress missing in about 100 of them. No wonder Sympa was short.

I double-checked by specifically requesting mailLocalAddress on the problematic server, and it was returned. But $big_server didn't volunteer that information.

I did some digging, and it seems this may be a bug in Sun Directory Server: it should be returning mailLocalAddress as one of the attributes. However, it does not do so for all entries, even when the querying user should have permission to see them. However, I'm unable to see the PR that the thread mentions, since we're not paying for support. (Thank you, Sun.)

Digging into Sympa, though, I found out that this was not the entire reason for the failure. Sympa uses Perl's Net::LDAP module to do its queries. It turns out that Net::LDAP wants a list when you're asking for particular attributes. But in List.pm's _include_users_ldap function, the search is created like so:

$fetch = $ldaph->search ( base => "$ldap_suffix",
                          filter =>; "$ldap_filter",
                          attrs => "$ldap_attrs",
                          scope =>; "$param->{'scope'}");

Changing one line:

$ rcsdiff -r1.1 List.pm
===================================================================
RCS file: RCS/List.pm,v
retrieving revision 1.1
diff -r1.1 List.pm
8646c8646,8647
<                                     attrs => "$ldap_attrs",
---
>                                     # attrs => "$ldap_attrs",
>                                     attrs =>  ["$ldap_attrs"],

meant that, instead of asking for the default attributes (which $big_server was calculating incorrectly), it was asking for mailLocalAddress and succeeding.

And now you know the rest of the story.

Tags: ldap

The pain

$ sudo -u sympa /opt/pkg/bin/perl /opt/pkg/sympa/bin/sympa.pl --help
Line 38, unknown field: bounce_path in sympa.conf
No web archives directory: /opt/pkg/arc

MHonArc is not installed or /usr/bin/mhonarc is not executable.
Language::SetLang(), missing locale parameter
Missing Return-Path in mail::smtpto()
Missing directory '/opt/pkg/bounce' (defined by 'bounce_path'
parameter)
Configuration file /opt/pkg/etc/sympa.conf has errors.

What this error message doesn't bother saying is that it has silently sourced wwsympa.conf as well as sympa.conf, and that the errors come from that file. And no, there is no explicit sourcing of wwsympa.conf in sympa.conf.

God, I hate this software.

Tags: rant

Scratch that

E280R takes different SCSI drives than the E220R. Serial ports and SCSI connectors: A Study in Nemesisssysadminss. Discuss.

Tags: hardware

f(220R) = 280R

At work, our mail server is an aging E220R. While underpowered for all it does, it has behaved well, more or less, until recently.

A couple of months ago it power cycled itself for no apparent reason. This weekend, it did the same thing. This is exactly the same behaviour I saw from another E220R at $other_university, and in that case it got progressively worse. Another sysadmin here says he's seen the same behaviour with two in his care. I'm preparing for the worst.

Part of that has meant preparing to move its functionality to another machine; this has been an excellent chance to delve into the bowels of our mail and list system. I've been steadily improving (read: creating) this for some time now, but this points out some bits I hadn't. So that's good.

Plan C is a loaner E280R from the other sysadmin (op cit.). I ran into trouble getting it working, though. First, I couldn't get a serial console working. (Getting a serial port working always seems to be a pain for me, no matter what the machine.) It has two of the old DB-25 ports; no problem, since I had a splitter and had got that working on the E220R. Except that it didn't work: no matter which port I hooked it up to, I couldn't see any output. I tried flipping the key around to diagnostic mode, but I still didn't see anything. (The manual said that you should be able to force output to ttyA by power-cycling the machine and hitting the power button twice when the amber service LED started blinking…but I never saw the blinking.)

This was especially weird to me because I had been able to get output from the RSC card using the same setup: OpenBSD laptop -> usb serial adapter -> DB-9 to RJ-45 adapter -> Cat 5 cable -> RJ-45 on RSC card. (The only difference was that, with the DB-25 port, the Cat5 cable had fit into the back of the DB-25 splitter.) But I couldn't log into the RSC card, and a quick Google turned up no easy way of resetting its password. (Putting it into the other E280 I have, which runs our database and website, was not an option.)

Out of desperation I finally hooked up the Cat5 to the DB-25 splitter on one side, and the console server on the other…and that worked. Damned if I know what was going on.

But then I had another problem: when it booted, I kept seeing line after line of I2C reset error; after a while, it would power-cycle itself and the pattern would start again. I remembered that op cit. had slotted the second CPU for me, so what the hell: I reseated it, and that did the trick.

Next up is detaching $failing_machine's second hard drive from the mirror and seeing if I can get it to boot in the 280. Let's hope.

In other news, LinuxFest Northwest is calling for papers. Were that not right around the due date of Project U-14, I might try submitting something and see what happens. Oh well...next beer in Jerusalem!

And there's the laptop battery...shoulda charged it at work.

Tags: hardware solaris

Power outage

We had a power outage today at work. The good news is, the UPS' worked. The bad news is, the servers were not set to shut themselves down automatically, and the UPS' ran out literally two minutes before the power came back on. Arghh.

Having a flashlight in the server room is a good thing. So is making sure that your servers are all connected to switches powered by the UPS. So is making sure that you have a laptop with a charged battery and a ready-to-use serial cable connected to your otherwise-accessible-through-SSH console server. So is Sun making an x86-based OS that doesn't hang every time it reboots badly.

In other news: as mentioned on the Dragonfly BSD digest, ICANN blogs (!). They've taken this moment to let us know that the address of L.ROOT-SERVERS.NET has changed. Now you know.

Tags: hardware

Social Message Transport Protocol

This is hilarious.

pkgsrc is still kicking my ass. The latest is a dupe of this bug; I can't tell right now if it's more weirdness with switching GCCs too soon, or something else.

OTOH, I came across MyReview today, and holy crap does it ever look like something my work could use. I've emailed the project thanking them profusely, and suggesting a Freshmeat page (am I the only one who turns there first when looking for Free software goodness?).

Tags: packagemanagement

Hiding_behind_the_desk

title: Hiding behind the desk date: Mon Nov 5 20:12:49 PST 2007

Every now and then it occurs to me that the great part of being a sysadmin, for me, is being able to hide behind the desk. I'm what you might call retiring (read: introverted) and for the most part I'm happy being by myself. I don't want to talk to people, most of the time; I want to stare at something and understand it, make it do neat things.

The last few weeks I haven't been doing that very much. The boss has taken an interest in the long-delayed upgrade to our website, and so that has become my priority. That means talking to people: soliciting proposals from contractors, talking with the communications person, talking to staff to figure out what's needed, what works, and what we'd like if money were no object.

I sometimes think that last part is exactly the wrong thing for me to be doing. I'm pretty comfortable with technology, I like the command line, and I don't do the work that other people do (filling out forms, dealing with money, writing theses, etc.). My needs are obvious to me but difficult to explain to someone not familiar with my job; that's no less true for an accountant, or an administrator, or a student.

It's hard for me to understand sometimes why Exchange really might be the best scheduling software for someone who doesn't have to take care of it. (The snide tone of that comment is made w/o any experience of administering an Exchange server, so please discount it.) Since I don't add records to the database all day, it can be hard for me to really be motivated to add that extra feature, rather than do the odd SQL insert every now and then. And since it's obvious to me that word processors cause chromosome damage, keeping up with the latest versions just doesn't appeal when (say) it's obvious that the firewall rules are in serious need of revision. (Actually I just took a look at them today and they're not as bad as I thought. Either my standards are slipping or my memory is.)

No great insight at the end of all this...

Tags:

pkgtool error: \"libtool: ar: not found\"

One of the things about pkgsrc is that it's very sensitive to paths and which compiler you use. (And fair enough; the whole process of bootstrapping a working set of tools for eight hundred thousand different OS' is ridiculous enough that it's a wonder it works at all. But I digress.)

Case in point: Solaris 10 machine today, installing pkgsrc on it for the first time. I successfully compiled gcc34, added GCC_REQD=3.4 to mk.conf, and then went to compile kile. During compiling of Mesalibs, one of its 3.2x10^6 dependencies, I got this error during the final linking phase:

/opt/pkg/bin/libtool: ar: not found

Naturally it was there in my path, so WTF?

I eventually came across a message to the pkgsrc user's list which suggested rebuilding libtool-base. This made a certain amount of sense to me, as I'd built that package using the bootstrap (ie, not-installed-from-pkgsrc) version of gcc to compile it; it was before I figured out the GCC_REQD directive. So I ran:

$ pkg_delete libtool
$ cd /opt/pkgsrc/devel/libtool
$ bmake clean && bmake install
$ cd /opt/pkgsrc/graphics/MesaLib
$ bmake clean && bmake install

and everything was right again.

Tags: packagemanagement solaris

Greylisting bug with Exchange

Earlier this week the boss forwarded some bounced emails to me and asked me to figure out what had gone wrong. The weird thing was that the email was being greylisted, so it shouldn't have bounced:

This is the Symantec Mail Security program at host
mail.globalsuite.net.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to <postmaster>

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

```
                    The Symantec Mail Security program

```

<example@example.com>: host smtpbackup.example.com said: 451
<example@example.com>: Recipient address rejected: Please
try sending again. (in reply to RCPT TO command)

Turns out that Symantec Mail Security is meant to sit in front of an Exchange server, and it turns out that Exchange has a bug (or had; I'm unsure if it's been fixed) where doesn't requeue email that's been greylisted, and later on bounces it back to the sender without ever having retried.

From what I can tell, globalsuite.net is run by guest-tek.com, which provides high-speed access for hotels…so I'm probably not the only one being asked to explain this bug. :-)

Tags: spam