Carousel is a lie!

...big news over here.

Birth is being induced. Baby's well, wife is well. Wish us luck.

When last Clara visited the doctor (Wed), Dr said that a routine checkup on the babby would be in order at some point this weekend -- Monday, maybe? Turned out to be today, around 2pm. During the ultrasound it further turned out that Babby/Clara had low levels of amniotic fluid. This means that they wanted to induce Real Soon Nowtm. This was begun about 6pm, about ten minutes before I made it to BC Women's. (Stupid bus drivers that don't stop at King Edward when they're asked -- but I digress.)

She's being kept for observation, which means taping big things to her belly and watching the strip of paper slowly come out of the machine that goes ping! when it runs out of paper. I've come back to the house to get things like the hospital bag and cheese, and to feed the catt. (Let this be a lesson to someone: when going to the hospital after your due date, always bring the bag. If they say you don't need it, hit them.)

Clara is doing well. The baby is doing well. No telling how long it could take before active labour starts; the nurses said they've seen it as quick as three hours, or as long as 24. We're hoping for a Canada Day babby, especially after hearing the story about the friend of the nurse who got free stuff for LIFE because the kid was born on July 1st. (Seriously. Government owes the kid a damned helicopter now.)

The next post will probably be made once we're back from the hospital; you should expect something like "Holy CRAP this thing's small!"

That is all.

So:

Arlo Maxwell Reginald Cristofaro, ne Trombone, was born on Saturday July 1st, 2006 at 2.26pm. Clara had a pretty damn good labour once things got going, and horsed him out after only 13 minutes of pushing. As labour stories are, by ancient right, public property, I'll let her post the details.

Both she and Trombone^WArlo are doing quite well. They've both learned how to nurse, and I've learned that the index finger does a lot to calm him down. We've managed to pick up a couple hours of sleep here and there, so we're not too punchy.

For those who haven't seen, here are a couple pix:

Oh, and you know what also calms him down? A slightly modified version of Fat Joe's Lean Back:

My Arlo, he don't know how to dance
He just leans back and he fills up his pants
He does the Rockaway! He does the Rockaway!

It also sends Clara into hysterics, so that's good too.

before I go back to changing diapers: from the ever-excellent Secrecy News comes a link to this report from retired US Army General Barry McCaffrey on his visit to the Guantanamo Bay prisons.

The report is well worth reading. As summarized in the newsletter:

"The JTF Guantanamo Detention Center is the most professional, firm, humane and carefully supervised confinement operation that I have ever personally observed," he stated.At the same time, "Much of the international community views the Guantanamo Detention Center as a place of shame and routine violation of human rights. This view is not correct. However, there will be no possibility of correcting that view.""There is now no possible political support for Guantanamo going forward," Gen. McCaffrey wrote.

McCafferey acknowledges in the report that "During the first 18 months of the war on terror there were widespread, systematic abuses of detainees under US control in Iraq, Afghanistan and Guantanamo. Some were murdered and hundreds were tortured or abused. This caused enormous damage to U.S. military operations and created significant and enduring damage to US international standing."

Yet nowhere in this report does he seem to realize that the U.S. also was condemned for its lawlessness:

The great value of the platform of Guantanamo was that it was a military space in which no Federal District Court had primary jurisdiction. For that reason alone, Gitmo has over the past 45 years been the location of choice for US migrant refugee operations (no appeal to the INS process) as well as other secret operations. No applicable foreign law, no foreign diplomatic intervention, no Federal Court civil orders, no nosy intervention by a US Ambassador -- only the exercise of unilateral military power and the tool of the Uniform Code of Military Justice. It was the perfect deal. No more.

The mourning of the loss of a place over which no court had jurisdiction, into which no "nosy " US ambassador could look, is entirely unbecoming of any democracy -- let alone one that views itself as the Great Vending Machine of Liberty. Yet this point flies right past the nose of a man who gives an otherwise straightforward and unblinking account of Gitmo's failures.

A few things.

First, Arlo is doing well:

Second, there's this.

Third, work has started on the world's most useless project: Theo Kills Your Pony, the aggressively destructive Unix-like system. (Thanks to Zen Render for the name!). I'm attempting to do things semi-right, and that means I've had to learn a bit more about how OpenBSD (and BSD in general) is put together.

Like what? Well, like the name for example. The OS is called TKYP, so that's what I want to show up everywhere. I figured I would start with the output of uname(1), since that's the most Thing is, this took a surprisingly long time to track down.

uname(1) is, as you might expect, a simple wrapper around the uname(3) libc function, which is in turn a pretty simple wrapper around a sysctl call. Through paths that, frankly, I'm still tracking down, you finally get to sys/conf/newvers.sh -- a simple shell script that creates a file called vers.c and sets the variables ostype, osrelease and osversion within it. (Paths are relative to /usr/src, BTW.) After that, the different sys/arch/*/conf/Makefiles compile it -- sys/arch/i386/conf/Makefile.i386, for example -- and then include it in SYSTEM_LD. After that, <handwave>I think these values are simply returned by sysctl(3)</handwave>.

Okay, so now I've tracked that down; I rebuild and install the kernel, then reboot. (QEMU rocks for this sort of thing.) And yay, it works:

-bash-3.1#  uname -a
TKYP tkyp-qemu 0.1 GENERIC#0 i386

Now to rebuild world, right? Wrong: first, Apache kept refusing to compile with an error about not being able to find -ldbm. Trolling through the mailing lists only found one message mentioning a similar problem, and no reply. The CVS tree showed that, since 3.9, a couple minor changes had been committed to the httpd Makefiles mentioning that OpenBSD has used its own dbm library for a while. I tried making a few changes, but couldn't get it to work. So I cheated: I removed httpd from usr.sbin/Makefile and moved on with my life.

Next problem: the GNU configure tools haven't heard of TKYP. (I'm sure I emailed RMS about this...). gnu/usr.bin/binutils is the first thing compiled in world that uses these tools, so that's where I'm looking first. A little judicious editing of config.guess (which guesses the OS and architecture), configure (which figures out what needs to be done for the OS/arch) and config.sub (which says it's a "configuration validation subroutine script"; I'm guessing a basic sanity check) takes care of thing. They're fairly simple changes, as it's pretty much just a matter of copying the OpenBSD entries.

And all this before I can even throw in anything nasty! I got big plans, of course -- SIGKILL replaced with SIGHUP, rot13 encryption for passwords, and the RTM worm pre-installed -- but I haven't even had a buildworld finish yet. Plus, there's the cautionary tale of MicroBSD to keep in mind...whatever else I do, I wanna make sure I piss off Theo for the right reasons. :-)

(Incidentally, the email to root is in etc/root/root.mail; etc/Makefile installs it in the right place. I thought for sure it'd be in share for some reason. newvers.sh mentions this file, plus a few others, that need to be changed to reflect new version numbers.)

...that I'd be wearing a Baby Bjorn and singing my kid to sleep with Joy Division's "Dead Souls". Clara and I dragged out a bunch of tapes yesterday, and man, I haven't listed to that one in years.

When my wife went to pick up the co-op car we'd booked for Sunday, she found that the mirror hanging off the door by the control cables. Fortunately, a little camo duct tape -- a Christmas gift from my parents -- took care of it (at least until the co-op can get it fixed):

And then there's this:

Thanks to John and Arwen for Arlo's shirt, and to Theo et al. for mine.

Woot! I managed to install OpenBSD 4.0 on my work laptop this afternoon while Arlo slept in my arms. Not only that, it automagically set up X and I figured out wireless + OpenVPN. Woot! Firefox is running, I've got Mozex and Adblock going...the only thing left is to figure out how to get IceWM to start up automagically.

This was Arlo's first time going all the way under water. He was definitely surprised by the whole operation, but he didn't panic or cry or anything. Such a trooper.

You have been warned.

I'm back from vacation, and a relaxing time it was. We got to enjoy the hospitality of a lot of family in Ontario, and sysadmin duties were pretty minimal. Hell, I didn't even check Slashdot the whole time.

I upgraded my dad's laptop to the newest version of Ubuntu, and got him a new wireless card that'll work in Linux (though with a restricted driver) as an early Father's Day gift. (If I had been able to buy him an old Orinoco somewhere, I'd've done that instead...as it is, I'll have to cringe under the wrath of my inner RMS. :-)

I also showed him how to FTP a new Wordpress theme to the server, and I have to say I'm impressed with how easy Gnome/Nautilus makes it for him. I'm starting to understand the appeal of a nice GUI, though I'm still sticking to my xterms for now.

As a bit of reciprocation, my dad gave me a 2GB SD card for the new camera we've got -- which was nice, because the old 256MB card was filling up very quickly.

I was happy to get back to work and find that, really, there wasn't that much to clean up. Coworkers had filled in nicely for me, and the worst that had cropped up was an SQL bug in a new credit card payment form; it was failing to update the second of two places that indicate someone has paid. (Yes, redundant, but to be fixed next year.) I'm a bit irritated by this, as the bug was an SQL statement, passed to PEAR's Db module, that said:

...set updated_by="foo form" form" ...

Yes, this is my typo, but why did PHP not report this error? What happened, and why wasn't it being caught?

Anyhow. Now that I'm back, relaxed, forced by funding to put off Big Website Rewrites 'til next year and mostly done with this year's web work, I'm finally able to contemplate upgrading our Big Server(tm) to Solaris 10. That will be a bear of a job, but it'll be nice to get it done.

On the home front, I'll be switching to Uniserve's ADSL shortly. They do allow servers, and offer static IPs for a small charge; that'll be nice. We used them at my last job, and the service was fine as logn as you didn't have to contact tech support.

Surprisingly, they also have this clause in the TOS:

65. UNISERVE shall have the right, without notice, to insert advertising
data into the Internet browser used by a UNSERVE customer, and
transferred to a UNISERVE customer over UNISERVE's network, so long as
this does not involve UNISERVE establishing the identity of the
customer to whom such data is sent.

In a previous life at an ISP, we started putting in machines run by a company called Adzilla. They were, as far as I could tell, proxy servers that replaced the ads on, say, CNN's website with ones for local businesses. I thought it was scummy, but couldn't persuade the bosses of this. I'm fairly certain this is the same thing, and probably the same company too. I still don't like it, but Uniserve is the best option I've got right now. And at least they admit they're doing this.

...is this picture of Arlo and me:

My god, the kid's cute:

Now that Clara's heading back to work, my schedule has changed a bit: I'm staying at home on Wednesdays to take care of Arlo, and then working from home on Saturdays to make up the time. I'm grateful to my boss for letting me do this, and I'm hopeful it will work out.

My first Wednesday (July 4th) went pretty darned well, really. Arlo ate, he played, he got vaccinated (Chicken pox; I had no idea they vaccinated for it), he napped and then he played some more. I didn't drop him, he didn't freak out and it was a great deal of fun.

As it happens I got to take care of him on Friday, too; my mother-in-law, who's going to be taking care of him two days a week, had a sudden trip to the emergency room. She's okay, but wasn't able to take care of him that day. (She was mad about it, too...) I called into work and let them know I wouldn't be in, then went in anyway just to make sure a few things were okay. I've got some karma built up and a fistful of sick days I rarely take, so all was well.

And then yesterday I worked from home. And man o man, did I get stuff done. Not quite as much as I wanted; I was hoping to use flar to duplicate a Solaris machine so I could test it, and ran into a bug that took a while to figure out. (If the patch I applied fixes the problem, I'll write it up here since there was only one other reference I could find.) But it was lovely to work for, like, four hours in a row on something and not be interrupted. Plus, there's the skipping of the 90-minute commute to enjoy.

My fondness for trivial patches continues. You may envy me.

For my own future reference, Otter Escaping North recently posted two excellent comments about being a geek parent in a recent Slashdot discussion about PC parental controls. The whole article is worth reading (though I always read at +3), but Otter's really resonated with me.

And from the world of obnoxious EULAs comes this gem from Live365's software player for Windows:

"You may not alter, merge, modify, adapt or translate the SOFTWARE
PRODUCT, or decompile, reverse engineer, disassemble, or otherwise
reduce the SOFTWARE PRODUCT to a human-perceivable form."

So hexdump -C is out, then? Or looking at it with less? Sigh…

I've had a bunch of ideas lately. I'm inflicting them on you.

The presentation went well...I didn't get too nervous, or run too long, or start screaming at people (damn Induced Tourette's Syndrome) or anything. There were maybe 30 or so people there, and a bunch of them had questions at the end too. Nice! I was embiggened enough by the whole experience that, when the local LUG announced that they were having a newbie's night and asked for presenters to explain stuff, I volunteered. It's coming up in a few weeks; we'll see what happens.

And then I thought some more. A few days before I'd been listening to the almost-latest episode of LugRadio (nice new design!), where they were talking about GUADEC and PyCon UK. PyCon was especially interesting to hear about; the organizers had thought "Wouldn't it be cool to have a Python conference here in the UK?", so they made one.

So I thought, "It's a shame I'm not going to be able to go to LISA this year. Why don't we have our own conference here in Vancouver?" The more I thought about it, the better the idea seemed. We could have it at UBC in the summer, where I'm pretty sure there are cheap venues to be had. Start out modest — say, a day long the first time around. We could have, say, a training track and a papers track. I'm going to talk about this to some folks and see what they think.

Memo to myself: still on my list of stuff to do is to join pool.ntp.org. Do it, monkey boy!

Another idea I had: a while back I exchanged secondary DNS service, c/o ns2exchange.com. It's working pretty well so far, but I'm not monitoring it so it's hard for me to be sure that I can get rid of the other DNS servers I've got. (Everydns.net is fine, but they don't do TXT or IPv6 records.) I'm in the process of setting up Nagios to watch my own server, but of course that doesn't tell me what things look like from the outside.

So it hit me: what about Nagios exchange? I'll watch your services if you watch mine. You wouldn't want your business depending on me, of course, but this'd be fine for the slightly anal sysadmin looking to monitor his home machines. :-) The comment link's at the end of the article; let me know if you're interested, or if you think it's a good/bad/weird idea.

The presentation also made me think about how this job has been, in many ways, a lot like the last job: implementing a lot of Things That Really Should Be Done (I hate to say "Best Practices) in a small shop. Time is tight and there's a lot to do, so I've been slowly making my way through the list:

• Improving backups (Bacula, Amanda)
• Automated install (FAI, Jumpstart)
• Monitoring services (Nagios)
• Monitoring performance (MRTG, Cacti)
• Ticket system (RT)
• Automating management (Cfengine)

Some of these things have been held up by my trying to remember what I did the last time. And then there's just getting up to speed on bootstrapping a Cfengine installation (say).

So what if all these things were available in one easy package? Not an appliance, since we're sysadmins — but integrated nicely into one machine, easily broken up if needed, and ready to go? Furthermore, what if that tool was a Linux distro, with all its attendant tools and security? What if that tool was easily regenerated, and itself served as a nicely annotated set of files to get the newbie up and running?

Between FAI (because if it's not Debian, you're working too hard) and cfengine, it should be easy to make a machine look like this. Have it work on a live ISO, with installation afterward with saved customizations from when you were playing around with it.

Have it be a godsend for the newbie, a timesaver for the experienced, and a lifeline for those struggling in rapidly expanding shops. Make this the distro I'd want to take to the next job like this.

I'm tentatively calling this Project U-13. We'll see how it goes.

Oh, and over here we've got Project U-14. So, you know, I've got lots of spare time.

Tonight, my 15.5-month-old son pointed at a hat and said "Hat!"

I said, "Yes, you're right. And what's that on the hat?"

"Guk?"

"It's a penguin. And what does a penguin say?"

"Inix."

It's a proud day.

Matthew Garret's presentation on Suspend-to-Disk make fun reading.

Arlo's sick with flu or something; I was up 'til 1am last night rocking him to sleep. Haven't done that in a while…

Telling detail: I'm about to blow away Debian testing on my desktop machine and install Ubuntu's Gutsy Gibbon. Partly it's because I'm tired of installing 80MB worth of updates every two weeks, and partly it's because it'll make setting up the printer a breeze.

I'll probably leave half the drive aside for good ol' Debian stable, but Ubuntu'll stay there for experimenting and so my parents, on their next visit, will not have to bring out their 4-tonne laptop.

I'll be reinstalling Ubuntu on my laptop as well; due to a stupid error, I installed Dapper, not Gutsy. I tried updating in one fell swoop, and after three days of apt-get -f install I finally got things working…except for the boot artwork, and GDM doesn't start one time out of three. Interesting experiment, but I think I'll take a do-over.

I may even install it twice, so that I can try out The Depenguinator, which appears to be a lot easier than trying to figure out PXE booting for FreeBSD. Unlike OpenBSD, there's no readily apparent "official way" of doing it, and the handful of HOWTOs I've found have contradicted each other. At this point I'm just too lazy to keep trying and seeing what I'm doing wrong.

Airport Toy X-Ray Machine, c/o Saint Schneier. Maybe I'll get one for Arlo.

Hah! Just listened to Season 5, Episode 13 of LugRadio, and they read my letter! It was about how, if you're working tech support, you usually have no idea whether or not the other person is actually clueful or not; thus, the inevitable "Is it plugged in?" questions which frustrate techies. Or, as Jono Bacon put it, you have no idea whether the customer is "chuffing up the bong pole". (And the giant round of Whiskey Tango Foxtrot? goes to…Jono Bacon! :-) So this is what fame feels like.

Incidentally, LugRadio Live USA looks like it's going to be freakin' sweet. I wish I could go, but Project U-14 is going to be in pre-release testing at that point...

As Project U-14 draws to a close, I've been spending a wonderful couple of hours on the couch playing with my laptop while Arlo naps.

Here's what I've been doing:

• Tracked down a bug that in gpsbabel that would make it crash if you gave it no filename to process. Turned out it was fixed late last year, and the new version will be in Lenny, but that was okay; the practice was good, it didn't take too long, and it turns out both the developers and I came to the same conclusion. I hope they know what they're doing, at least. :-) I'm unsure if I should file a bug in Debian or not; Lenny's coming soon, and I'm not sure of the usual practice in this case.
• Actually began work on Project U-13; I'm trying out fai-cd, rather than GRML. GRML's nice, but it's a way non-stock Debian install by the time it's done, and I'd like to stick as much as possible to stock Debian (OS of the OElder Goeds). We'll see how it goes.
• Read about how Dan Fucking Kaminsky managed to Rickrolled Facebook thanks to nasty DNS hijacking (similar to Verisign's Site Finder) by Earthlink and Comcast. No copy of the presentation up on his site, from what I can see, but I'm looking forward to it.
• Eating chocolate chip cookies my wife made. Yum.

...after a month off, and almost no emergencies in my absence. Sweet!

Now if only I could catch up on sleep. I remember this from the first kid: you never know just how much you can accomplish on so little sleep.

Flu sucks. I've been down with it for the last two days. Today I'm feeling a bit better — a little bit, mostly with lots of drugs. We'll see if I make it in to $WORK today or not.

What else have I been doing? Let's see…

• Our Roku SoundBridge M1001 has died after just over a year, which turns out to be a semi-common thing. Fortunately I've found a local electronic repair store that I'm hoping will fix things up. Other than that I've been fairly happy with it; it's definitely a lot handier with the SoundBridge Commander, though.
• Started playing with Xen on my web server. I'm going to try setting things up more complicated than they are now, but hopefully more secure: one instance to serve files, with /home (where all the sites are) mounted read-only; one with a database server; one for mail; one for SSH access; and one with the bag of passwords (LDAP, probably). Yeah, it's hopelessly baroque for a home server with, what, 8 websites, but it's a work in progress and it's fun to play. :-)
• Installed the Awesome window manager after reading about it for the nth time. Happy so far, but I'm still exploring.
• Tried making bread that rises overnight. I got into the habit of making my own bread at the beginning of the year, and I really got used to it. But two kids means much less time, and half an hour to mix and knead the dough is getting hard to find. So the latest trick has been using the KitchenAid and its dough hook (I did it all by hand previously), and fiddling with the amount of yeast so that, in theory, it'll rise overnight and be ready to bake when I get up at 5:30am to go to work. This used to work, but now I'm having problems with it collapsing. That may be because I'm using instant yeast rather than the regular, active stuff. Updates will follow as events unfold.

And that's that. Time to put together some lunch and see about going to work.

That's not quite my dad at c2k8, but damn if it wasn't enough to make me look twice.

Tuesday: youngest son (8 months old) up at 5:30am teething.

Wednesday: youngest son up at 5:15am teething.

Thursday: youngest son up at 5:30am teething. I'm so tired I go to bed at 8:30pm and fall asleep immediately.

Friday: youngest son up at 4:45am teething. At 5:45am he goes back to sleep. At 6am my phone tells me the DNS server at work is down; I can't raise it. I restore backed up zone files to a spare Xen instance (hurrah!), give it the DNS server's IP address and head into work. I restart the machine and shut down the Xen instance; can't figure out why the machine shut down in the first place. Then I discover a replication problem between two of our LDAP servers which is resulting in random bounced email for a newly created account.

I want to go home now. But there's a Very Important Meeting(tm) at 1pm, and I can't leave before then.

<headdesk>

Okay, so the other thing I was going to do was blog regularly. And now it's three days later.

But I've been meaning to mention another aspect of the new job as well. When, previous to working here, I'd thought about what I'd like my next job to be like, it was pretty consistent:

• Less desktop support (ideally, asymptotically approaching zero)
• More Unix server work
• Bigger place

The last point needs a bit of expansion. See, my first job in IT was on the helpdesk of a small ISP. There were three of us on helpdesk, one webmaster, one sysadmin, one database guy, one secretary and one manager; I got some mentoring from the sysadmin (who split his time betwen us and a sister company), but not lots. My second was at a startup company; the guy who hired me was a good mentor, and then after a while after he left I got to hire a junior and be a mentor to him. The job I just left was pretty much just me, though I'm lucky enough to have other people I could talk to; UBC's a big place, but I was in a small department.

So my next job was going to be bigger (as in a bigger installation — maybe a whole data centre, even) and have more people — because I really, really wanted to hang out with my peers and learn from them. I envied the people I'd met at LISA in 2006 who were part of a team, who had people to teach and people to learn from.

Well, at this job it's...just me. Sort of; the folks I've been working with for the last six months (one lab out of the five that make up the centre) are pretty technical. They know way more about Java and MySQL and web development and how the latest CPUs from Intel compare with AMD than I do. But I'm the sysadmin. There might be another in the future, but there isn't now.

But! But, there are two sysadmins on the floor above me who work in another department. For various reasons, we're going to be working closely for the forseeable future. On Friday, I went up to talk with them about how that was going to work out.

They knew stuff I didn't know -- no surprise there -- but it turned out I could show them a trick or two as well. We swapped war stories, discussed our very different backgrounds (saved for another entry), and just shot the shit. It was wonderful.

It's weird, because I'm an introvert, and not very socially apt. (Or ept. As in "opposite of inept".) But it's really, really nice to get together with people who like being a sysadmin the way I do.

(This entry brought to you by the number i, the letter Ve, and my youngest son's 90-minute nap.)

I can't believe it...my youngest son, after nearly three weeks of being up four or five times each night, slept nearly all the way through without a break: he only woke up at 1am and 5:15am, which is close enough to my usual wakeup time as makes no difference. It was wonderful to have a bit of sleep.

This comes after staying up late (11pm!) on Sunday bottling the latest batch of beer, a Grapefruit Bitter recipe from the local homebrew shop. You know, it really does taste like grapefruit, and even this early I'm really looking forward to this beer.

My laptop has a broken hinge, dammit. I carry it around in my backpack without any padding, so I guess I'm lucky it's lasted this long. Fortunately the monitor still works and mostly stays upright. I've had a look at some directions on how to replace it; it looks fiddly, but spending $20 on a new set of hinges from eBay is a lot more attractive than spending $100. Of course, the other consideration is whether I can get three hours to work on it….But in the meantime, I've got it on the SkyTrain for the first time in a week; it's been hard to want to do anything but sleep lately.

Work is still busy:

• I'm trying to get tinyMCE and img_assist to work with Drupal

  1. tinyMCE is no problem, but the img_assist part wasn't working with it. Turns out you need to get dev versions of the img_assist and WYSIWIG modules, because the latest version of tinyMCE (which is required for Drupal 6) broke some parts of img_assist (which, in turn, was in the middle of a rewrite anyhow). Eventually, the admin ass't will be able to work on the website w/o having to know HTML which == major win.

• Contacting vendors to look at backup hardware. So far we're looking at the Dell ML6010 and the Sun SL500. They're both modular, which is nice; we've got (low) tens of TB now but that'll ramp up quickly. The SL500 seems to have some weird things; according to this post, it takes up to 30 minutes to boot (!) and you can't change its IP address without a visit from the service engineer (!!). Those posts are two years old, so perhaps things have changed.

• Trying to figure out what we want for backup software, too. I'm used to Bacula (which works well with the ML6010) and Amanda, but I've been working a little bit with Tivoli lately. One of the advantages of Tivoli is the ease of restoring it gives to the users…very nice. I'm reading Backup and Recovery again, trying to get a sense of what we want, and reviewing Preston's presentation at LISA06 called "Seriously, tape-only backup systems are dead". So what do we put in front of this thing? Not sure yet…

• Speaking of Tivoli, it's suddenly stopped working for us: it backed up filesystems on our Thumper just fine (though we had to point it at individual ZFS filesystems, rather than telling it to just go), then stopped; it hangs on files over a certain size (somewhere around 500kb or so) and just sits there, trying to renew the connection over and over again. I've been suspecting firewall problems, but I haven't changed anything and I can't see any logged blocked packets. Weird.

Update: turned out to be an MTU problem:

• The Thumper supports Jumbo frames
• Our switch supports Jumbo frames
• Our firewall's inside interface, a GigE from Broadcom, does not support Jumbo frames
• Our switch will silently drop jumbo frames when directed to an interface that does not support it

I had no idea there were GigE NICs that did not support Jumbo frames. Though maybe that's just the OpenBSD driver for it. Hm.

Welp, after my training at LISA I finally got to start using SELinux. I was setting up a CentOS server with Mascot, search engine software for mass spectrometer software, and I thought I'd give it a try.

Mostly it turned out to be simple -- semanage fcontext to add some new httpd -friendly locations where the software had been installed, restorecon to set the labels. One thing that did take some tracking down was digging up exactly what this meant:

type=AVC msg=audit(1259021236.914:280): avc:  denied  { execstack}
for  pid=6845 comm="ld-linux-x86-64"
scontext=user_u:system_r:httpd_sys_script_t:s0
tcontext=user_u:system_r:httpd_sys_script_t:s0 tclass=process

This happened when the install script tested Perl to make sure everything was okay.

As described by Dan Walsh and Ulrich Drepper, this means that the Perl executable was marked as needing an executable stack. Not only is this a Bad Thing(tm), it's not usually necessary these days (what with the Internet and all). execstack -c cleared the flag, and things appeared to work after that; it was right at the end of the day, though, so it's possible problems will show up today.

And then when I got home...it was wonderful. The kids'd had two-hour naps each, there was a wild rice casserole in the oven (The Cheese Fairy is always amazing), and my parents had sent the kids a calendar full of pictures of Canadian wildlife. I got to tell Trombone how the beaks of different birds (great blue heron, snowy owl, cardinal) were adapted for eating different things; I think he was interested, and that was just flat out fascinating. Ah, domestic bliss.

Busy day:

• Wake up at 5am because the youngest son's teething and he's going to be up at 5.20am

• Clean up ZFS snapshots yet again; repeat "I must schedule this in cron" for the nth time

• Put in request to maintenance to look at server room humidifier; current block o' cold weather == 10% RH in there

• Find out why Mailman stopped working (probably permission problems on the logs), and how to monitor this (settle for web interface for now, since that wasn't working either; will probably need to make my peace with user accounts for Nagios on machines I'm monitoring)

• Figure out why Drupal is shitting cron.php files all over the place (still no idea)

• Fill out performance review for self

• Back up Windows 2003 server before tossing it over the fence to software installer

• Start writing article for SysAdvent at last on OCSNG/GLPI

• Struggle with cfengine tidy stanza that doesn't work; repeat "I must upgrade to cfengine 3 or puppet" for the nth time

Tonight, bed at 8.30pm. And there's no shame in that.

Happy 2010 everyone! Now that it seems to be well and truly under way, I feel I can say that safely.

It's been busy so far. All the stuff I didn't do in 2009 is still on my plate...which is obvious, right? but it still caught me by surprise after the 3 days doing Xmas maintenance on my own. It was easy to forget that there are, you know, people waiting to show up and do work.

Like the new students we've got for one of the faculty members. I'd upgraded OpenSuSE on their new workstations over the holidays, then when they came in yesterday the carefully-tweaked dual monitor displays weren't working. Arghh.

Or the guy who's let me know that he wants to get moving on the MySQL/PHP website he's building...which reminds me that I've still got to move the website to a virtualized machine. I'm tempted to do that RIGHT NOW and put his site in there, but I don't think that'll be the best way to do it.

Or the new project my boss is part of, which involves researchers from across Canada. For me, it's a new website, hardware recommendation and purchases, maybe a new LDAP server. I could add a new root suffix to the existing LDAP server, but

a. we don't need it yet a. that seems like it'll make it more difficult to move later a. while I can create one in the existing LDAP server (Fedora/389/CentOS DS), the cn=config tree seems suspiciously empty of any entries related to the new root...so I'm leery of trusting it.

I still haven't sat down yet and tried to plan my year. Partly I've been busy, partly my planning tools are a bit of a mess (daytimer + orgmode + RT). But at some point I need to get my priorities straight and oh, how I long to have them straight. I feel a bit like I'm spinning my wheels right now.

Ah well. In other news, Xmas was good; my kids got two guitars (one acoustic with an Elmo sticker, one fake double-neck electric) which makes four guitars they have now. Since they no longer have that to fight over, they've taken to fighting over a microphone (cardboard tube stuck in a toy that acts like a stand). But damnit, they're still cute.

Finally: Just for fun right now I did a word count of all my blog entries. I've been blogging since 2004, and I've got something like 158,000 words. Amazing. And there are still some entries I've got to grab from my old Slashdot journal.

Backups: Bacula has been giving me problems the last week or so. I've got this file server I'm trying to back up; it's got a 2TB partition, and I've been naively trying to just grab it all in one go. Partly that's because it hasn't been backed up before, and I figured this'd be the quickest, simplest way to get going.

What's happened is that after slurping 2 TB over a 100 Mbit connection (no, there's no way to make that quicker), which takes 53 hours, the writing to tape fails for reasons I've yet to figure out. Bacula doesn't say "Oh, the first bit worked so I can just grab that next time...." (To be fair, that's probably a much harder problem than I imagine.) And in the meantime, despite having two drives and two pools of tapes, backups for other stuff pile up behind this big backup and then don't work: they get put on spool space, but then despooling to tape fails.

Contact manglement: I've been looking for a contact management program for $WORK. Requirements:

• database (not LDAP), so I can do funky joins and such
• web-based
• FIAF
• members can belong to more than one group
• extensible (ideally w/o my intervention) without making me feel like I've returned to maintaining the database frontend at my last job

This turns out to be surprisingly hard to find, and not just because Freshmeat's interface is terrible. Applications appear to fall into n categories:

• Incomplete and not updated since 2004
• Complete but full of bletcherous hacks like direct SQL manipulation of the table (making future modification hard), or assuming that a person belongs to one-and-only-one group; often intended for personal use, not business, as evidenced by space for a birthday and contact fields for 30 different IM platforms
• SugarCRM
• Replacements for Access/Filemaker

So now I'm trying to decide between using Dadabik, which'll let me make a frontend w/o much work as long as I can come up with a schema, or modifying one of the complete-but-bletcherous apps and getting a prettier page. (I'm always paranoid about people refusing to use a web-based tool because it isn't pretty enough; I don't know how to make it prettier and it's not something I personally care about enough to do something about, so I'm caught between don't care and don't know how to fix it if I do care. As a result I panic.)

Family: Son #2 went to the hospital Sunday night with his mom; he's fine, but I was up 'til they got back at midnight. Still got up at 5:30am as usual, thinking I'd catch up last night. Then Son #1 had a bad nightmare last night and it took a while to get him calmed down. Spent a couple hours after that staring at the ceiling, trying to get myself calmed down. Still up at 5:30am as usual.

Dentist: Root canal didn't work. My former dentist, who is the second most graceless dentist I've ever seen, couldn't get through and referred me to an endodontist (someone who does root canals; thank you, Wikipedia). My appointment for them is on April 1st.

And that is that.

Wednesday, 5am: 2yr-old son wakes up and starts to shout. I go into his room to pick him up and step into a pool of cold vomit. He seems fine.

Thursday, 5:15am: 2yr-old son wakes up and starts to shout. I go into his room to pick him up and step into a pool of cold vomit. I stay home a few hours to clean up the carpet. He seems fine.

Friday, 6:10am: 2yr-old son wakes up and starts to shout. I go into his room to pick him up and turn on the light. I avoid stepping into a pool of cold vomit. 4yr-old son wakes up and starts to throw up; he throws up every half hour 'til noon. I stay home to take care of the kids. We put a towel on the carpet by the head of 2yr-old son's crib. 4yr-old son grey-green most of the day but picks up after nap time. 2yr-old son seems fine. We watch fourteen hours of television.

Saturday, 2am: 2yr-old son wakes up and starts to shout. Clara goes down and finds him in a pool of vomit within the crib. She sleeps in the rocking chair with him 'til 7am. 4yr-old son much better ("I feel MUCH better today!" he says). 2yr-old son whiny and miserable. Clara and I feel like shit and are whiny and miserable. Clara's parents take 4yr-old son for the day. I drive us all to the doctor, worried about how many things are going on right now (night vomits + bug + teething + what the hell else?); dr tells us almost certainly the same bug, but manifesting itself differently in all of us. Clara drives us home while I slump against the car window, cursing the sunlight. 2yr-old son refuses to nap so I hold him in a rocking chair for two hours while reading James Herriot. We watch sixteen hours of television and eat plain crackers dipped in filtered water.

Sunday: 2yr-old son feeling better. He has not thrown up in the night. 4yr-old son feeling even better than yesterday. Clara and I still feel like shit. We take the kids to the park in the morning. 2yr-old son naps on his own. Clara and I gradually improve through the day. We eat plain crackers with salt in the morning, plain crackers with peanut butter in the afternoon, then four pounds of leftover mac and cheese at 8pm. We watch "Office Space" and laugh weakly.

Monday, 4:45am: 2yr-old son wakes up and starts to shout but agrees to go back to sleep. 4yr-old son does the same at 5:45am.

I'm back from 3 weeks' vacation. My family and I drove (!) from New Westminster, BC to Brandon, MB to visit my brother and sister-in-law and my parents. We took our time driving out and back: 6 days out, 6 days back, and, um, 9? days there.

Originally our plan was to camp most of the way there and back, but after two days we realized that the kids were a tad young for that...between mosquitoes, not going to sleep on time, and long-ass days in the car, it made a lot more sense to stay in hotels. (Kids dug it a lot...every new Super-8/HoJo/Travellodge prompted a "Wow, TWO TABLES! This is the best room EVER!" from the oldest.)

Family was fun. The prairies were hot (who knew?). Hotels with swimming pools are to be celebrated. The Rockies were, as ever, amazing. Canmore has a good brewpub and is a very, very pretty town. Aylesbury, SK had a horse to look at, which made it a good place for a picnic with the kids. Regina is made entirely of ass. Saskatoon, home of The Dymunds, is as pretty as ever.

TThe Royal Tyrrell Museum was incredible, even in a visit abbreviated by two toddlers...first time I'd ever seen real dinosaur skeletons before ("and I saw a T Rex anna stegasaurus anna triceratops anna archaeopteryx anna...). I could have easily spent a week there. If you ever get the chance, visit by any means necessary.

Despite some last-minute panics before leaving, I only received one phone call while away, and that from a salesman. I managed to forget about work entirely, and I expect to have to be guided to my desk tomorrow. And that is the capstone of a good vacation.

My in-laws got us a family membership at Science World for Xmas last year. Yesterday I got to take my 4 year-old (how should that be hyphenated?) son for the morning. It was his third trip and my second.

We headed right for the Eureka room, which is aimed at the young 'uns, and he ran around showing me everything. "Daddy, here's a big tube where you can shoot out parachutes! And this air gun shoots balls up into the water!" We found out that you could stuff three plastic balls into the air gun at once (poom poom poom!).

Oh, and when we got home he wanted to do an experiment. He got some pennies and put them in a jar with water, to leave them for a few days and see if they would dissolve. I had a maple syrup candy in my pocket (no idea where I got it), so I threw that in too. The candy has dissolved and made the water brown, so I'm curious to see what he makes of that.

Science World is just incredible. I long to go see the grownup stuff, but even the kid stuff is enormously fun and moderately educational (though that's not my son's priority right now) (dang kids). I grew up in small towns as a kid, so trips to museums like this were rare, enormous fun. (And I never did get to go to Science North...) It's amazing to me that this stuff is right here, only a half hour away by transit. I'm still a little shocked we don't go, like, every weekend.

In other news, I've got a starter going for a batch of beer next weekend. It's a Belgian yeast, harvested from my January batch. The yeast was washed following these instructions, and the starter took off in about 18 hours. It seems to be doing quite nicely; I'll probably stick it in the fridge on Wednesday or so and cold-crash it.

The ingredients are pretty much whatever I have around the house: the last of my Gambrinus ESB, some biscuit and wheat malt, a bit of roasted barley, and the hops are Centennial, Goldings and Mt Hood. My father-in-law would call this a "ministrone" -- Italian not just for that kind of soup, but for "dog's breakfast" or "big ol' mixup". (I kind of like the idea of an Italian sounding like he's from Missouri.)

Still looking for a name; suggestions on a postcard, please. Sponsorship options are available. :-)

After that's in the bag, it's time to head back to Dan's for a shopping trip. This time, I think it'll be a 50-lb bag of plain ol' pale malt, and I'll see what difference that makes.

This week has been a writeoff. I took 2.5 days off sick (shoulda been 3), I slept for maybe four hours last night, and I've stared blearily at my work monitor more than I care to admit.

I did get some stuff done: updated one of my wireless routers to the latest version of OpenWRT (and promptly found problems), got njam working on the new incarnation of the MythTV box (the kids are thrilled), and listened, rapt, to my youngest son proudly show his friend around the house while I hid upstairs in bed, snuffling quietly. So there's that.

I've been reading "A History of Christianity". I long for footnotes, but more for comfort than anything else; other than that, it's pretty damn good. I've also got "Why Evolution Is True", and that's good too. I picked up Sue French's "Deep Sky Wonders", thanks to my ever-generous in-laws, and if the verdammt clouds ever clear up I hope to put it to good use. (Though I was proud, the last time the sky was clear, to have found NGC 1662 by Orion, which is mentioned in this book...I was surprised at how easy it was to find.)

Optional reading for the week: "Sun's Unified Storage 7210 -- designed to disappoint?". Bryan Cantrill is a class act.

Mandatory reading for the week: Terry Milewski's article on Section 34 of Bill C-30, which outlines the duties of inspectors, appointed by the minister under the act. Quote:

The inspectors may "enter any place owned by, or under the control of, any telecommunications service provider in which the inspector has reasonable grounds to believe there is any document, information, transmission apparatus, telecommunications facility or any other thing to which this Act applies."

...The inspector, says the bill, may "examine any document, information or thing found in the place and open or cause to be opened any container or other thing." He or she may also "use, or cause to be used, any computer system in the place to search and examine any information contained in or available to the system."

...The inspector -- remember, this is anyone the minister chooses -- is also empowered to copy anything that strikes his or her fancy. The inspector may "reproduce, or cause to be reproduced, any information in the form of a printout, or other intelligible output, and remove the printout, or other output, for examination or copying."

...Finally, note that such all-encompassing searches require no warrant, and don't even have to be in the context of a criminal investigation. Ostensibly, the purpose is to ensure that the ISP is complying with the requirements of act the but nothing in the section restricts the inspector to examining or seizing only information bearing upon that issue. It's still "any" information whatsoever.

Horrible. Email your MP today.

Sunday we all(1) went to the AAAS Family Science Day and ho boy, it was fun. The kids had fun, of course; the demos were aimed pretty much right at them, and there were stickers and lasers and more stickers and popcorn. Friends of ours showed up with their two kids in tow, and all the boys got to run around and shine lasers at each other's heads. My wife had fun cos hey! SkyTrain! that's fun. But also? I had fun. And by fun I mean ZOMG. Because among other things, the Physics Department(2) from UBC had a real, working cloud chamber. A CLOUD CHAMBER.

When I was a kid, I bought this book at a library book sale: a collection of Scientific American "Amateur Scientist" columns. There was all kinds of stuff in there, from how to build your own solid rocket motors to measuring the metabolism of rats (I think part of it was TAKING SAMPLES OF THEIR BLOOOOOOOD) to grinding your own telescope mirror. For a budding geek, this was simply endless entertainment.

But -- BUT -- there was also a column explaining how to build your own cloud chamber and watch cosmic rays decay before your eyes. There was also an offer to mail you a radioactive speck so that you'd see more exciting stuff, which I thought was possibly the coolest thing ever.

This may have been the beginning of my interest in particle physics, expressed in later years when I:

a) got a local welder to weld some copper pipe in a loop, in preparation for a sadly-never-completed circular particle accelerator (linear accelerators are for suckers and chumps);

b) sent a fan letter to Carlo Rubbia and was thrilled to get a personally-signed letter back;

c) went to university to take physics in preparation for a career at CERN (only to get distracted by the Internet and libraries that were open 'til midnight, and fail out two years later); and

d) was amazed that a coworker at my first sysadmin job was the son of the director of IT for CERN; I tried to get him to get me a job, and he said no, but he gave me a cool CERN swatch instead.

And in all that time, do you think I'd ever seen a real, live cloud chamber? Had I bollocks. I'd printed out pictures of bubble chamber tracks, read up on spark chambers, learned about emulsion tracks, but never actually seen a working cloud chamber.

This thing was cool. I had not realized how dynamic they were. They'd put in a radioactive source, of course -- maybe they had sent off for a radioactive speck back in the day -- but there were also cosmic rays and other natural sources of radiation leaving their marks. There were tracks appearing constantly, and then fading away; it was hypnotic to watch. This video shows exactly what I mean:

The woman giving the demo said that some of the tracks were muons. Dude! MUONS! I was seeing MUON TRACKS! And then I got embarassed 'cos I couldn't remember whether beta radiation was photons or helium nuclei (neither: they're electrons or positrons; gamma radiation is made of high-energy photons) and felt insecure about the whole prospect of being in the same room as all these scientists since I obviously needed a remedial course in shoelace tying if I couldn't remember what beta radiation was.

(My brain, it does not always make sense.)

And so but it turns out you can find a lot of instructions on how to build a cloud chamber (the Starbucks chamber gets big props for being a cool hack), forums, and even a company that will build one for you (starting price for museum-grade model: $48,000). Now I just gotta track down some dry ice.

(1) "my family"? "my wife and I and kids"? I think (by which I mean "worry", by which I mean "obsess") about these things.

(2) You damn betcha it's capitalized.

Last night was a rare semi-clear night (this month has been awful, grumble), so I was excited to see the Moon, Jupiter and Venus on my walk home from the bus stop after $WORK; it was kinda cloudy, but not completely, and anyhow the crecent moon was awful pretty through the haze. When I got home I asked my oldest son if he wanted to go out w/the telescope after supper. He was enthusiastic, so I put the 4.5" reflector outside to cool while we ate.

Forty minutes was enough to bring in more threatening clouds, but we could still see the three of them. I set up the scope and had a look at Theopilus. A couple years ago my son noticed its distinct appearance, and asked what its name was. I looked it up, and have been fond of it ever since. This time, though, he couldn't pick it out...but he was still interested, so that was good.

I'd looked on Heavens-Above to see if the ISS was due to fly overhead tonight, and it was -- just before 7pm, right when we were outside. Sweet! Sure enough, we watched closely and there it was, bright as anything and moving just past the moon. But wait, wasn't it supposed to go across the moon? What the...

...and then one minute later, there was the real ISS, and it was going across the Moon (very cool!). And there was the other satellite, almost as bright, moving on a different track. As far as I could tell, both stayed the same steady brightness -- so no tumbling for our mystery satellite. We watched both 'til they passed into the Earth's shadow, then headed inside for the night.

First thing I did, of course, was pull up Heavens-Above again to see what this other satellite was. And I couldn't find anything! There was simply nothing listed anywhere near the time the ISS flew over, let alone something that was supposed to be that bright. No Iridium flares either. Stumped, I reported to my son that I had no idea what it was.

But then I realized that I'd been looking at the listings that were supposed to be brighter than 3rd magnitude, rather than the fuller list that went down to 4.5. It was possible this thing was in the fuller list, but was brighter than predicted (because the predicted angle of reflection was wrong, say). So I pulled up the full list and started looking at tracks.

Sure enough, there were a bunch that were overhead at that time. The ISS was the most obvious one, but looking at the map tracks this Delta II rocket was the one we saw, which had launched the Globalstar 26. Here's the ISS pass:

And here's the Delta pass:

The times don't match up perfectly. The Delta was predicted to reach max altitude at 18:44 and enter shadow at 18:50; the ISS was predicted to reach 10 degrees altitude at 18:51, max at 18:54 and shadow at 18:57. I didn't note the time we saw the first one, since it was right around 18:50 and I thought it was the ISS.

I told my son about all this and -- being the son of a geek -- he thought it was pretty cool. :-)

Also -- and this is completely unrelated -- how did I not know about M-a in Emacs? Ordinarily it's "backward-sentence", but in programming modes, it moves to the beginning of non-whitespace on a line. ZOMG.

Today I gave some impromptu training at $WORK; the approximate topic was "Saving State in Linux". I've been meaning to do something like this for a while, but it was prompted by a conversation yesterday with one of the researchers who kept losing work state when shit happened -- Emacs window arrangements, SSH sessions to other machines, and so on. I found myself mentioning things like tmux, workgroups, and Emacs daemon mode...and after a while, I said "Let me talk to you about this tomorrow."

So today I found half an hour, decided to mention this to everyone in the lab, crowded into a meeting room, set up my laptop and the projector, and away I went. For a fly-by-the-seat-of-my-pants first attempt, I think it went relatively well. Best idea: asking people for questions. It hadn't occurred to me that people would want to know more basic stuff like "How do I split windows in Emacs?". I'm never sure what people already know, so I don't want to bore them...

Next time:

• Be more organized
• More notice for people so they can prepare questions
• Let the vi people know if it's going to be Emacs-heavy

In other news: finally converted my SVN repos to Git yesterday in a fit of pique. The big three -- my org-mode stuff, and the two Cfengine repos (Cf2 and -3) -- are already in use, as in that's where I'm checking stuff into. The rest (Nagios configs, for example) are being done as I get to them. It's really, really wonderful.

Family: holy house o' plague, Batman!

• Last Friday my older son pukes, stays home from school
• Last weekend my younger son comes down with bronchitis, demands compensation for being forced to take antibiotics
• My wife comes down with some kinda virus that makes her nauseous for a week
• I'm still fighting off a sinus infection
• Wednesday, my older son pukes again apropos of what we don't know

Gah. We're getting the house boiled next week. (Update, March 13: too late; I puked on Friday night and spent Saturday moaning in bed; my wife did the same thing Saturday night/Sunday. FUCK.)

Also? There's a Planet Lisp. Who knew?