November has hit. It's raining hard in YVR -- El Nino year -- and we are getting regular warnings from the weather office about dressing up warmly. I've taken this to heart by wearing two hoodies at a time, because apparently I'm getting old and need some soup.
Speaking of which, it has been 18 months since I've brewed. I've got some hops grown in my in-law's garden that I need to use up, and I'm thinking of doing an extract version of my Dubbel recipe. Not sure when I'll brew it -- Saturdays are getting filled up in advance at an alarming rate, by which I mean more than 0 -- but it would be nice to have some nice homebrew for the winter.
Random thing: My youngest son had soccer practice yesterday. He likes it a lot. Last night there were fireworks going off near the park where his team was playing. During a break, he tried to tell some of the other kids that the people setting off the fireworks weren't allowed to do that...but they were too busy clobbering each other to pay attention. He tried to tell them again, but the fistfight was too interesteding. It reminded me of my own childhood, not being able to talk to other kids -- not having anything in common.. Then I thought about how we see what we want to see in the world, and maybe that was why I saw this. Then I thought about how this is exactly the sort of thing I think, so of course I think it. That way lies recursion.
Couch: delayed. Will: still on track. Oncall: NOT GONNA JINX NOTHING.
There are days when I can't see the end of the TODO list. Patch the laptops; add SSL to Postfix, to try and keep the NSA away for one more year; take down the recycling, and the composting, and the garbage; take the empties back to Safeway; pick up the caulking for the bathtup, and a switchplate to replace the one the kids broke. Oh, and we're going to the in-laws too (they live just across town and they're wonderful, generous, loving people, so that is in no way a complaint). I wanted to brew, so I have to pick out a recipe and then some ingredients. I want to go out with the scope, so I need to pick some targets. And it's Saturday, so I'm taking the kids out for the morning (again, not a complaint).
In the middle of that, having to play with Eli, my younger son, can seem like a chore, like something that's taking away from The Things that Have to be Done. He senses that, too; he's five but he's no idiot. I'm mad, but I've got a secret weapon: while we were out this morning buying donuts (one each; their money, their afternoon treat) I bought myself a 10X magnifying glass (a loupe, basically). Now's the time to take it out. Sitting down on the front step, I see a sowbug exoskeleton. I pick it up, put it under the loupe and look. "Oh, wow."
"What is it?" He's cautious, but interested; he's pretty much over a phase where he was Not Interested in the things I wanted to show him. That was probably my fault as much as his. I wanted to show him things, all the wonders of the universe, and at some point he had enough, and he didn't hesitate to tell me so. We're both stubborn, a fact my parents grinned at knowingly when they were visiting a few weeks ago. At the time, Eli and I spent a certain amount of time glaring at each other. But now, he's watching me squat ungracefully on the sidewalk, trying to see what I see.
"It's a dead sowbug. Have a look."
He does, and it's cool, it's interesting. We look for other things: snails, ants, crystal rocks (defined as anything shiny). Eventually his brother comes home and they run upstairs to play together.
Later on I bribe him into helping me take back the empties to Safeway by promising him the deposit. On our way back we find a caterpillar on the sidewalk. He's yellow and black and fuzzy and looks like a toothbrush. We take him home, picking leaves from random trees along the way so he'll have something to east. One of the leaves we pick has the ladybug equivalent of a cocoon on it, and we decide that we should watch for it to hatch.
An hour later, I look at the leaves and realize the ladybug has hatched -- and there it is, bright yellow and looking like a tiny lemon. I call Eli over and we stare at it. There are no spots on it; they come in later, I explain, and Eli says "Huh!" in that way he has when he's actually interested in something. An hour after that we check again. The ladybug has spread its wings to dry, and its shell is showing a couple of green spots.
Despite the desperation, shit got done. The recycling went down. The cat did not vomit on the floor. I got an SSL certificate for the mail server. I watched a movie with my wife and kids. I picked up the ingredients for my next homebrew, a summer ale, and it's mashing now; it'll sit overnight, waiting for me to finish it tomorrow.
This was a busy-ass day, yo. Got up at 5.30am to make beer, only to find out that a server at work had gone down and its ILOM no longer works. A few hours later, I've convinced everyone that a trip to UBC would be lovely; I go in, reboot the server and we drive back. It's no 6000 km from Calgary to California and back like my brother does, but for us that's a long drive.
And then it's time to make beer, because I've left it mashing overnight. Boil, chill, sanitize, pitch, lug, clean, and we're one batch closer to 50 (50!). A call to my parents (oh yeah: Dad, you guys can totally stay here in May) and then its supper. And then it's time for astronomizing. Computers, beer and astronomy: this day had it all.
So tonight's run was mostly about trying out the manual setting circles. I don't have a tablet or smart phone to run something like Stellarium on, so for now I'm printing out a spreadsheet with a three-hour timeline, 15 minute intervals, of whatever Messiers are above the horizon.
How did it work? Well, first I zeroed the azimuth on Kochab (Beta UM) rather than Polaris, and kept wondering why the hell the azimuth was off on everything. I realized my mistake, set things right, and tried again. And...it worked well, when I could recognize things.
M42, for example, was easy. (It was the first thing I found by dialing everything in, and when I took a look there was a satellite crossing the FOV. Neat!) But then, it's big, easy to recognize, and i've seen it before. Ditto M45. M1? Not so much; I haven't seen it before, and I didn't have a map ready to look at. M35, surprisingly, was hard to find; M34 was relatively easy, and M36 was found mainly because I knew what to look for in the finder.
This should not be surprising. I've been tracking down objects by starhopping for a while now, so why I thought it would be easier now that I could dial stuff in is beyond me. It's my first time, and the positions were calculated for Vancouver, not New West (though I'm curious how much diff that actually makes).
There were some other things I looked for, though.
M51: found the right location via starhopping, and confirmed on my chart. But could I see it? Could I bollocks.
M50: Pretty sure I did see this; looking at sketches, they seem pretty similar to what I saw. (And that's another thing: it really does feel too easy, like I haven't earned it, and I can't be sure I've really found it.) Oh, and saw Pakan 3, an asterism shaped like a 3/E/M/W, nearby.
M65/M66: Maybe M65; found the location via starhopping and confirmed the position in my chart. Seems like I had the barest hint of M65 visible.
At the request of my kids:
Jupiter: Three bands; not as steady as I thought it would be.
Pleidies: Very nice, but I do wis I had a wider field lens.
M36: Eli suggested a star cluster, aso I went with this. Lovely X shape.
Betelgeuse: Nice colour. Almost forgot about this, and had to look at it through trees before I went home.
It's Xmas vacation, and that means it's time to brew. Mash was at 70 C, which was a nice even 5 C drop in the strike water temp. 7.5 gallons went in, and 6 gallons of wort came out. It was not raining out, despite the title, so I brewed outside:
My kids came out to watch; the youngest stayed to help.
The keggle was converted by my father-in-law, a retired millwright; he wrote the year (2009) and his initial using an angle grinder.
The gravity was 1.050, so I got decent efficiency for a change -- not like last time.
On a whim, Eli decided to make the 60 minute hop addition a FWH instead:
Ah, the aluminum dipstick. No homebrewer should be without one.
Eli demonstrated his command of Le Parkour...
and The Slide:
"Hey, it's Old Man Brown, sittin' on his porch eatin' soup an' making moonshine again!"
Eventually it was time to pitch the yeast. We took turns. I took this one of Eli...
...and he took this one of me:
Isn't it beautiful? Oh, and the OG was 1.062.
Yesterday I spent the morning in an unaccustomed role: that of the older, wiser homebrewer. My friends Ross and Wes were doing their first batch of homebrew, along with Kevin and Will, and wanted me along to help out. I was happy to oblige. My kids came along to play with their kids, and my wife to make homemade candy corn with the other wives (seriously, it's like a farmhouse social), and Wes brought beer for us all to drink, picked up from Central City Brewing the day before.
It was a damn good time and a shockingly warm and half-sunny day. The guys went all-grain for their first batch, and were using my homegrown hops for bittering and flavour. Ross rocked the clipboard while Wes and Kevin did the lifting:
A little bit of spilled wort on my shoes later ("Who the hell wears good shoes to a homebrew session?"), everything was in the fermenter and we were cleaned up. Gravity was a little lower than expected -- 1.038 -- but hey, we can clean up efficiency later, right? And word on the street is that the airlock was bubbling away later that night.
Awesome time. I might have to take this up professionally.
Got to brew yesterday, and my oldest son helped out:
Gonna be an amazingly bitter, session hefeweizen.
Random notes:
It's tedious setting Emacs' window layout just the way you like it -- splitting windows, adjusting their size, switching to the right buffers, etc. And even when it is set, it won't stay that way for long. On top of that, you can't save your window-configurations to disk, so you have to start over from scratch every time you restart Emacs.
There are solutions out there to parts of the problem -- elscreen, revive.el, window-configuration-to-register, etc. -- but none provide a complete solution. Workgroups does.
It is so choice. If you have the means, I highly recommend picking one up.
Samantha Wright is answering biology questions again on her Slashdot journal. She's smart, well-spoken (-written) and generous with her knowledge; it's well worth taking a look at all her entries.
A while back I brewed a double IPA that came in around 9% ABV. I used almost a pound (!) of hops, most of them Falconer's Flight. It turned out pretty well...but I made the mistake of bottling some of the sludge at the bottom with all the hop gunk. I thought it would settle out, but hop bits are surprisingly floaty, and I think I'm going to end up pouring it through a sieve of some kind. (Incidentally, I have not brewed at all this year. Yes, it's only 6 weeks in, but still.)
I am turning 40 tomorrow. (!!) I've taken the day off work, my wife and I are dropping off the kids at my in-laws' 'til tomorrow, and we're heading out for...well, an afternoon on the town :-), heading to Saint Augustine's to see what they've got on tap.
Last year (hah! last year!), by which I mean Xmas 2011, just two weeks ago, I did all my updates and disruptive work in the week BEFORE Xmas, rather than after. It's one of the perks of working for a university that I get a week off between Xmas and the new year, and I decided to actually take advantage of it this year.
(I could make that paragraph better, but it's beyond me right now.)
(The other advantage, of course, is free honey from honeybee researchers.)
I allowed myself three days, but was actually done in two. That's considerably better than last year, and in large part that's because I learned a (not the) right way to uninstall/reinstall proprietary ATI drivers. Unlike last year, in fact, it was practically painless.
This might explain why it was not terribly surprising to come back to a problem related to the upgrade: a user telling me that PyMOL no longer worked. And sure enough, when I tried running it over SSH, it crashed:
$ pymol
/usr/bin/pymol: line 2: 17723 Segmentation fault /usr/bin/python //usr/lib64/python2.6/site-packages/pymol/__init__.py "$@"
...which wasn't exactly what she was seeing, but I was pretty sure that it was only the tip of the iceberg.
A backtrace (caught by the wonderful catchsegv, which I only just
found out about) showed that it was failing at XF86DRIQueryVersion
,
which turns up in approximately 38% of all web pages. They're all
related to problems w/the proprietary ATI driver, how it eats ponies,
and how everything was sunshine and lollipops once they rolled back to
the Mesa-provided copy of libGL.
We are running the proprietary ATI driver -- we need the 3D performance -- so this made sense. And after last year's fiasco I was quite prepared to believe that ATI has a nasty appetite for BBQ. But much searching showed that before the Xmas upgrade, everyone'd been using the ATI-supplied libGL w/, presumably, no problems. I decided to prove it by reinstalling Mesa on a test machine. Yep, now it works fine. ATI hates the world!
...but I'd forgotten that I was running this over SSH. With X forwarding. And this made a difference. The Truth Hammer smacked me when I tried PyMOL on another workstation, from an actual sit-at-the-keyboard-and-use-the-coffee-holder X session, and it worked fine. I SSH'd to the original user's machine, and that worked fine.
I checked the version of libGL on my machine, and sure enough it was
different: 7.7.1 versus 7.8.2. My suspicion is that either the
XF86DRIQueryVersion
routine has changed enough that this causes
problems, or there was some other difference (32-bit vs 64bit? could
be...) between my machine and theirs (mine runs a different distro, so
there's lots of chances for interesting differences).
I simply did not expect there to be any problem debugging X programs over SSH; probably naive, but what the hell. Now I know.
Oh, and the user's problems? Wack PYTHONHOME
. Unset that and all
is well.
Happy new year, everyone!
Growing up was wall-to-wall excitement, but I don't recall
Another who could understand at all...
-- Sloan
Monday: day two of tutorials. I found Beth Lynn in the lobby and congratulated her on being very close to winning her bet; she's a great deal closer than I would have guessed. She convinced me to show up at the Fedora 14 BoF tomorrow.
First tutorial was "NASes for the Masses" with Lee Damon, which was all about how to do cheap NASes that are "mostly reliable" -- which can be damn good if your requirements are lower, or your budget smaller. You can build a multi-TB RAID array for about $8000 these days, which is not that bad at all. He figures these will top out at around 100 users...200-300 users and you want to spend the money on better stuff.
The tutorial was good, and a lot of it was stuff I'd have liked to know about five years ago when I had no budget. (Of course, the disk prices weren't nearly so good back then...) At the moment I've got a good-ish budget -- though, like Damon, Oracle's ending of their education discount has definitely cut off a preferred supplier -- so it's not immediately relevant for me.
QOTD:
Damon: People load up their file servers with too much. Why would you put MSSQL on your file server?
Me: NFS over SQL.
Matt: I think I'm going to be sick.
Damon also told us about his experience with Linux as an NFS server: two identical machines, two identical jobs run, but one ran with the data mounted from Linux and the other with the data mounted from FreeBSD. The FreeBSD server gave a 40% speed increase. "I will never use Linux as an NFS server again."
Oh, and a suggestion from the audience: smallnetbuilder.com for benchmarks and reviews of small NASes. Must check it out.
During the break I talked to someone from a movie studio who talked about the legal hurdles he had to jump in his work. F'r example: waiting eight weeks to get legal approval to host a local copy of a CSS file (with an open-source license) that added mouseover effects, as opposed to just referring to the source on its original host.
Or getting approval for showing 4 seconds of one of their movies in a presentation he made. Legal came back with questions: "How big will the screen be? How many people will be there? What quality will you be showing it at?" "It's a conference! There's going to be a big screen! Lots of people! Why?" "Oh, so it's not going to be 20 people huddled around a laptop? Why didn't you say so?" Copyright concerns? No: they wanted to make sure that the clip would be shown at a suitably high quality, showing off their film to the best effect. "I could get in a lot of trouble for showing a clip at YouTube quality," he said.
The afternoon was "Recovering from Linux Hard Drive Disasters" with Ted T'so, and this was pretty amazing. He covered a lot of material, starting with how filesystems worked and ending with deep juju using debugfs. If you ever get the chance to take this course, I highly recommend it. It is choice.
Bits:
ReiserFS: designed to be very, very good at handling lots of little files, because of Reiser's belief that the line between databases and filesystems should be erased (or at least a lot thinner than it is). "Thus, ReiserFS is the perfect filesystem if you want to store a Windows registry."
Fsck for ReiserFS works pretty well most of the time; it scans the partition looking for btree nodes (is that the right term?) (ReiserFS uses btrees throughout the filesytem) and then reconstructs the btree (ie, your filesystem) with whatever it finds. Where that falls down is if you've got VM images which themselves have ReiserFS filesystems...everything gets glommed together and it is a big, big mess.
BtrFS and ZFS both very cool, and nearly feature-identical though they take very different paths to get there. Both complex enough that you almost can't think of them as a filesystem, but need to think of them in software engineering terms.
ZFS was the cure for the "filesystems are done" syndrome. But it took many, many years of hard work to get it fast and stable. BtrFS is coming up from behind, and still struggles with slow reads and slowness in an aged FS.
Copy-on-write FS like ZFS and BtrFS struggle with aged filesystems and fragmentation; benchmarking should be done on aged FS to get an accurate idea of how it'll work for you.
Live demos with debugfs: Wow.
I got to ask him about fsync() O_PONIES; he basically said if you run bleeding edge distros on your laptop with closed-source graphics drivers, don't come crying to him when you lose data. (He said it much, much nicer than that.) This happens because ext4 assumes a stable system -- one that's not crashing every few minutes -- and so it can optimize for speed (which means, say, delaying sync()s for a bit). If you are running bleeding edge stuff, then you need to optimize for conservative approaches to data preservation and you lose speed. (That's an awkward sentence, I realize.)
I also got to ask him about RAID partitions for databases. At $WORK we've got a 3TB disk array that I made into one partition, slapped ext3 on, and put MySQL there. One of the things he mentioned during his tutorial made me wonder if that was necessary, so I asked him what the advantages/disadvantages were.
Answer: it's a tradeoff, and it depends on what you want to do. DB vendors benchmark on raw devices because it gets a lot of kernel stuff out of the way (volume management, filesystems). And if you've got a SAN where you can a) say "Gimme a 2.25TB LUN" without problems, and b) expand it on the fly because you bought an expensive SAN (is there any other kind?), then you've got both speed and flexibility.
OTOH, maybe you've got a direct-attached array like us and you can't just tell the array to double the LUN size. So what you do is hand the raw device to LVM and let it take care of resizing and such -- maybe with a filesystem, maybe not. You get flexibility, but you have to give up a bit of speed because of the extra layers (vol mgt, filesystem).
Or maybe you just say "Screw it" like we have, and put a partition and filesystem on like any other disk. It's simple, it's quick, it's obvious that there's something important there, and it works if you don't really need the flexibility. (We don't; we fill up 3TB and we're going to need something new anyhow.)
And that was that. I called home and talked to the wife and kids, grabbed a bite to eat, then headed to the OpenDNS BoF. David Ulevitch did a live demo of how anycast works for them, taking down one of their servers to show the routing tables adjust. (If your DNS lookup took an extra few seconds in Amsterdam, that's why.) It was a little unsettling to see the log of queries flash across the screen, but it was quick and I didn't see anything too interesting.
After that, it was off to the Gordon Biersch pub just down the street. The food was good, the beer was free (though the Marzen tasted different than at the Fairmont...weird), and the conversation was good. Matt and Claudio tried to set me straight on US voter registration (that is, registering as a Democrat/Republican/Independent); I think I understand now, but it still seems very strange to me.
My in-laws got us a family membership at Science World for Xmas last year. Yesterday I got to take my 4 year-old (how should that be hyphenated?) son for the morning. It was his third trip and my second.
We headed right for the Eureka room, which is aimed at the young 'uns, and he ran around showing me everything. "Daddy, here's a big tube where you can shoot out parachutes! And this air gun shoots balls up into the water!" We found out that you could stuff three plastic balls into the air gun at once (poom poom poom!).
Oh, and when we got home he wanted to do an experiment. He got some pennies and put them in a jar with water, to leave them for a few days and see if they would dissolve. I had a maple syrup candy in my pocket (no idea where I got it), so I threw that in too. The candy has dissolved and made the water brown, so I'm curious to see what he makes of that.
Science World is just incredible. I long to go see the grownup stuff, but even the kid stuff is enormously fun and moderately educational (though that's not my son's priority right now) (dang kids). I grew up in small towns as a kid, so trips to museums like this were rare, enormous fun. (And I never did get to go to Science North...) It's amazing to me that this stuff is right here, only a half hour away by transit. I'm still a little shocked we don't go, like, every weekend.
In other news, I've got a starter going for a batch of beer next weekend. It's a Belgian yeast, harvested from my January batch. The yeast was washed following these instructions, and the starter took off in about 18 hours. It seems to be doing quite nicely; I'll probably stick it in the fridge on Wednesday or so and cold-crash it.
The ingredients are pretty much whatever I have around the house: the last of my Gambrinus ESB, some biscuit and wheat malt, a bit of roasted barley, and the hops are Centennial, Goldings and Mt Hood. My father-in-law would call this a "ministrone" -- Italian not just for that kind of soup, but for "dog's breakfast" or "big ol' mixup". (I kind of like the idea of an Italian sounding like he's from Missouri.)
Still looking for a name; suggestions on a postcard, please. Sponsorship options are available. :-)
After that's in the bag, it's time to head back to Dan's for a shopping trip. This time, I think it'll be a 50-lb bag of plain ol' pale malt, and I'll see what difference that makes.
Monday night:
Los Angeles beckons the teenagers to come to her on buses
Los Angeles loves love
It is 5am, and you are listening to Los Angeles.
"Screewriter's Blues", Soul Coughing
Monday I met up with Donny and Ludmilla for supper...and who's there but Tobi Oetiker! Another chance for geekish hero worship, hurrah!
After thanking him for MRTG and RRDTool, I asked him what had happened to the call centre he had spent all that time debugging. He said that it was kind of in limbo: the troublesome app had been replaced by a web-based app and was slowly being rolled out...but since it didn't do everything the old app did the old one was being kept around and people were reluctant to upgrade. But because the old app was on the way out, no one wanted to spend money tracking down the problems with it. I have to say, I expect more neatly wrapped-up story endings from the people I admire. :-)
Also along were Walter and Kyle, two sysadmins from Boston's TERC. This was handy, because Kyle had lived once in Baltimore and was able to take us to DuClaw's brewpub, which was not too far from the hotel. The sampler included about 10 different beers:
Despite being from the German-speaking part of Switzerland, Tobi was not interested in drinking the beer, but appeared to be fascinated by the interest we took in it. Crazy Swiss, what are you gonna do?
Tobi also talked about coming to love JQuery and qooxdoo. Everyone kept asking him to repeat that name, and finally he wrote it down while we guessed how it was spelled. None of us were right, because we'd all been guessing crazy Dutch-German variations.
Kyle and Walter talked about their setup a bit. They're in kind of the same boat I am in that (being at an educational institute) funding is erratic yet the results (websites, curricula, etc) need to be around forever. Thus, they still have an NT4 web server which was only last month migrated to a VM. (Walter dulled the pain by asking the bartender to make him something sweet with rum. The procedure had to be repeated once, but then he was good to go.)
After that, we headed off to the James Joyce pub where OpenDNS was engaged in a COMPLETELY FUTILE attempt to gain my good will by buying the entire bar drinks all night. (Futile, do you hear?)
I didn't get to meet the OpenDNS folks, but that didn't stop Ludmilla from pasting OpenDNS stickers on everyone's shirt. And I did get to talk to another Norwegian sysadmin.
So he works for a Norwegian newspaper, whose website half of Norway starts their morning. (Apparently he went to a talk (previous LISA?) where Facebook was talking about their traffic levels; Facebook's traffic was less than their own and they used 1/5th the number of servers Facebook did.) They were using Squid in front of their webservers, but were looking for something to do better. Commercial/proprietary options didn't measure up. What to do?
Well, like any good Norwegian they decided to bring in a fellow Scandinavian. After determining that Linus Torvalds was not interested (not entirely sure how serious that part was), they asked Poul Henning-Kamp if he was interested; he wasn't. "I'm a kernel guy with 20 years of experience doing kernels," he said; "I'm just not interested in doing application work."
But then he comes back two weeks later and says, now that he's had some time to think about it, he is interested in the idea of a caching app that exploits the underlying OS to the hilt. N months later, Varnish was ready to go.
They roll it out at a big news conference, with The Register and others attending. Boss gives a speech while they watch the graph of request latency scroll across the screen; they throw the switch. The line go down from 300 ms to 30 ms and stays that way.
Also met Dan, who works for the U of Kansas Center for Remote Sensing of Ice Sheets. "I keep wanting to go down to Antarctica, but they keep not sending me there."
Full day:
I can't believe it...my youngest son, after nearly three weeks of being up four or five times each night, slept nearly all the way through without a break: he only woke up at 1am and 5:15am, which is close enough to my usual wakeup time as makes no difference. It was wonderful to have a bit of sleep.
This comes after staying up late (11pm!) on Sunday bottling the latest batch of beer, a Grapefruit Bitter recipe from the local homebrew shop. You know, it really does taste like grapefruit, and even this early I'm really looking forward to this beer.
My laptop has a broken hinge, dammit. I carry it around in my backpack without any padding, so I guess I'm lucky it's lasted this long. Fortunately the monitor still works and mostly stays upright. I've had a look at some directions on how to replace it; it looks fiddly, but spending $20 on a new set of hinges from eBay is a lot more attractive than spending $100. Of course, the other consideration is whether I can get three hours to work on it….But in the meantime, I've got it on the SkyTrain for the first time in a week; it's been hard to want to do anything but sleep lately.
Work is still busy:
I'm trying to get tinyMCE and img_assist to work with Drupal
Contacting vendors to look at backup hardware. So far we're looking at the Dell ML6010 and the Sun SL500. They're both modular, which is nice; we've got (low) tens of TB now but that'll ramp up quickly. The SL500 seems to have some weird things; according to this post, it takes up to 30 minutes to boot (!) and you can't change its IP address without a visit from the service engineer (!!). Those posts are two years old, so perhaps things have changed.
Trying to figure out what we want for backup software, too. I'm used to Bacula (which works well with the ML6010) and Amanda, but I've been working a little bit with Tivoli lately. One of the advantages of Tivoli is the ease of restoring it gives to the users…very nice. I'm reading Backup and Recovery again, trying to get a sense of what we want, and reviewing Preston's presentation at LISA06 called "Seriously, tape-only backup systems are dead". So what do we put in front of this thing? Not sure yet…
Speaking of Tivoli, it's suddenly stopped working for us: it backed up filesystems on our Thumper just fine (though we had to point it at individual ZFS filesystems, rather than telling it to just go), then stopped; it hangs on files over a certain size (somewhere around 500kb or so) and just sits there, trying to renew the connection over and over again. I've been suspecting firewall problems, but I haven't changed anything and I can't see any logged blocked packets. Weird.
Update: turned out to be an MTU problem:
I had no idea there were GigE NICs that did not support Jumbo frames. Though maybe that's just the OpenBSD driver for it. Hm.
I've been hlding off mentioning this 'til all my ducks were in a row, but at last it's settled. The job I've been working at part-time for the last six months will be my full-time job starting next Wednesday. w00t!
I've been spending my time at $job_1 making sure the documentation is complete, getting a spare workstation set up and ready to go, and dumping my brain into the sysadmin who will be helping fill in 'til a new person is hired (which might take a while).
I'm really excited about this. First off, I'll get my lunch hours back; I've been walking between the two offices (mornings at one, afternoons at the other, back to the first for the last half hour), and it'll be nice to have an hour to myself again. But the new job is exciting for me: nice big servers used for scientific computation, the chance to build an infrastructure from scratch, and some big projects. The people are friendly. The boss is nice. The place has funding for the next five years or so. It's all good. About the only thing missing is a rocket pack so I can cut down on this 90-minute commute.
And on top of all that, they're open to the idea of sending me to LISA this year. Now that would be nice…have to see if it works with the family, but I'm keeping my fingers crossed.
In other news:
The last few weeks, I've been setting up a small (5 racks) server room with the purchases that $OTHER_JOB recently made: 10 Sun X4140s, 2 — wait, 4 — X4240s, and one Thumper.
It's occupied a lot of my time, and before I lose the impulse, or fall asleep on my feet (second kid up at 4:30am for the last week or so; simultaneous discovery that at 4:30am I have a hard time getting back to sleep), I want to put down the things I learned.
ldap_cachemgr
does not like being told to connect to an IP address (via an entry in ou=Profiles) via SSL, and have the CN be a hostname instead. This took me a while to figure out.But...my first batch of homebrew beer has been bottled, and a second brew day is coming up on Saturday. And apparently I'm not the only sysadmin who brews...though I'm not nearly ready to do all-grain just yet.