Just upgraded my laptop to Debian Lenny with only minor hiccups (my own fault). Not only have I got the latest version of Iceweasel/Firefox without any GTK version nonsense, but I've got wicd working, including my Broadcom wireless and WPA2! (I never could figure out the settings to get encryption working with the various /etc/network/ files...) I'm happy…
title: I'm 37 today date: Wed Feb 11 09:53:33 PST 2009
Good god, it's the downhill run to 40. Weird.
(Oh, and you know who else's birthday it is? Alex Jones, who's two years younger than me. That's also weird.)
Tags:
"Phycicists are fun to be around. I was watching TV with one, and a commercial came on for OxyClean. The announcer's voice comes in, strong and deep, and says, What's the most powerful force in the universe? The guy I'm with starts pumping his fist and chanting, Strong nuclear force! Strong nuclear force! The announcer comes back and says, That's right, oxygen! Poor bastard looked like someone just shat in his ear."
(Conversation with a friend just now.)
Two things that didn't work:
Explanation: there's ou=Smith and ou=Jones, both of which are under ou=People,dc=example,dc=org. Smith wants to offer Jones the use of a few of his machines, which means setting up accounts for Jones and a few of his folks (cn=Alice, cn=Bob, and cn=Charlie). Obviously, these should be in ou=Jones, right? But if Smith's machines, through the wonders of pam_ldap, are set to check ou=Smith, how do Jones' folks log in?
(Digression: actually, Smith's machines right now check under ou=People — not ou=Smith,ou=People. Smith is the first one to use LDAP, so I stuck with that. I was going to change that at some point anyway, and I thought this might be a good chance to do just that.)
I thought I could try adding an alias, under ou=Smith, that'd point to cn=Alice,ou=Jones. And if I told LDAP that it was a posixAccount as well, then I could look at the account details with id and getent. But the logs showed that it just didn't work:
pam_ldap: error trying to bind as user "uid=Alice,ou=Jones,ou=People,dc=example,dc=org" (Inappropriate authentication)
Couldn't track down the error quickly, so went to plan B: stick with the current setup (machines checking ou=People) and put 'em under ou=Jones. I can always add host restrictions later on.
Explanation: Smith had a bunch of these machines at another location before getting server room space at UBC (and new servers). My access to them previously was via SSH only — there was no console access at all (sigh). Now they're at UBC, and one of 'em's gonna be my monitoring machine/second LDAP server ("The new server room: now with redundancy!") But while it was simple to turn on console redirection and choose PXE booting from the comfort of my office, I ended up borking the kickstart process and having to come back here anyway to set up the install. There's the BMC, which apparently I can access via the serial console if I so choose, but I'm still trying to figure out what that'll get me — ie, I can't find a manual in 11 seconds, so I'm putting that off for now.
Oh, and my new (work) laptop is in. Yay! It's a Dell D630, and aside from it's obscene footprint compared to my (ailing) C400, it's great. Ubuntu (Hardy for compatibility with the desktops here) is on so far, and CentOS (server work) and OpenBSD (instant firewall) aren't far behind.
Attended a talk today about the upgrade of UBC's network from supporting VLANs to supporting VRFs. Complicated but neat. I'm hoping that the presentation slides and video will make it to the IT Services website; I'll post a link if it does.
Also interesting is this external review of the IT department at UBC. It touches on some things I've been peripherally involved or interested in (funding models, culture and management); I've only skimmed it so far, but it's fascinating to read something so straightforward.
I can't believe it...my youngest son, after nearly three weeks of being up four or five times each night, slept nearly all the way through without a break: he only woke up at 1am and 5:15am, which is close enough to my usual wakeup time as makes no difference. It was wonderful to have a bit of sleep.
This comes after staying up late (11pm!) on Sunday bottling the latest batch of beer, a Grapefruit Bitter recipe from the local homebrew shop. You know, it really does taste like grapefruit, and even this early I'm really looking forward to this beer.
My laptop has a broken hinge, dammit. I carry it around in my backpack without any padding, so I guess I'm lucky it's lasted this long. Fortunately the monitor still works and mostly stays upright. I've had a look at some directions on how to replace it; it looks fiddly, but spending $20 on a new set of hinges from eBay is a lot more attractive than spending $100. Of course, the other consideration is whether I can get three hours to work on it….But in the meantime, I've got it on the SkyTrain for the first time in a week; it's been hard to want to do anything but sleep lately.
Work is still busy:
I'm trying to get tinyMCE and img_assist to work with Drupal
Contacting vendors to look at backup hardware. So far we're looking at the Dell ML6010 and the Sun SL500. They're both modular, which is nice; we've got (low) tens of TB now but that'll ramp up quickly. The SL500 seems to have some weird things; according to this post, it takes up to 30 minutes to boot (!) and you can't change its IP address without a visit from the service engineer (!!). Those posts are two years old, so perhaps things have changed.
Trying to figure out what we want for backup software, too. I'm used to Bacula (which works well with the ML6010) and Amanda, but I've been working a little bit with Tivoli lately. One of the advantages of Tivoli is the ease of restoring it gives to the users…very nice. I'm reading Backup and Recovery again, trying to get a sense of what we want, and reviewing Preston's presentation at LISA06 called "Seriously, tape-only backup systems are dead". So what do we put in front of this thing? Not sure yet…
Speaking of Tivoli, it's suddenly stopped working for us: it backed up filesystems on our Thumper just fine (though we had to point it at individual ZFS filesystems, rather than telling it to just go), then stopped; it hangs on files over a certain size (somewhere around 500kb or so) and just sits there, trying to renew the connection over and over again. I've been suspecting firewall problems, but I haven't changed anything and I can't see any logged blocked packets. Weird.
Update: turned out to be an MTU problem:
I had no idea there were GigE NICs that did not support Jumbo frames. Though maybe that's just the OpenBSD driver for it. Hm.
Another thing I'm trying to do at my new job is make/take more time for long-term planning. I've been dinged by mgt. for this in the past, and while it's not easy to hear I think there has been some validity to this. (My inclination is to concentrate hard on fixing the problems I'm faced with; giving up on something broken, even when doing so would make so much more sense and would free up resources to look for a replacement, just rankles and feels like...well, giving up.) Since the department I'm in is so new, it's even more important to pay attention to this.
Part of the problem is just recognizing that I need to make time. An hour a week to be isolated, and to (say) figure out what I'm going to need to do for the next month, is a habit I'm very conciously trying to adopt.
But another problem is how to keep track of all this. What I've done so far:
I'm a huge fan of Tom Limoncelli's Time Management for System Administrators, and his Cycle system has served me well. I've become a big fan of a paper organizer, so that's how I keep track of things. But it works est as a way of tracking day-to-day stuff; it's not so good at tracking a project that takes weeks, or months, or years.
I've read GTD, and that seems like a good system — but it's very different from The Cycle. I don't want to give up the Cycle, I want to graft on to it. And I'm not sure how well I can do that w/GTD.
I've tried org-mode in Emacs. I'm pretty happy with this, and in fact I switched to it for a while when I first started at this job back in July. It worked well for tracking day-to-day stuff, but I missed the flexibility and ubiquitousness of paper.
So where does that leave me? ATM, (paper planner Cycle) attempting
some longer-term project tracking w/org-mode. I figure the TODO bits
from org-mode will fit well with the planner, and the flexibility of
Emacs and org-mode (different from paper...oh, how I wish I could grep
paper) will work well for projects...the records for which should,
ideally, be suitable for pasting into wiki-based documentation.
If anyone has any suggestions, please let me know. If I make it to LISA this year, I'll be looking for a BOF about this. (Or maybe I'll just tackle Tom Limoncelli to the ground and holler "I love you, man!" a la "Say Anything".)
Moving on:
I really like TrueType font support in Emacs 23, and ttf-inconsolata in particular. Thanks to Emacs-fu for both suggestions.
I and a co-worker picked up the servers that had, for the last two years or so, been racked at BC Women's hospital (of all places...my sons were both born there). We both had the same reaction when we saw them on a cart, ready to be loaded into our truck: "They're so small!" Seven little 1U servers plus one disk array...you start to think of them as larger-than-life when you're not looking at them all the time, and it's easy to forget just how small they are.
Some interesting discussion on the Cfengine mailing list about how Cfengine should handle packages.
And now it is time for bed.
I knew I didn't like Vaio's very much, but I had no idea they were so awful — to the point of requiring hacking on your goddamn BIOS to enable VMX.
The flash demo for Dell's ML6000 tape library boasts that it's "completely self-aware". Not sure I want SkyNet running my backups…
O'Reilly has an upcoming webcast on -- deep breath -- "Advanced Twitter for Business". (At least they didn't call it a webinar. When I told my wife about this, she said "So...you and O'Reilly break up yet?"
Obviously not, because I've just ordered Backup and Recovery and Linux Clusters with Oscar, Rocks, OpenMosix and MPI. I had purchased B&R at my last job, but this is for me.
And did I mention the dream I had a while back about a Sun laptop that looked like an X4200 server folded in half? In the dream it ran nearly perfectly, except when you tried to go to a web page with flash; then it would crash, and a movie of Matt Stone would play, apologizing on behalf of Jonathan Schwartz and everyone else at Sun.
I'm playing with the CVS version of Emacs after reading about some of the new features in what will become Emacs 23. It's nice, but the daemon mode isn't quite multi-tty — you can run Emacs server, detached from any TTY, but if you try connecting to it with multiple emacsclient instances, the first one is where all the TTY action goes. Not sure what I'm missing.
Okay, so the other thing I was going to do was blog regularly. And now it's three days later.
But I've been meaning to mention another aspect of the new job as well. When, previous to working here, I'd thought about what I'd like my next job to be like, it was pretty consistent:
The last point needs a bit of expansion. See, my first job in IT was on the helpdesk of a small ISP. There were three of us on helpdesk, one webmaster, one sysadmin, one database guy, one secretary and one manager; I got some mentoring from the sysadmin (who split his time betwen us and a sister company), but not lots. My second was at a startup company; the guy who hired me was a good mentor, and then after a while after he left I got to hire a junior and be a mentor to him. The job I just left was pretty much just me, though I'm lucky enough to have other people I could talk to; UBC's a big place, but I was in a small department.
So my next job was going to be bigger (as in a bigger installation — maybe a whole data centre, even) and have more people — because I really, really wanted to hang out with my peers and learn from them. I envied the people I'd met at LISA in 2006 who were part of a team, who had people to teach and people to learn from.
Well, at this job it's...just me. Sort of; the folks I've been working with for the last six months (one lab out of the five that make up the centre) are pretty technical. They know way more about Java and MySQL and web development and how the latest CPUs from Intel compare with AMD than I do. But I'm the sysadmin. There might be another in the future, but there isn't now.
But! But, there are two sysadmins on the floor above me who work in another department. For various reasons, we're going to be working closely for the forseeable future. On Friday, I went up to talk with them about how that was going to work out.
They knew stuff I didn't know -- no surprise there -- but it turned out I could show them a trick or two as well. We swapped war stories, discussed our very different backgrounds (saved for another entry), and just shot the shit. It was wonderful.
It's weird, because I'm an introvert, and not very socially apt. (Or ept. As in "opposite of inept".) But it's really, really nice to get together with people who like being a sysadmin the way I do.
(This entry brought to you by the number i, the letter Ve, and my youngest son's 90-minute nap.)
Now that I've officially started my new job, I"m trying to develop/reinforce a couple of good habits.
One of them is sitting down and planning out my day. I started doing this after I got a copy of Time Management for System Administrators (the link throws Tom a few shekels), which saved my sanity. (If you haven't bought that book yet, do it now. It's that good.) However, I haven't been as diligent in going through the motions every day like I should. Today was a good chance to get back into the swing of things…especially since there were about 50 emails waiting for me when I got here! It was reassuring to see the list of completed tasks grow; it's very satisfying to cross something out in my planner.
The second is Daily Doc ("Cute names a specialty" — Aardvark Consultants). There has been a ton of stuff in the last little while that I've been wanting to document, but The One True Source (a wiki) has been in a state of flux until, well, today. I'm going to try my best to take 15 minutes at the end of every day and just write something down, or improve something that's already there.
There are other things I want to accomplish to, but that'll do for today. Gotta give you some reason to come back tomorrow…
Tags: reading
I'm listening to Obama's inauguration while in a server room, during my last day at $job_1, waiting for a server to boot up.
Tags: politics
I've been hlding off mentioning this 'til all my ducks were in a row, but at last it's settled. The job I've been working at part-time for the last six months will be my full-time job starting next Wednesday. w00t!
I've been spending my time at $job_1 making sure the documentation is complete, getting a spare workstation set up and ready to go, and dumping my brain into the sysadmin who will be helping fill in 'til a new person is hired (which might take a while).
I'm really excited about this. First off, I'll get my lunch hours back; I've been walking between the two offices (mornings at one, afternoons at the other, back to the first for the last half hour), and it'll be nice to have an hour to myself again. But the new job is exciting for me: nice big servers used for scientific computation, the chance to build an infrastructure from scratch, and some big projects. The people are friendly. The boss is nice. The place has funding for the next five years or so. It's all good. About the only thing missing is a rocket pack so I can cut down on this 90-minute commute.
And on top of all that, they're open to the idea of sending me to LISA this year. Now that would be nice…have to see if it works with the family, but I'm keeping my fingers crossed.
In other news:
A few quick notes about building Fedora Directory Server RPMs for CentOS:
$instance_dir points to /etc/dirsrv, not /etc/fedora-ds.(Partly a memo to myself, and partly to help anyone in the same boat; edits have been disabled in the FDS wiki, so I can't add this right now.)
Tags: packagemanagement linux ldap
Tuesday: youngest son (8 months old) up at 5:30am teething.
Wednesday: youngest son up at 5:15am teething.
Thursday: youngest son up at 5:30am teething. I'm so tired I go to bed at 8:30pm and fall asleep immediately.
Friday: youngest son up at 4:45am teething. At 5:45am he goes back to sleep. At 6am my phone tells me the DNS server at work is down; I can't raise it. I restore backed up zone files to a spare Xen instance (hurrah!), give it the DNS server's IP address and head into work. I restart the machine and shut down the Xen instance; can't figure out why the machine shut down in the first place. Then I discover a replication problem between two of our LDAP servers which is resulting in random bounced email for a newly created account.
I want to go home now. But there's a Very Important Meeting(tm) at 1pm, and I can't leave before then.
<headdesk>
Tags: geekdad
The Internet Storm Center writes about a new variant on malware that messes with your DNS: it installs a rogue DHCP server.
While not too sophisticated, the whole attack is very interesting. First, it's about a race between the rogue DHCP server and the legitimate one. Second, once a machine has been poisoned it is impossible to detect how it actually got poisoned in the first place - you will have to analyze network traffic to see the MAC address of thoese DHCP Offer packets to find out where the infected machine actually is.
In other news...all $job_2's new machines are set up and running. Kickstart is very nice…I really wish Debian had something similar; FAI is lovely, but Kickstart has the lovely feature of taking a hand-done installation you've just finished and turning that into a config file for a hands-off version. That saves a huge amount of time.
Next up: turn nscd back on (forgot I'd left it off for debugging LDAP
'til a simple find -exec chown was taking 10 minutes to finish);
relabel the machines with their new names; commit the documentation
I've been piecing together on my laptop; open up to others in the
group; look at either moving the LDAP server over to the server room,
or setting up a slave over there.
Tags: installation linux ldap
From a list of known issues with installation of Office 2008 for Mac. Number one:
Office 2008 updates cannot be installed if the Microsoft Office 2008 folder was moved, renamed, or modified Office Installer installs Microsoft Office 2008 for Mac in the Applications folder. If you move the Microsoft Office 2008 folder to another location on your computer, or if you rename or modify any of the files in the Microsoft Office 2008 folder, you cannot install product updates. To correct this issue so that you can install product updates, drag the Microsoft Office 2008 folder to the Trash, and then reinstall Office 2008 from your original installation disk.
Ah, hard-coded paths.
I can't download the volume license version of Office 2008 for Mac by using Safari Cause: Downloading the volume license version of Microsoft Office 2008 for Mac is unsuccessful when you use the Safari browser. Solution: We recommend that you use the latest version of Mozilla Firefox Web browser ( MozillaClick this link to open a browser window.http://www.mozilla.com) to download the volume license versions of the Microsoft Office 2008 for Mac suite or stand-alone applications.
That said, it turns out you don't need a license key for a volume copy of Mac Office 2008.
And now you know the rest of the story.
Tags: rant
I just spent the weekend (well, like an hour a day...kids, life, you know how it is) trying to track down why a bunch of new CentOS 5.2 installs at $job_2 couldn't pipe:
$ ls foo foo $ ls | grep foo $ echo $? 141
(Actually, I didn't think to look at the error code 'til someone else pointed it out…141 turns out to be SIGIPE) In the end, it would have been quicker if I'd simply searched for the first thing I saw when logging in:
-bash: [: =: unary operator expected -bash: [: -le: unary operator expected
This was particularly aggravating to track down because not every machine was doing this, and no matter what I thought to look at (/etc contents, /tmp permissions (those have a habit of going wonky on me for some reason), SELinux) I couldn't figure out what was different.
Turned out to be an upstream bug in nss_ldap. (The Bugzilla entry makes for some interesting reading, to be sure…) And I didn't see it on each machine because I hadn't upgraded after installation on all machines. (They're not yet in production, and I'm working on getting my kickstart straight.)
Man, it was gratifying to upgrade nss_ldap and see the problem go away…
title: Wow --- date: Thu Nov 20 11:47:14 PST 2008
Old news by now, but I just got pointed to Dave considered harmful by a posting to the SAGE mailing list. Kudos to Sun for the full and thoughtful explanation.
Tags:
I've since found a great deal more about multipath in Linux:
The trick was to search for "multipath" and "fstab".
Also, I contacted the installer from Sun who worked on our new machines, and he told me that the multipath driver download was lost during an upgrade of the download page; they're working on it, but in the meantime he's sent me a copy of the driver. Sweet!
Tags: linux
Just now from the window, over the sound of a stupid high-pressure washer, I heard a Canada goose fly by, honking its head off.
Tags: work
I'm in the process of setting up a bunch of new servers for $job_2. All but one are CentOS 5.2, kickstart installed and managed with cfengine. This is the third time I've goen thorugh a cfengine setup, and it always feels like starting from scratch each time. It seems -- and I'm not at all sure this is fair or accurate -- that each time I set up one of these systems, there's a lot that I've lost from the last time and have to relearn. I'm fortunate this time that I can refer to $job_1's setup to see how I did things last time, but if I didn't have that I'd be significantly further behind than I am.
I'm not sure what the solution is. Part of me thinks I should just be more aggressive about taking notes, or committing stuff to a private repository, or writing it down here more; part of me thinks that this might be a clue that cfengine is too low-level for my head. It feels like when I was trying to learn C, and couldn't believe that I had to remember all this stuff just to print something, or read a file, or connect to another machine over the Internet. By contrast, Perl (or any other scripted language) was such a relief...just print, or open, or use the Net::Telnet module, or whatever. The details are there and they are important, sometimes very much so; that doesn't mean I want to learn more metallurgy every time I need a fork. (No, I don't think that metaphor's tortured; why do you ask?)
Another thing is that I'm trying to get multipath connections working for the first time. We've got two database servers, each of which is connected via dual SAS HBAs to outboard disk arrays. (I don't think anyone else calls them "outboard", but I like the sound of it. See this hard drive? It's outboard, baby!) The arrays are from Sun and come with drivers, but the documentation is confusing: it says it's available for RHEL 5 (aka CentOS 5), but the actual download says it's only for RHEL 4.
As a temporary respite, I'm trying to see if I can get these working using Linux's own multipath daemon, and it's also confusing. The documentation for it is tough to track down, and I just don't understand the different device names: am I meant to put /dev/dm-2 in fstab, or /dev/mpath/mpath2p1? If the latter, why does the name sometimes change to the WWUID (/dev/mpath/$(cat /dev/random)) when I restart multipathd? (use_friendly_names is uncommented in the config file.) If the whole point of multipath is failover, why does this sequence:
(where /mnt is where I've got this array mounted, obvs) sometimes work, and sometimes end with "I/O error" being logged, and the filesystem being read-only? Is this the sort of thing that the Sun driver will fix? I can't find anything about this.
And I mentioned electrical problems. When we got our servers installed, the Sun guys told us they'd tripped breakers on the PDU and/or breakers in the room's electrical cabinet. Since it had a sign on it saying "100A", I figured we might be running up against power limtis -- either in the room as a whole, if my figures were 'way out, or on individual PDUs. Turns out I was probably wrong: I missed the bit on the sign that said 3-phase, which means (deep breath) we probably have 3 x 100A power available (I think).
It's more complicated than that, because some of it is in 120V, some of it is in twist-lock 220V 30A circuits, and so on. But I should've checked before emailing the faculty member who, in a year or two, will be going into this room (we're there as guests of the department) and happens to sit on the facilities committee. He had asked how we were doing, so I sent him an email -- nice, polite, and including a bit about how grateful we were for the room and the help of the local sysadmins (all of which is true).
I was under the impression that he was asking for info now, so that he could bring it up for action in a few months when we were out. Instead, two hours later when I'm swearing at multipath, in come the facilities manager and one of the sysadmins I was dealing with, looking to find out just how much power we were using anyhow. I apologized profusely, and they were very cool about it. But when the committee guy asks questions, people jump. I had not anticipated this. Welcome to University Politics 101. I emailed again and explained my mistake.
There are lots of remedial courses I could take. However, today I would most like to take "Electricity and wiring for sysadmins".
And on another note: Ack! My laptop's home partition is 93% full! How the hell did that happen?
And again: How did I not know about apt-file? This is perfect!
(Touch o' the hat to Tears For Fears and Steve Kemp; I'm moving closer every day to switching to Chronicle.)