The Life of a Sysadmin

I've been hlding off mentioning this 'til all my ducks were in a row, but at last it's settled. The job I've been working at part-time for the last six months will be my full-time job starting next Wednesday. w00t!

I've been spending my time at $job_1 making sure the documentation is complete, getting a spare workstation set up and ready to go, and dumping my brain into the sysadmin who will be helping fill in 'til a new person is hired (which might take a while).

I'm really excited about this. First off, I'll get my lunch hours back; I've been walking between the two offices (mornings at one, afternoons at the other, back to the first for the last half hour), and it'll be nice to have an hour to myself again. But the new job is exciting for me: nice big servers used for scientific computation, the chance to build an infrastructure from scratch, and some big projects. The people are friendly. The boss is nice. The place has funding for the next five years or so. It's all good. About the only thing missing is a rocket pack so I can cut down on this 90-minute commute.

And on top of all that, they're open to the idea of sending me to LISA this year. Now that would be nice…have to see if it works with the family, but I'm keeping my fingers crossed.

In other news:

• Though I'm starting to be a bit frustrated with its organization, the Postfix book has proved its worth again by helping me not only get SASL set up at $job_2, but understanding what it's doing. This is in contrast to the don't-touch-or-you'll-lose-a-finger approach I'd taken to getting this working at my current job.
• I'm growing increasingly comfortable with Fedora Directory Server, thanks in no small measure to being able to play with it in a couple of virtual machines. Man, it makes a difference to have that freedom. (And in other news, water continues to be wet.)
• Got my third batch of home-made beer (Grapefruit Bitter, from my local homebrew shop's recipe) into the fermenter on Saturday. My father-in-law is a retired millwright, and he made me an immersion chiller 'cos he's just that cool. Worked a treat; not so sure about the washed yeast. Here's hoping. My parents are coming for a visit in the summer; I figure my dad and I should be able to do a batch and bottle it before they leave, which'll give him a good souvenir.
• Enjoying the hell out of Anathem, which my wife got me for Xmas.
• Closer to migrating to Chronicle. The only thing I want is to not dump all generated HTML pages into one directory; shouldn't take too long to add. Wouldn't that be a lovely thing?

I'm listening to Obama's inauguration while in a server room, during my last day at $job_1, waiting for a server to boot up.

Now that I've officially started my new job, I"m trying to develop/reinforce a couple of good habits.

One of them is sitting down and planning out my day. I started doing this after I got a copy of Time Management for System Administrators (the link throws Tom a few shekels), which saved my sanity. (If you haven't bought that book yet, do it now. It's that good.) However, I haven't been as diligent in going through the motions every day like I should. Today was a good chance to get back into the swing of things…especially since there were about 50 emails waiting for me when I got here! It was reassuring to see the list of completed tasks grow; it's very satisfying to cross something out in my planner.

The second is Daily Doc ("Cute names a specialty" — Aardvark Consultants). There has been a ton of stuff in the last little while that I've been wanting to document, but The One True Source (a wiki) has been in a state of flux until, well, today. I'm going to try my best to take 15 minutes at the end of every day and just write something down, or improve something that's already there.

There are other things I want to accomplish to, but that'll do for today. Gotta give you some reason to come back tomorrow…

Okay, so the other thing I was going to do was blog regularly. And now it's three days later.

But I've been meaning to mention another aspect of the new job as well. When, previous to working here, I'd thought about what I'd like my next job to be like, it was pretty consistent:

• Less desktop support (ideally, asymptotically approaching zero)
• More Unix server work
• Bigger place

The last point needs a bit of expansion. See, my first job in IT was on the helpdesk of a small ISP. There were three of us on helpdesk, one webmaster, one sysadmin, one database guy, one secretary and one manager; I got some mentoring from the sysadmin (who split his time betwen us and a sister company), but not lots. My second was at a startup company; the guy who hired me was a good mentor, and then after a while after he left I got to hire a junior and be a mentor to him. The job I just left was pretty much just me, though I'm lucky enough to have other people I could talk to; UBC's a big place, but I was in a small department.

So my next job was going to be bigger (as in a bigger installation — maybe a whole data centre, even) and have more people — because I really, really wanted to hang out with my peers and learn from them. I envied the people I'd met at LISA in 2006 who were part of a team, who had people to teach and people to learn from.

Well, at this job it's...just me. Sort of; the folks I've been working with for the last six months (one lab out of the five that make up the centre) are pretty technical. They know way more about Java and MySQL and web development and how the latest CPUs from Intel compare with AMD than I do. But I'm the sysadmin. There might be another in the future, but there isn't now.

But! But, there are two sysadmins on the floor above me who work in another department. For various reasons, we're going to be working closely for the forseeable future. On Friday, I went up to talk with them about how that was going to work out.

They knew stuff I didn't know -- no surprise there -- but it turned out I could show them a trick or two as well. We swapped war stories, discussed our very different backgrounds (saved for another entry), and just shot the shit. It was wonderful.

It's weird, because I'm an introvert, and not very socially apt. (Or ept. As in "opposite of inept".) But it's really, really nice to get together with people who like being a sysadmin the way I do.

(This entry brought to you by the number i, the letter Ve, and my youngest son's 90-minute nap.)

• I knew I didn't like Vaio's very much, but I had no idea they were so awful — to the point of requiring hacking on your goddamn BIOS to enable VMX.

• The flash demo for Dell's ML6000 tape library boasts that it's "completely self-aware". Not sure I want SkyNet running my backups…

• O'Reilly has an upcoming webcast on -- deep breath -- "Advanced Twitter for Business". (At least they didn't call it a webinar. When I told my wife about this, she said "So...you and O'Reilly break up yet?"
• Obviously not, because I've just ordered Backup and Recovery and Linux Clusters with Oscar, Rocks, OpenMosix and MPI. I had purchased B&R at my last job, but this is for me.
• And did I mention the dream I had a while back about a Sun laptop that looked like an X4200 server folded in half? In the dream it ran nearly perfectly, except when you tried to go to a web page with flash; then it would crash, and a movie of Matt Stone would play, apologizing on behalf of Jonathan Schwartz and everyone else at Sun.
• I'm playing with the CVS version of Emacs after reading about some of the new features in what will become Emacs 23. It's nice, but the daemon mode isn't quite multi-tty — you can run Emacs server, detached from any TTY, but if you try connecting to it with multiple emacsclient instances, the first one is where all the TTY action goes. Not sure what I'm missing.

Another thing I'm trying to do at my new job is make/take more time for long-term planning. I've been dinged by mgt. for this in the past, and while it's not easy to hear I think there has been some validity to this. (My inclination is to concentrate hard on fixing the problems I'm faced with; giving up on something broken, even when doing so would make so much more sense and would free up resources to look for a replacement, just rankles and feels like…well, giving up.) Since the department I'm in is so new, it's even more important to pay attention to this.

Part of the problem is just recognizing that I need to make time. An hour a week to be isolated, and to (say) figure out what I'm going to need to do for the next month, is a habit I'm very conciously trying to adopt.

But another problem is how to keep track of all this. What I've done so far:

• I'm a huge fan of Tom Limoncelli's Time Management for System Administrators, and his Cycle system has served me well. I've become a big fan of a paper organizer, so that's how I keep track of things. But it works est as a way of tracking day-to-day stuff; it's not so good at tracking a project that takes weeks, or months, or years.
• I've read GTD, and that seems like a good system — but it's very different from The Cycle. I don't want to give up the Cycle, I want to graft on to it. And I'm not sure how well I can do that w/GTD.
• I've tried org-mode in Emacs. I'm pretty happy with this, and in fact I switched to it for a while when I first started at this job back in July. It worked well for tracking day-to-day stuff, but I missed the flexibility and ubiquitousness of paper.

So where does that leave me? ATM, (paper planner Cycle) attempting some longer-term project tracking w/org-mode. I figure the TODO bits from org-mode will fit well with the planner, and the flexibility of Emacs and org-mode (different from paper…oh, how I wish I could grep paper) will work well for projects…the records for which should, ideally, be suitable for pasting into wiki-based documentation.

If anyone has any suggestions, please let me know. If I make it to LISA this year, I'll be looking for a BOF about this. (Or maybe I'll just tackle Tom Limoncelli to the ground and holler "I love you, man!" a la "Say Anything".)

Moving on:

• I really like TrueType font support in Emacs 23, and ttf-inconsolata in particular. Thanks to Emacs-fu for both suggestions.
• I and a co-worker picked up the servers that had, for the last two years or so, been racked at BC Women's hospital (of all places…my sons were both born there). We both had the same reaction when we saw them on a cart, ready to be loaded into our truck: "They're so small!" Seven little 1U servers plus one disk array…you start to think of them as larger-than-life when you're not looking at them all the time, and it's easy to forget just how small they are.
• Some interesting discussion on the Cfengine mailing list about how Cfengine should handle packages.

And now it is time for bed.