the life of a sysadmin.
Carousel is a lie!

Linky:
[FSF Associate Member] LOPSA

Email: aardvark at saintaardvarkthecarpeted dot com

Presentation(s), conference, nagios exchange, Project U-13, Project U-14

Sat Sep 29 19:19:39 PDT 2007

I've had a bunch of ideas lately. I'm inflicting them on you.

The presentation went well…I didn't get too nervous, or run too long, or start screaming at people (damn Induced Tourette's Syndrome) or anything. There were maybe 30 or so people there, and a bunch of them had questions at the end too. Nice! I was embiggened enough by the whole experience that, when the local LUG announced that they were having a newbie's night and asked for presenters to explain stuff, I volunteered. It's coming up in a few weeks; we'll see what happens.

And then I thought some more. A few days before I'd been listening to the almost-latest episode of LugRadio (nice new design!), where they were talking about GUADEC and PyCon UK. PyCon was especially interesting to hear about; the organizers had thought "Wouldn't it be cool to have a Python conference here in the UK?", so they made one.

So I thought, "It's a shame I'm not going to be able to go to LISA this year. Why don't we have our own conference here in Vancouver?" The more I thought about it, the better the idea seemed. We could have it at UBC in the summer, where I'm pretty sure there are cheap venues to be had. Start out modest — say, a day long the first time around. We could have, say, a training track and a papers track. I'm going to talk about this to some folks and see what they think.

Memo to myself: still on my list of stuff to do is to join pool.ntp.org. Do it, monkey boy!

Another idea I had: a while back I exchanged secondary DNS service, c/o ns2exchange.com. It's working pretty well so far, but I'm not monitoring it so it's hard for me to be sure that I can get rid of the other DNS servers I've got. (Everydns.net is fine, but they don't do TXT or IPv6 records.) I'm in the process of setting up Nagios to watch my own server, but of course that doesn't tell me what things look like from the outside.

So it hit me: what about Nagios exchange? I'll watch your services if you watch mine. You wouldn't want your business depending on me, of course, but this'd be fine for the slightly anal sysadmin looking to monitor his home machines. :-) The comment link's at the end of the article; let me know if you're interested, or if you think it's a good/bad/weird idea.

The presentation also made me think about how this job has been, in many ways, a lot like the last job: implementing a lot of Things That Really Should Be Done (I hate to say "Best Practices) in a small shop. Time is tight and there's a lot to do, so I've been slowly making my way through the list:

  • Improving backups (Bacula, Amanda)

  • Automated install (FAI, Jumpstart)

  • Monitoring services (Nagios)

  • Monitoring performance (MRTG, Cacti)

  • Ticket system (RT)

  • Automating management (Cfengine)

Some of these things have been held up by my trying to remember what I did the last time. And then there's just getting up to speed on bootstrapping a Cfengine installation (say).

So what if all these things were available in one easy package? Not an appliance, since we're sysadmins — but integrated nicely into one machine, easily broken up if needed, and ready to go? Furthermore, what if that tool was a Linux distro, with all its attendant tools and security? What if that tool was easily regenerated, and itself served as a nicely annotated set of files to get the newbie up and running?

Between FAI (because if it's not Debian, you're working too hard) and cfengine, it should be easy to make a machine look like this. Have it work on a live ISO, with installation afterward with saved customizations from when you were playing around with it.

Have it be a godsend for the newbie, a timesaver for the experienced, and a lifeline for those struggling in rapidly expanding shops. Make this the distro I'd want to take to the next job like this.

I'm tentatively calling this Project U-13. We'll see how it goes.

Oh, and over here we've got Project U-14. So, you know, I've got lots of spare time.

(permalink) (comments)

On Being a Parent, and Obnoxious EULAs

Mon Sep 24 05:53:23 PDT 2007

For my own future reference, Otter Escaping North recently posted two excellent comments about being a geek parent in a recent Slashdot discussion about PC parental controls. The whole article is worth reading (though I always read at +3), but Otter's really resonated with me.

And from the world of obnoxious EULAs comes this gem from Live365's software player for Windows:

"You may not alter, merge, modify, adapt or translate the SOFTWARE
PRODUCT, or decompile, reverse engineer, disassemble, or otherwise
reduce the SOFTWARE PRODUCT to a human-perceivable form."

So hexdump -C is out, then? Or looking at it with less? Sigh…

(permalink) (comments)

Resume, laptop, presentation

Sun Sep 23 13:31:40 PDT 2007

Just updated my resume for the first time since starting my current job. It's nice to look back at what you've done and realize that, hey, there's been a lot.

In other news, I finally gave in to lust the other day and bought a Dell C400 on eBay. Nothing too special — 1.2GHz, 256MB, 30GB hard drive — but I was mainly after the 12" screen, so that I'd be able to (say) debug raw ethernet frames on my daily commute. About $280 when all was said and done; the strong Canuckistan peso was part of the incentive to buy now. Should be at the office in a week or so, and I can't wait.

It amazed me to see how many off-lease laptops were available, and just how cheap you could pick them up. A whilte back my boss got a D420; with extra memory and a few other things, it came in at about $1700 or so Canadian. But if you look around, there are plenty of D400s and D410s around for less than $500 — even less than $400 if you look hard. Add another $100 (say) for a working battery, and you're in pretty good shape.

Virtualbox has made it to Debian testing — hurrah! Only it won't run (Open)?Solaris. Dang.

On Tuesday, I'm giving a short presentation on my work's subnet at SNAG, the UBC System and Network Administrator's Group. I found Bruce in OpenBSD's ports tree on my laptop; the documentation is (ahem) thin, but it works. Wish me luck.

And there's Arlo up. Time to go get him.

(permalink) (comments)

Quickbooks, Bacula

Fri Sep 14 12:21:50 PDT 2007

I was able to get Quickbooks 2007 working with a non-admin account today…woot! Here's what I did:

  • Create a user (let's call it "quickbooks") and put the user in the admin group. Set a password.

  • Since our QuickBooks files are on a shared drive, I logged in as that user and mapped the share to a drive (let's say the Z: drive).

  • Still as the quickbooks user, open up Windows Explorer. Select Tools -> Folder Options -> View and select "Launch folder windows in separate process". Log out.

  • Log in as the ordinary user who needs to use Quickbooks and have them runas, using the quickbooks account: right-click on the Quickbooks icon, select Run As, then select the quickbooks account. Put in the password you set up.

  • You may need to browse to the file rather than opening it up from quickbook's list of recently-opened files.

  • I've also mapped the quickbooks drive in the ordinary user's account, and took care to map the drive to the same letter as in the Quickbooks account. I'm not sure if this is strictly necessary.

This isn't ideal — the explorer process in QB is still running privileged — but at least that's the only IE process running as admin.

And Bacula: tripped over a small thing. I'm running the btape utility to make sure our tape drive works with it. I ran bfill, rather than fill, then wondered why I got errors at the end. Turns out to be an old command that probably shouldn't be around anymore.

Now to run fill…another couple hours to go.

(permalink) (comments)

Why Use PHP?

If'n it ain't one thing…

Mon Sep 10 17:22:16 PDT 2007

…it's another. Busted CPU on a Sun 440 at the university across town meant I spent a bigger part of my day on the bus than usual. Remove the CPU card/assembly/whatever (god, they're mother huge) and we're back in business.

Incidentally, it amazes me that you can turn up fully spec'd V440s on Ebay for, like, $8000 US. 4 x 1GHz CPUs, 16 GB of RAM, 4 x 72GB drives…what's not to like?

(permalink) (comments)

Mail server

Sat Sep 8 18:29:36 PDT 2007

Just when I was about to sign off for the day, suddenly the mail server's down. No response to pings, no response on the console server even. It's an old E220R, and while it's underpowered for all we're asking from it, I haven't had problems with it before. (Well, except for the CDROM drive not powering up. But I can live with that.)

So drive into work with the wife and kid, on the off chance that it'll all be fine quickly. No such luck. It hadn't walked away, the cables were all still in place, and I had to power cycle it to get it to come back up. A lot of fscking later, and I'm waiting for it to finish booting. I can't remember what it was like the last time I rebooted it, but this time it seems rather ridiculous (20 minutes). More stuff to add to the documentation once I'm done…

And once more: sysadmin documentation MUST NOT depend on external services. (The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.)

Time for pizza.

(permalink) (comments)

Now here's a weird one…

Fri Sep 7 14:54:37 PDT 2007

One of the problems I've been working on since the upgrade to Solaris 10 has been the slowness of the SunRay terminals. There are a few different problems here, but one of 'em is that after typing in your password and hitting Enter, it takes about a minute to get the JDS "Loading your desktop…" icons up.

I scratched my head over this one for a long time 'til I saw this:

ptree 10533
906   /usr/dt/bin/dtlogin -daemon -udpPort 0
  10445 /usr/dt/bin/dtlogin -daemon -udpPort 0
    10533 /bin/ksh /usr/dt/config/Xstartup
      10551 /bin/ksh -p /opt/SUNWut/lib/utdmsession -c 4
        10585 /bin/ksh -p /etc/opt/SUNWut/basedir/lib/utscrevent -c 4 -z utdmsession
          10587 ksh -c echo 'CREATE_SESSION 4 # utdmsession' >/dev/tcp/127.0.0.1/7013

which just sat there and sat there for, oh, about a minute. So I run netcat on port 7013, log out and log in again, and boom! quick as anything.

/etc/services says:

utscreventd     7013/tcp                        # SUNWut SRCOM event deamon

which we're not running; something to do with smart cards. So why does it hang so long? Because for some reason, the host isn't sending back an RST packet (I presume; can't listen to find out) to kill the connection, like it does on $other_server.

So now I'm trying to figure out why that is. It's not the firewall; they're identical. I've tried looking at ndd /dev/tcp \? but I don't see anything obvious there. My google-fu doesn't appear to be up to the task either. I may have to cheat and go visit a fellow sysadmin to find out.

(permalink) (comments)

Goddammit!

Wed Sep 5 06:08:29 PDT 2007

And what do I see on Ben's blog but the new version of Solaris out — 8/07, not two weeks after this fiasco. Craptastic!

(permalink) (comments)

bsdstats.org

Wed Sep 5 06:01:54 PDT 2007

Came across a mention of BSDstats.org on the Dragonfly BSD Digest, and I've set it up on my home machine. There are a ton of FreeBSD machines, and only 64 OpenBSD clients reported…time to change that!

I'm reading the documentation for Bacula right now, and it's amazing. Clearly written, thorough and extensive — almost 800 pages long. I'm very impressed.

(permalink) (comments)

Emacs and a new beginning

Mon Sep 3 07:24:43 PDT 2007

Some fun Emacs stuff:

  • A nice tutorial on creating your own language mode. One of these days I'll get around to setting up something that indents SQL the way I want it.

  • Multi-tty Emacs. From the description: "Emacs is notoriously slow at startup, so most people use another editor or emacsclient for quick editing jobs from the console. Unfortunately, emacsclient was very awkward to use, because it did not support opening a new Emacs frame on the current virtual console. Now, with multi-tty support, it can do that. (Emacsclient starts up faster than vi!)" Must get me some of that. (Hey, I wonder if there's a way to forward the emacs client/server thing over SSH…emacsclient at the remote end, emacs here. Yeah, you could do it with Tramp or some such, but this'd be neat. Hm…)

I had a meeting with my boss at work last week (before a nice four-day weekend…the split schedule I've got means that sort of thing happens very rarely. But I digress) to set my priorities now that the upgrade has more or less been finished (lingering issues aside; see ahead).

One of the big things is getting Zimbra set up. This will be nice; we do not have a calendar for the office right now, and this is is getting to be a pain. My boss is open to the idea of something that's not Outlook/Exchange, and that's good.

The other thing is getting a bunch more Windows machines in. This is a small shop, so "a bunch" means another 15 or 20…which'll double the number we have. I'm not entirely happy about that, but because this is a longer-term project I've been given time to do this right. And to me, "right" means "using open-source tools whenever possible to manage Windows". Thus, I'll be getting the time to set up Unattended and wpkg, and possibly even digging up Windflower and seeing if it's worth continuing. I'm actually kind of excited about this.

It's a little strange having a manager take this much of a hand in setting priorities; I've worked in a series of small shops and, up 'til now, have been left more or less on my own nearly the whole time. It does feel good to get a bit of direction, though. I mean, I know what needs to be done and I'm doing it, but I've always felt a bit lost trying to decide what's most important for everyone once past the finger-in-the-dike stage.

Now to go try and get Multi-TTY working on this laptop…

Ack: Just realized I never described the lingering problems with Solaris 10. Fairly simple to describe: LDAP lookups take 'way longer than they should (ls -l /home/ can take 5 seconds per line sometimes), and JDS on the SunRays is slower in parts than it should be (click on the logout button, wait 60 seconds, message pops up saying "Are you shure you want to log out?"). I'm hopeful I can track those down without too much effort…

(permalink) (comments)