New server room ours at last

Given the recent hoo-ha about abandoned blogs, and my own tendency to lose interest in writing about something the longer I put it off (I haven't graphed it, but I suspect it's a nice exponential decay), I figured I should finally write up what I've been doing the last week: the move at $WORK to our new server room.

So: construction finally got finished on our new server room. Our UPS was installed, our racks set up, and the keys handed over (though they were to be changed again twice). Our new netblock was assigned, the Internet access at the new location was in place, and movers were booked.

Things I did in advance which helped immensely:

• Checklist in Org mode, plus printed copies; the ability to constantly edit a nice todo list, complete with checkboxes and statistics, was wonderful.
• Printed copies of the spreadsheet showing rack assignment, cabling requirements, VLAN changes, etc
• Tested new firewall with VMs (thus pointing out that "antispoof quick" is not a good thing to do with a bridging OpenBSD firewall)
• Cardboard for the floor of the new server room to lay the servers on (since we weren't going to be able to rack the machines as quickly as they came from the movers)

Last Thursday morning, it all started. I got the machines shut down (thank you, SSH and ubiquitous wireless access at UBC) before the two volunteers who were helping me showed up. We started getting machines unracked; since it was only about 20 machines, I figured it wouldn't take too long. While that was true, I had not counted on the rat's nest of power cables (our power requirements were such that we had to connect machines to PDUs in adjacent racks), or the fact that we wouldn't be able to disassemble that 'til we'd got the machines out.

There was one heartstopping moment: a 1U server, while extended on its rails, came off one of the rails while no one was supporting it. Amazingly the other rail held on while it rotated quickly through 90 degrees to bang loudly against the rack. "You swear quickly," the movers remarked. (Doubly amazingly, the machine seems to be fine, though the rails for the thing are shot.)

The movers were big and burly, which was wonderful when it came to moving the Thumper. I weigh more than it does, but not by much, and I'd had the bad fortune to screw up my back a week before the move. It was tricky trying to figure out how to remove it from the rails, but the movers' trick of supporting it with a couple of big blankets, while fully extended from the rack, made such considerations less urgent. Eventually we got it figured out. I don't know how that could have gone smoother, since we'd got Sun to rack the thing and, frankly, it's not like you spend a lot of time un- and re-racking something like that. Anyhow, a minor point.

The new location was right around the corner, which was handy. The movers had put the servers in these big laundry-like carts on wheels; in the end, we only had four of em. We got the machines unloaded, racked the Thumper with the movers help, signed the paper, then went off for lunch where we picked up two more volunteers.

After that, we started racking servers. Having only one sysadmin around (me) proved to be a bottleneck; the volunteers had not worked with rackmounted machines before, and I kept having to stop what I was doing to explain something to them. It would have been a great help to have another admin around; in fact, I think this is the biggest move I'd want to make without some other admin around.

Problems we ran into:

• Cage nut pullers are small and get lost easily. (Moral: designate one place for tools, just like it sez here)
• Mounting brackets didn't work. One of 'em, I just figured out today, we had in backwards. The other wasn't threaded for the bolts from APC, and I had only the right bolts — no cage nuts to fit. (Moral: photograph the racks for anything non-standard; if you have to ask, it's non-standard)
• One of the things we couldn't mount was a Very Important Disk Array. Fortunately it held a database which had been mirrored on another Very Important Disk Array, which also couldn't be mounted in its brackets. Instead, we used a rack shelf I happened to have around, and that worked well….but its advertised capacity wasn't enough to hold all four trays (2 trays per array), so we made do with one. (Moral: have a spare rack shelf or two on hand)
• The bolts from APC had these enormous heads, which would end up impinging on the rack unit above/below. This got to be a pain. Only today did I discover that there were plenty of bolts and cage nuts provided by the contractor who installed the racks. (Moral: dress rehearsal includes putting cage nuts and bots in adjacent holes to see how they fit)
• We had to re-hang the PDUs so they'd reach the power supplies. There were two in each rack, and both were on the right; the power supplies were all on the left, and I'd bought a bunch of 2' power cords to help with cable management. (Moral: Think about cable management for power, not just network)
• Another thing about the PDUs: The outlets don't stretch throughout the length of the bar, but instead are clustered such that there's a dead space at the bottom/top 8" or so. The power cables had to be chained together sometimes to reach the extremes. (Moral: dress rehearsal includes plugging things in)
• My plan to mount the switch in the middle of the rack with all the equipment has the advantages of shorter network cables (no running back to front, and no running top to bottom). But I should have noticed the middle empty spot in the PDUs adn mounted it there; as it is, there's a block of outlets in the PDUs I can't use because the power cables will get too close to the network cables. (Moral: think about cable management for network, not just power)
• Underestimated the amount of time it'd take to get things racked. I suppose this can only be bettered with experience.
• Underestimated the amount of time it'd take to get cables dressed; did not realize how important this was for working with things.
• Did not bring warm shirt for when the cooling was turned on. Mistake!
• Did not have lots of water on hand; did not figure out in advance where bathroom was (important in a building where you only have access to one room)
• Really could have used a phone in advance in the room; cel coverage was spotty
• Ratchet set very handy when tightening screws in awkward places (ie, behind power bar); last resort: hold bit in jaws of pliars/Leatherman. (Moral: dress rehearsal includes looking for tight corners and figuring out how you're going to work in them)
• Preserve all bits and label them; carry masking tape/removeable labels and sharpies; label anything and everything you haven't already; use ziplock bags for stuff and tape them to the machines they're associated with
• Firewall not modified to allow LDAPS to LDAP server from new netblock
• Monitoring machine came up with no ethernet interfaces; modprobe tg3 gave "probe of 0000:04:04.0 failed with error -22". (Moral: figure out how you're going to get information off a machine with no network)
• Anyone else notice that C13-C14 power cords are just plain wobbly in the PDU sockets? I had more than one pop out on me while moving cords around. (Moral: Andy Rooney lives!)
• Coulda used more printouts of the rack assignments.
• One cable was flaky: it worked for a while, then didn't. This was the cable that connected our firewall to the ILOMs for the servers, which meant I was unable to work from home on getting them up and running. This was probably for the best; I sorely underestimated just how wired I was when I went home. (Moral: you're more tired than you think)
• One of the racks was designated as the networking rack; however, since we didn't have that many switches to mount, I figured I'd use it for other stuff too. This turned out not to work: the distance between the front and back rails had been shortened to make room for network cables, and that meant the rack rails for the equipment I wanted to mount didn't fit.

Things that went well:

• Ripwrap is awesome. So are cordless drills that come with two batteries.
• The rack rails from Sun that just clip in are also awesome. Man, that makes things fast.
• There was good beer in the fridge when I got home. Thanks, Pre.
• Frankly, all the prep meant that things went pretty well overall. This was good.

I'm going to post this now because if I don't, it'll never get done. I may come back and revise it later, but better this than nothing at all.