Money City Maniacs

Hey you!
We've been around for a while.
If you'll admit that you were wrong, then we'll admit that we're right.

-- Sloan

After posting last night, a fellow UBCianiite and I went looking for drinks. We eventually settled on the bar at the Fairmont. The Widsomething Imperial IPA was lovely, as was the Gordon Biersch (spelling, I'm sure) Marzen...never had a Marzen before and it was lovely. (There was a third beer, but it wasn't very good. Mentioning it would ruin my rhythm.) What was even lovelier was that the coworker picked up the tab for the night. I'm going to invite him drinking a lot more from now on.

Sunday was day one of tutorials. In the morning was "Implementing DNSSEC". As some of the complaints on Twitter mentioned, the implementation details were saved for the last quarter of the tutorial. I'm not very familiar with DNSSEC, though, so I was happy with the broader scope...and as the instructor pointed out, BIND 9.7 has made a lot of it pretty easy, and the walkthrough is no longer as detailed as it once had to be.

Some interesting things:

• He mentioned not being a big believer in dynamic zones previously...and now he runs 40 zones and they're ALL dynamic. This is an especially nice thing now that he's running DNSSEC.

• Rackspace is authoritative for 1.1 million zones...so startup time of the DNS server is important; you can't sit twiddling your thumbs for several hours while you wait for the records to load.

• BIND 10 (did I mention he works for the ISC?) will have a database backend built right in. Not sure if he meant that text records would go away entirely, or if this would be another backend, or if it'd be used to generate text files. Still, interesting.

• DNSSEC failure -- ie, a failure of your upstream resolver to validate the records/keys/whatever -- is reported as SERVFAIL rather than something more specific. Why? To keep (say) Windows 3.1 clients, necessary to the Education Department of the fictional state of East Carolina, working...they are not going to be updated, and you can't break DNS for them.

• Zone signatures: root (.) is signed (by Verisign; uh-oh); .net is signed as of last week; .com is due next March. And there are still registrars that shrug when you ask them when they're going to support DS records. As he said, implement it now or start hemorrhaging customers.

• Another reason to implement it now, if you're an ISP: because the people who will call in to notify you of problems are the techie early adopters. Soon, it'll be Mom and Dad, and they're not going to be able to help you diagnose it at all.

• Go look at dnsviz.net

• Question that he gets a lot: what kind of hardware do I need to serve X many customers? Answer: there isn't one; too many variables. But what he does suggest is to take your hardware budget, divide by 3, and buy what you can for that much. Congratulations: you now have 3 redundant DNS servers, which is a lot better than trying to guess the right size for just one.

• A crypto offload card might be a good thing to look at if you have a busy resolver. But they're expensive. If your OS supports it, look into GPU support; a high-end graphics card is only a few hundred dollars, and apparently works quite well.

On why DNSSEC is important:

• "I put more faith in the DNS system than I do in the public water system. I check my email in bed with my phone before I have a shower in the morning."

• "As much as I have privacy concerns about Google, I have a lot more concerns about someone pretending to be Google."

On stupid firewall assumptions about DNS:

• AOL triggered heartburn a ways back when replies re: MX records started exceeding 512 bytes...which everyone knew was impossible and/or wrong. (It's not.) Suddenly people had weird problems trying to email AOL.

• Some version of Cisco's stateful packet inspection assumes that any DNS reply over 512 bytes is clearly bogus. It's not, especially with DNSSEC.

• If I rem. correctly (notes are fuzzy on this point), a reply over 512 bytes gets you a UDP packet that'll hold what it can, with a flag set that says "query over TCP for the full answer please." But there are a large number of firewall tutorials that advise you to turn off DNS over TCP. (My own firewall may be set up like that...need to fix that when I get back.)

When giving training on DNS in early 2008, he came to a slide about cache poisoning. There was another ISC engineer there to help him field questions, give details, etc, and he was turning paler and paler as he talked about this. This was right before the break; as soon as the class was dismissed, the engineer came up to him and said, "How many more of those damn slides do you have?" "That's all, why?" "I can't tell you. But let's just say that in a year, DNSSEC will be a lot more important."

The instructor laughed in his face, because he'd been banging his head against that brick wall for about 10 years. But the engineer was one of the few who knew about the Kaminsky attack, and had been sworn to secrecy.

Lunch! Good lunch, and I happened, along with Bob the Norwegian, to be nearly first in line. Talked to tablemates from a US gov't lab, and they mentioned the competition between labs. They described how they moved an old supercomputer over beside a new supercomputing cluster, and got the top 500 cluster for...a week, 'til someone else got it. And there were a couple admins from the GPS division of John Deere, because tractors are all GPS-guided these days when plowing the fields.

Sunday afternoon was "Getting it out the door successfully", a tutorial on project management, with Strata Rose-Chalup. This was good; there were some things I already knew (but was glad to see confirmed), and a lot more besides...including stuff I need to implement. Like: if startup error messages are benign, then a) don't emit them, and b) at least document them so that other people (customers, testers, future coders) know this.

QOTD:

• "Point to the wall, where you have a velvet-covered clue-by-four and chocolate. Ask, 'Which would you like to have your behaviour modified by today?'"

"What do you do if your product owner is an insane jackass?" "If your product owner is an insane jackass, then you have a typical product..." But srsly: many people choose to act like this when they feel they're not being listened to them. Open up your meetings and let them see what's on the table. Bring in their peers, too; that way their choice will be to act like a jackass in front of their peers, or to moderate their demands.

Tip from the audience: when faced with impossible requests, don't say "No". Just bring up the list of stuff you're already working on, and the requests/features/bugfixes that have already been agreed to, and ask them where this fits in. They'll either modify their request ('cos it's not that important to them), or you'll find a lot of other stuff moved out of your way ('cos that other stuff isn't that important to them).

After that was supper with Andy, who I hadn't seen since last year's LISA. We hit up a small Mexican place for supper (not bad), the Britannia Arms for a beer (where Matt tried to rope us into Karaoke and kept asking us to do "Freebird" with him), then the Fairmont hotel bar so Andy could get his Manhattan. (He's a bit intense about Manhattans.) It was a good time.