Sugar Free Jazz

Wednesday (cont):

 Put the fake goatee on
 And it moves as cool as sugar free jazz.

 "Sugar Free Jazz", Soul Coughing

During the break I got into a conversation with Ali and George about cfengine and Python. I recommended "Dive into Python", and George agreed; "There's no time for yet another 'hello, world!' programming book."

And then I met up with Noah from MIT. w00t! I hadn't known he was coming, but then on Monday he was called by the Rock Star Sysadmin o' the Year' contest guys, who asked if he was coming: "No, not in the budget this year." "Really? Are you sure you're not coming?" "Um..." So here he was. We ducked briefly into the GUru session on Zenoss, but it was not for us and we moved on to the papers session.

The first one was "Pushing Boulders Uphill: The Difficulty of Network Intrusion Recovery". And holy cow, they weren't kidding. The state of the art for intrusion recovery, as the presenter said, is wipe and reinstall from backups. Okay, maybe you can do that with one or two machines -- maybe even a few more than that. But what do you do when your system is massively compromised? When there aren't just some Code Red packets but when every single machine has a rootkit?

Reinstalling from backups is no longer satisfying, and yet no one wants to share solutions they might have come up with: "What, I should put it on my resume? 'Got pantsed in front of Slashdot.' I don't think so." So, without identifying the people involved, he shared the story for the purpose of "adding to the lore" (great term).

In a nutshell, an academic department at an American university had its gold server, from which they pushed updates to one thousand workstations, got compromised. Now the workstations had rootkits in them. They only found this out by accident when various processes were crashing in weird ways. And they found it out in the middle of December, right before exams and Xmas, right before half their IT staff was leaving for unrelated reasons. (You could hear gasps around the room as the story was told. Six of those were mine.)

So what do you do? Do you take everything offline and screw over the students? Do you reset passwords? They didn't know exactly when the compromise had occurred, so backups were out. That left reinstalling -- but with what? Same distro, when you don't know if it's vulnerable, or something else? How do you make sure it's all going to work? The state of the art addresses very little of this, and does nothing to help with the entirely reasonable gut-clenching panic.

(I admit I have not read the paper yet. But once I get some time, it's going to be one of the first.)

The second paper I tuned out of, only to hear Tom Limoncelli get up at the question time and say, "I think this paper is crazy. I think that's good, because LISA needs more crazy papers. But I wonder if you realize how crazy it is." The speaker nodded and said, "Oh, yes."

The third paper was a comparison of two big mail migrations...again, it had the feel of adding to the lore (a good thing). It was an entertaing story, well told, about how all the preparation they'd done had not covered every eventuality. The presenter mentioned that one of the reviewer's comments was "You must not have done enough testing." "And I thought: I know! I'm in the future now, too!" They finished their talk with a video of raised flooring packing foam air hockey...fun times.

During the break I talked to a woman who was attending the conference for free, in return for volunteering at the USENIX desk. She ran her own business, and with the economy tanking she'd had to lay off everyone but herself...which meant that she was the sysadmin, too. She has computer experience but no sysadmin experience, so she came here to learn. I sold her on joining LOPSA by talking about how much the mailing lists had helped me.

The talk on Eucalyptus was next, and man, do I have mixed feelings about this presentation. On the one hand, cool stuff: open-source implementation of the AWS API so that researchers can have an actual cloud (based on the only instance of a cloud that everyone agrees on) to do research. What could be wrong with that?

OTOH, the way this guy talked gave me the same feeling as when I read Marshall McLuhan: it's English, but not as I know it. The one example I wrote down (he spoke at about 300 wpm) was when he described a server as "an aggregated set of state updates." That said, my roommate (who's doing a Ph.D. in this sort of thing) thought he was brilliant, so I'm perfectly willing to admit I may have been out of my depth at times.

He was quite funny at times:

• "At the end of the first week after the release, there was a cadre of users who had root who wanted desperately to remove it from their machines." -- on the sysadmin-vs-researcher fight in grid computing (not the cloud stuff he's doing now.)

• "If you do an open-source project like this, people often want to tell you things. A lot. And they want to tell you at 4 am."

And one last thing: he said he was quite impressed with Amazon's API. He kept seeing cases where people would change the API, as Eucalyptus had implemented it, in an attempt to improve it; the changes would almost invariably lower the amount that Eucalyptus could scale.

The LOPSA meeting was that night, and it was interesting. They're up to about 500 members, but they need more -- partly to keep it growing and partly to get access to things like O'Reilly Safari. (The magic number for stuff like that is 1000 members.) They mentioned the ties they're making with other countries -- Australia, Ireland, a group in India, "and we've just been talking with someone who wants to start a converence in Vancouver."

Lightning talks! In the spirit of the thing, bullet-point summaries:

• mrepo -- update tool for RedHat I must check out
• selinux permissive domains -- not sure if this was the same as the targeted policy that Rik Farrow was talking about
• timestamps for web app -- guy from Yahoo saying that SSL depends on proper timestamps to prevent MITM attacks, and yet we're trusting the client for these...arghh! any ideas?
• openefs -- Trey Harris' project to keep software working by never, ever letting it change; a combination of symlinks and OpenAFS that's due to be open-sourced soon
• Beth's story of crazy
• Alva Couch asking if a Lessons Learned section for LISA would be good for next year; the whole room agreed. More about this later.

(If I've missed any, let me know.)

I talked to the organizer afterward and asked how many people he'd had sign up in advance; the answer was none, and he'd had to go after people in hallways to get them to present. I felt bad for not doing so...I had meant to but I got distracted. Next time, I will Do The Right Thing!(tm)

Rock Star Sysadmin of the Year award...first the good: both Matt and Noah got Finalist and Runner-Up awards (respectively). This is cool and all the winners are to be congratulated. There were cool prizes given out, and the grand prize winner donated his to charity. There was cake. Yay everyone!

Now the bad: my cheeseometer was pinned. As someone pointed out, the presenter looked like Guy Smiley; he had spiky marketer hair and was just smarmy. And the band, for reasons I can only guess at, was the pet band of a guy who's a cake chef/baker in Baltimore and has a TV show about cakes that he makes. I thought the music was awful (but then, Noah liked it a lot and he's the one with the sysadmin prize :-), but more than that it was loud. Fortunately I had earplugs or there would've been blood running out of my ears.

(No, you're old!)

Oh, and there were TV cameras (marketing material? next week's cake episode? memo to myself: must tape cake show) filming the women (who I think were there with the vendor but I could be wrong about that) dancing up at the front of the stage; what the cameras didn't show was that they were pretty much the only dancers up there.

There was an escape to the LOPSA suite. I signed up two more people, then headed off for the hotel bar with Noah and a few other folks. I meant to call it an early night, but that did not happen. Oh well.