Wednesday (cont):
Put the fake goatee on
And it moves as cool as sugar free jazz.
"Sugar Free Jazz", Soul Coughing
During the break I got into a conversation with Ali and George about
cfengine and Python. I recommended "Dive into Python", and George
agreed; "There's no time for yet another 'hello, world!' programming
book."
And then I met up with Noah from MIT. w00t! I hadn't known he was
coming, but then on Monday he was called by the Rock Star Sysadmin o'
the Year' contest guys, who asked if he was coming: "No, not in
the budget this year." "Really? Are you sure you're not coming?"
"Um..." So here he was. We ducked briefly into the GUru session on
Zenoss, but it was not for us and we moved on to the papers session.
The first one was "Pushing Boulders Uphill: The Difficulty of Network
Intrusion Recovery". And holy cow, they weren't kidding. The state
of the art for intrusion recovery, as the presenter said, is wipe
and reinstall from backups. Okay, maybe you can do that with one or
two machines -- maybe even a few more than that. But what do you do
when your system is massively compromised? When there aren't just
some Code Red packets but when every single machine has a
rootkit?
Reinstalling from backups is no longer satisfying, and yet no one
wants to share solutions they might have come up with: "What, I
should put it on my resume? 'Got pantsed in front of Slashdot.' I
don't think so." So, without identifying the people involved, he
shared the story for the purpose of "adding to the lore" (great term).
In a nutshell, an academic department at an American university had
its gold server, from which they pushed updates to one thousand
workstations, got compromised. Now the workstations had rootkits in
them. They only found this out by accident when various processes
were crashing in weird ways. And they found it out in the middle of
December, right before exams and Xmas, right before half their IT
staff was leaving for unrelated reasons. (You could hear gasps
around the room as the story was told. Six of those were mine.)
So what do you do? Do you take everything offline and screw over the
students? Do you reset passwords? They didn't know exactly when the
compromise had occurred, so backups were out. That left reinstalling
-- but with what? Same distro, when you don't know if it's
vulnerable, or something else? How do you make sure it's all going to
work? The state of the art addresses very little of this, and does
nothing to help with the entirely reasonable gut-clenching panic.
(I admit I have not read the paper yet. But once I get some
time, it's going to be one of the first.)
The second paper I tuned out of, only to hear Tom Limoncelli get
up at the question time and say, "I think this paper is crazy. I
think that's good, because LISA needs more crazy papers. But I wonder
if you realize how crazy it is." The speaker nodded and said, "Oh,
yes."
The third paper was a comparison of two big mail migrations...again,
it had the feel of adding to the lore (a good thing). It was an
entertaing story, well told, about how all the preparation they'd done
had not covered every eventuality. The presenter mentioned that one
of the reviewer's comments was "You must not have done enough
testing." "And I thought: I know! I'm in the future now, too!"
They finished their talk with a video of raised flooring packing foam
air hockey...fun times.
During the break I talked to a woman who was attending the conference
for free, in return for volunteering at the USENIX desk. She ran her
own business, and with the economy tanking she'd had to lay off
everyone but herself...which meant that she was the sysadmin, too.
She has computer experience but no sysadmin experience, so she came
here to learn. I sold her on joining LOPSA by talking about how much
the mailing lists had helped me.
The talk on Eucalyptus was next, and man, do I have mixed feelings
about this presentation. On the one hand, cool stuff: open-source
implementation of the AWS API so that researchers can have an
actual cloud (based on the only instance of a cloud that everyone
agrees on) to do research. What could be wrong with that?
OTOH, the way this guy talked gave me the same feeling as when I read
Marshall McLuhan: it's English, but not as I know it. The one
example I wrote down (he spoke at about 300 wpm) was when he described
a server as "an aggregated set of state updates." That said, my
roommate (who's doing a Ph.D. in this sort of thing) thought he was
brilliant, so I'm perfectly willing to admit I may have been out of my
depth at times.
He was quite funny at times:
"At the end of the first week after the release, there was a cadre
of users who had root who wanted desperately to remove it from their
machines." -- on the sysadmin-vs-researcher fight in grid computing
(not the cloud stuff he's doing now.)
"If you do an open-source project like this, people often want to
tell you things. A lot. And they want to tell you at 4 am."
And one last thing: he said he was quite impressed with Amazon's API.
He kept seeing cases where people would change the API, as Eucalyptus
had implemented it, in an attempt to improve it; the changes would
almost invariably lower the amount that Eucalyptus could scale.
The LOPSA meeting was that night, and it was interesting. They're up
to about 500 members, but they need more -- partly to keep it growing
and partly to get access to things like O'Reilly Safari. (The magic
number for stuff like that is 1000 members.) They mentioned the ties
they're making with other countries -- Australia, Ireland, a group in
India, "and we've just been talking with someone who wants to start a
converence in Vancouver."
Lightning talks! In the spirit of the thing, bullet-point summaries:
- mrepo -- update tool for RedHat I must check out
- selinux permissive domains -- not sure if this was the same as the
targeted policy that Rik Farrow was talking about
- timestamps for web app -- guy from Yahoo saying that SSL depends on
proper timestamps to prevent MITM attacks, and yet we're trusting
the client for these...arghh! any ideas?
- openefs -- Trey Harris' project to keep software working by never, ever letting it change; a
combination of symlinks and OpenAFS that's due to be open-sourced
soon
- Beth's story of crazy
- Alva Couch asking if a Lessons Learned section for LISA would be
good for next year; the whole room agreed. More about this later.
(If I've missed any, let me know.)
I talked to the organizer afterward and asked how many people he'd had
sign up in advance; the answer was none, and he'd had to go after
people in hallways to get them to present. I felt bad for not doing
so...I had meant to but I got distracted. Next time, I will Do The
Right Thing!(tm)
Rock Star Sysadmin of the Year award...first the good: both Matt
and Noah got Finalist and Runner-Up awards (respectively). This
is cool and all the winners are to be congratulated. There were cool
prizes given out, and the grand prize winner donated his to charity.
There was cake. Yay everyone!
Now the bad: my cheeseometer was pinned. As someone pointed out, the
presenter looked like Guy Smiley; he had spiky marketer hair and was
just smarmy. And the band, for reasons I can only guess at,
was the pet band of a guy who's a cake chef/baker in Baltimore and has
a TV show about cakes that he makes. I thought the music was
awful (but then, Noah liked it a lot and he's the one with the
sysadmin prize :-), but more than that it was loud. Fortunately I
had earplugs or there would've been blood running out of my ears.
(No, you're old!)
Oh, and there were TV cameras (marketing material? next week's cake
episode? memo to myself: must tape cake show) filming the women (who
I think were there with the vendor but I could be wrong about that)
dancing up at the front of the stage; what the cameras didn't show was
that they were pretty much the only dancers up there.
There was an escape to the LOPSA suite. I signed up two more people,
then headed off for the hotel bar with Noah and a few other folks. I
meant to call it an early night, but that did not happen. Oh well.
