maillog

Just came across maillog, which looks very cool. From TFM:

Maillog is a powerful tool for selecting and formatting entries from a
sendmail or postfix log. When a message is selected, it collects all
the mailer entries related to that message's queue id and formats them
in a more readable fashion. By default, the log fields that are
printed are: date, from, to, ctladdr, stat, and notes.

This is much better than my cobbled-together multiple-grep scripts. Rather surprised to not find it in Debian...

Tags: postfix toptip handytool

SELinux at last

Welp, after my training at LISA I finally got to start using SELinux. I was setting up a CentOS server with Mascot, search engine software for mass spectrometer software, and I thought I'd give it a try.

Mostly it turned out to be simple -- semanage fcontext to add some new httpd -friendly locations where the software had been installed, restorecon to set the labels. One thing that did take some tracking down was digging up exactly what this meant:

type=AVC msg=audit(1259021236.914:280): avc:  denied  { execstack}
for  pid=6845 comm="ld-linux-x86-64"
scontext=user_u:system_r:httpd_sys_script_t:s0
tcontext=user_u:system_r:httpd_sys_script_t:s0 tclass=process

This happened when the install script tested Perl to make sure everything was okay.

As described by Dan Walsh and Ulrich Drepper, this means that the Perl executable was marked as needing an executable stack. Not only is this a Bad Thing(tm), it's not usually necessary these days (what with the Internet and all). execstack -c cleared the flag, and things appeared to work after that; it was right at the end of the day, though, so it's possible problems will show up today.

And then when I got home...it was wonderful. The kids'd had two-hour naps each, there was a wild rice casserole in the oven (The Cheese Fairy is always amazing), and my parents had sent the kids a calendar full of pictures of Canadian wildlife. I got to tell Trombone how the beaks of different birds (great blue heron, snowy owl, cardinal) were adapted for eating different things; I think he was interested, and that was just flat out fascinating. Ah, domestic bliss.

Tags: selinux geekdad

More Son of LISA coverage

Thursday night (November 5th...god I'm behind) was NIGHT OF BoFs. (Dramatic music!) First up was my conference organizer's BoF. In a nutshell: I wanna start a conference; what do I need to know?

There were only a handful of people there, but hey, quality not quantity:

• Deb Nicholson from the FSF
• Scott Murphy, ArrowEye consulting, who has helped with BSDCan
• Ski Kacoroski from LOPSA
• Matt from Standalone Sysadmin, who (like me) was there to learn. (or possibly out of pity for me. Either way.)

Easiest part of organizing a conference: getting speakers. This surprised me, but everyone likes to talk about themselves. WIPs (work-in-progress posters/talks) will get everyone engaged.

Hardest part:

• defining the scope/theme of your event. This is important because a) you need your elevator pitch and b) otherwise it's just Saint Aardvark's Conference About Totally Interesting Stuff, and if you don't happen to be SAtC (poor you!) you may not be all that interested.

• the last week: death by a thousand papercuts + dread

Gotta have it:

• Swag bag. Contact local (or not!) sponsors early. For some reason I'm hung up on t-shirts being TOTES ESSENTIAL, but this is not necessarily the case.

• Chance to meet in advance; break the ice, get the newbies (and we're all newbies) to relax and make friends. If your event is on a Saturday, this is why Friday night was invented. Don't forget to have organizers working the floor.

• Everyone in the same room for meals -- either bring it in, or have one place close by designated and ready. You don't want people scattering to the four winds to eat...they'll never come back. And make the vegetarians/vegans happy; if all they get to eat is crackers and soy bologna, you will hear about it.

Random tips:

• Price the event according to what you aim to give people.

• Think about having a fun track beside one or two serious tracks.

• Record the sessions and offer Ogg/MP3 downloads. Don't forget slides and papers, too.

• Lead time: 9 months probably isn't enough time to organize an event with 300-400 attendees...but 6 months should do for 50 attendees. (That's more the scale I'm aiming at.)

• Careful with vendors; being sold at all day is a definite turnoff

• Re: sysadmin conference in particular: Survey local businesses and see what they need, what they'd send people to see.

• Always look for ways to delegate stuff, or you'll run yourself ragged.

Getting people back next year:

• Finish your closing speech with "See you next year!" ie, ask people to come back, and to spread the word.

• Meet within a month of finishing the conference with next year's organizers and start making plans. Put checklists and improvements on a wiki so that the info doesn't get lost.

• Get new blood every year, both attendees and in the organizing committee.

Also got various contacts and other suggestions from people...thanks very much!

After that came Matt's two BoFs: small infrastructure and bloggers. Unfortunately, my notes suck from these two events...but it was good talk at both. I was surprised to see how many people were there because they're professional writers; I keep thinking of this as just my way of scribbling on the walls.

Tags: lisa conferenceorganization

Serial console FAIL (somewhere...)

This is irritating...

We've got four new Dell R410 servers at work. Natch, I want 'em working with serial consoles so I don't have to sit in the server room. Three of them worked; the fourth did not, despite having identical BIOS/Grub settings.

The symptom was quite maddening: After getting past the various BIOS checks, the Grub menu would not appear unless you sat there and typed something. After that, you'd get the usual Grub entries and could boot as usual. If you did not hit a key, the machine would just hang -- no response to keypresses at all, and you'd have to power cycle.

I spent a stupid amount of time comparing BIOS and Grub settings but was unable to find anything different. Finally today I typed "grub console timeout serial dell" into Google and found this bug in Launchpad, with this comment as the last one:

Having the same hanging issue at the Grub 1.5 stage on brand new R200 Dell servers running OpenSuse 10.3. The terminal timeout is set to 10 and we get 10 press any key to continue messages and then a full system hang requiring a hard reboot.

If we do press any key on a connected console (using Dell's Serial Over Lan) or locally before then end of the timeout then it boots fine so seems to be a bug in continuing at the end of the wait time.

Removing the terminal line from /boot/grub/menu.1st seems to fix the issue on our servers. The console in this case is sent by BMC to both the local screen and the remote console with no timeout so works a treat. This may only work with Dell's BMC/SOL but thought I'd mention it in case anyone else has spent a day getting frustrated with this like we have.

This worked a treat, with the added bit of weirdness that I had two "terminal" lines:

terminal --timeout=2 serial console
serial --unit=0 --speed=9600
default=0
timeout=5
serial --unit=1 --speed=115200
terminal --timeout=5 serial console

and now I have one:

terminal --timeout=2 serial console
serial --unit=0 --speed=9600
default=0
timeout=5
serial --unit=1 --speed=115200
# terminal --timeout=5 serial console

Yes, I know that's redundant, but again: it worked on the other three machines.

I don't know if this is a problem with Grub, with Dell's firmware or something else, but Gott in himmell I hate bugs like this.

Tags: hardware bugs dell

Try the oven next time

As recycled by Bradley M. Kuhn on identi.ca, here's another tool for recovering a dead hard drive: a toaster oven.

Tags: hardware

Hall of Sigh

From a EULA I got recently:

Customer shall not in any manner or under any circumstances use, copy,
modify, enhance, merge, reverse engineer, reverse assemble, decompile,
or in any way alter the Software, Hardware or Documentation or any
copy, adaptation, transcription, or merged portion thereof or
otherwise attempt to derive Source Code therefrom; provided, however,
that, if any applicable laws (such as national laws implementing EC
Directive 91/250) expressly give Customer the right to perform any of
the aforementioned activities without Licensor's consent, Customer
shall, before exercising such right, notify Licensor of its intent to
exercise any such rights and only exercise such rights if Licensor has
not, within twenty (20) business days after Licensor's receipt of such
request, agreed to provide Customer with the result which Customer
would otherwise have obtained by exercising such rights (in which case
Customer shall pay Licensor its then-standard rates for such work).

Tags: freeasinfreedom

Ohio Linux Fest 2009 Audio

Okay, did you know that the Ohio LinuxFest has put up audio from their sessions at archive.org? I didn't, but I'm downloading it all now. (Along with a couple of NYLUG presentations on Rocks and Cobbler and Kexec/Kdump.)

Kudos to the organizers for such a great idea!

Tags: podcast linux

LISA Coverage Redux

Thursday afternoon:

First up was Elizabeth Zwicky's talk on distinguishing data from non-data, and how to deal with each when solving problems. She warned us that she was not a statistician, and what she was going to say would probably give a real statistician hives, but that it would be useful for dealing with computers -- "nothing with an ethics board."

Her talk was laced with examples from her career...like the time she tried to track down missing truck axles from a major defense contractor; this was complicated by their complete lack of data collection ("How many do you make in a week?" "The schedule calls for 100." "How many of those are completed by Friday?" "We're not collecting that data."). Or the time she broke into her CEO's office ("It has a lock!") by pushing up a ceililng tile, then reaching down with a coat hanger and pulling up the handle. Lesson learned: "If it stops at the ceiling, it's not really a wall."

Funny stories aside (and they were funny; I recommend listening to the talk), the point was the danger of assuming too much from initial observations -- we schedule X, so we must produce X; it looks like a wall, so it must be impervious. Data is observations, numbers with context -- not hearsay, or conclusions, or numbers without context. Again, listen to the talk; it's worth your time.

Hell, download every MP3 on this page and listen to them; that's what I'm going to do, and I've been to some of them.

Okay, after that came the refereed papers. Mostly I was there for the SEEdit paper, which describes the SEEdit tool (available on Sourceforge!) for editing/creating SELinux policy in a high-level language. After what Rik Farrow said about policy approaching his rule-of-thumb for human comprehension, I was interested to see if this could be used to generate/edit the existing policy. I tried asking this, but I don't think I made myself clear...and I meant to follow up with the presenter later, but I didn't. My bad.

The paper on the SSH-based toolkit was interesting, but it seemed complex; from what I could gather, you SSHd to a machine, then forwarded connections to (say) POP or SMTP over the tunnnel to a daemon at the other end, which would then forward it to the right destination. It kept seeming kludgy and complicated to me, especially compared to something like authpf plus the usual sort of encryption that should be on (say) POP or SMTP to start with. I asked him about this, and he wasn't familiar with authpf; he did say it was similar to another sort of tool, which I didn't write down in my notes. I'm guessing that I missed something.

With that the conference was over for the day; my roommate used my CD to install Ubuntu on his laptop (I knew bringing it along would come in handy!).

Tags: lisa selinux

Best stackoverflow answer EVER

Can regexes parse (X)HTML?

Tags: funny imnotcrazyinstitution perl

Aw hell, it's more LISA coverage

(Turns out you need at least three good, verbose albums to come up with that many quotable lyrics.)

Thursday morning (November 5 2009):

While waiting for the room to fill up for the Planck telescope talk, I had a ponies moment and realized that Tobi Oetiker has the coolest Beatles haircut ever. That is all.

The Planck (pronounced almost like "plonk") telescope is going to give the highest resolution maps of the cosmic microwave background, and it's going to be dealing with a metric fuckton (my words) of data -- on the order of 10^12 observations, or 10^8 sky pixels, or 10^4 power spectra (which is where the really interesting data is). To do this, you need a metric fuckton of computing power, and that's NERSC...which, the presenter said, has gone from being a data producer to a data sink, as more stuff comes in to be processed. (Even that has changed; scaling limits and other constraints have changed the math that they use to analyze the data.)

To handle all this data, they use a variety of techniques and hardware:

• They've got 60PB of storage in 10 Sun Ultrium 4tape libraries (but as he said later, that's a made-up number based on maximum capacity; in order to maximize retrieval times, they use a mix of Ultrium 3 and Ultrium 4)

• A 130 TB disk cache (!)

• About 400TB of storage in GPFS

• "One of the tricks to doing large data is: don't use I/O." Fast I/O is great, but avoiding it entirely is better. One byte/s of I/O is about 1000x the cost of one FLOP/s. It's easier to calculate it and keep it in memory than to look it up again.

• Having common data models across the community of users, to avoid duplication/remunging of data; it's a social challenge as much as a technical challenge, but addressing it early pays off.

• And remember: data from observations and experiments tends to increase in value over time (due to new analysis techniques), while data from simulations decreases in value over time (as computing capacity increases).

One question from the audience: Do you use GPU computing? A: No; lack of ECC is the biggest reason. PCI speed also a factor, but we already deal w/different speeds in different subsystems.

After that came the presentation for Anton, which is a specially-built supercomputer for molecular dynamics simulations. It was an interesting talk, and I'll be pointing one of the faculty members I work with at the slides and paper when they're available. Top quote: "Our user community is faster than our monitoring system."

Tags: lisa

Capirca

Google has just relased a new firewall generatool called Capirca. I'm in the middle of the presentation right now and it's very exciting. It not only generated firewal ACLs for Cisco, Juniper and iptables but it also will VALIDATE them against netflow info. No support yet for OpenBSD's pf but they say it should be easy to add. And (correction) Apache-licensed to boot!

Ha! Slides here!

Tags: lisa

Soundtrack to Mary

Wednesday:

Many miles wandering from room to room
Many trees slain just to write it to you...

"Soundtrack to Mary", Soul Coughing

Wednesday started with a test of the Emergency Viva System. My roommate had to defend a thesis with the University of Manchester, and they'd told him they were going to do it over the phone today at about mid-morning our time. What they didn't tell him was that they were going to call at 5am our time to make sure the phones worked.

So I got an early start to the day. I wrote yesterday's entry, then wandered down to the lobby to get coffee from the coffee shop (which had a sign saying "Now serving...Oatmeal and Grits". Hurl) and a free cinnamon bun from a sweet little old lady (no, really) in a hotel uniform. I met Matt and Bob the Norwegian (#6, I believe), where we discussed:

• Bumblebee tuna (holy crap, that's creepy)
• Free Enterprise
• Gummi Bear theme songs (Bob the Norwegian has 8 different languages on his music player, and more at home)
• And this exchange:

Matt: That's it, I give up. I've got eye cancer.

Bob the Norwegian: You've got eye cancer? You're crazy.

Me: ...said the guy with the 8 versions of the Gummi Bear theme song on his music player...

Bob the Norwegian: 8 languages. I have more versions than that at home. Want to hear the ska version?

Me: ...so you're in no position to throw the crazy brick around this room.

And then it was...opening time! As it happened I grabbed a seat right up at the front, and noticed Dr. Werner Vogel, CTO of Amazon.com, standing at the wall a couple feet to my left checking his email and waiting to give the keynote speech. "Oh...hello. I thought you'd be wearing a suit." "Nah." Jeans, Harley-Davidson t-shirt, denim long-sleeved shirt untucked.

Highlights from Adam Moskovitz' speech (he's the organizer):

• David Blank-Edelman got the SAGE Outstanding Achievement Award
• 815 attendees, and probably more what with late registrations
• 35% of papers submitted were accepted

Very quick speech; he knows his stuff.

And then it was the keynote. Dr. Vogel was talking about Amazon Web Services. This was interesting and entertaining and fascinating and all kinds of good gubbins. Highlights:

• Hanging out in the bar (missed this, dammit) w/LISA attendees; when he told them his original speech, they said "No, no, no! Tell us why Amazon is doing cloud stuff. Isn't it a book store?"
• "I sometimes introduced myself as the sysadmin for a large bookstore. But that would be disrespectful here; your job is a lot harder than mine." (Flattery will get you everywhere.)
• "Cloud computing is not my favourite term these days, because it includes almost anything...especially stuff that fails."
• "By now, we've learned that if major business magazines say Jeff Bezos is crazy, we're good."
• "For the first five years, Jeff's motto was 'Get big fast.' That's not a motto you should give to your engineers."
• "For some reason we put data centres near trailer parks. And trailer parks attract tornadoes."

He gave the example of Animoto, which is a startup that figured out how to detect rhythm and melody changes in music. They use it to automatically generate slideshows using slides submitted by users, or grabbed from their Flickr album. They offer a 30-second snippet, and then you can pay $x.95 to get the full version.

He showed one that used photos of him at a conference, and I forget what the music was but it was very disco-y and made the thing jaw-dropping, both because of the cheese and because the thing was utterly, completely addictive.

He showed a graph of their orders; it was climbing slowly from April 16th through the 18th, and then they released a Facebook app on April 19th. The app would grab pictures from a photo album, compose the slideshow, then notify all the user's friends that they had something cool to watch.

The graph went exponential. They had 25,000 customers signing up per hour. Their conversion rate is astonishingly high, because they ensured that the slideshow was available in 5 minutes or less.

And they own no servers at all: it's all done with Amazon virtual machines. They went from using 50 machines to a peak of 3500. "They're just a bunch of guys in New York with laptops; they use Amazon as their server park. Can you imagine going to VCs and saying, 'Give us $5 million 'cos we're going to release a Facebook app'?"

I thought it was really, really well done and interesting -- aside from one pretty noticeable hiccup. However, others disagreed. The USENIX summary is here. When the recordings/slides are up, I'll post a link.

Tags: lisa

Sugar Free Jazz

Wednesday (cont):

 Put the fake goatee on
 And it moves as cool as sugar free jazz.

 "Sugar Free Jazz", Soul Coughing

During the break I got into a conversation with Ali and George about cfengine and Python. I recommended "Dive into Python", and George agreed; "There's no time for yet another 'hello, world!' programming book."

And then I met up with Noah from MIT. w00t! I hadn't known he was coming, but then on Monday he was called by the Rock Star Sysadmin o' the Year' contest guys, who asked if he was coming: "No, not in the budget this year." "Really? Are you sure you're not coming?" "Um..." So here he was. We ducked briefly into the GUru session on Zenoss, but it was not for us and we moved on to the papers session.

The first one was "Pushing Boulders Uphill: The Difficulty of Network Intrusion Recovery". And holy cow, they weren't kidding. The state of the art for intrusion recovery, as the presenter said, is wipe and reinstall from backups. Okay, maybe you can do that with one or two machines -- maybe even a few more than that. But what do you do when your system is massively compromised? When there aren't just some Code Red packets but when every single machine has a rootkit?

Reinstalling from backups is no longer satisfying, and yet no one wants to share solutions they might have come up with: "What, I should put it on my resume? 'Got pantsed in front of Slashdot.' I don't think so." So, without identifying the people involved, he shared the story for the purpose of "adding to the lore" (great term).

In a nutshell, an academic department at an American university had its gold server, from which they pushed updates to one thousand workstations, got compromised. Now the workstations had rootkits in them. They only found this out by accident when various processes were crashing in weird ways. And they found it out in the middle of December, right before exams and Xmas, right before half their IT staff was leaving for unrelated reasons. (You could hear gasps around the room as the story was told. Six of those were mine.)

So what do you do? Do you take everything offline and screw over the students? Do you reset passwords? They didn't know exactly when the compromise had occurred, so backups were out. That left reinstalling -- but with what? Same distro, when you don't know if it's vulnerable, or something else? How do you make sure it's all going to work? The state of the art addresses very little of this, and does nothing to help with the entirely reasonable gut-clenching panic.

(I admit I have not read the paper yet. But once I get some time, it's going to be one of the first.)

The second paper I tuned out of, only to hear Tom Limoncelli get up at the question time and say, "I think this paper is crazy. I think that's good, because LISA needs more crazy papers. But I wonder if you realize how crazy it is." The speaker nodded and said, "Oh, yes."

The third paper was a comparison of two big mail migrations...again, it had the feel of adding to the lore (a good thing). It was an entertaing story, well told, about how all the preparation they'd done had not covered every eventuality. The presenter mentioned that one of the reviewer's comments was "You must not have done enough testing." "And I thought: I know! I'm in the future now, too!" They finished their talk with a video of raised flooring packing foam air hockey...fun times.

During the break I talked to a woman who was attending the conference for free, in return for volunteering at the USENIX desk. She ran her own business, and with the economy tanking she'd had to lay off everyone but herself...which meant that she was the sysadmin, too. She has computer experience but no sysadmin experience, so she came here to learn. I sold her on joining LOPSA by talking about how much the mailing lists had helped me.

The talk on Eucalyptus was next, and man, do I have mixed feelings about this presentation. On the one hand, cool stuff: open-source implementation of the AWS API so that researchers can have an actual cloud (based on the only instance of a cloud that everyone agrees on) to do research. What could be wrong with that?

OTOH, the way this guy talked gave me the same feeling as when I read Marshall McLuhan: it's English, but not as I know it. The one example I wrote down (he spoke at about 300 wpm) was when he described a server as "an aggregated set of state updates." That said, my roommate (who's doing a Ph.D. in this sort of thing) thought he was brilliant, so I'm perfectly willing to admit I may have been out of my depth at times.

He was quite funny at times:

• "At the end of the first week after the release, there was a cadre of users who had root who wanted desperately to remove it from their machines." -- on the sysadmin-vs-researcher fight in grid computing (not the cloud stuff he's doing now.)

• "If you do an open-source project like this, people often want to tell you things. A lot. And they want to tell you at 4 am."

And one last thing: he said he was quite impressed with Amazon's API. He kept seeing cases where people would change the API, as Eucalyptus had implemented it, in an attempt to improve it; the changes would almost invariably lower the amount that Eucalyptus could scale.

The LOPSA meeting was that night, and it was interesting. They're up to about 500 members, but they need more -- partly to keep it growing and partly to get access to things like O'Reilly Safari. (The magic number for stuff like that is 1000 members.) They mentioned the ties they're making with other countries -- Australia, Ireland, a group in India, "and we've just been talking with someone who wants to start a converence in Vancouver."

Lightning talks! In the spirit of the thing, bullet-point summaries:

• mrepo -- update tool for RedHat I must check out
• selinux permissive domains -- not sure if this was the same as the targeted policy that Rik Farrow was talking about
• timestamps for web app -- guy from Yahoo saying that SSL depends on proper timestamps to prevent MITM attacks, and yet we're trusting the client for these...arghh! any ideas?
• openefs -- Trey Harris' project to keep software working by never, ever letting it change; a combination of symlinks and OpenAFS that's due to be open-sourced soon
• Beth's story of crazy
• Alva Couch asking if a Lessons Learned section for LISA would be good for next year; the whole room agreed. More about this later.

(If I've missed any, let me know.)

I talked to the organizer afterward and asked how many people he'd had sign up in advance; the answer was none, and he'd had to go after people in hallways to get them to present. I felt bad for not doing so...I had meant to but I got distracted. Next time, I will Do The Right Thing!(tm)

Rock Star Sysadmin of the Year award...first the good: both Matt and Noah got Finalist and Runner-Up awards (respectively). This is cool and all the winners are to be congratulated. There were cool prizes given out, and the grand prize winner donated his to charity. There was cake. Yay everyone!

Now the bad: my cheeseometer was pinned. As someone pointed out, the presenter looked like Guy Smiley; he had spiky marketer hair and was just smarmy. And the band, for reasons I can only guess at, was the pet band of a guy who's a cake chef/baker in Baltimore and has a TV show about cakes that he makes. I thought the music was awful (but then, Noah liked it a lot and he's the one with the sysadmin prize :-), but more than that it was loud. Fortunately I had earplugs or there would've been blood running out of my ears.

(No, you're old!)

Oh, and there were TV cameras (marketing material? next week's cake episode? memo to myself: must tape cake show) filming the women (who I think were there with the vendor but I could be wrong about that) dancing up at the front of the stage; what the cameras didn't show was that they were pretty much the only dancers up there.

There was an escape to the LOPSA suite. I signed up two more people, then headed off for the hotel bar with Noah and a few other folks. I meant to call it an early night, but that did not happen. Oh well.

Tags: lisa

Moon Sammy

Tuesday morning:

And I wondered with great admiration...

"Moon Sammy", Soul Coughing

I got up this morning to find that the weather was absolutely gorgeous; blue sky, sun, and a wonderful look to the part of Baltimore that I could see: church spires, ship's masts, brick towers. I took a short walk around the harbour and found a wooden clipper ship tied up close by. I was hoping I could get to the Constitution, but I think it was further off than I thought.

Back to the conference and to Tom Limoncelli's morning class on time management. I've already devoured his book (seriously, if you don't have it you need to; the link throws Tom a few shekels) and I was looking forward to his course. It was a new approach to time management, based on the idea of looking ahead at your day and treating it accordingly. A day filled with meetings would be focused on making those meetings productive; a day without meeting would be focused on focus itself, making the most of those (blessed, blessed) long stretches of time and handling interrupts.

Some of the material was straight out of TMMSA; after all, the basics are in there. Also, the course was only a half day and that limited the amount of material that could be presented, new or not. And much of the material was aimed, I think, at much larger departments than my own (which == 1), which did limit some of the applicability to my situation.

But. Tom is a wonderful speaker and presenter, and it's well worth going to his course if you haven't before. The course was packed, as was his afternoon course, and I saw at least one guy who was attending Tom's course for the second time. And there was some new material in there that I noted for immediate use.

Some quotes:

• On the inapplicability of other time management systems to our profession: "System administration is not like real life."

• On the problem of mentoring: "If your boss is technical, she can't give you advice; she's just as screwed up as you."

• On using the term "meeting" in his training to mean any large, immovable block of time: "A change window is sort of like a meeting with a router."

• "The benefit of a paper planner is it can't play games. I check my calendar with my iPhone, and...let's play Tetris! The paper planner does have a Tic-Tac-Toe implementation, but it's single-user...it gets boring, so I quit. It'd be worse if I lost."

• On limiting distractions at your workstation: "I don't know what IM client you're using, but I bet it has a quit feature."

LISA does this every year where you have to go around getting signatures from people; it's a good ice-breaker. In 2006 the organizers had their pictures on the card; this year, it was a scavenger hunt. You had to find someone who had, say, a LISA t-shirt on, or was part of the program board, or supported more than 1000 users. Ten signatures got you a spin of the prize wheel at the registration desk.

At the beginning of his class, Tom asked people from the audience for help filling out his card. ("The trick to doing the card well is to have a PA system. But we'll be talking about abusing power later on.") As it turns out, I was his tenth, since I have a Hallowe'en costume (the OMG PONIES shirt; I'm going to be Slashdot from April Fool's Day 2006). He got to sign my card (he's a vendor, since he's with the Google presence here), and he was my tenth. Card buddies 4ever!

At the break I went to spin the wheel; there was a woman in front of me who actually won a free prize to next year's LISA, which is damned cool. I got the "Jump To Conclusions" mat....no, but it is a little keychain where you press a button and one of three lights comes on: ACK, NAK and EQN. (Gotta verify what EQN means; enquiry?) It's cute.

And during lunch I actually went and napped. I've been up late and up early every morning this week -- there are just so many people to meet here! -- and I'm starting to feel it.

Tags: lisa

City of Motors

Tuesday afternoon:

And I hear a rumbling
I hear transmission grind
I bear witness
I have the clutch now...

"City of Motors", Soul Coughing

Tuesday afternoon was another Tom Limoncelli class for me: "Design Patterns for System Administrators". I think of design patterns as being a step above algorithms in the abstraction scale. (Tom told us that the term was first used in architecture and city planning; I need to add the titles for the books and maybe look them up too.) DP was a way of capturing passive knowledge: the knowledge you only get from experience.

The course was interesting, and I will be keeping the slides handy for future reference. It was also crowded -- there was not a free seat in the house. However, some of the material was already familiar from Tom's books, and some of it just did not apply to me because it was aimed at much bigger departments.

At the break I talked with Ludmilla, who managed to cram into my brain a better understanding of cross-site scripting attacks; this has always been a mysterious subject to me.

Stopped by the LOPSA desk to ask if they'd be interested in helping me at all with my (still vague and nebulous) sysadmin conference for Vancouver. They pinged the IRC channel (horrible mix of metaphors) and said sure, send an email. We talked about some upcoming changes on the LOPSA website, and I suggested sending a feed to planetsysadmin.com

For supper I headed out to a nice Italian restaurant with a few folks. I heard complaints about Red Hat support; an upgrade from RHEL 4 to RHEL 5 produced massive disk corruption on their SAN. Red Hat and the disk vendor pointed their fingers at each other for a year. Finally the disk vendor came out with a beta/testing firmware upgrade, which fixed the problem, but a final release has not come out yet. He's left deeply unimpressed with RHEL support: they were paying buckets of money and were left in the lurch. And I've heard that from a number of people here.

We got back late, so we hung out in the hallways talking to folks. I ended up talking to a sysadmin from the University of Alberta who, it turns out, can practically touch the OpenBSD FTP server from his desk. He talked about a move on the campus to switch to Google Mail for the entire university.

This was controversial a while back, when Lakehead University in Ontario tried it; one of the groups on campus (teacher's union?) sued because they said it violated privacy restrictions to place their email w/in reach of the Patriot Act. So I was surprised to hear that they were giving it another try. THere were two things that made this a not-wasted effort: first, apparently Ontario's privacy commissioner had ruled that email is just not private, so it was okay. The second is that UofA has invited the Alberta privacy commissioner to participate, so they're hoping to avoid any problems from the start.

So why are they doing it? First off it's free; Google gives it away to universities. Second, there are something like thirty separate email systems at UofA and no unified calendaring system. These are good things but it's interesting to hear of a university-wide concern about this; UBC is balkanized/decentralized to the point that implementing a campus-wide system like this would be pretty much a non-starter.

After a while I headed up to the LOPSA suite. One of the members said, "Hey, are you the Vancouver guy interested in starting a convention there? How would one or two speakers work?" Cazart! I made it clear that it's still in my head and I don't know what I'm doing...but OTOH a recent IT re-organization at UBC means that HR there is interested in making a clear career path for IT folks there, both in the central department and the individual faculties, so they may be interested in helping with this. And of course, university == cheap space in the summer. Anyhow, it's all early days and I still need to email them to remind them, but still...woot!

And then there was the guy who drove five hours after a regular workday to get to LISA. He'd come up on his own dime to organize a BOF but more importantly to make contacts; he's unhappy at his current job and wants to jump ship. "Man, I'm gonna stay here as long as it takes and if I gotta drive all night to get back at 9am, I'm doing it."

Well, I'm here to tell you that within THREE MINUTES he had two different guys fighting over him ("What's your specialty?...Damn! Yeah, talk to that guy...dammit, dammit dammit...") It was the feelgood story of the evening, and he was a damn friendly guy to boot. And when I left for the night, he was talking to Bill Lefebvre ("Hey, do you know who this guy is? He wrote top!").

I worked my magic (hot-cha!) throughout the night; persuaded Matt (almost) to join the FSF, and one of the 8 Norwegian sysadmin's I've met to join LOPSA (on sale! $10 off the rest of the week!). I asked Tom Limoncelli about my idea for training on "The n things a sysadmin must know about development"; he thought it was a good idea, suggested I look at the open-source tools that exist to help w/the situations I described, mentioned that Strata Rose-Chalup had pitched a book about this (but sadly the deal fell through), and suggested I get experience doing training, and doing training on this, by volunteering at my local LUG.

Finally, I spent a good bunch of time -- in both senses -- talking to a manager about what the appeal of the job was for him. He confirmed what the tutorial instructor had said: it is really, really neat to help people improve, to make the environment that allows them to do that and keeps them happy, and to see them get better and climb the ladder. It's not always easy and there are not-fun, difficult decisions to make, but the rewards are there.

I asked him if he'd always known he'd want to climb the ladder, or if this was something he found out later on. He thought a bit, and said that when he was younger he'd had a false sense of what was important; that not having a family had allowed him to fucus on tech fun to the exclusion of all else. Now that he was older and had kids, the long nights spent on tech was shifted to family, and his focus had switched to helping his team -- which was much more rewarding.

Tags: lisa

Screenwriter's Blues

Monday night:

Los Angeles beckons the teenagers to come to her on buses
Los Angeles loves love

It is 5am, and you are listening to Los Angeles.

"Screewriter's Blues", Soul Coughing

Monday I met up with Donny and Ludmilla for supper...and who's there but Tobi Oetiker! Another chance for geekish hero worship, hurrah!

After thanking him for MRTG and RRDTool, I asked him what had happened to the call centre he had spent all that time debugging. He said that it was kind of in limbo: the troublesome app had been replaced by a web-based app and was slowly being rolled out...but since it didn't do everything the old app did the old one was being kept around and people were reluctant to upgrade. But because the old app was on the way out, no one wanted to spend money tracking down the problems with it. I have to say, I expect more neatly wrapped-up story endings from the people I admire. :-)

Also along were Walter and Kyle, two sysadmins from Boston's TERC. This was handy, because Kyle had lived once in Baltimore and was able to take us to DuClaw's brewpub, which was not too far from the hotel. The sampler included about 10 different beers:

Despite being from the German-speaking part of Switzerland, Tobi was not interested in drinking the beer, but appeared to be fascinated by the interest we took in it. Crazy Swiss, what are you gonna do?

Tobi also talked about coming to love JQuery and qooxdoo. Everyone kept asking him to repeat that name, and finally he wrote it down while we guessed how it was spelled. None of us were right, because we'd all been guessing crazy Dutch-German variations.

Kyle and Walter talked about their setup a bit. They're in kind of the same boat I am in that (being at an educational institute) funding is erratic yet the results (websites, curricula, etc) need to be around forever. Thus, they still have an NT4 web server which was only last month migrated to a VM. (Walter dulled the pain by asking the bartender to make him something sweet with rum. The procedure had to be repeated once, but then he was good to go.)

After that, we headed off to the James Joyce pub where OpenDNS was engaged in a COMPLETELY FUTILE attempt to gain my good will by buying the entire bar drinks all night. (Futile, do you hear?)

I didn't get to meet the OpenDNS folks, but that didn't stop Ludmilla from pasting OpenDNS stickers on everyone's shirt. And I did get to talk to another Norwegian sysadmin.

So he works for a Norwegian newspaper, whose website half of Norway starts their morning. (Apparently he went to a talk (previous LISA?) where Facebook was talking about their traffic levels; Facebook's traffic was less than their own and they used 1/5th the number of servers Facebook did.) They were using Squid in front of their webservers, but were looking for something to do better. Commercial/proprietary options didn't measure up. What to do?

Well, like any good Norwegian they decided to bring in a fellow Scandinavian. After determining that Linus Torvalds was not interested (not entirely sure how serious that part was), they asked Poul Henning-Kamp if he was interested; he wasn't. "I'm a kernel guy with 20 years of experience doing kernels," he said; "I'm just not interested in doing application work."

But then he comes back two weeks later and says, now that he's had some time to think about it, he is interested in the idea of a caching app that exploits the underlying OS to the hilt. N months later, Varnish was ready to go.

They roll it out at a big news conference, with The Register and others attending. Boss gives a speech while they watch the graph of request latency scroll across the screen; they throw the switch. The line go down from 300 ms to 30 ms and stays that way.

Also met Dan, who works for the U of Kansas Center for Remote Sensing of Ice Sheets. "I keep wanting to go down to Antarctica, but they keep not sending me there."

Tags: lisa beer

Blueeyed Devil

Monday afternoon:

Born to be a god among salesmen
Working the skinny tie
Slugging down fruit juice
Extra tall, extra wide

"Blueeyed Devil", Soul Coughing

Lunch time I talked with a gov't contractor who was in on the Hadoop tutorial. She talked about using a filesystem that was forty years old -- yes, that's a four zero -- which had lots of "warm" data (her term; I assume between hot caching and archiving to tape) cached to tape, but done very badly. The directory structure needs to be preserved, perhaps not at all costs but nearly; there are instances of old (maybe not 40 years old but close) documentation that refers to old paths that must not be broken. Interesting problem.

Also heard about this problem, which just gobsmacked me with its fullbore crazy.

Also, from other quarters, heard about a lab that lost its funding, which leaves it in a difficult position as it has a crapload of old G4s or G5s, watercooled, about half of which they discovered are leaking...

(Trying not to turn into Perez Hilton here. Not sure how well I'm doing.)

In the afternoon I took the Packaging for Sysadmins tutorial, which would have been much better (IMHO) handled as a hands-on workshop. I came back for the second half, but honestly it was a close thing...and yet when someone asked him, the instructor dropped gems of info about Func and Cobbler, which I'm going to be looking into as soon as I can.

During the break I talked with Derek, who's a sysadmin at a NYC trading firm. This was an absolutely fascinating talk, and only partly because I wasn't really aware of the whole high-frequency/low-latency trading...um, culture? algorithm? So:

• He has small data centres -- like, racks -- scattered across NYC in order to be rilly rilly close to the exchanges. Also works well for redundancy. The colos that are close to the exchanges are filled with fellow trading firms.

• The idea is that if you get your data from the exchange soon, analyze it soon, then get an order back to the exchange soon, you can make a lot of money. As a result, a 2.5 ms difference, like in swimming or 100-metre dashes, is absolutely huge.

• Improvements in speed are looked for all over the place: RT Linux, not running NTP on machines (partly because of the overhead it introduces, and partly because it doesn't have high enough resolution; better to take timing info from ethernet frames, which'll get you down to 7.03 nanosecond acccuracy), RT Java (which I didn't know existed), and even running apps on switches that run Linux (which, yes, may be slower than big servers, but are that much closer to the exchange and so it makes up for it0

• So his server rooms are small-ish and many (which if I was a better man I could turn into a full-on Dr. Seuss book), but get this: a trader's desk will have four workstations at it, each with four big-ass monitors sitting on top of their desk so that they can monitor the stocks they're trading. His power and cooling issues are at the desktop as well as in the server room. Madness.

After that back to my room, where my roommate (who's British) and I wondered at the madness of looking to the UK Conservatives for relief from a right-wing Labour agenda. Madness upon madness.

Tags: lisa

True Dreams of Wichita

Monday morning:

I've seen the rains of the real world come forward on the plains
I've seen the Kansas of your sweet little myth...
I'm half-drunk on babble you transmit
Through your true dreams of Wichita.

"True Dreams of Wichita", Soul Coughing

This morning I had the SELinux tutorial, held by Rik Farrow. I took a moment to shake hands with Rik Farrow, who's teaching this class, and tell him that ;login: magazine, like, changed my life, man, you know?. If you haven't picked up copies of that magazine/journal, you owe it to yourself to do so. (And if you have and you agree with me, send him an email -- he usually only gets email as editor when there's a problem.)

Matt was there, as was Jay, who I met back in 2006.

The course was quite interesting. Some choice bits:

• "How many of you are using SELinux?" (Two hands) "How many of you have disabled SELinux?" (a hundred hands and six tentacles; yes, even Cthulhu disables SELinux) "See, that's why I came up with this course; I kept seeing instructions that started with 'Disable SELinux' and I wanted to know why."

• Telling Matt about Jay's firewall testing script.

  Me:  So how to the big guys test their firewall changes?
  Matt:  I dunno...probably separate routers, duplicate hardware...
  Me:  Probably golden coffee cup holders, too.
  Matt:  Jerks.

• You don't write SELinux policy. SELinux policy is hard. It's NP-complete and makes baby Knuth cry. Instead, you use what other people have written, and make use of booleans to toggle different bits of policy.

• However, the size of the SELinux policy is big and is only getting bigger. There are something like 85,000 or more rules in recent versions of RHEL/CentOS. This is very close to RF's rule of thumb that a really, really smart and experienced person, who's been intimately involved in its creation, can only comprehend about 100,000 lines of code. This worries him.

• Also, the problem of using SELinux is complicated by a lack of up-to-date documentation; like everything else it's a fast-moving target, and a book published in 2007 is now half out-of-date.

• But this should not stop you from using SELinux now,; it's handy, it's here, get used to it. Example of SELinux stopping ntpd from running /bin/bash; the SELinux audit file was the only sign.

• "In a multi-level secure system, files tend to migrate to higher security levels, and the system becomes less unusable. But that's beyond the scope of this class."

• (On programs with long histories of serious security problems) "Flash is the Sendmail of -- what do we call this decade? the naughts?"

• (On the difficulty of trying to decode SELinux audit logs) "It says the program 'local' had a problem. 'Local'. What the heck is that? Part of Postfix. Oh, good. Thanks for the descriptive name, Wietse."

• Something I hope to quiz him further on: "Most Linux systems have a single filesystem." Really?

During the break I met a guy who works with the Norwegian Meteorological service. This was interesting. He's got 250TB in production right now, and increasing CPU power means that their models can increase their spatial resolution, which means increasing (doubling?) their storage requirements. He talked briefly about running into problems with islands of storage, but I got distracted before I could quiz him further...

...by his story of building a new server room where they were capturing the waste heat and using it to heat the building. Interesting; what kind of contribution would it be making to the overall heating budget? Probably not much, but it all just goes on the grid anyhow, like the hot water from the garbage dump. What?

Turns out that there is a city-wide network of hot-water pipes that collects heat from, among other places, water heaters powered by waste methane from rotting garbage. So they don't use the methane to make electricity and dump it in the electrical grid; they use it to heat hot water and dump that in the hot water grid, consisting of insulated water pipes buried in the ground, which places around the city (and beyond!) will use. We've got what you could call a steam grid at UBC and probably other universities, but I'd never thought of doing this city-wide.

Oh, and he signed my LISA card, which was the second time he got asked today; he was wearing a LISA t-shirt and so he was fair game.

At lunch I buttonholed Jay a bit. I asked him about his coworker's firewall unit testing scheme. He said he's no longer working at that place, but it ended up being a lot less useful than they thought it would be. When I asked why, he said that 90% worked but 10% didn't; that 10% was things like network isolation (to avoid problems with using real IP addresses), and the fact that the interface to the three machines was QEMU serial connections...less than ideal.

The conversation shifted to firewalling, and another guy who was there mentioned that he loved OpenBSD's pf, but had to use iptables because of driver problems that prevented getting full performance out of 10GigE NICs with OpenBSD. Jay said they'd looked at the same problem at his place o' work, and in his words "It was cheaper to throw 8 GigE NICs in a box and pay someone to make Linux interface bonding not suck."

Tags: lisa openbsd selinux

Super Bon Bon

QOTD:

Some kind of verb, some kind of moving thing
Something unseen, some hand is motioning to rise, to rise, to rise

Too fat fat, you must cut clean
You gotta take the elevator to the mezzanine
Chump change, and it's on, super bon bon
Super bon bon, super bon bon...

"Super Bon Bon", Soul Coughing

Tonight was a great deal of fun. I met up with Matt, who had invited me out for Turkish food earlier. I found that the group also included Tom Limoncelli and Doug Hughes, who is one of the Invited Talks coordinator and a very fun guy to boot.

We walked maybe 20 minutes across town to Cazbar on North Charles Street, and which I can recommend to anyone wanting good food. I had a lovely lamb and mozarrella Pide (like a pizza but more ethnic :-), did not like the Raki, but enjoyed the Sierra Nevada well enough.

Lovely food and fun conversation...like the guy who needed a Windows box to run Dell monitoring software, but decided to replace Explorer with Blackbox window manager and some kind of Apple Spotlight-like tool for Windows. My jaw dropped. "You've come this close to making Windows enjoyable for me."

After settling up the bill (non-trivial with 20 people, but we made it) we walked back again. I got to talk with Tom, which was neat (see 2006 entries from LISA re: accidental stalking); always fun to indulge in a little bit of hero worship.

Me: Oh, check it out: it's the Barnes and Noble store! Let's go party there!

Tom: What?

Me: Yeah, I've heard all about it! Free tequila shots at the door, cashiers dancing on top of their tills, DJs 'til 10am...

Tom: Oh, you're thinking of Borders.

I got to see the USS Constitution, which since I've been devouring the Master and Commander books over the last year or so I simply must visit. (Don't know when exactly...)

And so back to the bar. And so to bed. (tm Samuel Pepys.)

Tags: lisa dell

Sleepless

QOTD:

I got the will to drive myself sleepless
I got the will to drive myself sleepless
Sleepless....

"Sleepless", Soul Coughing

That time is how I feel, not the time it really is; not only is it Easter but it's Standard time, not DST, which means that the change caught me off guard this morning. I woke up my roommate thinking it was time for us to shift our asses, but no such luck. Oh well.

(Turns out that alarm clocks these days, at least of the sort that were developed for the DOD and have been provided under NDAs to major hotel chains, have a switch on the bottom for DST adjustments with three settings: On +1, Off, and Auto. That is one of the best ideas ever.)

8:35am and registration is good; I've got a cool IPv6 sticker and a copy of all the training material on a USB stick I'm going to try hard not to lose.

First day's training is an all-day course called "Management Skills, or Don't Panic!". It's not the sort of thing I'd usually sign up for -- soft skills, avoidance thereof -- but I figure it's probably a Good Thing for me to do, like exercise and eating right. It's interesting; there are some good anecdotes and quotes in there:

"How do you deal with a visionary-type manager? How do you get him to support your project?" Audience: "Tell him you read it in a Neal Stephenson book."

At the end of the course I had a question: I'd taken this course defensively, in order to pick up some skills that I lack -- but I enjoy the technical side of my job very much. I enjoy learning new things, but the problems involved in management seem best, to me, enjoyed in the abstract and at a distance. You give up your techie skills and joys; what compensating joys are there?

She had two answers. The first joy was seeing, and helping, people develop skills and at best exceed their teacher. The second was the fun of finding the problems that lay in organizations' way, no matter how many disparate groups or layers they might span (techies, mgt, suppliers, finance, cultural), and talking with those different groups/layers in order to solve those problems.

As I said, it was interesting. I'm still not entirely sold on management...but then there's the example of a friend of mine who's been doing this since '92. In a lot of ways, when it comes to technical problems he's been there and done that...so management is a (possible) way to keep it interesting for him.

On another topic: Lunch time I got into a very interesting discussion with a woman who figures that MS will lose majority market share on June 30, 2011. Her reasons:

First off, it was a two-year prediction made at a conference in June; had to come up with some kind of date. But also, MS only has majority market share in web browsers, PC OS and office suites. Of those, she figures the stats for web browsers are cooked for marketing purpose, and says that there is very little actual independent, large-scale data; however, data from W3 Schools shows increasing FF share. PC OS: less and less important as people move to Google docs and Gmail, which let's face it are plenty good enough for most home use. And the increasing ability of OpenOffice and other tools means that the domination of office suites is on the way down to.

Check out (her own? not sure) website at http://www.whatwillweuse.com.

After the course I met up with Matt and finally got to put a face to the face. He was there on Official Usenix Bizness(tm), as he's blogging for LISA and wanted to interview the instructor. Very friendly guy who's doing a lot to spread his knowledge around. And as it turns out he also got bit by DST, though worse than me. Poor bastard...

Tags: lisa