Sweet Odin's Raven!

I've just come across AsciiDoc, and this is SO CLOSE to what I want: Ascii-based markup, still intelligible, and rendered into pretty CSS-compliant whatnot.

For a while now I've been toying with the idea of leaving WordPress behind and just writing all my stuff in Emacs, the way RMS intended, and converting it all to pretty HTML through <handwave>some sort of script or Makefile</handwave>. But this...this is perfect. See this? If it were a black monolith orbiting Jupiter, I would say "My God, it's full of stars!" It's clean, it's spare, it looks good, and it does not require verdammnt patching to stop it from throwing in br tags every time it sees a newline. And you know what it requires? Python! That's it!

I know what you're saying: this is like wiki markup without the wiki. EXACTLY! It's easy to write, easy to read, it looks good and it's just static: no PHP remote inclusion waiting to happen, no heavy load, just simple plain text and html. Oh yes.

Tags: meta

There are certain things I would like to never re-compile again

• Atlas
• ACE/TAO
• Windows 2000

Actually, I shouldn't complain about that last one; MS actually uses Portage. Who knew?

Tags: windows

WWW::Mechanize and the values of testing

One of the great things about going to LISA is that you get the proceedings and/or training for everything on CD or dead tree. (Well, nearly everything...I've heard that some people didn't or couldn't make their training materials available (though I've not been motivated to confirm this yet), and some of the talks didn't do this (Tom, where are your slides?)). There is some wonderful stuff to be found in them...

...like WWW::Mechanize, which is just perfect for testing out this conference registration form I'm working on. Only I've run into a bug that comes when trying to specify which button to click on:

$agent->click_button(value => 'Okay to submit');

That li'l chunk gave me this error:

Can't call method "header" on an undefined value at /home/admin/hugh/perl/lib/perl5/WWW/Mechanize.pm line 2003.

One guy reported the same trouble, but got no response. And the RT queue is fulla spam.

But aha, I found out how to use the Perl debugger in Emacs (M-x perldb. Shhhh!) and was able to track things down. Turns out there are a couple things going on:

1. In the page that I'm parsing, there are actually two forms, not one; one sends you back to correct mistakes, one sends you forward to keep going. Since I was not specifying which one to use, it used the first...and in that one, there is no button labelled "Okay to submit". Once I specified the right form ($agent->form_number(2);) everything was good.

2. But of course, this sort of thing shouldn't happen, right? Right.

There are a couple subroutines/methods in this module that aren't testing for the right number of arguments. One of 'em is click_button, which has this loop:

    my $request;
    .
    .
    .
    elsif ( $args{value} ) {
        my $i = 1;
        while ( my $input = $form->find_input(undef, 'submit', $i) ) {
            if ( $args{value} && ($args{value} eq $input->value) ) {
                $request = $input->click( $form, $args{x}, $args{y} );
                last;
            }
            $i++;
        } # while
    } # $args{value}

    return $self->request( $request );

No test/case for not finding a button named whatever, so it just blithely returns $self->request( $request ). But of course, request does the same thing:

sub request {
    my $self = shift;
    my $request = shift;

    $request = $self->_modify_request( $request );

    if ( $request->method eq "GET" || $request->method eq "POST" ) {
        $self->_push_page_stack();
    }

    $self->_update_page($request, $self->_make_request( $request, @_ ));
}

Again, no test for the right number of arguments. And having just read the Test::Tutorial manpage, I'm all about unit testing and such, baby.

Tags: perl lisa

Streets of Fire

Come on, come out of the rain.

You're not oppressed, you're just too learned...

"Streets of Fire", The New Pornographers

Friday afternoon, a bunch of us were standing in the lobby. Jessica came by and said she was having problems getting into her home machine to get her boarding pass info. She was using the business centre, which only had locked-down Windows machines with no SSH client. The wireless was $87/hr or some such, and the free wireless set up by Usenix was way the hell over on the other side of the hotel. She was just about resigned to get up and go when a guy beside her piped up and said, "Hey, there's this tool that should help you out..."

"So I use it," she said, "and it turns out it tunnels SSH over DNS. It was the slowest connection I've ever used, but it was usable, and I got into my home machine."

I looked at her with wide eyes. "Was that...was that Dan Kaminsky who helped you?"

"I dunno," she said, "I've never meen him before. What does he look like?"

Normally I suck at descriptions, but I had this one down. "He looks like Brendan Frasier," I said confidently.

She shrugged. "I dunno, I don't think that was him...oh wait, there's the guy there."

We all turned to see Dan Kaminsky grinning. "That's one of the few times I've seen that tool actually be useful," he said.

Turns out he's a very friendly and funny guy, and if I heard him right he was roommates with the guy who started Friendster, who Jessica also knew. I foamed at the mouth for a bit in fanboyish wonder, then told him about IPoD and William Shatner's rap of the "Friends, Romans, Countrymen" speech from Free Enterprise. And of course, he wore the tracking monkey:

After that we split up for a bit, then re-united for supper. We hit FIXME, where we found a cute Mongolian waitress ("How many times can you say that?" asked Andy) and Bill Clinton burgers. We hit The Angry Inch in search of Angry Ale, which they no longer sold. Andy bought a t-shirt ("I'm never coming back to this place. And the last time I said I wasn't coming back to a place, I bought the place a round. This is cheaper").

Then we headed back to the the final LISA party. It was in the original hotel building, and it was the biggest goddamned suite I've ever seen. It had to be bigger than any two apartments I've lived in put together. There were lots of people there. I drank toasts with Wout (Cisco IT guy from Belgium; friendly, funny and BEST NAME EVAR) and Noah to Strata Rose Chalup, drinking this godawful Romanian plum moonshine...oh god, it was harsh. I spent a good 15 minutes with one of the board members of LOPSA trying to figure out the purpose of one of the suite's alcoves (we were stumped). And natch, I got more pix of the tracking monkey with William Lefebvre (top, 'member?):

and many, many others.

Eventually it came time to go home, so I said goodbye and collapsed in my suite.

Quotes I missed earlier:

• "I've got one user who considers 7-bit ASCII a luxury compared to what you can get from 5 or 6 bits."
• "'Cooperative collaboration.' Yeah, it's part of our vision statement."
• "90% of being a sysadmin is typing at computers. 10% is typing at users." -- Mark C.
• Pretty sure I saw Theodore T'so grinning (or at least had a confused look on his face) when I did Logan's Run.

Tags: lisa

Chump Change

I stole a page from your book, and a line from your page

And flew into a lesbian rage...

"Chump Change", The New Pornographers

Friday morning was Dan Fucking Kaminsky's talk, which I'd really been looking forward to. I dragged Ricky to it, telling him he rilly rilly needed to go, kthxbye.

My notes could not possibly do justice to his presentation, which was both funny and awe-inspiring. Anyway, Dan also makes the best slide shows I've seen; they're a whole textbook on their own. Go read all his stuff. And go see him talk! He's intelligent and friendly on rye bread.

Some random observations/quotes:

• When he takes questions from the audience, he thinks about what they're saying for a minute before replying. One question prompted the reply, "Man, you're gonna put me on an entire research path."
• When he mentioned the (FIXME: include link) auto paper generation tool, he described it as "a fuzzer that exploited a conference."
• On why SSH host keys suck: "You're looking at a bunch of random characters, comparing them, and if you're one character off that's it. One character off!
• On how easy it is to include a bank (say) login form, so that you end up posting to an HTTP form instead of an HTTPS form: "My grandmother could do that. She'll 0wn you." (laughter) "You laugh, but she's been to the last three Black Hat conferences. Have you?" (Note: I had originally conflated this point and the last one (SSH keys), but Wout set me straight. Next time I promise not to take so long to make corrections. :-)
• "Remember, there's nothing a bank wants less than hearing from you or seeing you in person."
• "Humans seem to have hardware acceleration for remembering names."

Ricky allowed as how Dan Fucking Kaminsky might have been worth getting up early for.

Okay, but after that the bitter pill of (FIXME: full name, title) Dmitri. This was a depressing, scary talk about network threats and how they're driven by very, very successful criminals. I'd heard this before, but the facts and stats he brought in were enough to just crush your soul.

The usual list:

• Spammers probably want anti-spam companies to stay in place. That protects the channel they're abusing. Otherwise, like a parasite that goes too far, they'll end up killing the thing they're exploiting.
• Most trojans/bots/whatever just ask the user to click...and it works. You don't need to go looking for a zero-day if you don't want to.
• Trojans that are sent out in small numbers will almost certainly never get sent to an AV company for analysis...which means AV software will almost certainly never detect it.
• Speculation about future uses of zombie networks: distributed computing, or distributed file systems.
• Image spam is already defeating many anti-spam programs. And to get around it, it wouldn't take much more than something like this: (FIXME: Add image)
• Many/most zombie networks will be active (spamming, say) for a few hours...then go silent for another month. Good luck trying to detect that.
• Some zombies will check blacklists before spamming, to see if their IP is listed. If the list supports it, they'll submit a request to get their IP delisted.
• His company is working on a way of filtering traffic, not just email, based on reputation. Push the responsibility to the user: your bank says "We're not accepting traffic from you because your IP address has a bad rep."

Dan Kaminsky asked if maybe the answer was to abandon persistence on the desktop, and just hand out Knoppix disks to everyone. Dmitri replied that would just push the attack to web databases and such that held the user's settings. DK pointed out that would mean a much smaller number of machines to secure, which Dmitri conceded.

Q: I work for a web farm; what can we do? A:: watch your netflows carefully and learn your normal traffic. (cf Dan Klein's presentation).

Q: I use fuzzy OCR plugin for SA and it works fine. A: you might not be seeing adaptation yet, but you will. OCR is bound to fail; too easy to trick.

He closed his talk by saying the obvious: he's very, very pessimistic, he sees no magic bullet, and he can't see any light at the end of the tunnel.

Tags: lisa

The Laws Have Changed

Introducing for the first time, Pharoah on the microphone!

Sing: All hail what will be revealed today

From the fear of the great unknown, from the line to the throne.

"The Laws Have Changed", The New Pornographers

Thursday night was the USENIX Carnival Of Fun: lots of carnival games that got you more tickets for the door prizes (which were a huge pile of No Starch Press books plus a Monty Python box set). I wandered around for a while, looking at the huge crowd and fighting the temptation to run to the balcony and shout, "Carousel is a lie! You can LIVE!"

I talked for a while to a woman I'd been running into the whole week, a sysadmin at a defence contractor. She had been to Andy's talk as well. One difference between her job and Andy's is that she's responsible both for classified and unclassified networks. One effect of this is that she's able to contact more people for support...but there are limits.

For example, she had to send off logs from one app that was failing to the vendor for them to pore over. The app was on a classified computer; she was forbidden to copy any data from that machine directly to an unclassified network, so that meant no SSH, no ftp, no USB disk, no burning of CDs, nothing. What did she do? She printed out the logs, verified that nothing in there was classified, then put them through a scanner and used OCR to munge the images back into text.

Later, an engineer from another vendor came to poke at an app running on an unclassified computer, and it was her job not just to supervise him, but to run the big K-Mart Special flashing blue light to let everyone around her know that there was someone without clearance in the room, and to watch their mouths and adjust their monitors appropriately. In other situations, she's had to sit at the keyboard and type what the engineer told her to...because without clearance, you're not allowed to touch the machine.

I wandered on, and picked up a tracking monkey. There was a security consultant with a huge bag of stuffed monkeys that were meant to wrap around your arm or shoulder or something. I couldn't make that work, so I wrapped it around my neck. A little tight, but it was worth it: when people would ask what it was or where I'd got it, I'd fix them with a stern look and ask suspiciously, "Where's your tracking monkey, citizen?"

Eventually I hooked up with Noah (CSAIL) and Deb (FSF). Deb made us smack things (Noah won the strength test) and throw things (she cheated at skeeball, but I managed to win another ticket so that was okay). When the draw came over I dragged over Ricky the Bostonian/iite/aniananan for luck, since at least 8 people who'd been w/in 70 feet of him had won. However, turns out his luck function really peaks at 70 feet, and at 4 feet away it's pretty minimal. Oh well.

We went to check out the Google BOF, but on the way out Deb dared me to play Logan. I dragged her up to the balcony overlooking the ball room and yelled my line, but sadly it got lost in the noise. The lineup for the Google BOF was insane; someone told us that they were giving away a MacBook Pro. <post-hoc rationalization> We decided to form a Bass BOF and headed to the bar.</post-hoc rationalization> (Sorry I couldn't make your scotch BOF, Jessica!)

There was massive talk about salting the cod (which just sounds like the best euphemism anywhere, and I really want everyone to pick up on that, so go!), places to drink in Boston (incl. one place that has 100 beers on tap), and many, many other things. After a while we headed to the LOPSA room, where a lot of people ended up. I talked briefly to Andy, the guy who talked about Command and Control:

I got a lot of pictures with the tracking monkey, including Tom Limoncelli:

and dkap and Melanie Rieback:

And when the night wound down, we went back down to the bar to verify that their supplies were still good. (They were.) Man, it's been a long time since I've closed a bar. :-)

Tags: lisa

Electric Version

Sound of tires, sound of God...

"Electric Version", The New Pornographers.

Thursday morning came far too early. My roommate offered some of his 800mg Ibuprofins, and I accepted. First thing I attended was the presentation "Drowning in the Data Tsunami" by Lee Damon and Evan Marcus. It was interesting, but seemed to be mostly about US data regulations (HIPPA/SOX et al.) and wasn't really relevant to me. I had been expecting more of an outline of, say, how in God's name we're going to preserve information for, say, a hundred years (heroic efforts of the Internet Archive notwithstanding). There was mention of an interesting approach to simply not accumulating cruft as you upgrade storage (because it's easier than sorting through to see what can be discarded; "Why bother weeding out 200MB when the new disk is 800GB?"): a paper by Radia Perlman (sp?) (she of OSPF fame) that proposes an encrypted data storage system (called The Ephemerizer) combined with key escrow that, to expire data, simply deletes the key when the time is up. Still, I moved on before too long.

...Which was good, because I sat in on Alva Couch's presentation on his and Mark Burgess' paper, "Modelling Next-Generation Configuration Management Tools". Some very, very confusing stuff about aspects, promises and closures -- confusing because the bastard didn't preface his talk with "This is what Hugh from Vancouver will need to know to understand this." (May be in the published paper; will check later.) Here's what I could gather:

• System administration could be described as the Pinky and the Brain problem: "What are we going to do tonight, Pinky?" "Same thing we do every night, Brain: try to take over the world!"
• IOW, the problem is too big -- and in the meantime you have all these competing theories (aspects from Luke/Puppet (I think), promise theory from Burgess (which I had heard about) and closures from the bcfg2 people) that need to be integrated, but currently aren't.
• Many tools model/modify configuration, not behaviour -- and implicit in there is the (unproven?) assumption that correct behaviour emerges from correct configuration as if by magic. There is no understanding in cfengine of outside forces.
• A promise, in sysadmin terms, is promise to do something. For example, an NFS server promises to make certain files available over the network. A client mounting a disk from the server promises to access some of those files.
• Closure is the whole of the problem: in the case of the NFS server, it's DNS plus routing plus mountd running plus nfsd running plus proper ACLs (which I only found out at this conference that nearly everyone pronounces "ackles" rather than "ay see ells").
• His model: closures encompass promises encompass aspects. By dividing up the problem this way, you no longer have to take over the whole world.
• His model accounts for site policy by designating it a soft aspect.

I will do the right thing and read his paper, and I may update this later; these are just my notes and impressions, and aren't gospel. Couch is an incredibly enthusiastic speaker, and even though I didn't understand a lot of it I ended up excited anyway. :-) He gave another talk later in the week that Ricky went to, about how system administration will have to become more automatic; as a result, we'd all better learn how to think high-level and to be better communicators, because more and more of our stuff will be management -- and not just in the sense of managing computers. I'm going to seek out more of his stuff and see if it'll fit in my head.

After the break was a talk on "QA and the System Administrator", presented by a Google sysadmin. I went because it was Google, and frankly it wasn't that interesting. One thing that did jump out at me was when he described a Windows tool called Eggplant, a QA/validation tool. It has OCR built-in to recognize a menu, no matter where it is on the screen. This astounded me; when you start needing OCR to script things, that's broken. I don't doubt that it's a good tool, and I can think of lots of ways that would come in handy. But come on. I mean, a system that requires that is just so ugly.

I went out to lunch with Jay, a sysadmin from a shop that's just got permission from the boss to BSD a unit-testing program they've come up with for OpenBSD firewalls: it uses QEMU instances to fully test a firewall with production IP addresses, making sure that you're blocking and allowing everything you want. It sounds incredibly cool, and he's promised to send me a copy when he gets back. I can't wait to have a look at it.

After that was the meet-the-author session. I got to thank Tom Limoncelli for "Time Management for System Administrators", and got an autograph sticker from him and Strata Rose Chalup, his co-author for Ed 2. Sadly, I didn't get a chance to thank Tobias Oetiker (who I nearly ran into at lunch the day before).

Next up was the talk from Tom Limoncelli and Adam Moskovitz (Adam's looking for a job! Somebody hire him!) about how to get your paper accepted at LISA. Probably basic stuff if you've written a paper before, but I haven't so it was good to know. Thing like how to write a good abstract, what kind of paper is good for LISA, and how you shouldn't say things like "...and if our paper is accepted, we'll start work right away on the solution." Jay asked whether a paper on the pf testing tool would be good, and they both nodded enthusiastically.

Must Google:

• When talking about papers that go over the same subject, a paper from a previous LISA was mentioned that surveyed 8 years of papers on data storage and found identifiable cycles from "Oh no, we've got more data than disks!" to "Oh no, we've got more data than tape!" (This made me feel better about skipping out on the 9am talk.)
• Apparently, Sun reimplemented cat(1) and improved performance 10x.

Quotes from the talk:

• Tom: "You're not supposed to publish your paper on your website until it's published at LISA. And if you're cool, you'll do that with a cron job."
• From an audience member at another conference presentation: "At any point, did you step back and look at your work? And if so, were you sufficiently disgusted?"
• Tom again, on how audience criticism is a good thing: "Every theory paper needs someone to go up to the mic and say, 'Okay, Buck Rogers, but I live in reality.'"

At this point I started getting fairly depressed. Part of it was just being tired, but I kept thinking that not only could I not think of something to write a paper about, I could not think of how I'd get to find something to write about. I wandered over to the next talk feeling rather sad and lost.

The next talk was from Andy Seely on being a sysadmin in US Armed Forces Command and Control. Jessica was there, and we chatted a bit about how this talk conflicted with Tom Limoncelli's Time Management Guru session, and maybe ducking over to see that. Then Andy came over and asked Jessica to snap some picture, so she ended up staying. I was prepared to give it five minutes before deciding whether or not to leave.

Well, brother, let me tell you: Andy Seely is one of the best goddamned speakers on the planet. He was funny, engaging, and I could no more leave the room than I could get my jaw to undrop. Not only that, his talk was fascinating, and not just because he's a sysadmin for the US Armed Forces while simultaneously having a ponytail, earrings and tattoos. You can read the article in ;login: (FIXME: Add link) that it was based on, but he expanded on it considerably. Let me see what I can recall:

• One slide, a computer display of a map of the Middle East with lots of dots: "This is a map of people dying." This is what a screw-up or a service outage means in his job: people across the planet die.
• "There are databases where you can search on anyone in Afghanistan named Mohammed. It's an entertaining database optimization problem, let me tell you."
• On deadlines: "The more you work with government, the more you find dates...well, they're filled with humour."
• "We've got headquarters with systems everywhere -- no surprise, where haven't we invaded yet?" (laughter) I yell out "Canada!" "We're thinking about it. But we're looking for some place that'll fight back." (more laughter) "I'm sorry, that came out wrong. But it was funny." This made it to IRC, which prompted Ricky and others to ditch what they were doing and come over to this talk. (I met Andy later on and he apologized profusely, saying that he meant Canada was an ally, so why would the US invade them in the first place? We had a duel, my shot grazed his shoulder, and Canada's honour was regained.)
• Having to support an app where there's strong debate over whether it's written in C, Ada or Java, or whether it uses UDP or TCP.
• Being told that an app that keeps failing is single-threaded, so throwing more CPUs at it won't do anything; it's RAM that it needs. Later investigation confirms that, in fact, it's multi-threaded and needs more CPUs, not RAM...which the vendor eventually confirms.
• He can't install a compiler, or a debugger, or anything that doesn't come with a default install of Solaris 8, or 7, or 2.x. That would be a huge security offence.
• A Sun E4000 mainboard blows up in the Middle East. Getting one through regular channels would take too long, so where do you go? That's right: Ebay. He's a contractor, so he has no budget...but he does have a government credit card with a $2500 limit. So he calls up the guy selling it and cuts a deal to buy the thing for $2500 (shipping was billed separately). Put it on a C130, and off she goes.
• Not being allowed to write a program...but he is allowed to string shell commands together...and sometimes those commands get written down in a file for reference purposes. If he's lucky, Perl's on the machine as well.

Longer story: Because of the nature of his work, he's got boxes that he has to keep working when he knows next to nothing about what they're meant to do. Case in point: a new Sun box arrives ("and it's literally painted black!"), but the person responsible for it wants to send it back because it doesn't work -- which means that when they click the icon to start the app it's meant to run, it doesn't launch and there's no visible sign that it's running. There's no documentation. And yet he's obligated to support this application. What do you do?

Even tracking down the path to the program launched by the icon is a challenge, but he does, tracks down the nested shell scripts and finally finds the jar that is the app ("Aha! It is Java!"). He finds log files which are verbose but useless. He contacts the company that wrote it, and is told he needs a support contract...which the government, when putting together the contract for the thing, did not think to include. So he calls back an hour later, talks to the help desk and tells them he's lost the number -- "Can you help a brother out?" They do, but they're stumped as well, and say they've never seen anything like this.

Time to pull out truss, which produces a huge amount of output. Somewhere in the middle of all that he notices a failing hard read of a file in /bin: it was trying to read 6 bytes and failing. Turns out the damned thing was trying to keep state in /bin, and failing because the file was zero bytes long. He removed the file, and suddenly the app works.

Andy also talked about trying to get a multiple GB dump file from Florida to Qatar. Physical transport was not an option, because arranging it would take too long. So he tries FTPing the file -- which works until he goes home for the day, at which point the network connection goes down and he loses a day. So he writes a Perl script that divides the file into 300MB chunks, then sends those one at a time. It works!

At this point, someone yells out "What about split?" Andy says, "What?" He hadn't known about it. There was a lot of good-natured laughter. He asked, "Is there an unsplit?" "Cat!" came the response from all over the room. He smacked his forehead and laughed. "This is why I come to LISA," he said. "At my job, I've been there 10 years. People come to me 'cos I'm the smart one. Here, I'm the dumb one. I love that."

There are two things I would like to say at this point.

First off, Andy is at least the tenth coolest person on the entire Eastern seaboard. No, he didn't know about cat -- but not only did he reimplement it in Perl rather than give up, he didn't even flinch when being told about it in the middle of giving a talk at LISA. I would probably have self-combusted from embarassment ("foomp!"), and I would have felt awful. Andy's attitude? "I learned something." That's incredibly strong. (Although he told a story later about being in the elevator with some Google people. They recognized him and said, "Hey, it's the 'man cat' guy!")

Second, when he said, "Here, I'm the dumb one. I love that" I sat up straight and thought, "Holy shit, he's right." Here I am at LISA for the first time ever. I've met people who can help me, and people I can help. I've made a crapload of new friends and have learned more in one week than I would've thought possible. And I'm worried 'cos it might be a few years before I can think about presenting a paper? That's messed up. I tend to set unreasonably high goals for myself and then get depressed when I can't reach them. Andy's statement made me feel a whole lot better.

During Q & A I asked what he did for peer support, since his ability to (say) post to a mailing list asking for help must be pretty restricted. He said that he's started a wiki for internal use and it's getting used...but both the culture and the job function mean that it's slow going. He's also started a conference for fellow sysadmins: 100 or so this year, and he's hoping for more next year.

In conclusion: if you ever get the chance to go see him, do so. And then buy him a beer.

Tags: lisa cfengine

The Bleeding Heart Show

You looked as though I'd picked your name out of a hat

Next thing I know, you're fast asleep in someone's lap...

"The Bleeding Heart Show", The New Pornographers

Small shops BOF is coming up tonight, not last night. Wednesday's BOFs were:

• Should you roll your own config tool? I was actually looking for Tobias Oetiker's (!) (who received an award here for MRTG and RRDTool, and who I nearly tripped over at lunch yesterday) BOF on his tools but wandered into this one by mistake. I left after a few minutes, as a lot of the concepts were over my head. A shame, because apparently I missed a big discussion between Luke (puppet), the bcfg2 devels and (if I remember right) Mark Burgess (cfengine). Mark Burgess lost the fight and committed seppeku, with William Lefebvre as his second. Blood everywhere. USENIX is gonna lose the damage deposit for sure.

• Splunk: Someone directed me here because this was where the beer was. Again, all that was left was Bud (Lite). Splunk isn't really my bag, although the guy I've met who's gotta deal with 50GB of logfiles a day (or some such) was quite interested.

• LOPSA: Interesting, not least because someone else asked the question I was afraid to: What the hell is up with LOPSA and SAGE? Short answer: SAGE is about advancing the profession (research, training); LOPSA is about advancing you (professional development, support, fellowship...). Long answer: politics and tax laws.

• Streaming media at universities: Seven people including me. Everyone else was streaming terabytes of data with multiple servers; I wasn't. That was pretty much it.

Tags: lisa

To Wild Homes

To wild homes we go,

To wild homes we return,

To wild homes we go.

"To Wild Homes", The New Pornographers

This morning was the keynote address by Cory Doctorow on "Hollywood's Secret War On Your NOC". Excellent stuff...lots of stuff I was already familiar with, but some specifics that were incredible and/or funny:

• "Nobody puts a rootkit on media you get over eMule. That only happens if you're dumb enough to go into the store and plunk down $15 for it." (Cute quote, but arguable.)
• "Small tip: If you want to improve your Star Wars: Episode 1 experience, switch the language to Italian and pretend it's opera."
• Comcast PVR: apparently, Comcast pushed a patch that deletes already-recorded material two weeks before the DVD is due to come out. Must track that down.
• "Nobody sets the security slider to less."
• Casting the DRM debate in terms of hardware that treats its user/owner as an attacker.
• EULAS: "agreements you've agreed to by moving through time or space" ("by entering this building, you agree...")
• "The WIPO has the same relationship to dump copyright laws that Sauron has to evil in Middle Earth." (The man knows his audience.)
• Turning down an engagement at Disney to speak to their execs about DRM. Why? Because at the last minute, he was given a EULA containing a clause that said "You agree to never use the word 'Disney' in print again." When he asked about this, he was told not to worry, as they'd almost certainly never use it.
• When he and a bunch of other NGO reps got access to WIPO meetings, they naturally published their notes twice a day on websites. This was in sharp contrast to the official WIPO minuts that a) took 6 months to come out, and b) had been vetted by everyone involved, giving them a chance to retract anything they didn't want to have on the record. Very shortly, various government reps started getting calls from their capitals saying, "You agreed to what?" When their replacements showed up, they had read the unofficial notes -- they were more timely, and more useful, than the official minutes.

Must Google:

• Owner-override model for DRM that makes it more user/owner-friendly.must Google.
• World of Warcraft -- URL/content trading in speech bubbles to get around the Great Firewall of China.

Whew! Met up with the Boston sysadmin again, and I pointed him to Windflower -- he's a small enough shop that it may actually be useful for him. Good stuff. Picked up a ribbon that said "Blogger", another that says "Newcomer", and a third that says "Usenix Baby" for Arlo.

After that came technical papers on spam. First up was a paper by Brent Kang et al. on Privilege Messaging (FIXME: Add link). Third-hand, but: allegedly, as of last year, phishing is making more money than drug smuggling. A cite would be really nice for that, but he didn't have one. He also mentioned a recent paper (again, need cite) showing that spam coming from Gmail accounts (not forged, but real accounts) had rised from 1% at the start to 10%...interesting to think of how that might indicate a failure of friend-of-a-friend. OTOH, maybe that's an indication of success of FOAF, since...

...the next paper, on the experience of an Italian research network, showed that their percentage of legit mail (not caught by the spam filters) had, over the last few months, gone as low as 8%. That's fucking incredible. However, he's having excellent results with Bayes and SpamAssassin, so maybe there's some hope.

After that was "A Forensic Analysis of a Distributed Two-Stage Web-Based Spam Attack" by Daniel Klein. Very interesting: showed how regular monitoring of his systems and looking at the graphs it produced let him notice -- the second time it happened -- a very subtle attack that let 5,000 messages go out the door because of a subtle, simple CGI bug. As at least some (and probably most) of the attacks were through web proxies, I asked him (knees knocking; I was very nervous) if he thought it would be worth looking for this sort of traffic, or this sort of traffic on certain pages. He pointed out that actually, this sort of traffic -- distributed, small requests, high in numbers -- was exactly what you wanted from a website, so it was extremely hard to analyze as it happened.

After that, I talked with Noah, a Debian security guy and senior sysadmin at MIT's Artificial Intelligence lab. ! We talked about spam, getting depressed about DRM (him) vs spammers (me), and moving the AI lab to a new building after 40 years (me. no, wait). Very interesting stuff, and a good guy.

The afternoon was taken up with data closet/centre setup training. Very, very good stuff once everyone got talking -- the slides were 'way thin, but my notes filled the rest of the book. Since I've learned what I know about this by making mistakes, it was good to think of maybe shaving a mistake or two off my list from the future.

And then...then the vendor exhibit. Beer (yay!), Budweiser (boo!), and a chance to pick up the cable modem hacking book from No Starch Press' table. I also got a chance to talk with the FSF folks, up/down from Boston, and pick up a t-shirt. No luck convincing a fellow attendee to join, but I'll keep working on him. Splunk had the best booth babes (or so I heard), but Google by far had the most people around their table. Interesting.

Now off to the BOFS. Quite looking forward to the one on life at small shops.

Tags: lisa

My Streets

Cities and circles drawn perfect, complete

These are the fables on my street, my street, my street

"My Street", The New Pornographers

Okay, my (lawyers, please note) TOTALLY ACCIDENTAL stalking of Tom Limoncelli continues. I met another sysadmin from Boston (who, BTW, is into LISP. Call that accidental? 'Cos I don't) (alsoplus he's the third guy I've met from a small shop, which is damned reassuring in a conference full o'people from multi-continent corporations/teams) who invited me along to the LOPSA hospitality room. I talked to David Parter from LOPSA about why I should join. He also gave me the sad news that the Burritos-as-big-as-your-head place in Madison, WI is closed. Noooooo!

Nice bunch of people, who'll probably be getting a membership fee from me post-haste. Totally unrelated to the free beer. I met a guy from a Scandinavian hosting company that has, like, 300,000 domains (!). We talked about spam for a while, and PHP's ability to include files remotely (he's a big fan. Oh, wait, no) ("When I meet the guy who put that in..." "You'll punch him in the cock?" "Oh, that's just the start of it."), and Perl vs. C vs. LISP vs. Dvorak keyboards vs. I don't know what all.

And who else is in the room AND stared at my badge trying to figure out who the hell I was? That's right, Tom! Still no chance to lean over casually and say, "So I hear Google's trying to figure out what to about TCP scalability bringdown. 'Cos, like, my enterprise-fu PHP taint mode will totally nebbish your gubbins. Scalable. Solution. Moving forward. Come back!"

Also went to the: Free Beer and Ice Cream BOF, PGP/CACert BOF, and the Bash scripting BOF. Last challenge: using Bash built-ins only, check to see if a given TCP port on a given host is open. Welp, I did know about Bash's built-in /dev/tcp/host/port, but totally foundered on syntax. We were told to email our scripts to polvi.net...which sounded familiar, and it should, 'cos was Alex Polvi, who works at Oregon State University Open Source Lab, they who provide bandwidth to such as Gentoo, Mozilla and Kerneltrap. At one point, a few friends of his came in and sat down close to where I was, and he came over and talked to them during one of the challenges. "I think everyone would get freaked out if they knew a Google recruiter was here," he said, laughing. Worked for me.

And, BTW, I thought I was at least quarter-decent at Bash. Hah! It is to laugh.

Tags: lisa

Letter From An Occupant

What the last ten minutes have taught me:

Bet the hand that your money's on

"Letter From An Occupant", The New Pornographers

Attended my first BOF last night on wikis for sysadmin documentation -- amazingly fun and informative. I even managed to contribute to the conversation. And when I told the war story about recovering my wiki from spammers (that's right! because PHPWiki sucks!) I got a gratifying look of sympathy from the audience.

Today's talk was "Habits of Highly Effective Sysadmins". It was aimed at folks like me who've been mostly self-taught, and I thought they hit the mark extremely well. (I've heard lots of people here say that they'll go see anything put on by Lee Damon or Mike Ciavarella just on principal (principle?).) Very, very informative and great teachers, too.

I found out today that Tom Limoncelli's name is pronounced "li-mon-sell-ee", not "li-mon-chell-ee". W/luck, this will save me embarassment later.

Tonight the BOFs start in earnest, including the one that offers free beer and ice cream. Sadly, I will be attending the one on pet counting instead. I will die a little bit inside.

Tags: lisa

Choose It

Two sips from the cup of human kindness, and I'm shit-faced

Just laid to waste

If there's a choice between chance and flight,
Choose it tonight.

"Choose It", The New Pornographers

Just got back from a whirlwind walk from the Lincoln Memorial to the Washington Monument to the White House. Beautiful, all of it...though a) the White House is small and b) there was something being filmed/videotaped in the courtyard, which made me think of Vancouver.

Training again. AFrisch was good, convering Cfengine quite well; would've liked to see more info about expect. (Apparently there are Perl/Python bindings...I had no idea.) Afternoon course was "Interviewing For System Administrators" by Adam Moskowitz and that was great -- lots of things I didn't know, lots of tips on doing it better next time.

Saw Tom Limoncelli in the hall during a break. Managed to restrain myself. I have the reputation for quiet restraint of a nation to uphold.

Very tired now. Time to go get beer.

Tags: lisa cfengine

Bones of an Idol(2)

As we sift through the bones of an idol

We dig for the bones of an idol

When the will is gone

'Cause something keeps turning us on

"Bones of an Idol", The New Pornographers

Today was Solaris 10 Administration, an all-day course that introduced all the nifty features of Solaris 10. I've only worked with Solaris since July, but I've been reading so much about Solaris 10 that most of the stuff presented (dtrace, SMF, zones) was familiar to me. OTOH, the course was aimed at admins of older versions of Solaris (2.veryearly through 8 and 9), and so the explanation of the differences assumed a lot more familiarity with Solaris than I had. It was a curious sensation.

Still, though, it was worth going to. Good quote: "Oracle DBAs are the most Kool-Aid drinking people I've ever met." And another: "Zones are the most controversial thing we'll be talking about today, and spending the most time on. I saw someone carrying two cups of coffee -- that's the right attitude." Also, Bill Lefebvre, the man I was going to accuse of stealing my underwear, wrote top(1).

Oh, and it's a good thing I brought a second wireless network card; the onboard one in the laptop kept dying, with an entry in syslog that read "fatal firmware error". Now I've got an Orinoco Gold in here, and it's working just fine.

Met a sysadmin today who works in the VOIP department of a phone company; they've moved most of their stuff from racks and racks of old-style Alcatel equipment to one rack of Solaris machines acting as soft switches. I was curious about the difference in reliability and uptime; my understanding is that the demands on telecom equipment are worlds above anything that can be provided by COTS Unix, and asked him how it worked for them.

He said that, yes, you'd get situations where a phone call would be delayed because of a system crash: instead of taking one second to connect, it might take two or even three. And if that was anything beyond a small fraction of their customers, that would be a big problem. However, the soft switches had much better failover ability than the old stuff; the old stuff would be up much longer, but when it failed everything would cascade and the whole system would come tumbling down, at which point a customer would hear "Your call cannot be completed as dialed."

Met another guy who was very excited about ZFS, because of an app at his work that writes 4 TB of data in individual 4 KB files. The best they've heard from their current storage vendor of choice is a block size of 8 KB...which means doubling their storage requirements just to deal with filesystem overhead.

I had alligator jumbalaya. It's official: it tastes like salty chicken.

Tags: lisa

Jackie

Jackie, you yourself said it best when you said

 There's been a break in the continuum 

The United States used to be lots of fun... 

"Jackie", The New Pornographers 

10am CST: Welp, I'm in the air on my way to Chicago, and from thence to Washington for LISA. The laptop is running well (stress-tested by Sloan, The New Pornographers and Yo La Tengo), and I'm using my time to skip watching "Lady in the Water" (not how I want to see this film for the first time) and work on AsciiDoc. I think this is going to work pretty well for my plan: to start having my blog in just plain text for source, and plain HTML for output. I like it a lot, and the less PHP I have to audit the happier I am. (Not that I *do* audit PHP. But I feel guilty when I don't.)

Turned out I was rather stupidly cautious at the airport. The flight left at 6.15am PST, and I was there at 3.45am. What I didn't realize is that the ticket counter didn't open til 4.30am, and customs not til *5am*, thank you. But once they got started, everyone moved along pretty quickly.

I did get pulled over for extra searching, but nothing serious: where was I going, could I open the bag, where do I work. Once that was done, the officer was quite friendly; he urged me to take time to go see the sights, since work was paying for this. I expected worse.

But man, I don't know when I'll have the time. Training starts tomorrow with a full day of Solaris 10, and it just keeps going from there. Plus, of course, there's the free beer and ice cream. The time, she flies, no?

I need to get a haircut. I haven't shaved my head in two weeks, so I've got a damned dirty commie hippie head of hair at the moment.

Wow...over somewhere midwestern now, and the patchwork of land is neat to look at. Not half as beautiful as a city at night from 3000 metres, though...man, that's God's own set of Xmas lights.

12.30pm CST: Later...In O'Hare at Chicago, taking advantage of the free electrical outlets for charging laptops. The wifi access is charged-for, though, same as in Vancouver. And me without OzymanDNS...

10.20pm EST: Now in my hotel room. No wireless from USENIX up here, but it does work in the lobby where there's simply an amazing amount of very dressed-up corporate types. I think it's some sort of Xmas party. The contrast between them and the t-shirts-and-jeans crowd (not to mention me typing away alone on my laptop) is stunning. (Incidentally, my grandmother was both shocked *and* appalled to learn that not only was I not purchasing a new suit for this conference, I would not be wearing a suit at all.)

My luggage, I found out after an hour of waiting, is currently wending its way here from Chicago; I imagine some sort of Die Hard 2-esque leap across the tarmac that failed, but only barely. Allegedly United expected it here at 7pm and will courier it over Real Soon Now. We'll see.

By the time I finally made it to the hotel and checked in, it was 6.30pm . It had been a long time since I'd had anything but Mountain Dew (SPECIAL CAFFEINATED US VERSION!) to eat, so I was just starving enough to go for the -- wait for it -- $13 (US!) cheeseburger in the lobby. That and two Guinesses pretty much blew my budget for the week; at this point, I'm looking into the carb count in a BSSID beacon frame. (Yes, I'm making that term up.) Worth it, though; my roommate and I exchanged war/horror stories with a Sony engineer/sysadmin from San Francisco over the beer. Good times.

I'm pretty sure I saw Aeleen Frisch in the lobby. I think I saw William LeFebvre, the program chair, at the airport picking up baggage from the SAME BAGGAGE CAROUSEL where my stuff was supposed to be. There's this thing called USENIX bingo, where they give you cards with organizers' photos in it and you're supposed to get them to sign it. I think I'm going to tackle LeFebvre and ask him where my underwear is, then get him to sign my card to affirm that he didn't steal it.

I have not yet seem Tom Limoncelli, and I wouldn't recognize Dan Kaminsky if he queried my DNS server via avian carrier, so my plans to see what they've done with my underwear are, as yet, hazy. If my underwear doesn't show up, I may have to go shopping. I think the nearest Wal-Mart is in Tennessee.

Tags: lisa

Canada_post_sucks_(updated)_

title: Canada Post Sucks (UPDATED) date: 2006-11-25 10:18:15

Update Jan 7, 2007: After sending a snail-mail copy, slightly edited, to Canada Post, I got a letter back from their Customer Service department. It reads in part:

At the outset, I would like to offer our sincere apologies for the service failure. Our delivery personnel are expected to always issue a delivery notice if no one is at the residence to accept an item. Furthermore, a final delivery notice should be mailed from the postal outlet five days later if the item has not been claimed. I am sorry that proper procedures were not followed with your parcel. I have asked the Supervisor at [your local delivery centre] to review this incident with our delivery person. As well, I have brought the failure to issue a final notice to the attention of our Retail Business Manager who oversees the [local retail outlet where the parcel was held]. The [local delivery centre] is the pickup location for all postal items for addresses near you. (You could not find an outlet for the ID 0000268453 because the number belongs to the [local delivery centre]; items are never carded to delivery depots. The message they inputted -- "Item has been transferred tothe Post Office for pick up as a carded item" -- refers to the action they took rather than to their location.) The enclosed information from our website shows how to locate postal outlets, although if proper procedures had been followed, your delivery notice would have informed you of the location. I regret the lack of timely assistance on our part to help locate your parcel and the obvious inconvenience you experienced. If you pre-paid the shipping charges for the returned parcel, please send me the invoice and I will arrange for a refund.

So they did screw up and I got angry; they're sorry and they're making up for it, and I'm satisfied. I don't like deleting old posts, but if I did I'd delete this one. As it is, I'm closing comments. This was an angry post, I'm getting angry comments, and I don't want that. My fault.

The original post follows:

SERVICE, At Mailposte.ca wrote on November 24, 2006:
> Hello Hugh Brown,
> 
> Thank you for your message to Canada Post.
> 
> This message is simply to confirm that our delivery confirmation system now 
> shows this item as being successfully delivered on 11/07/2006.
> 
> Regards,
> 
> Julie Pich?
> Customer Service

Thanks very much for your response.  

The web tracking page for this parcel now says that, as of 11:08
(presumably AM, no idea which time zone) on November 24th, "Item was
unclaimed by Receiver and will be returned to Sender".  It also says
that delivery was attempted, not completed, on November 7th.  As
Canada Post's website says that the information available there is the
same as is available to its customer service representatives, I'm
unsure why I'm being told now that delivery has succeeded.

I never received the first notice.  I did not receive any subsequent
notice.  The link on the tracking page that was meant to show me where
I could pick up the item simply said, "No outlets found for outletID:
0000268453".

I will be contacting the shipper to let them know that, when the
parcel finally makes it back to them, they should ship it to me via
courier: FedEx or UPS, but not Canada Post or Purolator.

I am extremely disappointed with Canada Post.  I never received notice
telling me that a parcel was available for pickup; I only found out
that delivery had been attempted by checking the website.  The
location of the retail outlet where my parcel sat could not be found
on Canada Post's website.  After waiting for a second notice to be put
in my mailbox, I contacted Canada Post via the website and asked
specifically about these problems.  The response, which came 7 days
later, had no response to these questions.  The information it
contained was wrong.  I am now told that my package is being returned
to the sender as it is unclaimed.

I will not use Canada Post in the future for any important delivery.
(Exceptions may be made for bill payments, and other items for which I
want plausible deniability.)  I will strongly recommend to friends and
family that Canada Post be avoided for any important deliveries.

Thank you again for your time.

Yours sincerely,
Hugh Brown

> 
> ----- Original Message -----
> From:         aardvark@vcn.bc.ca 
> Sent:         November 17, 2006 02:04
> To:           SERVICE, At Mailposte.ca
> Subject:      Delivery Notice Card
;> 
> 11/17/2006
> Name:Hugh Brown
> Comments:According to the tracking number I've been given (9184 1445 5359
> 2002), a notice was left November 7th.  I never received this notice.  It
> is now November 16th and I have not received a second notice.  As well,
> the info on Canada Post's website for this tracking number says that the
> parcel is being held at a New Westminster post office, yet the link that
> is supposed to give directions to this post office says  No outlets found
> for outletID: 0000268453 .  Where should I pick up this parcel?
> Customer #:
> Company Name:
> Mailing Address: [snip]
> City:New Westminster
> Province/State:BC
> Country:Canada
> Postal/Zip Code:V3L 5V8
> Email:aardvark@vcn.bc.ca
> Telephone Number: [snip]
> Fax:
> Issue Type:No card received
> Notice Type Received:No notice
> Notice No.:
> Delivery Agent on Card:
> Name on Card:Hugh Brown
> Is this your regular postal outlet:
> Date Card Received:-1 -1, -1
> Date of Last Pickup Attempt:-1 -1, -1
> Date on Card:-1 -1, -1
> Item Type:
> Item Serial # on Card:
> Item Available:
> How is your mail delivered:Apartment
> Time of Delivery Attempt:
> Name of Postal Outlet on Card:
> Address of Postal Outlet on Card:
> Amount Due:
> Item held at Post Office at request of sender:
> Please specify if there is a mark in the box - Other:
> 
> 

Tags:

Stupid,_uninformative_errors_

title: Stupid, uninformative errors date: 2006-11-21 15:41:24

Spent half the day trying to figure out why a Sun Directory Server had suddenly lost its ability to replicate over SSL. The logs said:

[21/Nov/2006:00:01:00 -0800] - INFORMATION - NSMMReplicationPlugin - conn=-1 op=-1 msgId=-1 -  Replication over SSL FAILED as SSL is not enabled. Check that the attribute nsslapd-security in cn=config is on.
[21/Nov/2006:00:01:00 -0800] - ERROR<8318> - Repl. Transport  - conn=-1 op=-1 msgId=-1 -  [S] Bind failed with response: SSL configuration error  (808).
[21/Nov/2006:00:01:00 -0800] - ERROR<8221> - Incremental Protocol - conn=-1 op=-1 msgId=-1 -  Failed and requires administrator action [ldap.example.com:636]
[21/Nov/2006:00:01:00 -0800] - ERROR<8221> - Incremental Protocol - conn=-1 op=-1 msgId=-1 -  Failed and requires administrator action [ldap.example.com:636]

Google turned up nada. In the end, it turned out that the last time the directory server had been started, the security token had not been provided. Restarted the server, typed in the token on standard input, and replication works again.

Yes, this is a job for expect -- but this approach has failed for coworkers in the past. I'll have to look into it.

Tags:

Openbsd_netboot_problems_ _unknown_error_code_72_

title: OpenBSD netboot problems - unknown error code 72 date: 2006-11-18 21:50:28

(Note: edited to actually be correct this time. :-)

While trying to get a Sparc machine to boot disklessly so I could install OpenBSD on it, I kept getting these errors:

Boot: bsd.rd
Automatic network cable selection succeeded : Using TP Ethernet Interface 
Using BOOTPARAMS protocol: ip address: 192.168.23.25, hostname: roark
root addr=192.168.23.10 path=/home/aardvark/openbsd-sparc64/chroot
open /sbus@1f,0/ledma@e,8400010/le@e,8c00000/bsd.rd: Unknown error: code 72

tcpdump showed that the machine was trying to contact the NFS server (192.168.23.10) by udp on port 0; the server kept responding with an ICMP port unreachable error. Googling turned up one other person back in '99 (!) who had the same problem, but no fix.

What was weird was that this had worked during an earlier install -- only the running of MAKEDEV hadn't completed (don't ask), so I didn't have /dev/console when I booted up, which meant no nothing once it tried to mount the root directory.

I started looking at the traffic in greater detail, and saw that the packet to port 0 was, according to Ethereal^WWireshark, a nicely formed NFS call trying to get the filehandle for the kernel (/bsd). Well, what would make it send it there? After all, mountd was listening on the same port it'd been contacted on a moment ago...

Looking at the call to portmap on port 111, I saw that the client was asking for the port for nfsd, but was being told that there was no such thing -- that the port number was zero. What the...I checked rpcinfo -p and saw that, yep, there was no nfsd...and then realized my mistake: mountd only deals with mount requests; it's nfsd that actually reads/writes files, gives information about their size and modification times, and so on. I'd been starting the NFS stuff by hand since this was a one-off, and had totally forgotten to start nfsd. I did so, and suddenly all went well. PEBCAK.

Tags:

Bones of an Idol

Thursday: Go to The Other University to do some prep for the move coming up next week. Check in with their computer store (where you pretty much have to buy things) to see how the order on the console server is going. The guy behind the counter looks up the order, frowns, and tells me that it seems their supplier does not have one in any of their three Canadian warehouses. Okay, so how long will it take to get one in? He looks at me earnestly and says that, sometimes, they never come in. I ask at what point I can count on the supplier a) giving up and b) informing me of that fact. He frowns again, and suggests that I check back in a couple weeks (four weeks after I've placed the order) just to be safe.

Friday: Get email from contractor/university liason for new building to say that network and electrical connections will not be ready in time because the requests were received so very late. While The Other Guy was supposed to get them in long ago, I should've been on top of this.

Monday, a stat in Canada: Go to the old building to do a serverectomy on a soon-to-be-formerly shared rack. The Other Guy mentions that the new server room has water on the floor. I go over to look, and it's a rapidly evaporating puddle, irregular in shape and maybe two metres across at its widest. I can't figure out where it's coming from. Turns out there's some other stuff that should become formerly shared as well, so I spend time poring over Sun Enterprise 1 workstations (which I like) and old inkjet cartridges for printers that may no longer be around (which I don't like). Ask The Other Guy, who's been involved with the move a lot longer than I have, what electrical connections he's asked for him and for me (long story) in the new building. He says that he gave them the model number of the Sun rack he's got (which has built-in, and very nice, PDUs) and asked them to figure out what he needs.

Tuesday: Moving day. As expected, network and electrical are not present; we've got 2 x 15A 120V circuits. Also, the leak is back, and we can see that it's coming from a small leak in the concrete roof. I move my rack into another room; The Other Guy spreads a blanket over his rack. The liason promises us that the contractors are on the job to fix the roof. The network connections (two fiber, two Cat5) get terminated, so I call the local network folks to get that taken care of. The university wireless network is not present in the new building.

Wednesday: The contractors show up to start fixing the leak. The network connections have been set up. The contractors have put in a big tube of plastic sheeting, taped to the roof at one end and a 40-gallon recycling barrel at the other. The Other Guy decides things are good enough and starts setting up his rack; I elect to hold off another day.

Thursday: The contractors say the roof is fixed, so I move the rack in and start hooking things up. The new OpenBSD firewall comes up nicely -- thank you, pf developers -- as does the main Sun server. Next up is the SunRays in the lab, only they're not. I take my laptop in and try to verify connectivity. I can't. The Other Guys suggests that the VLANs on my new switch are the problem and suggests just simplifying things. I do and keep testing. Traffic from the laptop's RFC 1918 address just never makes it to the server. In a fit of desperation I try using an address in our routable subnet, and it works. This takes me until 8pm to figure out. I email various bosses explaining how far I've got, and the campus network folks to ask if they're filtering this subnet in some way. (This isn't completely out of the question; this place has a reputation for a pretty locked-down network.)

Friday: I buttonhole the guy at the campus network office and ask him about this. He considers this and realizes that while he's forgotten to unblock DHCP (told you it was pretty locked down), the other behaviour I'm seeing can be explained if I've somehow got my interfaces crossed. I'm doubtful but give it a try, which is a good thing because suddenly everything works. I don't understand it or what I did wrong, but assume that I was simply too tired the previous night and thank him profusely for taking the time to talk to me. I am now where I should have been twenty hours before. Mighty battles emerge with Sun's DHCP and Sunray servers. In the end, I have to delete the Sunray configuration, delete all DHCP configurations, and then add the Sunray configuration back. This works, which annoys me; why are there all these opaque configurations around? Not a single plain-text file in sight. I manage to get a printer working, then another. DHCP is modified so that laptops work as well. I call it a night and head home.

Tags: hardware network fail

8_o'clock,_the_lights_were_on_at_shea

title: 8 o'clock, the lights were on at Shea date: 2006-11-11 21:52:43 tags: openbsd, geekdad

Woot! I managed to install OpenBSD 4.0 on my work laptop this afternoon while Arlo slept in my arms. Not only that, it automagically set up X and I figured out wireless + OpenVPN. Woot! Firefox is running, I've got Mozex and Adblock going...the only thing left is to figure out how to get IceWM to start up automagically.

Tags:

The_universe_occasionally_says_"fuck_you"_

title: The universe occasionally says "Fuck you" date: 2006-11-10 12:09:26

If your machine has hard drives that are, in theory, removeable because they have a front catch, but in practice require you to open up the case to disconnect the SCSI and power cables, that's not a server.

If your machine's CD drive fails and it takes you fifteen minutes of searching to find the unlabelled holes on the bottom of the case that allow access to the screws that are attached to the bottom of the drive so that you can actually remove the drive, that's not a server.

For $399 US, thank you, for the Academic edition of MathMagic, I expect better goddamned installation instructions than this:

• Windows - MathMagic Pro Edition Full installer with some old versions of fonts and Plugin (Please run this full installer first.) [ a url ] - MathMagic Pro Edition v3.5 (application only. The latest version.) (Please use this v3.5 application, instead of v3.0 installer by the installer, after moving it into /Program Files/MathMagic Pro/ folder) [ another url ] - new CS & CS2 plugin (Please use this Plug-in, instead of the Installer installed one. Copy it into InDesign's Plug-ins/Equations folder.) [ whee! lookit alla urls! ] - new fonts for PDF embedding (If you want to embed MathMagic fonts in your PDF documents on Windows, please download the new MathMagic TrueType font set, and replace the preinstalled ones(remove the old MathMagic fonts from Windows Fonts directory and copy these new fonts into Windows Fonts folder). [ sale! sale on urls! ]

The email goes on to suggest that installation instructions can be found on their website (but neglect to mention that it only covers the Mac version), or "in User Guide documents that you can find after installation." What a crock.

Tags: