Carousel is a lie!

So a couple of weeks ago, a coworker said to me: "Hey, you going to LinuxCon? It's here in Vancouver." I had no idea. I took a look at the schedule (Greg Kroah-Hartman! Matthew Garret! Linus his own bad self!) and started to think about it. I hadn't budgeted for it, and it was the last three days of my vacation...but yeah, I wanted to go.

I checked with my lovely wife and she said it was okay with her. I checked with my lovely boss, and he said it was okay. I checked my budget and decided I (well, work) could afford it. And now, I'm busy checking the schedule to see when I should be out the door to arrive (looks like about 7am).

There's a lot that I want to see. The FreeIPA talk, Puppet and OpenStack, Linux and Finance...it's going to be a good time. I haven't seen a lot of Twitter traffic on LinuxCon, but I'm afraid I won't be able to contribute much. However, I will be writing up the day's experience as I do with the LISA conferences. So, you know, sharpen up your copy of Lynx, 'cos this text-only layout won't optimize itself.

In other news:

• 3- and 5-year old kids are surprisingly adaptable. When we say something like, "It's 4.30 in the afternoon and we're bored, wanna drive to a beach?" they cheer. And then they keep their cool for the half hour of preparation and the 20 minute drive. We would have KILLED for this level of spontaneity two years ago.

• I got a new scope! It's an 8"/200mm SkyWatcher dob from Craigslist. The owner stored it outside ("but only in the summer") so it wasn't in the greatest shape. But it works, and I'm happy with it. And I was able to see Jupiter's Great Red Spot yesterday morning with it, which is hard -- it's quite pale, and not bright red at all these days.

• I'm taking my oldest son (5 years) to camp out overnight at the Aldergrove Park Star Party this Friday. We're both looking forward to this. Originally I was just going to stay out late and drive back, but then I found our old tent when I was doing some cleaning...and camping out just makes a lot more sense. It's going to be a long day for me -- duck out of LinuxCon, come home, and drive out to Aldergrove -- but hopefully a lot of fun. I'll power up on Red Bull or something.

Morning came and I found myself at the beautiful Airport Hilton^W^WVancouver Hyatt, standing in line to register for LinuxCon. Ponytails, beards; jeans and t-shirts, but also jeans and open-neck dress shirts. (OH in line: "Yeah, we really have to leverage first-line early adopters to get that community buildup...") Coffee, starchy sweets and free stickers, then gawking at maddog and thinking that forgetting my FSF pin was worse than forgetting my business cards. Some signs of Scary Viking Sysadmins, which reassured me.

Sign on the window: "Don't miss the Complimentary Morning Yoga at Vancouver Corporate Yoga at the Royal Centre!"

Then at 8:55am, the BELLS OF GOD rang in the lobby, interrupting the conversation I was having with the Oracle folks ("No, we have no plans to close-source VirtualBox. No, we cannot let you fly Larry Ellison's jet.") to let me know it was time to go to THE OPENING KEYNOTE ZOMG THIS WAY. I clutched my hands to my head and staggered into the ballroom, bleeding from my ears, took a complimentary Band-Aid and found a seat.

I overheard someone behind me saying "Dude, you can see maddog here and Linus and all these people you only read about!" I turned around, noticed that they were from the Oregon State University Open Source Labs, and said "Dude! You work at a place that hosts Mozilla! And Gentoo! I'm not worthy!" Geek love...is it wrong?

The first presentation was from Jim Zemlin, pres. of the Linux Foundation. As the head of a non-profit, he was in the mandatory uniform of jeans and a grey hoodie, and his theme song was "Eye of the Tiger" (no, really). His talk: where would we be w/o Linux? Well, we wouldn't have that creepy as hell IBM commercial about Linux that I saw described as "Children of the Corn-like". Seriously, it was disturbing.

Next up, the CEO of Red Hat in jeans and open-neck dress shirt, and his theme song was that goddamn "Tonight's Gonna Be a Good Good Good Good Good (Good Good) Good Ni-i-i-i-ight" by the Black Eyed Peas. His talk: where is Linux going to be in 20 years? Spoiler: he doesn't know. I just saved you a) 30 minutes of a semi-interesting speech (he started with Slackware in the 90s, he says) and b) sorting through 8,000 goddamn retweets of that TechCrunch article("RH CEO: I Have No Idea What's Next") while looking for any, ANY interesting tweets re: the conference. (I'm spoiled from LISA.) (Note: they showed up later in the day...but they're still thin on the ground.)

Assertion from RH CEO: Google, Facebook et al. would be nowhere w/o Linux, for only Linux provides easy, cheap prototyping without bureaucracy/licensing/etc. Sorry, what happened to the BSDs? Did I miss something? And another example: "Could Facebook have taken off if they were using Oracle Solaris with Sparc Servers, charging $10 per user to register?" Notice a) ORACLE ORACLE ORACLE and b) ignoring the actual FB business model of selling your data to the highest bidder.

Also, he used the phrase "leading business thought leader" to describe someone.

Okay, so after more sugary starch and not enough coffee, I wen to the FreeIPA talk. This is definitely an interesting project, and I think I should have been using this a long time ago. Benefits:

• Everyone does LDAP organization differently, which makes mergers hard. Also, there's no enforcement of relationship integrity. FreeIPA handles of those things (standard layout + automatic relationship updates).

• XML-RPC, CLI, Web interfaces, all of which do everything.

• System Security Services Daemon -- offline credential caching FTW.

• Kerberos + NTP + DNS automagic.

(Contrary to rumour, Larry Ellison was NOT at the back shouting "This does NOTHING that NIS 4 won't do!" I don't know how these things get started.)

I left early, because while I agreed that I shoulda used FreeIPA a long time ago, it wasn't telling me much more than that. I ducked over to Greg Kroah-Hartman's talk on the stable kernel, and basically showed up in time for him to say "No more questions? Okay, then, thanks everyone." I wanted to buttonhole him afterward about how in hell Broadcom was persuaded to release wireless drivers under the GPL, but he was being buttonholed by other actual developer types...I figured I could stalk him later. (Incidentally, his voice is quite deep; someone in the audience had an app on their phone that showed it was 0.8 metric BarryWhites.)

I needed coffee, so I headed to the Starbuck's in the hotel. And who should be in line in front of me but Linus Torvalds his own bad self? True! Not only that, but he was trying to grift the poor cashier with the ol' San Francisco Shuffle. He was paying with a gift card, but there was a minimum order of $10 and he was $0.37 short, so can't you just add a small service charge...he walked away with his coffee, $53 in cash and the keys to the staff bathroom. If you ever meet him in person, hold on to your wallet with both hands.

Next up was James Turnbull's talk on OpenStack and Puppet. (ObSartorialNote: sat next to jeans + polo shirt + black & red running shoes, talking to jeans + open-neck white dress shirt; I'm guessing startup manager + mid-level VC.) This was quite a cool talk. If you're not familiar w/OpenStack (and I wasn't), it's meant to be a way of managing all your cloudy stuff (VMs, storage, networking, etc) no matter who the provider is. Think high-level API for spinning up/down instances, with low-level plugins (or something) worrying about how to do it with AWS, Rackspace, Eucalyptus...

So far they've got Nova (compute instances), Glance (image service) and Swift (simple blob storage, very little metadata); coming RSN is authorization, dashboard, block storage, message queueing, database and load balancing (code name Atlas, which is just the coolest thing ever). But even with just those three components production-ready (or nearly so), PuppetLabs is getting ready to migrate something like 20 VMs they use for testing over to OpenStack.

Interesting points:

• Written in Python; an advantage of that is that it allows sysadmins and other non-ninja-programmer-types to contribute. It's all API-driven, with an HTTP/REST interface.

• Currently supports Ubuntu; working to get CentOS/RHEL in there, but it's a while off.

• His demo was using OpenStack within a VM to create another VM, then configure it with Puppet. He introduced this with a slide of five turtles all piled on top of each other in a pond.

• Puppet is used for managing the new VMs; nothing prevents you using Cfengine. That said, they're building modules (is that the right term?) for Puppet that will be able to manipulate OpenStack directly.

• One of PuppetLab's customers is quite interested in OpenStack. They are -- get this -- a mortgage broker company that drives an 18-wheeler full of computers to state fairs and such, then sell mortgages on the ground to punters. Right now, fixing problems with the computers/servers is a PITA because they're in back of beyond with no spare parts, service folks, etc. The ability to just magic VMs out of the ground, and destroy/re-create them if there are problems, is very, very appealing to them.

I talked to him after the talk. He insists that there is no bad blood between Luke and Mark Burgess, and that rumours of cage matches are completely unfounded. (I don't know how these things get started, I really don't.)

I asked him about packaging support in Puppet; Cf3 basically washes its hands of the matter, saying "I'll run your stupid installation commands but don't come crying to me if everything breaks." He said that this is a subject of much debate between Cf3, bcfg2 and Puppet; Mark's feeling is that it's simply not solvable, and his (James') own feeling is that it's merely non-trivial. He's trying to find a way to inhale the package manager's graph of dependencies and merging it with Puppet's own, but myriad differences in package manager behaviours are making this difficult.

After that was "What's inside benchmarks" by Oracle. I stuck around for a while, but it was simply not that interesting. I moved on to the "PowerNap your data centre" presentation by Dustin Kirkland, and this was definitely better. PowerNap is a Python script that will watch for activity (processes, disk or network IO, whatever) and lower power consumption if it thinks the machine has been idle long enough. Matthew Garrett was there, and offered to help put this in the kernel (if I understood his questions correctly).

At my work we don't pay for power (it's a university) so that incentive is out; instead, we worry about capacity, and this might help. A friend of mine who works with render farms was interested in modding the code so that it would throw an idle machine into the render farm, but return it to interactive use if someone sat down at it.

Oh, and PowerNap version 3 will be a client/server thing -- client says "Hey, looks like I'm idle...tell me what to do"; server will say "it's before 5pm, so stay fully powered no matter what."

I headed to the FreedomBox talk next. Eben Moglen was in the audience, and I took the opportunity to thank him for his speech (I think it was this one). (Hands shook: RMS, Eben Moglen; Linus next?)

The talk was interesting, as is the project itself. The goal is a personal server, running Free software, that creates and preserves privacy. Personal == something like a plug computer; if it's in your home, some legal jurisdictions treat data very differently than they do if it's on an external server. Privacy == enabling privacy-respecting/creating apps to replace current privacy sinkholes like Facebook et al. They're starting with Debian, due to long history with it, an eventual goal of creating FreedomBoxen easily with "apt-get", and to ensure that their work survives the project.

I'm going to be keeping an eye on this, and I suggest you do too.

I went to the Q&A with Linus, and it was interesting. He said he'd been asked by people to skip version 3.1 because of bad memories, but was still considering naming 3.11 "Linux for Workgroups".

He got asked about the Google/Android dispute. He said it'll probably happen, it's a couple years out at least, the Google team is relatively small and oversubscribed...and anyhow, he's not afraid of forks.

And after that, it was beer o'clock with Paul. Fun times.

Thursday morning was the keynote from Dr. Irving Wladawsky-Berger at IBM. His memories of Linux ascendancy were interesting...possibly because of the cheerleading/"We would simply prevail" feeling I felt. But his speculation on what would come was fuzzy and handwavy...slides with things like "Smart retail / Smart traffic/ Smart cities / Smart regions / Smart planet / Intelligent oil field technology" (wait, what happened to smart?) and graphs of Efficiency vs. Transformation, with a handy downward-sloping line delineating "Reinventing Business" from "Rethinking IT", just made THE RAGE come on.

The HP speech that came after wasn't much better, so I ducked out after five minutes...perhaps a mistake, in retrospective. I will say, though, that it amazes me that multitasking, in 2011, is something to brag about.

Next up was the presentation from IBM on "Improving Storage in KVM-based clouds". Despite teh buzzwords, it boiled down to an interesting war story about debugging crappy FS performance, from verifying ("Yes, the users are right when they say it sucks") to fixes ("This long-term kernel project will add the feature we need to stop sucking!"). If I can find the slides, I highly recommend reading them...there's a lot of practical advice in there.

Next up was a presentation by the mysteriously-employed Christoph on Linux in the world of finance. It was a short presentation -- a lot of presentations at LinuxCon have been short -- but he made up for it with a lively Q&A afterward. (To be fair, he explained at the beginning that he was used to a much more hostile/loud audience and a much more interactive presentation style, and actively solicited questions.)

Right, so: Linux is used in finance a lot, because it's fast and very, very tweakable. He describes this as "Linux hotrodding", that seems to capture the attitude very well. Sadly, a lot of this stays in-house because these tweaks are considered part of the "secret sauce" that makes them money.

I asked if the traders were involved in the technical side of things, or if it was more like "Let me know when my brilliant algorithm is sufficiently fast." Answer: no, traders are very, very technical (some give keynotes at tech conferences), and there is very tight integration between the two. I asked if the culture was as loud, macho and aggressive as the stereotype. Answer: yes. Someone asked why Solaris usage had declined. Answer: neither traders ("You got bought! You're a loser!") nor techies ("Oracle kills MySQL and puppies!") liked Oracle buying Sun.

And now for an opposing view.

I spoke after the talk to three sysadmins from the same trading company, and they disputed some of Christoph's points. First, their company contributes back to open source/Free software; their CTO says it's a moral imperative. They've open-sourced their own trading software, though not the algorithms ("algos" if you're a trader type) that make them money. They admit that this makes their company unusual; in their industry, secrecy is the rule.

Second, they said the culture varies from company to company, and that anyhow it's very different now that MIT PhDs and such are being hired. It's not all "Wall Street".

And one bit they confirmed: hotrodding. Things like overclocking their chips -- but to the degree that the vendors phone them up to say "You'll burn out your CPU in a week!" Response: "Okay." Because it'll make more money in the first hour it's running than the CPU costs.

I had lunch with Chris, who I used to work with, and caught up on everything. Then I hung out in the vendor area a bit. The PandaBoard was neat: Ubuntu 10.10, playing a 1080p movie trailer and drawing less than two watts. Incredible.

I buttonholed the FreeIPA guy; complimented him on the talk, and asked some questions. Master-slave in FreeIPA LDAP server? No, multi-master only. Doesn't that make you nervous? No. Doesn't keeping config information for the LDAP server in LDAP, rather than a plain text file, make you nervous? Shrug; if you can't read LDAP, you're probably hosed anyway. Oh, and btrfs is coming to Fedora 17, probably RHEL 7. Doesn't that make you nervous? No. (Conclusion for the home listeners: I am a misinformed worrywart.)

And Rik van Riel was there, but I forgot to hug him.

In the afternoon I went to a two-hour introduction to KVM-based virtualization. This was excellent; while I'm using KVM at the moment, I'm not familiar with the tools available. (Which probably means I shouldn't be using it....) He covered tools like virt-p2v, KSM, and how to monitor performance of VMs from the host, even if you don't have root privileges. Good stuff.

Friday morning I pretty much skipped the keynotes. While waiting for the tutorials to start up, I got buttonholed by Kurt von Fink of the MariaDB project. Nice guy, and he pretty much convinced to take a serious look at it. I'd been dithering about whether to attend the talk on it in the afternoon, and decided it was worth my while.

First tutorial, though, was the Filesystem Tuning talk by Christoph Hellwig. After some minor problems (geek presentation tech problems mean 47 people in the audience shouting out suggestions for xrandr flags), the talk began...and ho boy, I took notes furiously during this one. He concentrated on ext4 and XFS, adding the disclaimer that he's an XFVS developer/fan. Hopefully his slides will be up soon; if not, I'll type up my notes RSN.

Okay, one quote/summary: do not buy a cheap RAID controller -- ie, anything without a battery-backed cache. Why not? Because you'll be doing stuff you could do with mdadm and running it on an undebugged RTOS, which itself will be running on a tiny, underpowered ARM chip.

In the afternoon was the Bufferbloat talk. Here are some links for you to look at while I come up with more summarizationalism of the talk itself:

• Slashdot article
• Wikipedia article
• I, Cringely article

I've got a lot of reading to do there.

At work, I'm about to open up the Rocks cluster to production, or at least beta. I'm finally setting up the attached disk array, along with home directories and quotas, and I've just bumped into an unsettled question:

How the hell do I manage this machine?

On our other servers, I use Cfengine. It's a mix of version 2 and 3, but I'm migrating to 3. I've used Cf3 on the front end of the cluster semi-regularly, and by hand, to set things like LDAP membership, automount, and so on -- basically, to install or modify files and make sure I've got the packages I want. Unlike the other machines, I'm not using cfexecd to run Cf3 continuously.

The assumption behind Cf3 and other configuration management tools -- at least in my mind -- is that if you're doing it once, you'll want to do it again. (Of course, there's also stuff like convergence, distributed management and resisting change, but leave that for now.) This has been a big help, because the changes I needed to apply to the Rocks FE were mostly duplicates of my usual setup.

If/when I change jobs/get hit by a bus, I've made it abundantly clear in my documentation that Cfengine is The Way I Do Things. For a variety of reasons, I think I'm fairly safe in the assumption that Cf3 will not be too hard for a successor to pick up. If someone wants to change it afterward, fine, but at least they know where to start.

OTOH, Rocks has the idea of a "Restore Roll" -- essentially a package you install on a new frontend (after the old one has burned down, say) to reinstall all the files you've customized. You can edit a particular file that creates this roll, and ask it to include more files. Edited /etc/bashrc? Add it to the list.

I think the assumption behind the Restore Roll is that, really, you set up a new FE once every N years -- that a working FE is the result of rare and precious work. The resulting configuration, like the hardware it rests on, is a unique gem. Replacing it is going to be a pain, no matter what you do. There aren't that many Rocks developers, and making it Really, Really Frickin' Nice is probably a waste of their time.

(I also think it fits in with the rest of Rocks, which seems like some really nice bits surrounded by furiously undocumented hacks and workarounds. But I'm probably just annoyed at YET ANOTHER UNDOCUMENTED SET OF HACKS AND WORKAROUNDS.)

And so you have both a number of places where you can list files to be restored, and an amusing uncertainty about whether the whole mechanism works:

I found that after a re-install of Rocks 5.0.3, not all the files I asked for were restored! I suspect it has to do with the order things get installed.

So now I'm torn.

Do I stick with Cf3? I haven't mentioned my unhappiness with its obtuseness and some poor choices in the language (nine positional arguments for a function? WTF?). I'm familiar with it because I've really dived into it and taken a course at LISA from Mark Burgess his own bad self, but it's taken a while to get here. But it is the way I do just about everything else.

Or do I use the Rocks Restore Roll mechanism? Considered on its own, it's the least surprising option for a successor or fill-in. I just wish I could be sure it would work, and I'm annoyed that I'd have to duplicate much of the effort I've put into Cf3.

Gah. What a mess.

So: the quota file on an ext3 filesystem contains usage information ("How much disk space is this user using?"). It's updated when quotacheck is run, typically at boot time. After that the kernel has up-to-date info on quotas but doesn't write it to disk for performance reasons. So the kernel will deny/allow writes as necessary.

But the userland tools used -- particularly by users -- to monitor or report on quota state ("How much space am I allowed to use?") only uses those files. And those aren't updated unless quotacheck is run...which is either at boot time, or when called from cron. And to run it on a live system, you've got to turn off quotas to prevent corruption.

Bleah.

UPDATE: Near as I can tell, I was checking quotas while not realizing that quotas had been turned off for the filesystem I was checking, and thus quota reported bogus data.

CONCLUSION: I am on crack. BIG CRACK. Ignore me.

either:

a) financial regulations are deliberately made NP-complete, in order to ensure that prosecutions can prove that a human deliberately subverted them, or

b) lobbyists push to keep financial regulations deliberately simple, ostensibly to increase efficiency but in actuality to facilitate computer-assisted subversion without having to invest in AI?

ObXKCD.