I Am Not Afraid Of You And I Will Beat Your Ass

Thank you to our sponsors for the title.

Good news: I'm going to LISA! I convinced my employers to heavily subsidize my trip. I've booked a double room at the hotel; I'll be posting to the roomshare mailing list shortly, but feel free to comment or email if you wanna split the cost.

Bad news: I somehow borked X on my desktop at work yesterday. The symptoms are quite strange, and mostly involve not being able to click on a window and have focus move there. It's IceWM, and I haven't changed focus model, and the symptoms persisted over multiple restarts of KDM (ctrl-alt-backspace). I looked for open files, running processes and even removed .gconf* and .gnome* on principle; nothing. The only thing that was different was running, for the first time, the new(ish - 1.5.0.2) version of Firefox after d/l it from the Mozilla site. The machine is running SuSE 10, and for various reasons I can't update it right now. In the end, I got desparate enough to try a reboot, and of course that fixed it...which is NO FUCKING WAY to solve problems, dammit.

(Interesting how this pokes holes in my manly command-line-only stance; yes, I was able to get some work done by going to the console, but frankly I've become very very used to managing terminals and a browser with IceWM and it's hard to switch back. Damn.)

Weird news: A while back I came across a problem with a Solaris 10 machine: lpq just hung, and eventually timed out with an error (that I haven't written down, so I suck). Eventually figured out it was trying to contact the lpd service on the machine's main interface (handwave goes here about BSD-compatibility printing commands), which should've been run by inetd. Okay, but inetd is now taken care of by inetadm and svcs, not /etc/inetd.conf anymore. And while the command is called in.lpd, it's actually called svc:/application/print/rfc1179. Which is in maintenance mode, so start it up only it doesn't and I cannot figure out why: no log files I can see (the scattering of log files in a default Solaris install is really driving me nuts), no reason given, nothing. I ask another sysadmin who admits he's stumped by it but just for fun tries putting in an entry in /etc/inetd.conf and then running inetconv, the way you're not supposed to have to do except for weird legacy stuff that hasn't been moved to svcs yet. And damnitall, it works. Again, no idea why.

And that is it for now. I am tired beyond belief, having moved up my annual snifter of port from Xmas to go out with coworkers last night. I stopped drinking at 7pm and I'm still tired today. Pathetic. Arlo would be so disappointed in me.

Tags: lisa solaris

Two Recent Examples of My Handiwork

When my wife went to pick up the co-op car we'd booked for Sunday, she found that the mirror hanging off the door by the control cables. Fortunately, a little camo duct tape -- a Christmas gift from my parents -- took care of it (at least until the co-op can get it fixed):

And then there's this:

Thanks to John and Arwen for Arlo's shirt, and to Theo et al. for mine.

Tags: geekdad

Two Top Tips

1. Trying to NFS mount something (Solaris client and server) and getting error 7 (RPC: Authentication error)? Check /etc/nsswitch.conf on the server and make sure that things like auth_attr, netgroups and so on are not set to use (say) ldap [NOTFOUND=return]. Doubly so if you've just run ldapclient -v init the night before and forgot that, surprise! it changes nsswitch.conf.

2. Starting up OpenOffice on Solaris 9/Gnome only to see gibberish in the title bar rather than the file name? Log out, and in the whatever-DM login screen go to Language and select C - POSIX. Log back in. Works like a charm!

These Top Tips brought to you by the number Pi and the beverage Beer.

Tags: solaris

Spec Bebop

I just love clever network hacks.

Speaking of which, I think I'm going to ask my boss if she'll send me to LISA. I didn't realize I had sysadmin heroes 'til I started looking at the program: Æleen Frisch! Michael Lucas! Tom Limoncelli (who's working at Google now, natch)! W. Curtis Preston! But also Dan Fucking Kaminsky, that's who:

I like big graphs and I can't deny...You other hackers can't deny...when a packet routes in with an itty bitty length and a huge string in your face you get sick...cuz you've fuzzed that trick...

...who's going to be presenting the results of a worldwide SSL scan among lots of other stuff.

I think it'd be great to attend, but it's a long shot. Wish me luck.

Tags: lisa network

Samba Problems, or Don't Forget Your SID!

This is going to be a long story, but I hope it'll be instructive. Bear with me.

Back at my last job, we had a Samba server, running on FreeBSD, acting as a Primary Domain Controller for around 35 W2K machines. The same machine also acted as NIS master for a similar number of FreeBSD machines. It also did printing, mail, DNS, and half a dozen other things. This machine was getting old; it's CPU usage was often pegged by a large print job, it was running out of disk space, and I was beginning to be worried about the inevitable day of death. I began planning for the upgrade: a new machine, faster and bigger hard drives, more memory and gigabit ethernet for the day we all moved to GigE. Oh, and rack-mounted...definitely rack-mounted.

The opportunity was taken to upgrade much of the software on the machine, including Samba. I decided to move from 2.2 to the 3.0 series; the speed differences seemed pretty impressive. I also wanted to get as many of the big upgrades done at once as possible: the prospect of going through the upgrade repeatedly did not appeal.

Of all the upgrades I was doing, Samba made me the most nervous. I read through the excellent (and Free) Samba HOWTO and made notes: how to move to the tdsam password database, changes in configuration options, and so on. I had the new server for a while, so I was able to run through many tests: getting a Windows machine to log on, DNS queries, and so on.

Finally, the big day came. I went in on a Saturday and made the move. Most of the rest of the day was spent testing, chasing down the inevitable mistakes, and testing some more. I tested by logging into machines after they'd joined the domain, and making sure that everyone could still log into their workstations. All told, things went pretty damned well, and I congratulated myself on a job well done.

Later though, a few things began to crop up that I haven't been able to explain. I could no longer add new domain accounts to SSH under Cygwin. A shared printer wasn't being shared any longer. In fact, shares weren't working at all. I banged my head against this for a while, but since the problems were pretty erratic they tended to fall to the wayside in favour of explaining, one more time, why the words "spare computer" were self-contradictory.

Finally, though, I put some more time into it. And it's a little hairy, especially for this Unix guy, so bear with me.

(Incidentally, I couldn't have figured out half of this without the help of Clarence Lee, a co-op student working with us. Sure, he uses IIS, but he firewalls it with OpenBSD and he got an internship at Microsoft. He's a good guy.)

The shared printer: could not figure out what was going on here. Guy who had it could print to it, no problem. Used to work for everyone, no problem. Now it wouldn't work. Broke the problem down to the point where I was using smbclient on FreeBSD, or net view on W2K, to try and list the shares, and that didn't work. Not any of them -- not IPC$ or anything. I was fairly sure this wasn't supposed to be happening.

There was a machine in limbo (not the same as spare, thenk yew!) while a coop student became permanent. I got it using the other networked printer, and tried sharing it. Again, command-line utilities would simply not list the shares. What's more, when I tried getting other people to log into the machine (I was fairly irritated at this point, and not at my most rational), they couldn't log in. WTF? I could log in, and there had been no complaints from the person whose machine it had been.In a moment of irritation, I got the test machine to rejoin the domain...and suddenly, everything was working: I could list shares on it, other people could list shares on it, people could log in, and everything. Yay! It's so simple! Rejoin the domain! Everything will be great!

Ha! It is to laugh. Profiles were not coming in when people logged in. My Documents was empty, they got that stupid, evil, vile "Let's take a tour of Windows! And let me help you set up your network! DO IT!" popup window. I couldn't figure it out.

Clarence and I banged out heads against it some more, and finally came to a conclusion.

When you migrate Samba, you're meant to take the old SID with you using net(8) GETLOCALSID and SETLOCALSID. The SID is meant to be a world-unique string/number that identifies a domain, or an account -- think something like the DN in LDAP, or NIS domainname + UID in Unix. (A user's SID has a part that belongs to the domain, and another, smaller part that is unique to that user.) I didn't do that -- screwup -- and so the Samba server had generated a new SID. As far as Windows is concerned, the identity of your domain is solely determined by the SID; the name is their just for your convenience. (Insert snide remark here about how magic invisible numbers have no business being that important.)

As a result, the machines that were present at the migration didn't know where their Primary Domain Controller (PDC-- the machine officially in charge of the domain) had gone, and were running on cached credentials, profiles and so on. (This is the same thing that allows you to log into a Windows laptop that belongs to a domain, even when you've taken it home and aren't able to reach your PDC any more.) Printing and shared resources from the Samba server continued to run because of open permissions or credentials (ie, user name and password) that don't depend on SIDs.

This also explained why I could log into the machines without problems: because, as sysadmin, I'd logged into all of them before to do maintenance. My credentials were cached, so the machines were able to authenticate me w/o consulting with their (now missing) PDC. And of course, everyone was able to log into their own workstations for the same reason.

So: machine rejoins the domain and people can log in, because now the machine can find its PDC and verify their passwords. But profiles aren't showing up because the profile's NTUSER.DAT -- the user's hive, loaded into the registry at HKEY_CURRENT_USER when they log in -- belonged to/was marked with/was owned by the account's old SID, and Windows refused to load it and lots of stuff broke or was missing.

After some more searching, I finally figured out the way around this.

First, you need to use the profiles(1) tool in Samba to change the SID on NTUSER.DAT, which'll be wherever Samba keeps profiles. You should check their SID in Samba by using pdbedit(8), though odds are the user ID/group ID part will have remained the same.

Second, you need to take care of the profile. There are a few ways of doing this. The easiest way is to copy the modified NTUSER.DAT to their profile directory, then log into the machine as Administrator and join the new domain, then get the user to log in. Their profile will be copied over, just as if they'd logged into a machine for the first time. However, this can cause problems with certain programs who haven't been informed about the change.

To illustrate: if the domain name is named EXAMPLE, and the user account is jdoe, then their profile will usually be at C:\Documents and Settings\jdoe (let's just call that D&S\jdoe for short). However, D&S\jdoe will belong, after joining the new domain, to an old account that's no longer around, which means that Windows will put their profile somewhere else -- probably something like D&S\jdoe.EXAMPLE. Odds are, though, that the old path will still be in the registry or other files, which means a lot of cycles of "Why-did-that-break-let-me-fix-it". Another option is simply to move D&S\jdoe out of the way, so that paths can remain the same. Finally, you can also change ownership recursively to the new account once you've joined the domain; this will take a while, but it's probably quicker than copying the profile over wholecloth if they've got a lot of files. If you do this, it's best to remove the machine's copy of their NTUSER.DAT file; it'll just be copied over from the server.

This took a lot of work, of course, and usually there were things like Outlook.pst to screw things up further. But after much work, I finally got everyone moved over to the new domain, and things were good again.

Lessons learned:

1. Take the new SID with you.
2. Learn how something works, even if it stinks.
3. Testing the usual is good and necessary. So is testing things that wouldn't ordinarily happen.
4. You can never know too many people on the other side of the fence.

Tags: samba warstory windows migration

Alt-bucky-cokebottle

I had a problem with X recently; after a restart, the Alt and Windows/Menu keys on my Microsoft Natural keyboard did not seem to work with X.org. I'm running Debian testing and I upgrade weekly, yet an X session might not restart for a long time...so it's hard to be sure when this started.

I finally managed to track it down to an error that could be shown like so:

$ setxkbmap -print | xkbcomp - $DISPLAY

Error:            No Geometry named "microsoft" in the include file "pc"
                  Exiting
                  Abandoning geometry file "(null)"

This error also showed up in /var/log/Xorg.0.log. After much, much, much searching I finally came across this ten-year old posting to comp.os.linux.misc from who I'm pretty sure is this guy. And it worked a treat: by adding the line

Option  "XkbGeometry"   "pc"

to the keyboard section, poof! all my windows-menu-alt keys have come back.

Tags:

rpmbuild and SuSE

Actually, might be other RPM-based distros, but I keep working on SuSE. Anyhow: reminder to myself to move .bashrc out of the way, because /usr/sbin/Check is run toward the end of the build, and SuSE keeps sourcing it. If there are any errors (like, say, an alias that's not valid on that machine because it refers to some program that isn't installed) it'll fail and your build will fail and Theo will kill your pony.

Tags:

Now_this_is_just_cool:

title: Now this is just cool: date: 2006-09-04 13:35:12

OpenBSD works under Xen (mostly), thanks to Google's SoC. Coolness! I'm pulling a copy of the repository now, and maybe I'll be able to get TKYP working this way too. My devel box, a nice P4, appears to be borked: it's currently shutting itself down every 20 or 30 minutes. This'd let me run near-native on my desktop machine, a much slower (but actually running) P3.

Between OpenBSD and OpenSolaris, I might be running a lot more on this machine. Of course, I still have to follow Alioth's advice and start running my webserver under it...

Tags:

Never thought...

...that I'd be wearing a Baby Bjorn and singing my kid to sleep with Joy Division's "Dead Souls". Clara and I dragged out a bunch of tapes yesterday, and man, I haven't listed to that one in years.

Tags: geekdad

Theo Kills Your Pony

A few things.

First, Arlo is doing well:

Second, there's this.

Third, work has started on the world's most useless project: Theo Kills Your Pony, the aggressively destructive Unix-like system. (Thanks to Zen Render for the name!). I'm attempting to do things semi-right, and that means I've had to learn a bit more about how OpenBSD (and BSD in general) is put together.

Like what? Well, like the name for example. The OS is called TKYP, so that's what I want to show up everywhere. I figured I would start with the output of uname(1), since that's the most Thing is, this took a surprisingly long time to track down.

uname(1) is, as you might expect, a simple wrapper around the uname(3) libc function, which is in turn a pretty simple wrapper around a sysctl call. Through paths that, frankly, I'm still tracking down, you finally get to sys/conf/newvers.sh -- a simple shell script that creates a file called vers.c and sets the variables ostype, osrelease and osversion within it. (Paths are relative to /usr/src, BTW.) After that, the different sys/arch/*/conf/Makefiles compile it -- sys/arch/i386/conf/Makefile.i386, for example -- and then include it in SYSTEM_LD. After that, <handwave>I think these values are simply returned by sysctl(3)</handwave>.

Okay, so now I've tracked that down; I rebuild and install the kernel, then reboot. (QEMU rocks for this sort of thing.) And yay, it works:

-bash-3.1#  uname -a
TKYP tkyp-qemu 0.1 GENERIC#0 i386

Now to rebuild world, right? Wrong: first, Apache kept refusing to compile with an error about not being able to find -ldbm. Trolling through the mailing lists only found one message mentioning a similar problem, and no reply. The CVS tree showed that, since 3.9, a couple minor changes had been committed to the httpd Makefiles mentioning that OpenBSD has used its own dbm library for a while. I tried making a few changes, but couldn't get it to work. So I cheated: I removed httpd from usr.sbin/Makefile and moved on with my life.

Next problem: the GNU configure tools haven't heard of TKYP. (I'm sure I emailed RMS about this...). gnu/usr.bin/binutils is the first thing compiled in world that uses these tools, so that's where I'm looking first. A little judicious editing of config.guess (which guesses the OS and architecture), configure (which figures out what needs to be done for the OS/arch) and config.sub (which says it's a "configuration validation subroutine script"; I'm guessing a basic sanity check) takes care of thing. They're fairly simple changes, as it's pretty much just a matter of copying the OpenBSD entries.

And all this before I can even throw in anything nasty! I got big plans, of course -- SIGKILL replaced with SIGHUP, rot13 encryption for passwords, and the RTM worm pre-installed -- but I haven't even had a buildworld finish yet. Plus, there's the cautionary tale of MicroBSD to keep in mind...whatever else I do, I wanna make sure I piss off Theo for the right reasons. :-)

(Incidentally, the email to root is in etc/root/root.mail; etc/Makefile installs it in the right place. I thought for sure it'd be in share for some reason. newvers.sh mentions this file, plus a few others, that need to be changed to reflect new version numbers.)

Tags: theokillsyourpony geekdad

One more thing

before I go back to changing diapers: from the ever-excellent Secrecy News comes a link to this report from retired US Army General Barry McCaffrey on his visit to the Guantanamo Bay prisons.

The report is well worth reading. As summarized in the newsletter:

"The JTF Guantanamo Detention Center is the most professional, firm, humane and carefully supervised confinement operation that I have ever personally observed," he stated.At the same time, "Much of the international community views the Guantanamo Detention Center as a place of shame and routine violation of human rights. This view is not correct. However, there will be no possibility of correcting that view.""There is now no possible political support for Guantanamo going forward," Gen. McCaffrey wrote.

McCafferey acknowledges in the report that "During the first 18 months of the war on terror there were widespread, systematic abuses of detainees under US control in Iraq, Afghanistan and Guantanamo. Some were murdered and hundreds were tortured or abused. This caused enormous damage to U.S. military operations and created significant and enduring damage to US international standing."

Yet nowhere in this report does he seem to realize that the U.S. also was condemned for its lawlessness:

The great value of the platform of Guantanamo was that it was a military space in which no Federal District Court had primary jurisdiction. For that reason alone, Gitmo has over the past 45 years been the location of choice for US migrant refugee operations (no appeal to the INS process) as well as other secret operations. No applicable foreign law, no foreign diplomatic intervention, no Federal Court civil orders, no nosy intervention by a US Ambassador -- only the exercise of unilateral military power and the tool of the Uniform Code of Military Justice. It was the perfect deal. No more.

The mourning of the loss of a place over which no court had jurisdiction, into which no "nosy " US ambassador could look, is entirely unbecoming of any democracy -- let alone one that views itself as the Great Vending Machine of Liberty. Yet this point flies right past the nose of a man who gives an otherwise straightforward and unblinking account of Gitmo's failures.

Tags: geekdad politics

The Great Unix Software Upgrade Flowchart

Note that the snide comment about NetBSD is just a joke...couldn't come up with anything else to say. Everything else, of course, is the gospel truth.

Last day at my old job was Friday, and as a going-away present I got not only a lovely universal gift certificate from my co-workers, but this t-shirt from the sysadmin I hired a little while back:

Arlo and Clara are doing well:

I have been peed on twice now, which I'm told is fairly good for the first week of a new parent.

Tags: unix

Arlo Maxwell Reginald Cristofaro

So:

Arlo Maxwell Reginald Cristofaro, ne Trombone, was born on Saturday July 1st, 2006 at 2.26pm. Clara had a pretty damn good labour once things got going, and horsed him out after only 13 minutes of pushing. As labour stories are, by ancient right, public property, I'll let her post the details.

Both she and Trombone^WArlo are doing quite well. They've both learned how to nurse, and I've learned that the index finger does a lot to calm him down. We've managed to pick up a couple hours of sleep here and there, so we're not too punchy.

For those who haven't seen, here are a couple pix:

Oh, and you know what also calms him down? A slightly modified version of Fat Joe's Lean Back:

My Arlo, he don't know how to dance
He just leans back and he fills up his pants
He does the Rockaway! He does the Rockaway!

It also sends Clara into hysterics, so that's good too.

Tags: geekdad

Update

When last Clara visited the doctor (Wed), Dr said that a routine checkup on the babby would be in order at some point this weekend -- Monday, maybe? Turned out to be today, around 2pm. During the ultrasound it further turned out that Babby/Clara had low levels of amniotic fluid. This means that they wanted to induce Real Soon Nowtm. This was begun about 6pm, about ten minutes before I made it to BC Women's. (Stupid bus drivers that don't stop at King Edward when they're asked -- but I digress.)

She's being kept for observation, which means taping big things to her belly and watching the strip of paper slowly come out of the machine that goes ping! when it runs out of paper. I've come back to the house to get things like the hospital bag and cheese, and to feed the catt. (Let this be a lesson to someone: when going to the hospital after your due date, always bring the bag. If they say you don't need it, hit them.)

Clara is doing well. The baby is doing well. No telling how long it could take before active labour starts; the nurses said they've seen it as quick as three hours, or as long as 24. We're hoping for a Canada Day babby, especially after hearing the story about the friend of the nurse who got free stuff for LIFE because the kid was born on July 1st. (Seriously. Government owes the kid a damned helicopter now.)

The next post will probably be made once we're back from the hospital; you should expect something like "Holy CRAP this thing's small!"

That is all.

Tags: geekdad

Soon!

Birth is being induced. Baby's well, wife is well. Wish us luck.

Tags: geekdad

Stupid, stupid Internet

Up until today, I would've told you that the stupidest thing I'd read on the Internet was a white paper titled "Is PowerPoint An E-Learning Solution?" But OMG ponies, the bar has been raised.

Precisely why a made-up word making it to Google should be considered news is never really explored. Wired's whole-hearted gushing about someone who "has registered freedbacking domains and plans to aggregate freedbacking comments on a new website next week" is also a nice touch -- way to accelerate the IPO! Finally, you've got the thoughtfully-placed-last obligatory OTOH about how "consumer ignorance and laziness could also keep the value of the suggestions low."

<headdesk /> <headdesk /> <headdesk />

Tags: rant

BlogFS/ifconfig up

So Pouxie, my new OpenSolaris box, started displaying the same let's-shut-down-randomly-'cos-it's-Friday problems it previously did -- guess it's not the case after all. No problem, 'cos I happen to have a spare mobo and CPU that I've been itching to try out.

As it happens, it's got an onboard Intel ethernet interface which is detected just fine (iprb0, thank you) by Belenix/OpenSolaris, but fails to be brought up properly during boot. The problem is that while the interface is assigned an IPv4 address, it's not actually up, which means that adding the route fails, and /lib/svc/method/net-physical (which surprised me by being a simple shell script) declares failure. (I think it's just the route command that fails, but I should check this out.)

No idea why this happens on iprb0 and not nfo0, but what the hell. Looking around the script shows that it does do ifconfig plumb up on IPv6 interfaces -- but when I tried touching /etc/hostname6.iprb0 and running the script again (yeah, I know, probably a horrible thing that makes Bill Joy cry) it created a duplicate iprb0 interface with only an IPv6 interface. It was up, the IPv4 version was still down, and the IPv4 route command failed.

In the end I just edited the script to make it run ifconfig plumb up like it does with IPv6, and it seemed to do the trick just fine. I'm currently trying to see if there's a similar bug already filed on OpenSolaris.org; looks like I have a lot of slogging.

In other news, I thought I'd be posting this using BlogFS, but I'm running into library problems. First, I had to change import xmlrpc to importxmlrpclib. No biggie, even I can do that, but now I'm getting this when I try to create the directory that would mount the blog:

#  mkdir foo:bar@saintaardvarkthecarpeted.com/blog/xmlrpc.php
mkdir: cannot create directory `./foo:bar@saintaardvarkthecarpeted.com/blog/xmlrpc.php': No such file or directory

Not sure what's going on.

Tags: solaris meta

Third install!

In preparation for my new job, I've installed OpenSolaris on Pouxie, my wife's old desktop machine (a nice 2GHz Athlon). I've used Belenix, a live CD that includes a driver for Pouxie's onboard NForce ethernet interface.

So far I'm having a lot of fun. It took me three hours (spread over four days...damn this commute) to get a static IP address assigned to the thing, and then to get DNS working. But after a reinstall (a newer version of Belenix had come out that included the Sun packaging tools, which should let me use Blastwave to grab Emacs...a good first project, I think), I had it up and running in just a few minutes. Progress!

For those playing the home game, here's what I had to do:

1. modinfo | grep nfo: yep, the module has been loaded.
2. ifconfig -a | grep nfo0: Not there.
3. dladm show-link: But it is here.
4. echo "192.168.23.40 pouxie-2" >> /etc/inet/hosts
5. echo "pouxie-2" > /etc/hostname.nfo0 ; echo "netmask 255.255.255.0" >> /etc/hostname.nfo0
6. echo "192.168.23.254" > /etc/defaultrouter
7. reboot -- -r: to get Solaris to find the new interface (?)
8. ifconfig -a: Now it shows up configured.
9. svcadm --disable svc:/network/inetmenu: Otherwise, it interferes with the change to nsswitch.conf I'm going to do up ahead.
10. svcadm --enable svc:/network/dns/client: I long to know what this actually turns on.
11. cp /etc/nsswitch.dns /etc/nsswitch.conf
12. echo "nameserver 192.168.23.254" >> /etc/resolv.conf
13. ping www.saintaardvarkthecarpeted.com: It's alive!

Happy birthday, OpenSolaris!

Tags: solaris work

Holy crap, I made the Globe and Mail!

Tags: work meta

Everything Is Good

Let's not worry now<br /> Let's not worry now<br /> Cos we're right<br /> And they're wrong<br /> And it's over.<br />

Sometimes it's just all about Swell.

Tags: