Samba Problems, or Don't Forget Your SID!

This is going to be a long story, but I hope it'll be instructive. Bear with me.

Back at my last job, we had a Samba server, running on FreeBSD, acting as a Primary Domain Controller for around 35 W2K machines. The same machine also acted as NIS master for a similar number of FreeBSD machines. It also did printing, mail, DNS, and half a dozen other things. This machine was getting old; it's CPU usage was often pegged by a large print job, it was running out of disk space, and I was beginning to be worried about the inevitable day of death. I began planning for the upgrade: a new machine, faster and bigger hard drives, more memory and gigabit ethernet for the day we all moved to GigE. Oh, and rack-mounted...definitely rack-mounted.

The opportunity was taken to upgrade much of the software on the machine, including Samba. I decided to move from 2.2 to the 3.0 series; the speed differences seemed pretty impressive. I also wanted to get as many of the big upgrades done at once as possible: the prospect of going through the upgrade repeatedly did not appeal.

Of all the upgrades I was doing, Samba made me the most nervous. I read through the excellent (and Free) Samba HOWTO and made notes: how to move to the tdsam password database, changes in configuration options, and so on. I had the new server for a while, so I was able to run through many tests: getting a Windows machine to log on, DNS queries, and so on.

Finally, the big day came. I went in on a Saturday and made the move. Most of the rest of the day was spent testing, chasing down the inevitable mistakes, and testing some more. I tested by logging into machines after they'd joined the domain, and making sure that everyone could still log into their workstations. All told, things went pretty damned well, and I congratulated myself on a job well done.

Later though, a few things began to crop up that I haven't been able to explain. I could no longer add new domain accounts to SSH under Cygwin. A shared printer wasn't being shared any longer. In fact, shares weren't working at all. I banged my head against this for a while, but since the problems were pretty erratic they tended to fall to the wayside in favour of explaining, one more time, why the words "spare computer" were self-contradictory.

Finally, though, I put some more time into it. And it's a little hairy, especially for this Unix guy, so bear with me.

(Incidentally, I couldn't have figured out half of this without the help of Clarence Lee, a co-op student working with us. Sure, he uses IIS, but he firewalls it with OpenBSD and he got an internship at Microsoft. He's a good guy.)

The shared printer: could not figure out what was going on here. Guy who had it could print to it, no problem. Used to work for everyone, no problem. Now it wouldn't work. Broke the problem down to the point where I was using smbclient on FreeBSD, or net view on W2K, to try and list the shares, and that didn't work. Not any of them -- not IPC$ or anything. I was fairly sure this wasn't supposed to be happening.

There was a machine in limbo (not the same as spare, thenk yew!) while a coop student became permanent. I got it using the other networked printer, and tried sharing it. Again, command-line utilities would simply not list the shares. What's more, when I tried getting other people to log into the machine (I was fairly irritated at this point, and not at my most rational), they couldn't log in. WTF? I could log in, and there had been no complaints from the person whose machine it had been.In a moment of irritation, I got the test machine to rejoin the domain...and suddenly, everything was working: I could list shares on it, other people could list shares on it, people could log in, and everything. Yay! It's so simple! Rejoin the domain! Everything will be great!

Ha! It is to laugh. Profiles were not coming in when people logged in. My Documents was empty, they got that stupid, evil, vile "Let's take a tour of Windows! And let me help you set up your network! DO IT!" popup window. I couldn't figure it out.

Clarence and I banged out heads against it some more, and finally came to a conclusion.

When you migrate Samba, you're meant to take the old SID with you using net(8) GETLOCALSID and SETLOCALSID. The SID is meant to be a world-unique string/number that identifies a domain, or an account -- think something like the DN in LDAP, or NIS domainname + UID in Unix. (A user's SID has a part that belongs to the domain, and another, smaller part that is unique to that user.) I didn't do that -- screwup -- and so the Samba server had generated a new SID. As far as Windows is concerned, the identity of your domain is solely determined by the SID; the name is their just for your convenience. (Insert snide remark here about how magic invisible numbers have no business being that important.)

As a result, the machines that were present at the migration didn't know where their Primary Domain Controller (PDC-- the machine officially in charge of the domain) had gone, and were running on cached credentials, profiles and so on. (This is the same thing that allows you to log into a Windows laptop that belongs to a domain, even when you've taken it home and aren't able to reach your PDC any more.) Printing and shared resources from the Samba server continued to run because of open permissions or credentials (ie, user name and password) that don't depend on SIDs.

This also explained why I could log into the machines without problems: because, as sysadmin, I'd logged into all of them before to do maintenance. My credentials were cached, so the machines were able to authenticate me w/o consulting with their (now missing) PDC. And of course, everyone was able to log into their own workstations for the same reason.

So: machine rejoins the domain and people can log in, because now the machine can find its PDC and verify their passwords. But profiles aren't showing up because the profile's NTUSER.DAT -- the user's hive, loaded into the registry at HKEY_CURRENT_USER when they log in -- belonged to/was marked with/was owned by the account's old SID, and Windows refused to load it and lots of stuff broke or was missing.

After some more searching, I finally figured out the way around this.

First, you need to use the profiles(1) tool in Samba to change the SID on NTUSER.DAT, which'll be wherever Samba keeps profiles. You should check their SID in Samba by using pdbedit(8), though odds are the user ID/group ID part will have remained the same.

Second, you need to take care of the profile. There are a few ways of doing this. The easiest way is to copy the modified NTUSER.DAT to their profile directory, then log into the machine as Administrator and join the new domain, then get the user to log in. Their profile will be copied over, just as if they'd logged into a machine for the first time. However, this can cause problems with certain programs who haven't been informed about the change.

To illustrate: if the domain name is named EXAMPLE, and the user account is jdoe, then their profile will usually be at C:\Documents and Settings\jdoe (let's just call that D&S\jdoe for short). However, D&S\jdoe will belong, after joining the new domain, to an old account that's no longer around, which means that Windows will put their profile somewhere else -- probably something like D&S\jdoe.EXAMPLE. Odds are, though, that the old path will still be in the registry or other files, which means a lot of cycles of "Why-did-that-break-let-me-fix-it". Another option is simply to move D&S\jdoe out of the way, so that paths can remain the same. Finally, you can also change ownership recursively to the new account once you've joined the domain; this will take a while, but it's probably quicker than copying the profile over wholecloth if they've got a lot of files. If you do this, it's best to remove the machine's copy of their NTUSER.DAT file; it'll just be copied over from the server.

This took a lot of work, of course, and usually there were things like Outlook.pst to screw things up further. But after much work, I finally got everyone moved over to the new domain, and things were good again.

Lessons learned:

1. Take the new SID with you.
2. Learn how something works, even if it stinks.
3. Testing the usual is good and necessary. So is testing things that wouldn't ordinarily happen.
4. You can never know too many people on the other side of the fence.