The Life of a Sysadmin

The Life of a Sysadmin http://saintaardvarkthecarpeted.com/blog Carousel is a LIE! en Copyright 2006 Tue, 30 Dec 2008 15:00:59 -0800 Tue, 30 Dec 2008 15:00:59 -0800 aardvark@saintaardvarkthecarpeted.com Building Fedora Directory Server RPMS for CentOS A few quick notes about <a href="http://www.directory.fedora.redhat.com/wiki/Howto:BuildRPMsForCentOS_RHEL">building Fedora Directory Server RPMs for CentOS</a>: <ul> <li> You need to download and install <a href="http://directory.fedoraproject.org/yum/dirsrv/fedora/6/i386/SRPMS/fedora-ds-dsgw-1.1.1-1.fc6.src.rpm">fedora-ds-dsgw</a>, as it's required by fedora-ds; this has been omitted from the list o' SRPMs to install. </li> <li> You need to patch the spec file for fedora-ds itself. It requires fedora-ds-admin-console. That package was renamed in 1.1.2 from fedora-admin-console, but updated SRPMs for the renamed package were not provided (at least, they're not there as of today) for Fedora 6 (which is what's used to build for CentOS 5.2). Given the age of F6, it seems unlikely that updated SRPMs will be provided, so the simplest thing is to edit the spec file. </li> <li> Also, the <a href="http://ficherosfede.googlepages.com/mmr.pl">mmr.pl</a> script mentioned <a href="http://directory.fedoraproject.org/wiki/Howto:MultiMasterReplication">here</a> needs to be modified so that <tt>$instance_dir</tt> points to <tt>/etc/dirsrv</tt>, not <tt>/etc/fedora-ds</tt>. </li> </ul> (Partly a memo to myself, and partly to help anyone in the same boat; edits have been disabled in the FDS wiki, so I can't add this right now.) http://saintaardvarkthecarpeted.com/blog/2008-12/building_fedora_directory_server_rpms_for_centos.html So tired Tuesday: youngest son (8 months old) up at 5:30am teething. Wednesday: youngest son up at 5:15am teething. Thursday: youngest son up at 5:30am teething. I'm so tired I go to bed at 8:30pm and fall asleep immediately. Friday: youngest son up at 4:45am teething. At 5:45am he goes back to sleep. At 6am my phone tells me the DNS server at work is down; I can't raise it. I restore backed up zone files to a spare Xen instance (hurrah!), give it the DNS server's IP address and head into work. I restart the machine and shut down the Xen instance; can't figure out why the machine shut down in the first place. Then I discover a replication problem between two of our LDAP servers which is resulting in random bounced email for a newly created account. I want to go home now. But there's a Very Important Meeting(tm) at 1pm, and I can't leave before then. <headdesk> http://saintaardvarkthecarpeted.com/blog/2008-12/so_tired.html By which I mean scary The <a href="http://isc.sans.org/diary.html?storyid=5434">Internet Storm Center</a> writes about a new variant on malware that messes with your DNS: it installs a rogue DHCP server. <div class="listingblock"> <div class="content"> <pre><tt>While not too sophisticated, the whole attack is very interesting. First, it's about a race between the rogue DHCP server and the legitimate one. Second, once a machine has been poisoned it is impossible to detect how it actually got poisoned in the first place - you will have to analyze network traffic to see the MAC address of thoese DHCP Offer packets to find out where the infected machine actually is.</tt></pre> </div></div> In other news…all $job_2's new machines are set up and running. Kickstart is very nice…I really wish Debian had something similar; FAI is lovely, but Kickstart has the lovely feature of taking a hand-done installation you've just finished and turning that into a config file for a hands-off version. That saves a huge amount of time. Next up: turn nscd back on (forgot I'd left it off for debugging LDAP 'til a simple <tt>find -exec chown</tt> was taking 10 minutes to finish); relabel the machines with their new names; commit the documentation I've been piecing together on my laptop; open up to others in the group; look at either moving the LDAP server over to the server room, or setting up a slave over there. http://saintaardvarkthecarpeted.com/blog/2008-12/by_which_i_mean_scary.html Because I am amused by childish things. Why do you ask? From a list of known issues with installation of Office 2008 for Mac. <a href="http://www.microsoft.com/mac/itpros/default.mspx?CTT=PageView&clr=99-15-0&target=5eb4aafa-5457-4ce0-8aa3-4b58cfdeeddc1033&srcid=b7d8f7fc-b588-47fd-8e47-afdf1dd18b0a1033&ep=7">Number one</a>: <div class="listingblock"> <div class="content"> <pre><tt>Office 2008 updates cannot be installed if the Microsoft Office 2008 folder was moved, renamed, or modified Office Installer installs Microsoft Office 2008 for Mac in the Applications folder. If you move the Microsoft Office 2008 folder to another location on your computer, or if you rename or modify any of the files in the Microsoft Office 2008 folder, you cannot install product updates. To correct this issue so that you can install product updates, drag the Microsoft Office 2008 folder to the Trash, and then reinstall Office 2008 from your original installation disk.</tt></pre> </div></div> Ah, hard-coded paths. <a href="http://www.microsoft.com/mac/itpros/default.mspx?CTT=PageView&clr=99-15-0&target=780887ce-689a-4124-9cbe-f060a0e6ea1b1033&srcid=b7d8f7fc-b588-47fd-8e47-afdf1dd18b0a1033&ep=7">Number two:</a> <div class="listingblock"> <div class="content"> <pre><tt>I can't download the volume license version of Office 2008 for Mac by using Safari Cause: Downloading the volume license version of Microsoft Office 2008 for Mac is unsuccessful when you use the Safari browser. Solution: We recommend that you use the latest version of Mozilla Firefox Web browser ( MozillaClick this link to open a browser window.http://www.mozilla.com) to download the volume license versions of the Microsoft Office 2008 for Mac suite or stand-alone applications.</tt></pre> </div></div> That said, it turns out <a href="http://blogs.technet.com/migreene/archive/2008/02/28/2941140.aspx">you don't need a license key</a> for a volume copy of Mac Office 2008. And now you know the rest of the story. http://saintaardvarkthecarpeted.com/blog/2008-11/because_i_am_amused_by_childish_things_why_do_you_ask.html Well, *that* took a long time to track down I just spent the weekend (well, like an hour a day…kids, life, you know how it is) trying to track down why a bunch of new CentOS 5.2 installs at $job_2 couldn't pipe: <div class="listingblock"> <div class="content"> <pre><tt>$ ls foo foo $ ls | grep foo $ echo $? 141</tt></pre> </div></div> (Actually, I didn't think to look at the error code 'til someone else pointed it out…141 turns out to be <a href="http://en.wikipedia.org/wiki/SIGPIPE">SIGIPE</a>) In the end, it would have been quicker if I'd simply searched for <a href="http://bugs.centos.org/view.php?id=2964">the first thing I saw when logging in</a>: <div class="listingblock"> <div class="content"> <pre><tt>-bash: [: =: unary operator expected -bash: [: -le: unary operator expected</tt></pre> </div></div> This was particularly aggravating to track down because not every machine was doing this, and no matter what I thought to look at (/etc contents, /tmp permissions (those have a habit of going wonky on me for some reason), SELinux) I couldn't figure out what was different. Turned out to be <a href="http://rhn.redhat.com/errata/RHBA-2008-0611.html">an upstream bug</a> in nss_ldap. (The <a href="https://bugzilla.redhat.com/show_bug.cgi?id=448014">Bugzilla</a> entry makes for some interesting reading, to be sure…) And I didn't see it on each machine because I hadn't upgraded after installation on all machines. (They're not yet in production, and I'm working on getting my kickstart straight.) Man, it was gratifying to upgrade nss_ldap and see the problem go away… http://saintaardvarkthecarpeted.com/blog/2008-11/well_that_took_a_long_time_to_track_down.html Wow Old news by now, but I just got pointed to <a href="http://geeklondon.com/blog/view/dave-considered-harmful">Dave considered harmful</a> by a posting to the <a href="http://www.sage.org">SAGE</a> mailing list. Kudos to Sun for the full and thoughtful explanation. http://saintaardvarkthecarpeted.com/blog/2008-11/wow.html So that's where they're keeping it I've since found a great deal more about multipath in Linux: <ul> <li> <a href="http://people.redhat.com/nayfield/storage/RHEL4Storage.html">http://people.redhat.com/nayfield/storage/RHEL4Storage.html</a> </li> <li> <a href="http://geekyschmidt.com/2008/09/15/multipath-and-redhat-linux-5/">http://geekyschmidt.com/2008/09/15/multipath-and-redhat-linux-5/</a> </li> <li> <a href="http://www.tuxyturvy.com/blog/index.php?/archives/42-RHEL4-dm-multipath-on-root-HOWTO.html">http://www.tuxyturvy.com/blog/index.php?/archives/42-RHEL4-dm-multipath-on-root-HOWTO.html</a> </li> <li> <a href="https://www.redhat.com/archives/nahant-list/2006-November/msg00044.html">https://www.redhat.com/archives/nahant-list/2006-November/msg00044.html</a> </li> </ul> The trick was to search for "multipath" and "fstab". Also, I contacted the installer from Sun who worked on our new machines, and he told me that the multipath driver download was lost during an upgrade of the download page; they're working on it, but in the meantime he's sent me a copy of the driver. Sweet! http://saintaardvarkthecarpeted.com/blog/2008-11/so_thats_where_theyre_keeping_it.html I love working at UBC Just now from the window, over the sound of a stupid high-pressure washer, I heard a Canada goose fly by, honking its head off. http://saintaardvarkthecarpeted.com/blog/2008-11/i_love_working_at_ubc.html This is The Working Hour; we are paid by those who learn by our mistakes I'm in the process of setting up a bunch of new servers for $job_2. All but one are CentOS 5.2, kickstart installed and managed with cfengine. This is the third time I've goen thorugh a cfengine setup, and it always feels like starting from scratch each time. It seems — and I'm not at all sure this is fair or accurate — that each time I set up one of these systems, there's a lot that I've lost from the last time and have to relearn. I'm fortunate this time that I can refer to $job_1's setup to see how I did things last time, but if I didn't have that I'd be significantly further behind than I am. I'm not sure what the solution is. Part of me thinks I should just be more aggressive about taking notes, or committing stuff to a private repository, or writing it down here more; part of me thinks that this might be a clue that cfengine is too low-level for my head. It feels like when I was trying to learn C, and couldn't believe that I had to remember all this stuff just to print something, or read a file, or connect to another machine over the Internet. By contrast, Perl (or any other scripted language) was such a relief…just print, or open, or use the Net::Telnet module, or whatever. The details are there and they are important, sometimes very much so; that doesn't mean I want to learn more metallurgy every time I need a fork. (No, I don't think that metaphor's tortured; why do you ask?) Another thing is that I'm trying to get multipath connections working for the first time. We've got two database servers, each of which is connected via dual SAS HBAs to outboard disk arrays. (I don't think anyone else calls them "outboard", but I like the sound of it. See this hard drive? It's outboard, baby!) The arrays are from Sun and come with drivers, but the documentation is confusing: it says it's available for RHEL 5 (aka CentOS 5), but the actual download says it's only for RHEL 4. As a temporary respite, I'm trying to see if I can get these workign using Linux's own multipath daemon, and it's also confusing. The documentation for it is tough to track down, and I just don't understand the different device names: am I meant to put /dev/dm-2 in fstab, or /dev/mpath/mpath2p1? If the latter, why does the name sometimes change to the WWUID (/dev/mpath/$(cat /dev/random)) when I restart multipathd? (use_friendly_names is uncommented in the config file.) If the whole point of multipath is failover, why does this sequence: <ul> <li> touch /mnt/1 </li> <li> remove first cable </li> <li> rm /mnt/1 </li> <li> replace first cable </li> <li> touch /mnt/2 </li> <li> remove second cable </li> <li> rm /mnt/2 </li> <li> replace second cable </li> </ul> (where /mnt is where I've got this array mounted, obvs) sometimes work, and sometimes end with "I/O error" being logged, and the filesystem being read-only? Is this the sort of thing that the Sun driver will fix? I can't find anything about this. And I mentioned electrical problems. When we got our servers installed, the Sun guys told us they'd tripped breakers on the PDU and/or breakers in the room's electrical cabinet. Since it had a sign on it saying "100A", I figured we might be running up against power limtis — either in the room as a whole, if my figures were 'way out, or on individual PDUs. Turns out I was probably wrong: I missed the bit on the sign that said 3-phase, which means (deep breath) we probably have 3 x 100A power available (I think). It's more complicated than that, because some of it is in 120V, some of it is in twist-lock 220V 30A circuits, and so on. But I should've checked before emailing the faculty member who, in a year or two, will be going into this room (we're there as guests of the department) and happens to sit on the facilities committee. He had asked how we were doing, so I sent him an email — nice, polite, and including a bit about how grateful we were for the room and the help of the local sysadmins (all of which is true). I was under the impression that he was asking for info now, so that he could bring it up for action in a few months when we were out. Instead, two hours later when I'm swearing at multipath, in come the facilities manager and one of the sysadmins I was dealing with, looking to find out just how much power we were using anyhow. I apologized profusely, and they were very cool about it. But when the committee guy asks questions, people jump. I had not anticipated this. Welcome to University Politics 101. I emailed again and explained my mistake. There are lots of remedial courses I could take. However, today I would most like to take "Electricity and wiring for sysadmins". And on another note: Ack! My laptop's home partition is 93% full! How the hell did that happen? And again: How did I not know about <a href="http://linux.derkeiler.com/Mailing-Lists/Debian/2004-12/4313.html">apt-file</a>? This is perfect! (Touch o' the hat to Tears For Fears and <a href="http://blog.steve.org.uk/">Steve Kemp</a>; I'm moving closer every day to switching to <a href="http://www.steve.org.uk/Software/chronicle/">Chronicle</a>.) http://saintaardvarkthecarpeted.com/blog/2008-11/this_is_the_working_hour_we_are_paid_by_those_who_learn_by_our_mistakes.html How handy is *that*? I mean, are *those*? DNS and Emacs: <ul> <li> M-x dns-mode </li> <li> M-x dig </li> <li> <a href="http://www.mail-archive.com/debian-emacsen@lists.debian.org/msg00833.html">dig-browser.el</a> (Can't find a proper web page for it) </li> </ul> http://saintaardvarkthecarpeted.com/blog/2008-11/how_handy_is_that_i_mean_are_those.html