The Life of a Sysadmin

Tag: scripting

Here's a fun game: create a large (>1GB) file in your home directory called core and start Firefox. Have a look at this part of run-mozilla.sh:

if [ -x "$crc_prog" ]
then
        DEBUG_CORE_FILES=1
fi
if [ "$DEBUG_CORE_FILES" ]
then
        crc_old=
        if [ -f core ]
        then
                crc_old=`$crc_prog core | awk '{print $1;}' `
        fi
fi
##
## Run the program
##
"$prog" ${1+"$@"}
exitcode=$?
if [ "$DEBUG_CORE_FILES" ]
then
        if [ -f core ]
        then
                crc_new=`$crc_prog core | awk '{print $1;}' `
        fi
fi
if [ "$crc_old" != "$crc_new" ]
then
        printf "\n\nOh no!  %s just dumped a core file.\n\n" $prog
        printf "Do you want to debug this ? "
        printf "You need a lot of memory for this, so watch out ? [y/n] \"

Care to guess what'll happen? That's right: Firefox will take 10 seconds to start up because its busy md5summing a big-ass core file. The user will think that it hasn't launched at all and will click again. Rinse and repeat, with more and more clicking every time. By the time I figured out what was going wrong, the system load was about 200. Fortunately, its a simple thing to add DEBUG_CORE_FILES= judiciously (not DEBUG_CORE_FILES=0; I keep forgetting that a simple [ $FOO ] simply tests whether $FOO is empty, not whether its non-zero).

Also: the advantage to being in a small shop is that if youre the only one running Linux on the desktop, you can just go ahead and add things like the latest version of Firefox (now without the amusing bug that makes a search work on some other random tab, instead of the one you're looking at) and the MySQL DBD connector for Perl. It's really incredible how much irritation those two things are gonna save me.

Finally: this is just plain cool. As he did during the Bash scripting BOF,Wout takes me to school. Didn't know about: ssh -t, COLUMNS/LINES environment variables, tput, or just how much Applescript can do.

I have just spent two and a half hours trying to track down the reason a page on a Joomla site suddenly started saying "You are not authorized to view this resource". In the end it turned out to be a known problem with the OpenSEF plugin, but it took me a stupidly long time to even guess that might be the problem. (Probably shoulda searched for the error message first…)

There has to be a better way to do this. The only way I could figure to trace the problem was by sprinkling lots of print "FIXME: Made it here\n"; throughout the code. I know, it's a terrible way of doing it, but running php index.php didn't seem to work — I couldn't get the code to see the arguments I was trying to pass on. What am I missing?

At work, one of our visitors had a problem: his browser kept crashing whenever he visited crashing whenever he visited his university's webpage. That's what caused this problem, with the ginormous core files; Firefox would start grabbing memory at a rate of about 10-20MB/s, and then Solaris would kill it off once it got to around 2.5GB. Truss showed that it kept opening a copy of the Times Roman font over and over again, but I couldn't figure out why.

I tried duplicating it under my own account, but couldn't — so I moved my .mozilla out of the way and set up a new profile, which did have the problem. At first I figured it must be Javascript; I hate Javascript and almost always turn it off (but thanks to the NoScript plugin it's easy to toggle it for individual sites), so that must be it, right? Wrong. Okay, so what about prefs.js? Nope.

After copying over nearly all the files in my original profile, FF was still grabbing memory…until finally, out of desperation, I copied over the Flash plugin. And by the beard of Zeus, it worked!

I did some digging around (there are a truly depressing number of Mozilla bugs that mention Flash) and found mention (sorry, lost link) that some methods of detecting whether a browser has the Flash plugin can end up using all the memory available to the browser. That sounds similar enough to what I saw that I think I'm going to call that the problem-designate for now.

So this is 'way late. But a while back someone sent me a link to their Flash movie on www.elfyourself.com: a picture of their head on an elf, dancing and singing Xmas songs. Of course, at the end of it they offered you the chance to create one of your own. I was about to, when I noticed the terms of service. Check it out:

 GRANT OF RIGHTS.  By submitting a photograph or any other material
 (including, without limitation, vocal messages, text messages, or
 text) (each a "Submission"), I hereby grant to OMX, Inc., its
 subsidiaries and affiliated companies and each of their respective
 licensees, successors and assigns (collectively, "OfficeMax"), the
 unlimited right and permission to use the Submission or any part
 thereof (including, without limitation, my name, screen name, instant
 message name, or email address) throughout the universe, in
 perpetuity, in any manner or venue and for any purpose whatsoever,
 including, without limitation, for purposes of advertising, promotion
 or trade in promoting and publicizing OfficeMax and its products and
 services, by means of any and all media and devices whether now known
 or hereafter devised, which includes, without limitation, the
 unlimited right and permission to post the submission on the
 [reindeer.com, elf.com, pole.com, etc.] websites (collectively, the
 "Websites").  OfficeMax shall have the right, in its sole discretion,
 to edit, composite, morph, scan, duplicate or alter the Submission in
 any manner for any purpose which OfficeMax deems necessary or
 desirable, and I irrevocably waive any and all so-called moral rights
 I may have in the submission and I agree I shall have no right of
 approval, no claim to compensation, and no claim (including, without
 limitation, claims based upon invasion of privacy, defamation or right
 of publicity) arising out of any use, blurring, alteration, editing,
 morphing, distortion, illusionary effect, faulty reproduction,
 fictionalization, or use in any composite form of my name, picture,
 likeness, voice, and biographical information.

 GENERAL RELEASE AND LIMITATIONS ON LIABILITY.  By submitting my
 Submission, I agree that: (1) any and all disputes, claims, and causes
 of action arising out of or connected with the Submission shall be
 resolved individually, without resort to any form of class action; (2)
 any and all claims, judgements and awards shall be limited to actual
 out of pocket costs incurred, including costs associated with
 submitting the Submission, but in no event will attorneys' fees be
 awarded or recoverable; and (3) under no circumstances will I be
 permitted to obtain any award for, and I hereby knowingly and
 expressly waive all rights to seek, punitive, incidental or
 consequential damages and/or any other damages, other than actual out
 of pocket expenses, and/or any and all rights to have damage
 multiplied or otherwise increased.  I acknowledge and agree that
 OfficeMax is not responsible for any costs, injuries, losses, or
 damages of any kind arising from or in connection with: (i)
 transmission, or technical problems, failures, or malfunctions of any
 kind, whether originating with sender, with OfficeMax or otherwise,
 that may limit my ability to upload my Submission; and/or (ii) any
 injury or damage resulting from making the Submission (including,
 without limitation, claims, costs, injuries, losses and damages
 related to personal injuries, death, damage to, loss or destruction of
 property, or any claims, costs, injuries, losses or damages related to
 or based on my rights of publicity or privacy, or a claim that I have
 somehow been defamed or portrayed in a false light).

So: Office Max can take your picture and paste it into one of an Al-Qaida meeting. They can forward it on to the FBI. Not only that, they retain rights to the picture "throughout the universe, in perpetuity, in any manner or venue and for any purpose whatsoever". They can also spam you into submission long after you're dead. They could do this to the entire population of, say, Alberta, and you're not allowed to join a class-action lawsuit. And if you get sent to Guantanamo to rot out the rest of your existence, that's just tough; Office Max will not accept responsibility for that.

Incredible. Absolutely fucking incredible.

There is nothing worse than a problem that goes away once you restart the program. Case in point: ls -l /home ran atrociously slow (slowly?) on a Solaris 10 machine at work today. It's running Sun's DS 5.2 (or whatever they're calling it these days).

I've come across this problem before when I was trying to figure out how to get the thing to bind to itself by default as an LDAP client, rather than to one of the remote servers that're meant to be backups.

This time, though, that simply wasn't the problem: no traffic was going to the other machines at all. All I saw was looooooooooong lookup times for simple passwd stuff. Error logs showed nothing. Access logs swore blind that access times were on the order of zero nanoseconds. Truss showed it kept mmap() ing things; dtrace showed a whole lotta reads. I couldn't figure out more than that (which, natch, is my fault, not the tools).

In the end and out of desperation I restarted the server…which did the trick but left me frustrated that I'm no closer to figuring out what's going on with the damn thing.

As you can see, I've changed my blog a bit. I'm now using Gnu make, Perl, AsciiDoc and Emacs to generate everything. The old blog can still be found here, though i'll eventually be turning off comments for it.

Which brings me to another thing: comments on the new one are going to be a little funny, at least at first. Comments will be emailed to me; while I'll be scripting it eventually, for right now I'll be applying comments by hand. I'll write later about exactly what I'm trying to accomplish with all this, but right now my wife wants her laptop back. ("Are you writing in your new blog yet? Is it dreamy? Are you going to tell anyone where it is?" Aye, it's a fine marriage. :-)

Inna meantime, if you notice any problems please email me: aardvark at saintaardvarkthecarpeted dot com. In particular, I've tried to make sure that RSS continues to work with the old links; let me know if you run into problems.

From openbsd-misc:

 Do you have any idea how fucking insane the h.323 protocol is?  Anyone
 who runs a h.323 should get shoved out a window, beaten, flayed,
 spanked, shot, disembowled, hung, and forced to listen to hummpa music.  If
 you  want to firewall h.323, go commit yourself to an asylum with
 straight jackets and with padded walls -- at least you'll be in common
 company with the other linux wacko's.

Solved a ghostscript problem at work yesterday; not a big deal in itself, but I'd always had this impression that GS crashes were dark, nasty, impenetrable things that I could not possibly understand. I mean, c'mahn, look at this error:

$ ps2pdf report06w5060.ps
 Error: /invalidfont in findfont
 Operand stack:
Fi   87   --nostringval--   55   45   --nostringval--   65   74
 74   111   74   83   46   65   65   83   83   83   83   120   46   2
 --nostringval--   4
 6   83   83   46   74   83   74   83   83   12   --nostringval--   92
 83   101   1   --nostringval--   101   120   1   --nostringval--   138
 4   --nostring
 val--   120   120   101   101   120   111   101   101   19
 --nostringval--   55   42   1   --nostringval--   83   2
 --nostringval--   55   35   --nostringv
 al--   83   83   2   --nostringval--   --nostringval--   45   166.044
 Times-Italic   Font   Times-Italic   496086   Times-Italic
 --nostringval--   Times-It
 alic   NimbusRomNo9L-ReguItal   (NimbusRomNo9L-ReguItal)
 NimbusRomNo9L-ReguItal   (NimbusRomNo9L-ReguItal)
 NimbusRomNo9L-ReguItal
 Execution stack:
%interp_exit   .runexec2   --nostringval--   --nostringval--
 --nostringval--   2   %stopped_push   --nostringval--
 --nostringval--   --nostringval--   f
 alse   1   %stopped_push   1   3   %oparray_pop   1   3   %oparray_pop
 1   3   %oparray_pop   1   3   %oparray_pop   .runexec2
 --nostringval--   --nostring
 val--   --nostringval--   2   %stopped_push   --nostringval--
 --nostringval--   74   4   %oparray_pop   75   4   %oparray_pop
 --nostringval--   --nostringv
 al--   --nostringval--   --nostringval--   --nostringval--   false   1
 %stopped_push   78   5   %oparray_pop   --nostringval--
 --nostringval--   --nostring
 val--   5   -1   1   --nostringval--   %for_neg_int_continue
 --nostringval--   --nostringval--
 Dictionary stack:
--dict:1046/1123(ro)(G)--   --dict:0/20(G)--   --dict:75/200(L)--
 --dict:103/300(L)--   --dict:17/17(ro)(G)--
 --dict:1046/1123(ro)(G)--
 Current allocation mode is local
 Last OS error: 2
 Current file position is 95763
AFPL Ghostscript 8.00: Unrecoverable error, exit code 1

Then, in desperation, I JFGI and found the problem: for some reason, the fonts had disappeared. This is an old install with lots of overlapping installs of everything, so it's hard to tell why it might've happened. However, it should just be a matter of either getting rid of the old install (rm /opt/bin/gs* (and yes, I know that's bogus)) or setting GS_FONTPATH and GS_LIB appropriately. (Or figuring out why they got borked…hm.)

OTOH, on the same machine I've got The Case Of The Missing Java:

$ java
There was an error trying to initialize the HPI library.
Please check your installation, HotSpot does not work correctly
when installed in the JDK 1.2 Solaris Production Release, or
with any JDK 1.1.x release.
Could not create the Java virtual machine.

instead of (same version of Solaris, too):

$ java
Usage: java [-options] class [args...] (to execute a class)
or  java -jar [-options] jarfile [args...] (to execute a jar file)

which kind of worries me since its, like, Solaris and all, and java really should be working. Sigh.

Hi everyone...in case you haven't noticed, I've changed the software I use for my blog. This is just here for archival purposes; no comments or trackbacks or pings are allowed. The new blog is where all the action'll be. C'mon over and have a look!

libpst is a command-line tool that converts Outlook .pst files into standard mbox files, the way T&R intended. Wish I'd known about this before…

One of the outstanding feature requests is listing and extracting individual messages. Maybe I'll take a look at this.

In other news, I borked my home machine (Debian testing) by trying to extend a partition w/ReiserFS. That gave me a perfect excuse to upgrade to a bigger disk and reinstall Debian.

Next up is maybe looking at replacing my venerable copy of Slackware 9 with a Debian install, too; the ease of installing and upgrading Debian packages is just too good to pass up.

I did consider other OSs:

• FreeBSD: Even after three years, port-upgrade still scares me.
• NetBSD: meh, what's exciting about that?
• OpenBSD: Secure, yes, and God knows I'd like to use pf. But not easy to upgrade, either ports or releases.
• Dragonfly: Not yet.
• BSDs in general: I want a journalled FS.
• OpenSolaris: If I wanted to upgrade everything by hand, I'd stick with Slackware.

And yes, I realize I'm damned ignorant, and that a server should not be exciting. But I'm convinced that a big part of running a server successfully is ease of upgrading, whether security fixes or new app versions, and Debian is just wonderful.

People have been calling me out on my last post, and that's good; I love a good argument^Wdebate, and doubly so when it comes from people w/more experience than me. So I'm going to start responding to the comments, laying out where I'm wrong and where I still think I'm right.

I said:

OpenSolaris: If I wanted to upgrade everything by hand, I'd stick with Slackware.

Bzzt! As I found on on a recent episode of BSDTalk, NetBSD's pkgsrc is available for over nine hundred thousand operating systems, including Solaris and Slackware Linux. Tha's right, both premises in that statement were wrong.

Not only that, pkgsrc can be tucked out of the way so that it doesn't interfere with the rest of the system…so I could even throw it on Thornhill right now, Slackware and all, and start using it instead of my own half-assed build script for Apache/SSH/PHP/OpenSSL/mod_ssl (which, in my own defence, works pretty darned well).

In fact, tomorrow I'm heading out to The Other University to set up two new X4200 servers, and I'm seriously considering adding pkgsrc to them — if only to avoid having to compile (and botch) Lapack and Blas. If that goes well, I may start adding it to the main server here so that we can easily get more up-to-date versions of Firefox et al. (Though I could probably get them from Blastwave…this has been a low enough priority for me so far that I haven't really looked into all my options.)

That is not to say it's perfect:

• While it has six thousand packages or so in its tree, only (…) something like two or three thousand compile.
• Upgrading is less than perfect; as you'd expect, the process is basically remove-and-recompile…and since that goes for dependencies as well (at least in the default case), it can potentially be a while before it all gets back to a useable state:

It is possible, and in the case of updating a package with
hundreds of dependencies, arguably even likely that the process will
fail at some point. One can fix problems and resume the update by
typing make update in the original directory, but the system can have
unusuable packages for a prolonged period of time. Thus, many people
find 'make update' too dangerous, particularly for something like glib
on a system using gnome.

To use binary packages if available with "make update", use
"UPDATE_TARGET=bin-install". If package tarball is not available in
${PACKAGES} locally or at URLs (defined with BINPKG_SITES), it will
build a package from source.

To enable manual rollback one can keep binary packages. One method is
to always use 'make package', and to have "DEPENDS_TARGET=package" in
/etc/mk.conf. Another is to use pkg_tarup to save packages before
starting.

From the Swedish NetBSD wiki.

It's nice that manual rollback is doable; that's always my big paranoia when it comes to source-based upgrades.

• Upgrading Gnome in particular is a fucking bear.

That last complaint is not as fair as it could be. I mean, I'm not going to be upgrading Gnome on either Thornhill or the two new Sun machines. And at around 80 packages, it would be damned difficult to try and recompile it all without starting with a clean slate. But this sort of nonsense with Gnome is what put me off the ports tree in FreeBSD.

(I was going to put in something about how Debian doesn't need that sort of thing, but I should research that first.)

• Word-by-word diff of GPLv2 and v3
• New version of Unix Backup and Recovery, by W. Curtis Preston. The first was amazing; this one covers Windows and Mac OS X as well. WANT.
• Good reading and good suggestions. I need to alias ls, fg, mutt, sudo, and to shorten my alias for SSHing to my webserver.
• I think I finally figured out what was going on when my desktop machine at work had a suddenly borked mouse/X relationship: double-clicking was inconsistent, single-clicking was inconsistent, but the keyboard still worked. It happened again a few days ago, and I managed to run xev and verify that things really were messed up: the mouse and all its clicks and motions just weren't registering at all. I managed to figure out what was (probably) going on when I came across this link. In the end, what worked for me was switching to a console and running cat /dev/input/mouse1. Don't know why (yet!) but that seemed to reset everything.

Still to come: Why upgrading is the most important thing EVAR.

Brian Cain told me to get my ass in gear and try out a Nexenta (I'm elaborating on his words a little) and I'm glad he did. I've installed it on my desktop machine on a second hard drive, and I have to say I'm impressed so far. Debian plus Solaris…damn, girl. Damn.

Everything is a .deb package, including all the SUNW stuff, and there appear to be a ton more packages available. Mutt and Emacs are there, as is procmail and fetchmail; I may see if I can get a package going for icewm, which'd be just about all I need. (Of course, ratpoison's already available…) (Update: someone's already working on it; haven't made it to the end of the thread yet, but it may already be done.) There was a lot of Gnome stuff installed that I don't want, but that's okay; Nexenta's deliberately emulating/duplicating Ubuntu, and anyway the install disk (which comes with Tetris, btw) has a minimal option which I suspect'd be right up my alley (for server or desktop).

I'm in the process of creating a zone right now, into which'll go Apache2 and MySQL. I did trip over these bugs in the process, but apt-get dist-upgrade fixed the first and some judicious editing of /usr/lib/nexenta-zones/elatte-unstable.bootstrap fixed the second (I'm guessing they haven't made a new package since the fix). (Update: My own damn fault for not noticing that the new version was in unstable, not testing. I'm upgrading now.) /export/home was set up with ZFS, and I've made a snapshot already. The GRUB menu entry was not correct — it pointed at the primary IDE drive (hd0) instead of the second (hd1) — but again, that was easily fixed. I should file a bug on that.

I still have some questions. I'd like to know (and will try to watch to find out) how often they update their packages, especially security fixes. I'm curious to see how closely they follow OpenSolaris.org development…though since I only have a hazy idea how OS.org do it, I'm not really sure what to look for. And of course, this is an unstable distro; I might want to hold off on replacing the server with it.

But for desktop use and/or experimentation, this is neat stuff. I can always get my mail on my firewall if need be. :-)

I installed RT at work a couple days ago using pkgsrc. This was the first time I'd ever used pkgsrc, and I have to say I'm impressed. Yes, it's just like a portable ports tree — but it's just like a portable ports tree, and I'm starting to think that's a very, very powerful idea.

RT went well except for the final install, where it complained and died. Fortunately, it turned out to be susceptible to exactly the sort of one-line patch that I have an affinity for. Not as cool as correcting Theo de Raadt's code, mind you :-) but still a good feeling.

Ah...RT, I've missed you.