$ ls foo foo $ ls | grep foo $ echo $? 141
I mentioned that I've been having problems with Bacula recently. These have been aggravated by the fact that the trigger seems to be a job that takes 53 hours to finish.
Well, I think I've got a handle on one part of the problem. See, when Bacula is doing this big job, other jobs stack up behind it -- despite having two tape drives, and two separate pools of tapes, and concurrent jobs set up, the daily jobs don't finish. The director says this:
9279 Full BackupCatalog.2010-02-20_21.10.00_10 is waiting for higher priority jobs to finish 9496 Full BackupCatalog.2010-02-23_21.10.00_13 is waiting execution 9498 Full bigass_server-d_drive.2010-02-24_03.05.01_15 is running 9520 Increme little_server-var.2010-02-24_21.05.00_38 is waiting on Storage tape 9521 Increme little_server-opt.2010-02-24_21.05.00_39 is waiting on max Storage jobs
but storage says this:
Running Jobs: Writing: Full Backup job bigass_server-d_drive JobId=9498 Volume="000031"
pool="Monthly" device="Drive-0" (/dev/nst1) spooling=1 despooling=0 despool_wait=0 Files=708,555 Bytes=1,052,080,331,191 Bytes/sec=11,195,559 FDReadSeqNo=22,294,829 in_msg=20170567 out_msg=5 fd=16
Writing: Incremental Backup job little_server-var JobId=9508 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0) spooling=0 despooling=0 despool_wait=1 Files=156 Bytes=3,403,527,093 Bytes/sec=72,415 FDReadSeqNo=53,041 in_msg=52667 out_msg=9 fd=9
Writing: Incremental Backup job little_server-etc JobId=9519 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0) spooling=0 despooling=0 despool_wait=0 Files=9 Bytes=183,606 Bytes/sec=3 FDReadSeqNo=72 in_msg=50 out_msg=9 fd=10
Writing: Incremental Backup job other_little_server-etc JobId=9522 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0) spooling=0 despooling=0 despool_wait=1 Files=5 Bytes=182,029 Bytes/sec=3 FDReadSeqNo=45 in_msg=32 out_msg=9 fd=19
Writing: Incremental Backup job other_little_server-var JobId=9525 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0) spooling=0 despooling=0 despool_wait=0 Files=0 Bytes=0 Bytes/sec=0 FDSocket closed
Out of desperation I tried running "unmount" for the drive holding the daily tape, thinking that might reset things somehow...but the console just sat there, and never returned a prompt or an error message. Meanwhile, storage was logging this:
cbs-01-sd: dircmd.c:218-0 <dird: unmount SL-500 drive=1 cbs-01-sd: dircmd.c:232-0 Do command: unmount cbs-01-sd: dircmd.c:596-0 Try changer device Drive-0 cbs-01-sd: dircmd.c:617-0 Device SL-500 drive wrong: want=1 got=0 skipping cbs-01-sd: dircmd.c:596-0 Try changer device Drive-1 cbs-01-sd: dircmd.c:612-0 Found changer device Drive-1 cbs-01-sd: dircmd.c:625-0 Found device Drive-1 cbs-01-sd: block.c:133-0 Returning new block=39cee10 cbs-01-sd: acquire.c:647-0 JobId=0 enter attach_dcr_to_dev
...and then just hung there. "Aha, race condition!" I thought, and sure enough a bit of searching found this commit in November: "Fix SD DCR race condition that causes seg faults". No, I don't have a segfault, but the commit touches the last routine I see logged (along with a buncha others).
This commit is in the 5.0.1 release; I wasn't planning to upgrade to this just yet, but I think I may have to. But I'm going on vacation week after next, and I'm reluctant to do this right before I'm away for a week. What to do, what to do...
While trying to figure out why Nagios was suddenly unable to check up on our databases, I suddenly realized that the permissions on /dev/null were wrong: 0600 instead of 0666. What the hell? I've had this problem before, and I was in the middle of something, so I set them back and went on with my life. Then in happened again, not half an hour later. I was in the same shell, so I figured it had to have been a command I'd run that had inadvertantly done this.
Yep: don't run the MySQL client as root. Yes yes yes, it's bad
anyway, I'll go to sysadmin hell, but this is an interesting bug. The
MYSQL_HISTFILE is set to
root...and when you exit the client, it sets the permissions for the
history file to 0600. So, you know, don't do that then. (Still no
fix committed, btw...)
Just spent the better part of five hours cleaning up four old, out-of-date Wordpress installations after they got infected with this worm. I host nine sites on my home server for friends and family; I'm cutting that down to three (just family), and maybe looking at mu-wordpress, as of Real Soon Now.
Happy Labour Day, everyone!
Update: I meant to add in here a few things I looked for, because this info was hard to track down.
I found extra admin-level users in the wp_users table; some had their email address set to "firstname.lastname@example.org", some had random made-up or possibly real addresses, and some had the same email address as already-existing users.
On one blog (possibly infected much earlier) I found 42,000 (!!) approved, spammy comments.
I searched for infected posts using a query from here:
SELECT * FROM wp_posts WHERE post_content LIKE '%iframe%' UNION SELECT * FROM wp_posts WHERE post_content LIKE '%noscript%' UNION SELECT * FROM wp_posts WHERE post_content LIKE '%display:%'
I just spent the weekend (well, like an hour a day...kids, life, you know how it is) trying to track down why a bunch of new CentOS 5.2 installs at $job_2 couldn't pipe:
$ ls foo foo $ ls | grep foo $ echo $? 141
(Actually, I didn't think to look at the error code 'til someone else pointed it out…141 turns out to be SIGIPE) In the end, it would have been quicker if I'd simply searched for the first thing I saw when logging in:
-bash: [: =: unary operator expected -bash: [: -le: unary operator expected
This was particularly aggravating to track down because not every machine was doing this, and no matter what I thought to look at (/etc contents, /tmp permissions (those have a habit of going wonky on me for some reason), SELinux) I couldn't figure out what was different.
Turned out to be an upstream bug in nss_ldap. (The Bugzilla entry makes for some interesting reading, to be sure…) And I didn't see it on each machine because I hadn't upgraded after installation on all machines. (They're not yet in production, and I'm working on getting my kickstart straight.)
Man, it was gratifying to upgrade nss_ldap and see the problem go away…
Heads up for those of you using Blastwave and CUPS: after upgrading to the latest stable version, printing stopped working for me (and a few users :-). I eventually tracked it down to the movement of two files: suddenly
were moved to
resulting in many error messages like Unsupported format text/plain! and
Hint: is ESP ghostscript installed?. Moving them both back into place and restarting CUPS fixed things just fine.
One of the problems I've been working on since the upgrade to Solaris 10 has been the slowness of the SunRay terminals. There are a few different problems here, but one of 'em is that after typing in your password and hitting Enter, it takes about a minute to get the JDS "Loading your desktop…" icons up.
I scratched my head over this one for a long time 'til I saw this:
ptree 10533 906 /usr/dt/bin/dtlogin -daemon -udpPort 0 10445 /usr/dt/bin/dtlogin -daemon -udpPort 0 ``` 10533 /bin/ksh /usr/dt/config/Xstartup 10551 /bin/ksh -p /opt/SUNWut/lib/utdmsession -c 4 10585 /bin/ksh -p /etc/opt/SUNWut/basedir/lib/utscrevent -c 4 -z utdmsession 10587 ksh -c echo 'CREATE_SESSION 4 # utdmsession' >/dev/tcp/127.0.0.1/7013
which just sat there and sat there for, oh, about a minute. So I run netcat on port 7013, log out and log in again, and boom! quick as anything.
utscreventd 7013/tcp # SUNWut SRCOM event deamon
which we're not running; something to do with smart cards. So why does
it hang so long? Because for some reason, the host isn't sending back
an RST packet (I presume; can't listen to find out) to kill the
connection, like it does on
So now I'm trying to figure out why that is. It's not the firewall;
they're identical. I've tried looking at
ndd /dev/tcp \? but I don't
see anything obvious there. My google-fu doesn't appear to be up to
the task either. I may have to cheat and go visit a fellow sysadmin to
I tried ripping a CD recently on my desktop machine running Debian testing. grip seemed to hang, and I kept getting this error message in the logs:
hdc: write_intr: wrong transfer direction!
Google didn't turn up much that helped, except to suggest a simpler test case (
cdparanoia -d /dev/hdc 1-1). Data CDs seemed to work just fine.
Finally tried upgrading the kernel package, from
kernel-image-2.6.8-2-386 (the default kernel after installation) to
linux-image-2.6.18-4-686, and that did the trick nicely.
Disgust is the feeling you get when you realize that the sudden and
mysterious failures you've been tracking down for the last half hour
happened because you typed
$debug == 1;.
At work, one of our visitors had a problem: his browser kept crashing whenever he visited crashing whenever he visited his university's webpage. That's what caused this problem, with the ginormous core files; Firefox would start grabbing memory at a rate of about 10-20MB/s, and then Solaris would kill it off once it got to around 2.5GB. Truss showed that it kept opening a copy of the Times Roman font over and over again, but I couldn't figure out why.
I tried duplicating it under my own account, but couldn't — so I moved
.mozilla out of the way and set up a new profile, which did have
plugin it's easy to toggle it for individual sites), so that must
be it, right? Wrong. Okay, so what about
After copying over nearly all the files in my original profile, FF was still grabbing memory…until finally, out of desperation, I copied over the Flash plugin. And by the beard of Zeus, it worked!
I did some digging around (there are a truly depressing number of Mozilla bugs that mention Flash) and found mention (sorry, lost link) that some methods of detecting whether a browser has the Flash plugin can end up using all the memory available to the browser. That sounds similar enough to what I saw that I think I'm going to call that the problem-designate for now.