A while back I migrated Bacula, along with my tape library, from a Linux machine to a FreeBSD machine. The FreeBSD server is a whitebox with a bunch of hard drives, and those + ZFS == 30 TB of backup space. Which is sweet.
This lets me try doing disk-to-disk-to-tape with Bacula. Mostly working, but one thing I notice is that it is damn slow getting to the end of a tape to append data when I want to migrate -- like, half an hour or more. This is way longer than the Linux machine ever took, and that's doubly weird because I moved over the FC cards with the migration so it's the same hardware (though perhaps different drivers). So WTF?
Turns out, I think, that this is a shortfall in FreeBSD. Follow the bouncing ball:
Tape drives write one EOF (end-of-file) marker for the end of a file.
Bacula writes its jobs in chunks -- I have it set up to write 8GB chunks -- and the tape driver marks each one with an EOF.
When a tape is loaded, Bacula needs to get to the end of a tape. That means lots of "please look for the next EOF marker", which takes a while.
There's a shortcut, though: the MTIOCGET ioctl can (I think) keep track of how many EOFs are on the tape. That means the driver can just tell the tape drive "please fast-forward to the nth EOF marker."
Linux implements MTIOCGET.
FreeBSD does not.
Now, this is all stuff I've put together from various mailing list posts, so I might be wrong...but it seems to explain what's going on. There's also a suggestion that I'm wrong about this -- that FreeBSD does support MTIOCGET. So much for conclusions. Stay tuned.
Did you know there was a fork of Bacula named Bareos? Not I. Not sure whether to pronounce it "bar-ee-os" or "bear-o-s". Got Kern Sibbald, Bacula's creator, rather upset. He promises to bring over "any worthwhile features"...which is good, because there are a lot.
Post by Matthew Green titled "How does the NSA break SSL?". Should be reading that now but I'm writing this instead.
I have not read The Phoenix Project, which makes me a bad person for reacting so viscerally to things like "A personal reinterpretation of the three ways" and the self-congratulatory air of the headshot gallery. I'm trying to figure out why I react this way, and whether it's justified or just irrational dislike of people I perceive as outsiders. Seriously, though, the Information Technology Process Institute?
Got Netflix at home? Got IPv6? That might be why they think you're in Switzerland and change your shows accordingly. In my case, they thought I was in the US and offered to show "30 Rock" and "Europa Report"...until I tried to actually stream them and they figured out the truth. Damn.
Test-Driven Infrastructure with Chef. Have not used Chef before, but I like the approach the author uses in the first half of the book: here's what you need to accomplish, so go do it. The second half abandons this...sort of unfortunate, but I'm not sure explaining test infrastructure libraries (ServerSpec, etc) would work well in this approach. Another minor nitpick: there's a lot of boilerplate output from Chef et al that could easily be cut. Overall, though, I really, really like this book.
Mencius Moldbug, one of the most...I mean...I don't even. Jaw-droppingly weird. Start with "Noam Chomsky killed Aaron Swartz".
I have a love-hate relationship with Bacula. It works, it's got clients for Windows, and it uses a database for its catalog (a big improvement over what I'd been used to, back in the day, from Amanda...though that's probably changed since then). OTOH, it has had an annoying set of bugs, the database can be a real bear to deal with, and scheduling....oh, scheduling. I'm going to hold off on ranting on scheduling. But you should know that in Bacula, you have to be explicit:
Schedule {
Name = "WeeklyCycle"
Run = Level=Full Pool=Monthly 1st sat at 2:05
Run = Level=Differential Pool=Daily 2nd-5th sat at 2:05
Run = Level=Incremental IncrementalPool=Daily FullPool=Monthly 1st-5th mon-fri, 2nd-5th sun at 00:41
}
This leads to problems on the first Saturday of the month, when all those full backups kick off. In the server room itself, where the backup server (and tape library) are located, it's not too bad; there's a GigE network, lots of bandwidth, and it's a dull roar, as you may say. But I also back up clients on a couple of other networks on campus -- one of which is 100 Mbit. Backing up 12 x 500GB home partitions on a remote 100 MBit network means a) no one on that network can connect to their servers anymore, and b) everything takes days to complete, making it entirely likely that something will fail in that time and you've just lost your backup.
One way to do that is to adjust the schedule. Maybe you say that you only want to do full backups every two months, and to not do everything on the same Saturday. That leads to crap like this:
Schedule {
Name = "TwoMonthExpiryWeeklyCycleWednesdayFull"
Run = Level=Full Pool=MonthlyTwoMonthExpiry 2nd Wed jan,mar,may,jun,sep,nov at 20:41
Run = Level=Differential Pool=Daily 2nd-5th sat at 2:05
Run = Level=Differential Pool=Daily 1st sat feb,apr,jun,aug,oct,dec at 2:05
Run = Level=Incremental IncrementalPool=Daily FullPool=MonthlyTwoMonthExpiry 1st-5th mon-tue,thu-fri,sun, 2nd-5th wed at 20:41
Run = Level=Incremental IncrementalPool=Daily FullPool=MonthlyTwoMonthExpiry 2nd Wed jan,mar,may,jun,sep,nov at 20:41
}
That is awful; it's difficult to make sure you've caught everything, and you have to do something like this for Thursday, Friday, Tuesday...
I guess I did rant about Bacula scheduling after all.
A while back I realized that what I really wanted was a queue: a list of jobs for the slow network that would get done one at a time. I looked around, and found Perl's IPC::DirQueue. It's a pretty simple module that uses directories and files to manage queues, and it's safe over NFS. It seemed a good place to start.
So here's what I've got so far: there's an IPC::Dirqueue-managed queue that has a list of jobs like this:
I've got a simple Perl script that, using IPC::DirQueue, take the first job and run it like so:
open (BCONSOLE, "| /usr/bin/bconsole");
print BCONSOLE "run job=" . $job . " level=Full pool=Monthly yes";
close (BCONSOLE);
I've set up a separate job definition for the 100Mbit-network clients:
JobDefs {
Name = "100MbitNetworkJob
Type = Backup
Client = agnatha-fd
Level = Incremental
Schedule = "WeeklyCycleNoFull"
Storage = tape
Messages = Standard
Priority = 10
SpoolData = yes
Pool = Daily
Cancel Lower Level Duplicates = yes
Cancel Queued Duplicates = yes
RunScript {
RunsWhen = After
Runs On Client = No
Command = "/path/to/job_queue/bacula_queue_mgr -c %l -r"
}
}
"WeeklyCycleNoFull" is just what it sounds like: daily incrementals, weekly diffs, but no fulls; those are taken care of by the queue. The RunScript stanza is the interesting part: it runs baculaqueuemgr (my Perl script) after each job has completed. It includes the level of the job that just finished (Incremental, Differential or Full), and the "-r" argument to run a job.
The Perl script in question will only run a job if the one that just finished was a Full level. This was meant to be a crappy^Wsimple way of ensuring that we run Fulls one at a time -- no triggering a Full if an Incremental has just finished, since I might well be running a bunch of Incrementals at once.
It's not yet entirely working. It works well enough if I run the queue manually (which is actually tolerable compared to what I had before), but Bacula running the "baculaqueuemgr" command does nto quite work. The queue module has a built-in assumption about job lifetimes, and while I can tweak it to be something like days (instead of the default, which I think is 15 minutes), the script still notes that it's removing a lot of stale lockfiles, and there's nothing left to run because they're all old jobs. I'm still working on this, and I may end up switching to some other queue module. (Any suggestions, let me know; pretty much open to any scripting language.)
A future feature will be getting these jobs queued up automagically by Nagios. I point Nagios at Bacula to make sure that jobs get run often enough, and it should be possible to have Nagios' event handler enqueue a job when it figure's it's overdue.
For now, though, I'm glad this has worked out as well as it has. I still feel guilty about trying to duplicate Amanda's scheduling features, but I feel a bit like Macbeth: in blood stepped in so far....So I keep going. For now.
After a lot of faffing about, I've accomplished the following on the backup server at $WORK:
Broken out /var/lib/mysql to a separate, mirrored, Linux software
raid-1 275 GB partition; it's using about 36 GB of that at the
moment, which is 15% -- the lowest it's been in a long, LONG-ass
time.
Migrated the Bacula catalog db to Innodb.
Shrunk the raid-0 spool partition to about 1.6 TB, down from 2
TB; did this to free up the two disks for the mirrored partition
Ensured that MySQL will use /dev/shm as a temporary area
Sped up the restoration of files (which was mostly because of
earlier "analyze" commands on the File table while it was still
MyISAM)
innodbfilepertable is on; innodbbufferpoolsize=10G; ``` defaultstorageengine=InnoDB
I encountered the following problems:
* The stupid raid card in the backup server only supports two RAID drives --
thus, the mirrored drive for /var/lib/mysql is Linux software raid. I'd have preferred to keep things consistent, but it was not to be. ```
The many "analyze" and "repair" steps took HOURS...only to turn
out to be deadlocked because it was running out of tmp space.
I had to copy the mysql files to the raid-0 drive to have enough
space to do the conversion.
Knock-on effects included lack of sleep and backups not being run
last night
Basically, this took a lot of tries to get right, and about all ``` of my time for the last 24 hours.
I learned:
* The repair of the File table while MyISAM, with tmp files in
/dev/shm, took about 15 minutes. That compares with leaving it overnight and still not having it done. ```
You have to watch the mysql log file for errors about disk space,
and/or watch df -h to see /tmp or whatever fill up.
You can interrupt a repair and go back to it afterward if you
have to. At least, I was able to...I wouldn't do it on a regular
basis, but it gives me cautious optimism that it's not an
automatic ticket to backups.
Importing the File.sql file (nominally 18 GB but du shows 5 ``` GB...sparse?), which converted it to InnoDB, took 2.5 hours.
I still have to do these things:
* Update documentation.
* Update Bacula checks to include /var/lib/mysql.
* Perhaps up pool_size to 20 GB.
* Set up a slave server again.
* A better way of doing this might've been to set up LVM on md0, then use snapshots for database backup.
* Test with a reboot! Right now I need to get some sleep.
A while back I upgraded the MySQL for Bacula at $WORK. I tested it afterward to make sure it worked well, but evidently not thoroughly enough: I discovered today that the query needed to restore a file was taking forever. I lost patience at 30 minutes and killed the query; fortunately, I was able to find another way to get the file from Bacula. This is a big slowdown from previous experience. Time for some benchmarking and exploratory tweaking...
Incidentally, the faster way to get the file was to select "Enter a list of files to restore", rather than let it build the directory tree for the whole thing. The fileset in question is not that big, so I think I must be doing something pretty damned wrong to get MySQL so slow.
On Tuesday I attempted migrating the Bacula database at work from MyISAM to InnoDB. In the process, I was also hoping to get the disk space down on the /var partition where the tables resided; I was runnig out of room. My basic plan was:
Here was the shell script I used to split the dump file, change the engine, and reload the tables:
csplit -ftable /net/conodonta-private/export/bulk/bacula/bacula.sql '/DROP TABLE/' {*}
sed -i 's/ENGINE=MyISAM/ENGINE=InnoDB/' table*
for i in table* ; do mv $i $(head -1 $i | awk '{print $NF}' | tr -d '`' | sed -e's/;/.sql/') ; done
for i in $(du *sql | sort -n | awk '{print $NF}') ; do echo $i; mysql -u bacula -ppassword bacula < $i ; done
(This actually took a while to settle on, and I should have done this part of things well in advance.)
Out of 31 tables, all but three were trivially small; the big ones are Path, Filename and File. File in turn is huge compared with the others.
I had neglected to turn off binary logging, so the partition kept filling up with binary logs...which took me more than a few runs through to figure out. Eventually, late in the day, I switched the engines back to MyISAM and reloaded. That brought disk space down to 64% (from 85%). This was okay, but it was a stressful day and one that I'd brought on myself for now preparing well.
When next I do this, I will follow this sequence:
I've got Nagios set to alert me if Bacula jobs go too long without running. Then I re-run them. This is a pain, so I made a better way to do this. I've got a Mutt macro that looks like this:
macro index ,b "|rerun_from_nagios_or_bacula.sh\n\n" "Rerun bacula job."
And in turn, the shell script looks like this:
#!/bin/bash
awk '/PROBLEM.*backup is/ {print "run job=" $6 " level=" $7 " yes"}' |
sed -e's/level=backup//; s#=.*/#=#' |
ssh backup-server -t bconsole
Of course, a better way of having all this work would be an actually-working scheduler in Bacula. But this makes the pain bearable for one more day.
A user at $WORK was running a series of jobs on the cluster -- dozens at any moment. Other users have their quota set to 60 GB, but this user was not (long story). His home directory is at 400GB, but it was closer to a terabyte not so long ago....right when we had a hard drive and a tape drive fail at the same time on our backup server.
We do backups every night to tape using Bacula. Most backups are incremental (whatever changed since the last backup, usually the day before) and are small...maybe tens of GB per day. But backups for this user, because of the proliferation of logs from his jobs, were closer to the size of his home directory every day -- simply because all these log files were being updated as each job progressed.
Ordinarily this wouldn't be a problem, but the cluster of hardware failures have really fucked things up; they're better now, but I'm very slowly playing catchup backups. Eating a tape or more every day is not in my budget right this moment.
I asked him if any of the log files could be excluded from backups without any great loss. After talking it over with him, we came to this agreement:
This would exclude lots of other files like "1rep2.foo", "8rep9.log", etc, and would cut out about 200 GB of useless churn every day.
Bacula has the ability to do this sort of thing...but I found its methods somewhat counterintuitive, so I want to set down what I did and how I tested it.
First off, the original, let's-include-everything FileSet looked like this:
FileSet {
Name = "example"
Include {
File = /home/example
Options {
signature = SHA1
}
}
Exclude {
File = /proc
File = /tmp
File = /.journal
File = /.fsck
File = /.zfs
}
}
We back up everything under /home/example, we keep SHA1 signatures, and we exclude a handful of directories (most of which are boilerplate, applied to every FileSet by default).
In order to get Bacula to change the FileSet definition, you have to get the director to reload its configuration file. But some errors -- not all -- cause a running bacula-dir process to die. So before I started fiddling around, I added a Makefile to the /opt/bacula/etc directory that looked like this:
test:
@/opt/bacula/sbin/bacula-dir -t && echo "bacula-dir.conf looks good" || echo "problem with bacula-dir.conf"
reload: test
echo "reload" | /opt/bacula/sbin/bconsole
Whenever I made a change, I'd run "make reload", which would test the configuration first; if it failed, bacula would not be reloaded. (The "@" symbol, in a Makefile, discards standard output.)
Next, I needed a listing of what we were backing up now, before I started fiddling with things:
echo "estimate job=fileserver-example listing" | bconsole > /tmp/listing-before
The "estimate" command gets Bacula to estimate how big the job is; the "listing" argument tells it to list the files it'd back up. By default it gives you the info for a full backup. (You can also append a joblevel, so you can see how big a Differential or Incremental; I didn't need that here, but it's worth remembering for next time.)
After that, I made another Makefile that looked like this:
test: estimate shouldwork shouldfail
estimate:
@echo "estimate job=fileserver-example listing" | bconsole > /tmp/listing-after ; wc -l /tmp/listing*
shouldwork: estimate
grep rep0 /tmp/listing-before | grep projects/output | while read i ; do grep -q $$i /tmp/listing-after || exit 1 ; done
shouldfail:
grep rep2 /tmp/listing-before |grep projects/output | while read i ; do grep -q $$i /tmp/listing-after && exit 1 ; done ; true
This is a little hackish, so in detail:
The estimate target gets an updated listing of what Bacula will back up; the line count lets me eyeball how it compares to the old, all-inclusive listing.
The shoudwork target gives me a quick way to make sure that all the files with "rep0" in the name and "projects/output" in the path are still in that updated listing. We grep for these files in the new listing; it either works or exits with error code 1, which make will catch and declare an error.
The shouldfail target is similar, except I'm making sure that files with "rep2" in the name are excluded from the new listing and we're short-circuiting the loop if any line is found. The "true" at the end is there to give make a final success; we only make it to that command if the entire loop has not found anything, which is what we want. It's there to make this test a "MUST NOT". (That's probably not explained very well.)
Anyhow: after each change, I'd run "make reload" as root to make sure that the syntax worked. After that, I'd run "make test" as an ordinary user (no need for root privileges) to make sure that I was on the right track. After a while, I got this:
FileSet {
Name = "example"
Include {
File = /home/example
Include {
Options {
signature = SHA1
Wilddir = /home/example/projects/output
Exclude = yes
}
}
}
Include {
File = /home/example/projects/output
Options {
WildFile = "*rep0*"
Signature = SHA1
}
Options {
Exclude = yes
RegexFile = ".*"
}
}
Exclude {
File = /proc
File = /tmp
File = /.journal
File = /.fsck
File = /.zfs
}
}
Again, this is a little counterintuitive to me, so here's how it works out.
The first "Include" stanza is the same, except that in the "Options" section we're excluding "/home/example/projects/output". That's what the "Wilddir" and "Exclude = yes" directives are for.
The second "Include" stanza puts the "/home/example/projects/output" back in, but modified with two "Options" sections: the first to include "rep0" (a simple fileglob) and the second to exclude everything. What ends up being included by this stanza is the union of those two options: only files named "rep0" in the directory "/home/example/projects/output".
Last, the third stanza is our standard "Exclude" boilerplate.
After I was confident that I had the right set of files excluded, I sent the user a list of files to confirm that all was well:
cat /tmp/listing_before | while read i ; do grep -q $i /tmp/listing_after || echo $i ; done > /tmp/excluded
Now, I'm the first to admit that that is ugly. Diff, useless use of cat...lots of objections to raise. But it's been a long day and I got what I wanted. I pointed the user at it, made sure it was okay, and committed the changes.
All in all, this gave me a good loop for testing: it caught fatal errors before they happened, it let me be sure I was excluding the right things, and I was able to work in a stepwise fashion to get where I wanted.
This is an attempt to lay out my problems with Bacula, and to be explicit about what I hope to achieve by replacing it (if, in fact, I do go ahead with that). If I'm wrong, correct me.
Too many long jobs monopolize spool space, storage job slots, and generally hold up production.
My largest jobs right now are around 1-2 TB -- and in order to accomplish that, I need to manually split up filesystems using a messy syntax. A job running that long will cycle through
many, many times. During spooling, a slot of storage space jobs is used. During despooling, no other job can despool to that tape drive. Often, this ends up holding up a lot of other jobs. If there's a problem, I'm faced with a choice between killing a job that's been running for days, or letting lots of other stuff go swithout backups until/unless it finishes.
More generally, I'm faced with a choice between letting everything run forever at the beginning of the month (because it's simplest to schedule fulls for the first Saturday or some such), or juggling schedules manually to stagger things (which I'm doing now, and leads to schedules like FullBackupSecondSundayAfterLent).
Possible fixes:
Bacula seems to get confused easily about what tapes are available for use.
Bacula's storage daemon seems to often hold on to outdated info about what tapes are in what state.
Example: the daily pool is full, so jobs are halted. Status storage shows it's waiting for a drive to be created for the daily pool. I move a volume from another pool, then have to attempt to mount it manually in the appropriate drive -- the storage daemon doesn't pick up on this change automatically.
Sometimes this works, and sometimes it doesn't. Sometimes both are waiting for a tape from the same pool; creating one doesn't let the jobs queued up on the other drive run on that new tape, but rather you need to create a second new tape and mount it. On top of that, sometimes the jobs hang around on the storage daemon still waiting for a new tape -- or something...because they don't get out of the way, and let other jobs run in their place, unless they're cancelled (and sometimes only when bacula-sd is restarted).
This may be fixed with the upgrade to 5.2.6. However....
The new version of Bacula crashes when I run too many jobs at once.
That's 5.2.6, upgraded to from 5.0.2 (time got away on me, yes). And by too many I mean, like, 50. That's not too many! I'm not sure what the hell's going on, though at least now I have a backtrace. I'm seriously pissed off about this point. Yes, I'll file a bug, but this is annoying.
All in all, I spend far too much time babysitting Bacula.
It's extremely high maintenance, and that's pissing me off. Understand, this is coming after a long weekend spent babysitting it, trying to make sure some jobs got written. There are other problems at work, yes, but this is not meant to be so hard.
Periodically I remove tapes at $WORK from our tape library to keep them somewhere else. getmonthlytapes is a Perl script that helps me do just that. Released under the GPL; share and enjoy!
Just compacted the Bacula catalog, which we keep in MySQL, as the partition it was on was starting to run out of space. (Seriously, 40 GB isn't enough?)
First thing I tried was running "optimize table" on the File table; that saved me 3 GB and took about 15 minutes. After that, I ran mysqldump and reloaded the db; that saved me another 300 MB and took closer to 30 minutes. Lesson: "optimize table" does just fine.
I've got a tape library at work with two tape drives. Today, one of the drives was doing (full) backups and the second was free for a restore job. However, when that restore job ran, I got this error:
JobId 62397: Forward spacing Volume "000039" to file:block 7:0.
JobId 62397: Error: block.c:1016 Read error on fd=7 at file:blk 3:0 on device "Drive-0" (/dev/nst1). ERR=Input/output error.
JobId 62397: End of Volume at file 3 on device "Drive-0" (/dev/nst1), Volume "000039"
JobId 62397: Fatal error: acquire.c:72 Acquire read: num_writers=1 not zero. Job 62397 canceled.
JobId 62397: Fatal error: mount.c:844 Cannot open Dev="Drive-0" (/dev/nst1), Vol=000039
JobId 62397: End of all volumes.
JobId 62397: Error: Bacula cbs-01-dir 5.0.2 (28Apr10): 03-May-2011 12:09:20
The problem wasn't that it encountered the end of the volume -- the job spanned a number of volumes, so that was okay.
No, the problem was that after the restore job had run, a number of other regular backups had started. These were incrementals, and thus were unable to use the first drive. When the restore job ran into the EOM on the first volume, it appears to have released the drive -- at which point the incrementals started up and denied the use of the second drive to the restore job. The restore job promptly gave up and called it an error.
As I was in a hurry, I tried killing off the incrementals and re-running the restore job. This worked just fine. Arguably it's a bug, but I suspect I just need to tweak the priority for restore jobs instead.
(Two entries in one day...woot!)
I came across this tip on an old posting to the Bacula mailing list. To determine if exclusions in a fileset are working, run these commands in bconsole:
@output some-file
estimate job=<job-name> listing level=Full
@output
The file will contain a list of files Bacula will include in the backup.
(Incidentally, I came across this while trying to figure out why my
excludions weren't working; turned out I needed to remove the trailing
slash in my directory names in the Exclude section.
I've been off on vacation for a week while my parents have been visiting. It's been fun, relaxing, and generally a good time. But today it was back to work.
I found out that backups have not been running since the day after I went on vacation. There were something like 650 jobs in Bacula stacked up, waiting to run; the storage daemon was stuck trying to do something, and nothing was getting done.
Near as I can tell, the storage daemon was stuck in a deadlock. I've got some backtraces, I've posted to the devel mailing list, and it looks there's a bug, recently fixed, that addresses the problem. Inna meantime I've put in Nagios monitoring for the size of the run queue, and I'm figuring out how to watch for the extra bacula-sd process that showed up.
Tonight, at least, the same problem happened again. This is good, because now I have a chance to repeat it. Got another backtrace, killed the job, and things are moving on. Once things are finished here, I think I'm going to try running that job again and seeing if I can trigger the problem.
Sigh. This was going to be a good day and a good post. But it's late and I've got a lot to do.