11 Apr 2012
This is an attempt to lay out my problems with Bacula, and to be
explicit about what I hope to achieve by replacing it (if, in fact, I
do go ahead with that). If I'm wrong, correct me.
Too many long jobs monopolize spool space, storage job slots, and
generally hold up production.
My largest jobs right now are around 1-2 TB -- and in order to
accomplish that, I need to manually split up filesystems using a messy
syntax. A job running that long will cycle through
- spool for hours
- despool for hours
many, many times. During spooling, a slot of storage space jobs is
used. During despooling, no other job can despool to that tape
drive. Often, this ends up holding up a lot of other jobs. If
there's a problem, I'm faced with a choice between killing a job
that's been running for days, or letting lots of other stuff go
swithout backups until/unless it finishes.
More generally, I'm faced with a choice between letting everything run
forever at the beginning of the month (because it's simplest to
schedule fulls for the first Saturday or some such), or juggling
schedules manually to stagger things (which I'm doing now, and leads
to schedules like FullBackupSecondSundayAfterLent).
Possible fixes:
- Disk -> Disk -> Tape
- More breaking up of jobs
- More/faster spool space
- Tweaking max jobs
- Smart ZFS snapshot scripts (so I can restart a job by pointing it at the snapshot; may be yet more baroqueness)
Bacula seems to get confused easily about what tapes are available
for use.
Bacula's storage daemon seems to often hold on to outdated info about
what tapes are in what state.
Example: the daily pool is full, so jobs are halted. Status storage
shows it's waiting for a drive to be created for the daily pool. I
move a volume from another pool, then have to attempt to mount it
manually in the appropriate drive -- the storage daemon doesn't pick
up on this change automatically.
Sometimes this works, and sometimes it doesn't. Sometimes both are
waiting for a tape from the same pool; creating one doesn't let the
jobs queued up on the other drive run on that new tape, but rather you
need to create a second new tape and mount it. On top of that,
sometimes the jobs hang around on the storage daemon still waiting
for a new tape -- or something...because they don't get out of the
way, and let other jobs run in their place, unless they're cancelled
(and sometimes only when bacula-sd is restarted).
This may be fixed with the upgrade to 5.2.6. However....
The new version of Bacula crashes when I run too many jobs at once.
That's 5.2.6, upgraded to from 5.0.2 (time got away on me, yes). And
by too many I mean, like, 50. That's not too many! I'm not sure what
the hell's going on, though at least now I have a backtrace. I'm
seriously pissed off about this point. Yes, I'll file a bug, but this
is annoying.
All in all, I spend far too much time babysitting Bacula.
It's extremely high maintenance, and that's pissing me off.
Understand, this is coming after a long weekend spent babysitting it,
trying to make sure some jobs got written. There are other problems
at work, yes, but this is not meant to be so hard.
Tags:
backups
bacula
06 Apr 2012
Full moon, impending cold, lack of sleep -- don't care. It's a clear
night, it's a long weekend, and it's time to get out.
Mars was wonderful. Saw detail easily -- much more easily than I
did last time. Near as I can tell, I saw Mare Acidalium in the
north, and Sinus Meridiani/Mare Arythraeum (which, hey, is reasonably
close to where Opportunity is in the south. Oh, and the polar
ice caps. Got even clearer as the night wore on; not sure if that was
the seeing or the scope cooling off.
M36/M37/M38 were pretty easy to find, which is nice -- these gave me
a lot of trouble previously. Saw M38's Chi shape.
For shits and giggles decided to try and find M81/M82 to see how a
full moon in a suburban location compares with a semi-dark site on a
nearly new moon. Well, damn me if I didn't find them -- with
binoculars for M82, and both of 'em in the scope. No, the view wasn't
spectacular, but it was incredible to me that I could see them.
The Moon was lovely, even full; I could stare at it for hours.
Aristarchus was pretty, as was Copernicus and Tycho. Tried
sunglasses and that worked well for the 40mm view -- which is almost
my favourite -- but not so much for higher magnifications.
Back in at 11.25. It's really nice observing so close to home.
Tags:
astronomy
27 Mar 2012
Periodically I remove tapes at $WORK from our tape library to keep
them somewhere else. getmonthlytapes is a Perl script that
helps me do just that. Released under the GPL; share and enjoy!
Tags:
bacula
backups
24 Mar 2012
Last night was the first clear night in a ridiculous amount of time.
It was also a nearly new moon. On top of all that, the kids were over
at their grandparents for the night, and I could stay up as late as I
wanted. IOW, the astronomy gods were saying "GO."
I decided that I'd go to Mount Seymour. There's a parking lot about
halfway up that folks congregate at; it's darker than the local park I
usually walk to. I'd only been to it once before; I don't like going
regularly because it's a long drive and gas, she is expensive. But
this seemed like a special occasion.
I packed up tea and a sandwich, picked out some targets for the night,
and started off. I got there at 8.15pm, distracted only slightly by
Jupiter and Venus along the way. There were a half dozen people there
already, including the woman I met last June ("Have I met you before?
Yeah, and I probably asked you then if I'd met you before...") I set
up and waited for the scope to cool down.
Well, no, I didn't wait. Who could wait? Venus was a lovely half
circle, even if it was shimmery and wobbly. Jupiter was just heading
below the trees and I got a very brief look; SEB, I think, present,
and NEB not very noticeable.
Then Mars -- the first time I'd looked at it through the Dob. Very
nice. I'd been preparing myself to see no features at all, but slowly
some came out; it held my attention much longer than I anticipated. I
think I saw hints of the north polar ice caps, and a band just south
of that surrounding the pole. I came back to it a few times over the
night, and magnification held up to 320x (Barlow'd 7.5mm Plossl;
nothing special, but works for me). (I'd just traded my 4.3mm
eyepiece in earlier in the day, and I was a little bit sad I didn't
have a chance to try that...)
I got a chance to check out a number of things through a 6"
AstroPhysics refractor, with a 100-degree TeleVue, that another person
had brought. The Double Cluster and M42 were simply stunning, and I
know now just how wonderful those supernormowide eyepieces are. I was
surprised, though, how much colour fringing showed up on Mars. I envy
him the deep sky stuff, but I'm happier w/my view of Mars.
M35, M36, M37 and M38 were all perfectly visible through binoculars --
nice! Took a quick look at M35 through the scope; pretty as always.
I took my usual run at M81 and M82. I've been searching for these for
a year or more, and it's always been a fruitless search. After 30
minutes I gave up in frustration. I got talking to the woman I'd met
earlier, and she showed me them through her scope (10" Dob). She also
pointed out where they were in the sky w/her laser pointer. Inspired,
I tried again, found out where I should be looking, and sweet Cthulhu
on a pogo stick FOUND THEM! It was immensely satisfying, and very
pretty -- faint, but pretty. Amazing to think I can hold two galaxies
in one FOV.
I had to give up at 11.15pm; my feet were cold as hell. I hadn't got
to half of my list (Comet Garradd, M46, M47, M67, NGC 2261, NGC 2264,
X Cancri (Carbon star) and the Leo triplet), but what the heck. I
packed up, came home, drank some therapeutic wine and got to bed at
1am. And then SLEPT IN 'til 8am. Wonder of wonders. What a night.
Tags:
astronomy
21 Mar 2012
If you don't follow his blog, it's worth it. Two snippets from
his post on the 2.6.32 kernel. First:
We all drifted back to our companies, and planted the seeds that maybe
something like the 2.6.32 kernel would be a nice one to do our product
on. This planting worked so well, I had to refrain from fits of
laughter in one meeting where a project manager got up and said, "We
decided that the 2.6.32 kernel would be the best for our product, what
does engineering think about this?"
This successfully cumulated in the release of SLE11 SP1, Debian
"Squeeze", RHEL 6, Oracle Linux 6, and Ubuntu 10.4 LTS, all based on
the 2.6.32 kernel.
Hacking the business models of these different and competing groups,
to coordinate on this specific kernel, was one of the (previously)
unsung successes of how the community really can achieve remarkable
things if they decide to do it.
Two:
I would personally like to thank the Debian kernel developers,
specifically Ben Hutchings, Maximilian Attems, Dann Frazier, Bastian
Blank, and Moritz Muehlenhoff. They went above and beyond what any
"normal" developer would have done, ferreting patches out of the
kernel.org releases and the different vendor kernels and bug tracking
systems, backporting them to the 2.6.32 kernel, testing, and then
forwarding them on to me. Their dedication to their user community is
amazing for such a "volunteer" group of developers.
I firmly believe that without their help, the 2.6.32 kernel would not
have been the success that it was. The users of Red Hat and SuSE
products owe them a great debt.
Buy them a beer the next time you see them, they more than deserve it.
Tags:
linux
16 Mar 2012
Here's my attempt at getting a working Foswiki/TWiki mode for Emacs
based on Org-mode. I've started taking this good work and
mangling it; my excuse is that I'm only just starting to program
Emacs.
What works so far: cycling visibility with tabs and detecting
levels. Hey, it's a start. :-) Next up is getting Meta-left/-right
to work to promote/demote headings; I'm not sure if I should be doing
this with Emacs advice, or if there's some other way to locally
override a function's definition. Answers on a postcard to aardvark
[ta] saintaardvarkthecarpeted [tod] com 'til I get comments sorted out.
Tags:
emacs
foswiki
15 Mar 2012
When using magit, I came across errors that looked much like
these:
Unpulled commits:
fatal: ambiguous argument 'HEAD..exoplanet/compute_server': unknown revision
or path not in the working tree.
Use '--' to separate paths from revisions
(Here, "exoplanet" is the name of a machine, and "compute_server" the
name of a branch.)
The problem turned out to be that .git/config looked like this:
fetch = +refs/heads/*:refs/remotes/origin/*
url = exoplanet:~/dotfiles-git-master
[branch "compute_server"]
Note how the branch lists the remote as "exoplanet", but there's no
"exoplanet" remote config stanza -- only one called "origin".
Changing the remote listed under the branch to be "origin" worked.
Here's the working config:
fetch = +refs/heads/*:refs/remotes/origin/*
url = exoplanet:~/dotfiles-git-master
[branch "compute_server"]
Tags:
git
emacs
14 Mar 2012
Part of how I'm beginning to work with Org and RT:
(defun x-hugh-boxquote-yank-and-indent ()
"My attempt to combine boxquote-yank and indent.
The car/cdr bits are from the docstring for boxquote-points. It's a bit silly to run it twice, but it was simple."
(interactive)
(save-excursion
(boxquote-yank)
(next-line)
(indent-region (car (boxquote-points)) (cdr (boxquote-points)))))
(global-set-key (kbd "<f9> y") 'x-hugh-boxquote-yank-and-indent)
Tags:
emacs
rt
14 Mar 2012
Hacking Is Important:
The end result of successful hacking is product, and that product
needs to grow by building more things. The more you grow, the more
things you have, and the more you need people whose job is simply to
coordinate the increasingly interdependent building activities. These
people, called managers, don't create product, they create process.
Hackers are allergic to process not because they don't understand the
value; they're allergic to it because it violates their core
values. These values are well documented in Zuckerberg's letter: "Done
is better than perfect", "Code wins arguments", and that "Hacker
culture is extremely open and meritocratic". The folks who create
process care about control, and they use politics to shape that
control and to influence communications, and if there is ever a
sentence that would cause a hacker to stand up and throw his or her
keyboard at the screen, it's the first half of this one.
Tags:
13 Mar 2012
I need to add a post-commit hook to my RT git trees to email the day's
logs to RT; send 'em as a comment so they show up automagically in the
ticket. Looks like the default should do nicely.
Tags:
git
rt
13 Mar 2012
At $WORK I'm trying to install tmap on 64-bit CentOS 5. Here's
how it goes:
Built RPM for tmap, and it works -- but not using tcmalloc.
What's tcmalloc? Part of Google-perftools; a faster malloc.
We really want tcmalloc.
Found RPM for google-perftools installed, but includes only the
32-bit version of tcmalloc due to dependence of other parts of
perftools on libunwind.
installing libunwind on 64-bit CentOS 5 a big PITA and I decide to
try working around it.
Conveniently, tcmalloc can be compiled on 64-bit platform;
produces libtcmalloc_minimal, which documentation says is perfectly
valid malloc.
tmap does not come, out of the box, configured to look for (in
configure script) tcmallocminimal, but there is an commented-out
option to do so. You can remove the comment and run autogen.sh,
and then configure will look for libtcmallocminimal.
...but this fails because the way I compiled/built rpm for
tcmalloc does not include libtcmallocminimal.so; includes
libtcmallocminimal.so.4.
and so my half-assed RPM/devel skillz come back to bite me in the
ass again.
Tags:
sysadmin
centos
13 Mar 2012
Bill C-11 is in committee right now. I've sent (paper) letters
to the committee members and emailed them. I'm not holding out a lot
of hope, despite my dream (true story) a few weeks ago that I managed
to convince Stephen Harper, while he was visiting my house, that as a
parent I should be allowed to back up DVDs. When Harper kept
James Moore in place after the lection, he kept somone who (whom? damn
English) I've come to accord a grudging respect to as a pit bull of the
first order.
Ugh. There's always the next election.
Update: Not four hours later, it's back to Parliament for 3rd
reading and Senate approval. I'd better start backing up those DVDs now.
Tags:
copyright
09 Mar 2012
Today I gave some impromptu training at $WORK; the approximate topic
was "Saving State in Linux". I've been meaning to do something like
this for a while, but it was prompted by a conversation yesterday with
one of the researchers who kept losing work state when shit happened
-- Emacs window arrangements, SSH sessions to other machines, and so
on. I found myself mentioning things like tmux, workgroups,
and Emacs daemon mode...and after a while, I said "Let me talk to
you about this tomorrow."
So today I found half an hour, decided to mention this to everyone in
the lab, crowded into a meeting room, set up my laptop and the
projector, and away I went. For a fly-by-the-seat-of-my-pants first
attempt, I think it went relatively well. Best idea: asking people
for questions. It hadn't occurred to me that people would want to know
more basic stuff like "How do I split windows in Emacs?". I'm never
sure what people already know, so I don't want to bore them...
Next time:
- Be more organized
- More notice for people so they can prepare questions
- Let the vi people know if it's going to be Emacs-heavy
In other news: finally converted my SVN repos to Git yesterday in a
fit of pique. The big three -- my org-mode stuff, and the two
Cfengine repos (Cf2 and -3) -- are already in use, as in that's where
I'm checking stuff into. The rest (Nagios configs, for example) are
being done as I get to them. It's really, really wonderful.
Family: holy house o' plague, Batman!
- Last Friday my older son pukes, stays home from school
- Last weekend my younger son comes down with bronchitis, demands
compensation for being forced to take antibiotics
- My wife comes down with some kinda virus that makes her nauseous for
a week
- I'm still fighting off a sinus infection
- Wednesday, my older son pukes again apropos of what we don't know
Gah. We're getting the house boiled next week. (Update, March 13:
too late; I puked on Friday night and spent Saturday moaning in bed;
my wife did the same thing Saturday night/Sunday. FUCK.)
Also? There's a Planet Lisp. Who knew?
Tags:
linux
emacs
sysadmin
geekdad
07 Mar 2012
Random rpmbuild stuff:
Here's how to create symlinks when building an RPM
If you're packaging a statically-built binary (I know, I know) and
it suddenly craps out with "unexpected reloc type in static binary",
put this in your spec file:
%define __os_install_post %{nil}
(Found here)
- To see what macros are defined for your RPMs, run:
- The best way I found to make a symlink in an RPM is like so:
pushd $RPM_BUILD_ROOT/opt/bin
ln -sf ../package/bin/foo .
- If an rpmbuild invocation fails, don't forget to remove
$RPMBUILDROOT. OTOH, that might be my fault for not including
that as part of the setup...not sure.
Tags:
rpm
centos
toptips
07 Mar 2012
I received this bit of spam a few days ago:
From: AUW-RSVP <melud.halasa@example.com>
To: undisclosed-recipients: ;
Subject: hi
Organization: Gen. Melud
My name is Gen. Melud Massoud Halasa,I was a Libyan army General in the
military force of Gardaffi in Libya,i have $23 Million Dollars hidden in
Libya,i need your assistance to move this money out of Libya to your
country,i have resigned from the army and i want to go into business in your
country as your partner.If interested REPLY ONLY VIA MY PERSONAL EMAIL
melud.halasa@example.org for more details.
I forwarded it off to a friend of mine and asked him if he knew
anything about it. His reply:
Well, Melud's been trying to move that cash for about six months now but no
one will help him. The thing is, it's actually a physical pallet of $100
bills that we had used to buy his loyalty back during the uprising. That's
why he's starting to reach out farther and farther afield, trying to find
someone who will come to Libya and help him carry the danged pallet to the
local Western Union branch. I'd say it's not worth it ... that Western Union
is at least a mile from where he has the pallet stashed in his mom's house.
I value my lower back more than that.
And, of course, by sending this email back and forth we've guaranteed that
it's being read by some automated spy system that is trying to determine if
this counts as terrorist "chatter." In the interests of being friendly and
polite, I'd like to say hello to that automated system, and perhaps to any
actual human analyst who stumbles across it.
I'd just like to say: that is the funniest thing I hope to read all
week. Maybe all month. And definitely on my top ten for the year.
Tags:
spam
06 Mar 2012
Saw this guy at work today, hammering away on a street light:
It's a Northern Flicker, and it's a type of woodpecker. The male's
the one with the red moustache on the right.
Apparently they do this to scare away other males and attract mates;
in the wild, they use hollow logs. From Wikipedia:
This bird's call is a sustained laugh, ki ki ki ki ..., more congenial
than that of the Pileated Woodpecker. One may also hear a constant
knocking as they often drum on trees or even metal objects to declare
territory. Like most woodpeckers, Northern Flickers drum on objects as
a form of communication and territory defense. In such cases, the
object is to make as loud a noise as possible, and s why woodpeckers
sometimes drum on metal objects. One Northern Flicker in Wyoming could
be heard drumming on an abandoned tractor from a half-mile away.
Oh, and also saw the standard pair of bald eagles circling overhead.
Tags:
06 Mar 2012
From the Emacs manual:
35.6 Mail Amusements
M-x spook adds a line of randomly chosen keywords to an outgoing mail
message. The keywords are chosen from a list of words that suggest you
are discussing something subversive.
The idea behind this feature is the suspicion that the NSA1 and other
intelligence agencies snoop on all electronic mail messages that
contain keywords suggesting they might find them interesting. (The
agencies say that they don't, but that's what they would say.) The
idea is that if lots of people add suspicious words to their messages,
the agencies will get so busy with spurious input that they will have
to give up reading it all. Whether or not this is true, it at least
amuses some people.
Hee hee. And then there's the Jargon file:
NSA line eater: n.
The National Security Agency trawling program sometimes assumed to
be reading the net for the U.S. Government's spooks. Most hackers
used to think it was mythical but believed in acting as though
existed just in case. Since the mid-1990s it has gradually become
known that the NSA actually does this, quite illegally, through
its Echelon program.
And now this:
The Department of Homeland Security monitors your updates on social
networks, including Facebook and Twitter, to uncover "Items Of
Interest" (IOI), according to an internal DHS document released by
the EPIC. That document happens to include a list of the baseline terms
for which the DHS -- or more specifically, a DHS subcontractor hired to
monitor social networks -- use to generate real-time IOI reports. (Although
the released PDF is generally all reader-selectable text, the list of
names was curiously embedded as an image of text, preventing simple
indexing. We've fixed that below.)
Epic.org info here. Some random keywords, thanks to Animal New
York:
- United Nations
- Disaster management
- Security
- Industrial spill
- Public Health
- Telecommunications
- Yuma
- Target
- Earthquake
- China
- Social Media
Tags:
politics
emacs
06 Mar 2012
...now with extra synergy!
No. My idea is that if you cd into a particular directory, you'd
automagically start a new history file w/in that directory. This
would let you keep history when working on a particular task (which in
turn would assume you have a different directory for each task). This
is sort of related to how I'm beginning to organize work on tickets.
Here's a walkthrough:
- When I start working on a new ticket, I already run something like this:
```
function setup_rt {
TICKET=$@
cd ~
mkdir rt_$TICKET
cd rt_$TICKET
mkdir build test
touch notes.org
git init
git remote add remote remote:/opt/git/rt_${TICKET}.git
ssh remote -t "git init --bare /opt/git/rt_${TICKET}.git"
}
```
- And now the magic bit, where
cding to that directory automatically
saves history in ~/rt_$TICKET/.bash_history.
The magic part is, of course, tricky. I can think of a few ways of
accomplishing this:
```
function start_work_on_ticket () {
cd ~/rt_$TICKET
bash --histfile=~/rt_$TICKET/.bash_history
}
```
- Fiddling with history recording to include PWD each time, and then
periodically record by running
grep rt_$TICKET $HISTFILE >>
~/rt_$TICKET/.bash_history
And then of course you'd commit it to git (and birds would sing and
did I mention it's sunny outside? Because it is). Even better would
be a way for RT to automagically link to git-based repos with all this
stuff in them.
Of course, for some/most tickets this is overkill. But there are
times when I find myself two weeks into fixing something because it's
turned out to be surprisingly involved, and I have no good
recollection or record of what's been going on. And it's usually
right then that someone comes by and says "Oh hey, turns out we gotta
do that all over again because [blah]."
Tags:
29 Feb 2012
Because it's awesome, that's why.
http://realprogrammers.com/jump_start/mutt/
Tags:
mutt