So a while back I wrote about how it feels, sometimes, to have so much on the go that there's no possible way to do it all. One of the things I mentioned was an idea for a program to mirror your Github repos: all the things you've ever starred, or forked, or watched, or whatever. That was a serious project (though not necessarily a serious goal), and in the grand tradition of missing the point entirely I started writing it. It's now good enough for me to use from cron (though it's not without its bugs and missing features), so here it is: landle, a Small but Useful(tm) utility to mirror your Github repos. You can grab it from Github (natch) or my own repo (double-natch).
I imagine two use cases. First, someone wants an up-to-date copy of their Github stuff on their laptop, available for hacking when offline. Second, maintaining an up-to-date copy of their stuff in case of takeover by Oracle (the Githubpocalypse).
To be clear: this is a straight-up ripoff/rewrite of ghsync (right down to the directory layout). ghsync is elegant and wonderful, but I was unable to get its dependencies to work for me. I had the choice of persevering and learning Python better, or rewriting it in Perl and learning development a bit better. I chose the latter. (Side note: not an easy choice! I hate leaving problems unsolved, for one, and I really do want to do more with Python. But this was also a good chance to start a real, though bite-sized, project, and to try hard to do the Right Thing.)
In the grand tradition of all ambitious rewrites, I would not have called the current release "1.3", but instead "0.99pluralZalpha-rc1" or some such. But modulo the TODO list, it's pretty good as it is...a good, solid beta.
So at work there's this program called CHARMM that's my responsibility to build. It's not the worst thing I've ever had to compile, but it's close:
the build script is an interactive tcsh script; you have to tell it where to find libraries, and can't pass these on as arguments;
the whole thing is insanely sensitive to what compiler and library you use, but often that'll only turn up when you actually go to use the damn thing;
CHARMM will often just crash if it it doesn't like you or your input, and the reaction on the forums is often, "Oh yeah, you should just patch that";
there are about 40 different build options, of which an amusing fraction are documented (and an even more amusing fraction documented correctly), and of course the combinations just explode; the wrong combinations will make CHARMM crash, and the reaction on the forums is often, "Oh yeah, they've been meaning to fix the documentation on that."
To get around this I built a script that collects arguments, applies a patch, unpacks the tarball, echoes the right args to the build script (which means, of course, that I am damned for all eternity) and then builds the damned thing. Hurrah! Everything's fine!
Except now I have to rebuild. And invoking the script is becoming complicated:
I'm using modules to use different compilers;
I want to name the destination directory after some combination of compiler, options, and patches;
I'm writing qsub scripts that set all this up and then invoke my script, which invokes the build script;
And there are, at last count, two versions of CHARMM, two patches that are required for one of those versions, and two sets of options. That's 8 scripts.
So I'm looking at all this and thinking, "Self, you're telling me I need a templating system for my templating system. Have I got that right?" And I think, "Self, you've got it." So I punch myself in the face. And that's why it hurts to build CHARMM.
At $WORK I'm moving a web server to a new machine. Natch, I'm getting the new one ready, testing as I go, and when things are good I'll change DNS to point to the new machine. I found myself testing a lot of Apache Alias directives -- we've accumulated rather a lot -- my usual routine:
...was getting damned tiresome. Perl to the rescue! WWW::Mechanize, Test::More and Apache::Admin::Config are damned useful, and when the authors weren't looking I bent them to my will.
So: thornhill, a Small but Useful(tm) script to check URLs mentioned in Apache config files. Here's what it does:
We'll call this release 0.1 ("works for me!"). You can download it here; be sure to note the bugs and assumptions.
Share and enjoy!
Update: Problems have arisen.
A while back I was looking for a script that would email warnings if a user was over their disk quota. Surprisingly, I couldn't find one, so I wrote one.
Here's quota_check.pl, for what it's worth: A Small but Useful(tm) utility to check quotas and send emails periodically. Depends on Perl's Quota module. Meant to be called from cron like so:
for i in /filesystem/* ; do [ -d $i ] && quota_check.pl -u $(basename $i) -f $(dirname $i) ; done
It will check whether the user is at or above the warning level (default: 80%) their quota for inodes or blocks. If so, it will see if a warning has been sent recently (default: 7 days) by looking for the file ".quotawarningsent" in their home directory. This file is maintained by the script, and hold the time (in seconds since the epoch) the last warning was sent. If it's time to send another warning, or if one was never sent in the first place, it'll send it and update this file.
Released under the GPLv3. Share and enjoy!
I came across a problem compiling GotoBLAS2 at work today. It went well on a practice cluster, but on the new one I got this error:
gcc -c -O2 -Wall -m64 -DF_INTERFACE_G77 -fPIC -DSMP_SERVER -DMAX_CPU_NUMBER=24 -DASMNAME=strmm_ounncopy -DASMFNAME=strmm_ounncopy_ -DNAME=strmm_ounncopy_ -DCNAME=strmm_ounncopy -DCHAR_NAME=\"strmm_ounncopy_\o
../kernel/x86_64/gemm_ncopy_4.S: Assembler messages:
../kernel/x86_64/gemm_ncopy_4.S:192: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:193: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:194: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:195: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:197: Error: undefined symbol `WPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:345: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:346: Error: undefined symbol `RPREFETCHSIZE' in operation
../kernel/x86_64/gemm_ncopy_4.S:348: Error: undefined symbol `WPREFETCHSIZE' in operation
The solution was simple:
gmake clean
gmake TARGET=NEHALEM
The problem appears to be that newer CPUs (Intel X5650 in my case) are
not detected properly by the CPU ID routine in GotoBlas2. You can
verify this by checking the contents of config.h in the top-level
directory. Without TARGET=NEHALEM, I saw this line:
#define INTEL_UNKNOWN
But with TARGET=NEHALEM, this becomes:
#define NEHALEM
The problem with gemm_ncopy_4.S arises because it defines
RPRETCHSIZE and WPREFETCHSIZE using #ifdef statements depending
on CPU type. There is an entry for #ifdef GENERIC, but that was not
set for me in config.h.
In addition, if you type "gmake TARGET=NEHALEM" without "gmake clean" first, you get a little further before you run into a similar error:
usr/bin/ld: ../libgoto2_nehalemp-r1.13.a(ssymv_U.o): relocation R_X86_64_32S against `PREFETCHSIZE' can not be used when making a shared object; recompile with -fPIC
../libgoto2_nehalemp-r1.13.a(ssymv_U.o): could not read symbols: Bad value
If I was a better person, I'd have a look at how the sizes are defined
and figure out what the right value is for newer CPUs, then modify
cpuid.c (which I presume is what's being used to generate
config.h, or at least this part of it. Maybe another day...
Here's how to piss me off:
Refuse to update your software so that it uses a Makefile. Yes, I know you're not only 'way smarter than I am but you've got a source tree going back to 1977. I don't care; editing six different shell scripts is not the way to do things.
Sprinkle those six scripts with assumptions about which software is present and where the source code is being compiled. Document most of them.
Run tests but carefully delete all the results; don't include an option to save them. That way I have to edit your scripts to figure out what the hell went wrong.
Assume the presence of csh for everything, rather than POSIX-standard sh.
Put configurable options inside shell scripts, rather than in a configuration file or allowing them to be set by arguments to those scripts.
Include directions like "Perhaps change foo, bar and baz", without explaining why or what they're set to. When tests later fail because you didn't properly set foo, bar and baz, don't explain where these are set or how they affect the tests.
Set a hard-coded location for temporary output. Die silently when those locations aren't present, rather than explaining why or offering to create them or using /tmp. Refuse to overwrite already-present files, but don't explain this anywhere; instead, say that they might be useful next time.
Have important variables, like the hard-coded location for temporary output, set in two or more different places. Suggest editing some of them.
Have test failure indicated by "!!FAILED", ensuring a moment's confusion about whether that means "FAILED!", "NOT FAILED!" or "NOT NOT FAILED".
_Update, April 20th 2010: