The Life of a Sysadmin

I'm listening to CCC: Revolved right now. It's not bad; She Said Traffic and Close To No One are quite good. But you just know they're a lawsuit away from being yanked from the net. Why not just publish the source code for these mash-ups? Assume the person making it finishes mixing Iron Maiden's Run To The Hills with Modest Mouse's It's All Right On Ice, Alright. Instead of writing an Ogg, their editor publishes a Makefile: take the first eleven seconds of this song and loop it while playing the last eighty seconds of that song through an echo plugin, then play the bit from Dirty Harry where he says "well, do ya, punk?" You download the Makefile and compile your own Ogg using your own sound files. I'm sure the record companies would find a way to complain (ie, sue), but this would partly be the point: are you allowed to do things with your own media or not?

I want one of these.

I am fucking pissed off. Over the last few weeks, I've been noticing attempts to spam the wiki on my website. The spammers would create a new page similar to one already existing, and fill it full of links to Russian linkfarms (right term? who cares?). It was annoying, and I figured it would only get worse, but I didn't get too worried. I deleted the pages, blocked the IP address (it was all coming from one open proxy), and watched the changes page for further action. Last night I checked the changes page again. It was late (well, sort of; it had been a long day) and I was making one last check before going to bed. Just to make sure that everything was okay, you know? Every single fucking goddamned page had been vandalized. Every single page that I had put up had been replaced with spam, and there were a dozen new pages with even more spam. Over the course of maybe four hours, all my work had been removed. My only consolation is that Google had not visited the wiki since the changes had been made. There were maybe a hundred pages to revert. And PHPWiki, the software I was using, sucks ass through straws when it comes to reverting changes. Check this out, ladies and germs:

• There is no easy, documented way to revert to a specific revision of a page using the web interface. The version I was using (1.3.4) forces you to go edit an old version, then save that version. The new version I tried upgrading to (1.3.10) allegedly has "action=revert", but I was unable to get this to work: it appeared to do nothing different from "action=edit". To be fair, this may be because the spammer seemed to edit most pages multiple times, perhaps to get around action=revert. But why couldn't I find any documentation on this? All I could find was this page and the words "See action=revert".
• There is no easy way to revert to a specific revision of a page using the database directly. Check it out: The database appears to store metadata in a column dedicated to compressed, cached markup. That's right: instead of breaking out metadata like revision, author IP and so on into a separate table, it's stored in the middle of a big gzipped, serialized PHP object. This means I can't do something like "delete from version where versiondata like '%10.0.0.1%'"; going to the page I've done this on hits an assert in the code that appears to check that the revision listed in the cache column is available in the pagedata table. Whee! Let's get all our programming ideas from MS Office!

As a result, I'm pulling a backup of the database from Friday in order to get the old pages back. I'm going to dump the pages to HTML, figure out how to script whatever changes I want to make, then leave PHPWiki forever the fuck behind me. Shame, really, 'cos I do like the ease of use of Wikis. But I do not have time for this fucking nonsense. Shame on me for not remembering these words:

Someone challenged me, Well, how am I supposed to continue hosting these low-barrier discussions? I’m sorry, but I don’t know. To quote Bruce Schneier, “I feel rather like the physicist who just explained relativity to a group of would-be interstellar travelers, only to be asked, ‘How do you expect us to get to the stars, then?’ I’m sorry, but I don’t know that, either.”

Those of you looking for info on the NWR04B, please continue to leave comments on my blog. I'll get the documentation from the wiki back as soon as I can.

Reminder for myself. So you've got some backed-up MySQL table files (if that's the right term), rather than a proper dump. Untar them somewhere, and note the path to the data files -- say, /home/foo/mysql_recovery/data. Copy /etc/my.cnf to your home directory. Edit it and change the port to something different -- say, 3307. Run: /usr/local/mysql/bin/mysqld --defaults-file=/home/foo/my.cnf --datadir=/home/foo/mysql_recover/data Then run: mysqldump -P 3307 --opt -u foo -p database > recovery.sql Of course, all this could be prevented if you were running mysqldump nightly instead of just copying the data directories...

I'll be away for a week, so no updates on the router. I'll be turning off comments on the blog to keep asshole spammers at bay...sorry about that. For the record, here's where things stand: I've tried booting my own linux kernel but have failed; spammers got the wiki, which is where all the info was about serial ports and such; I've recovered the wiki markup from backups, and the next step is to make it HTML again.

...getting closer to getting my wiki pages back, through a nasty combination of awk, sed and perl. I'll put up the script once I'm happy with it, in case it's useful to anyone in the future. After that, the next thing is to try again with the kernel for the NWR04B; it's been sitting on the back burner for a while now. Though it's hard to tell if it's working or not...I might have to try to get a JTAG adapter working first. Another thing to do is upgrade my wife's computer, maybe. Right now she's running RH9, and of course there's no more RH releases. I could try moving to Fedora Core, but bleah...I never really learned to like RH anyway. I'm downloading Ubuntu right now: install and live ISOs of the Hoary preview, for Debian goodness with a Gnome desktop. We'll see how it goes.

I'm finally working again on the NWR04B. Right now my focus is trying to get a kernel booting, but I'll be satisfied with any kind of response from the damn thing. Right now, this is as far as I get: Verifying product code......PASS
Boot Product Code!!! And there it sits until I power cycle the thing. Crap. I've got a pretty steep learning curve here. First off, I haven't worked with the ARM architecture before. Second, I haven't ported Linux (or anything) to another architecture before. (I'm not really porting stuff here -- the hard work was already done by the HRI and Codeman. But the experience would definitely help.) Third, I know very little about assembly; I've got a copy of a really good ARM assembly guide, but I'm just not used to thinking at such a low level. Fourth, I still have not disassembled the bootloader that comes with the existing, vendor-supplied firmware, so I really don't know what state everything's in when the kernel comes up. Fifth, I don't have a JTAG adapter on this thing. As a result, things are going slowly. I started by assuming this sequence of events:

1. The bootloader sets up the serial port, and decompresses application.bin.
2. Application.bin is copied to RAM.
3. The memory map is flipped. (This is in the datasheet. Before, flash memory starts at 0x00000000 and RAM starts at 0x20000000; afterward, it's the other way around.)
4. The CPU jumps to 0x0, and execution continues from there; this is the Linux kernel initialization and decompression routine.

By disassembling the compressed Linux kernel, I can see that it should work -- ie, there's no need to (say) jump to some random address within the kernel to start working. (It's good to confirm these things...) But the lack of any response at boot time, even with verbose kernel debugging messages turned on, is disheartening. I had a look at the uClinux file arch/armnommu/boot/compressed/head.S, and realized that it might be missing some definitions for putc; this is architecture-dependent, and everything's wrapped in if 0. I tried putting in this:

#elif 1 /* my attempt at cx84200 serial debugging -- assuming that the address for mov is uart0*/
                .macro  loadsp, rb
                mov     \rb, #0x90000000
                .endm
                .macro  writeb, rb
                strb    \rb, [r3, #0]
                .endm

According to the datasheet, the byte at 0x90000000 is where "UARTDR, data read or written from the interface" goes. I'm assuming that means you put a byte there, then magic happens, then that byte is written to the UART. Still no response. I tried taking out the #if/#endif statements around debugging statements, to make it all as verbose as possible -- still nothing. However, with the judicious use of dd I've been able to cobble together a silly little "Hello, world!" in ARM assembly, and I'm able to get that to boot (well, print). This confirmed I had the basic sequence of events correct. What's more, I was able to insert this little bit into various places at the beginning of the kernel, and confirm how far along things were going. The answer is: not very. I've been following along in head.S, and I can see where the debugging information should be printed -- but it just doesn't. What's strange is that by mistake, I inserted helloworld at a non-four-byte boundary -- at byte 70, not 72 -- and then I got a response from a routine in head.S that prints out the first 256 bytes of the uncompressed kernel...and then nothing after that. So close!

I've started to post information on the Network Everywhere NWR04B recovered from the wiki here. Pretty rough at the moment, but I'm working on it. If you've contributed something to the wiki and would like your name in credits in the revived pages, please let me know.

Here as a reminder to myself, and in case it'll save anyone else some time. Ugly, but it did work:

find /cdrom/ -name *.mp3 | \
        sed -e'h;  # Copy file name to holding space
               s/ /\\ /g;  # Escape the spaces
               x;  # Exchange the holding space with the working space
               y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/;  # stupid sed lower-case trick
               s/ /_/g; # space to underscore
               s/_\([-\.]\)_/\1/; # change _-_ or _._ to - or .
               s/_\./\./; # almost the same
               s#^/cdrom/##;
               s/\(.*\)-\(.*\)\.mp3/\2-\1.mp3/; # song-artist.mp3 -> artist-song.mp3, the way God intended
               H; # append to holding space...
               g;  # then grab the holding space back into the working space
               s/\n/ /; # H appends with a newline, so take it out
               s/^/cp /;' > /tmp/foo ; sh /tmp/foo  # Sheer ugliness

This copied a bunch of files on a CDROM named things like A BIG LONG FILE NAME - BAND NAME .mp3`` to ``band_name-a_big_long_file_name.mp3. Ordinarily I'd do this sort of thing in perl, but I got started in sed and was too obstinate to stop. In other news, I've got some weekend work coming up. There's a bunch of upgrades I haven't had a chance to get to lately -- going away for a week doesn't help, admittedly -- and it's time to get cracking. This Saturday it'll be replacing three dumb, crash-prone 24-port Linksys switches with three 50-port (48 10/100, 2 10/100/1000) Procurve switches. This'll give us VLANs, more MRTG graphs, and 50-odd badly-needed ports...not that we have the drops for it. Next up is some rack rationalization. I've got three racks, two of which belong to us and one of which is good (it's from APC, who have rocked so far). Unfortunately, the good one is going to have to be 'way far away from the Procurves...simply don't have the room or the electrical 'nads to hook it up. (In case it's not clear, I'm dealing with some serious growing pains with our network.) Some servers'll be moved around, and I rather suspect that'll be next Saturday -- the Saturday after MS releases their next security update. Dodged a bullet in March, but what are the odds it'll happen 2x in a row? That's what I thought. Hopefully sooner than that is some work on the Promise VTrak 15100; the order for the SCSI cards got dropped on the floor, so they're not coming in 'til tomorrow or Friday. I'll be able to get a better idea of how much I like it at that point, but I'm not impressed with Promise The Company(tm) so far. Here's why:

1. The web page for the 15100 originally said it was manageable by SSH. BZZT -- telnet and HTTPS only. They've corrected it since I brought it to their attention.
2. Still waiting to get the SNMP OIDs. It's no good running snmpwalk if you don't know what you're looking at.
3. They run Linux on the thing and use the busybox utility, but they're not distributing source. I've talked to a manager and need to call him back, so this is very prelinary, but: so far I've been told that the changes they've made to the kernel (they keep ignoring busybox) are proprietary and so I can't have the source. To be fair, the mgr. I've been talking to has promised to follow up on this, so we'll see how it goes.

None of this leaves me with a good feeling so far.