The Life of a Sysadmin

• (This is OpenVPN version 1.6, btw.)
• Bridging can be done pretty easily. Use the device tap0 directive in your config file, then set these sysctl variables:
  net.link.ether.bridge_cfg: dc2,tap0 -> dc2,tap0 net.link.ether.bridge: 0 -> 1 This assumes, of course, that dc2 is the inside interface you want bridged over the VPN.
• If OpenVPN exits for some reason, you need to turn bridging off by setting the net.link.ether.bridge sysctl to 0 before starting OpenVPN again. Near as I can figure, this is because the new tap0 interface is different from the old one, even though it has the same name, because it was created by a different PID.
• No problems getting OpenVPN to run under djb's daemontools so far, as long as I toggle bridging as part of the startup script.
• On one machine, the inside interface being bridged was sis0, an onboard ethernet interface. This led to worlds of hurt: if OpenVPN went down then started again, everything would look good but the bridge wouldn't work -- I could no longer ping that interface from the other side of the bridge. Furthermore, trying to ping from that machine to the other side of the bridge got me this error: ping: sendto: no buffer space available Googling for this turned up many, many references to this problem. It seems to be a long-standing bug in FreeBSD, present to this day (!) and with no fix (!!), that may or may not have something to do with particular ethernet drivers. I thought I would have to install Linux and rebuild it -- this was on Friday afternoon, and the VPN is meant to be set up this Monday -- but I took a chance and bought some different ethernet cards (Linksys LNE100TX-CA ver 5.1, which in FreeBSD show up as dc(4) devices; thank you, London Drugs) and so far I haven't seen the problem again.
• The machine with the problematic sis0 interface is a 2GHz P4; the other side is a 3GHz P4. Both have 512MB of RAM. If I restart OpenVPN on the slower machine, pings take about 30-40 seconds to start working again; if I restart the faster side, it's almost instantaneous. (Both sides know the address of the other side, and both are set to re-initiate the connection if it goes down.)
• For testing, the setup is something like this:
  Client A -- Slow machine -- (OpenVPN here) -- Fast machine -- Client B Copying a big-ass file from Client B to Fast Machine takes 1 minute; if Client A copies the same file to itself, it takes about 1 minute 20 seconds; if Slow Machine copies the same file to itself, it takes 1 minute 40 seconds; and finally, if Client B copies that same file from Slow Machine to itself, it takes 1 minute 10 seconds. (Copying is done by scp in all cases; Client A and Slow Machine are connected by a crossover cable, as is Slow Machine and Fast Machine; Fast Machine and Client B are connected through by a switch or two.)

A friend of mine sent me this link, and Jesus Christ it looks like me. Family Guy on tonight. Woohoo!

Welp, the wireless link got set up yesterday. We've now got two Tsunami 100s (unfortunate name, that) installed and pointing at each other from our office windows. Rough guess is that it's something like 150 metres between the two units. OpenVPN worked right away, which makes me happy. Ping times were good between the two units -- 4 or 5 ms. I tested it by copying the same big-ass file from one end to the other that, over a crossover cable, took less than 2 minutes to copy -- and was shocked to find the ETA listed as 16 hours and climbing. What the...but then I visited the other side and found that the ethernet cable had been connected to the 10Mb/s jack, not the 100Mb/s jack. This made things better, but still crappy: 30 or 40 minutes was the ETA. Near as I can figure, we're getting about 1.5Mb/s on this thing, which isn't much different from ADSL. The units are half-duplex, and I suppose that could be affecting things somehow (besides the obvious, I mean), but that still seems terribly low to me. I spent most of yesterday getting other things in the new digs working, so I didn't have a chance to call the company who set it up; that'll be today's job. Another gotcha from the move: turns out that our voicemail system (Norstar something or other) actually requires a phone to be physically present in order to send a call to voicemail after n rings. That was a bit of a surprise. We're currently scrambling to get some extra phones to leave hooked up. (Don't even ask why voicemail or their direct lines doesn't work at the new office.)

It was a duplexing problem, though not like I thought. The radio link in a Tsunami is duplex, and the ethernet interfaces also do duplex, but they don't do auto-negotiation. Setting the interfaces on the computers at each end to 100Mb/s full-duplex worked a treat, and now I'm getting something between 20 and 40 Mb/s -- completely respectable.

I followed Cyberdyne's suggestion and looked at the link options for the kernel I'm making for the NWR04B. So far, it looks promising, though I'm not that much better off. The problem was that the argument for puts, which should've been the address of some text to print to the screen, was 'way off and as a result I was seeing garbage. A closer look (with some paying attention this time) showed that, instead of being passed the address 0x28a0 (where you'd see EXMIF -- FIXME backwards), it was being passed the argument 0x428a0. And sure enough, in arch/armnommu/boot/Makefile, whaddawe see but this:

ifeq ($(CONFIG_CX84200_SMC),y)
#ZRELADDR        = 0x00040000
#ZTEXTADDR       = 0x00000000
ZRELADDR        =0x00008000
ZTEXTADDR       =0x00040000
INITRD_PHYS     =0x00700000
endif

This page told me that ZTEXTADDR is, basically, the address in memory where the kernel should expect to start -- or in this case, where the decompressor (I'm doing make zImage here) should expect to start. That sounds like something that would affect where things get put, all right, so I tried changing ZTEXTADDR to just 0x0 -- and sure enough, the argument passed to puts has the right address this time. But still no joy: when I load the image, I still don't see that EXMIF, but just a single character (which is better than the 416 characters of crap I was seeing previously) of uncertain ancestry (because for some reason the capture of serial port output to a file upon which I could run hexdump was not working). And furtherly furthermorish, that 416 characters of crap I was talking about were found in the original image starting at 0x418a0 -- an offset of 0x3F000, or off by a thousand from what I would have expected. So, like, what, memory is starting at -0x1000? Arghhh.

Following the instructions here, I was able to get my $14-at-Walmart-in-Oregon-four-years-ago IBM USB webcam seeing infrared without too much difficulty. Since getting the IR filter out proved to be a bit tricky, and since the note on this camera on Geoff's pages just said "Tricky to find", I thought I'd put up some pix and instructions on how to do this. I'll be curious to see how well it works tonight...I live above a busy street, and I'm hoping I'll be able to get some neat pix.

Just came across the Open Source Labs' photo album. There are some pretty funky pictures in there. Oh, and what look like some servers labelled in Braille. Man, I gotta work at a university.

...it's long-haired, shouty, short pregnant people in the streets. I'm totally voting for Lorne.

From here, via As Days Pass By.

Finally figured out two things:

1. The macros debug_reloc_start and debug_reloc_end in head.S are not called by default -- which is why I haven't been seeing any output from them. Duh.
2. If I put them in and close a comment properly, I can print out pc -- which as near as I can figure is set to 0x1008 more than it should be if the image was being run from the beginning of memory.

Currently trying a truly horrible hack (.rept 4112 instead of .rept 8 at the beginning of the image) to see what heppens.

I've been meaning to put this up for a couple weeks now, but haven't had a chance... I got an email from the customer service manager at Promise, and it's good news. As required by the GPL, they've finally released the source for the Linux kernel and the Busybox utility. They're used in the firmware for the VTrak SATA-to-SCSI drive arrays. MD5 checksums for the tarballs match those that I had downloaded (and mirrored) before. Hurray!

This is the coolest thing I have ever seen.

My uncle forwarded me an email from Tiger Direct, and I was sorely tempted to purchase the this. It's the Asante FR1104-G 802.11G router, and it's only 27 Canuckistan pesos. What's more, there is the rebate listed on Asante's website -- $20 US!, which would have almost made me a profit on the damned thing. Then I realized the rebate program had ended in April (damn!). And I couldn't find anyone else who'd tried to hack the thing, or run Linux on it. (Why else would I buy it?) The datasheet mentioned a 32-bit RISC microprocessor, but nothing more...almost certainly an ARM. And I remembered that I still have to get thing working with the wireless router I do have. (Dropped my crappy serial adapter the other day, so now I have to fix it.) Still, good deal if anyone wants one...

Arghh...the crappy little RS232 adapter I hacked together for the NWR04B got dropped the other day, and now I just see garbage. I spent two hours this afternoon re-soldering various connections, then gave up and ordered two of these (the 233 adapters, 3V version, in the DB9 shell). Even ordered 'em assembled. Work on that'll be on hold, though I may have some notes to put up. In the meantime, I'll be trying to get my PVR-500MCE working. Whee!

First off, The London Times has published secret UK government minutes from a 2002 meeting on the coming war in Iraq:

C reported on his recent talks in Washington. There was a perceptible shift in attitude. Military action was now seen as inevitable. Bush wanted to remove Saddam, through military action, justified by the conjunction of terrorism and WMD. But the intelligence and facts were being fixed around the policy. The NSC had no patience with the UN route, and no enthusiasm for publishing material on the Iraqi regime's record. There was little discussion in Washington of the aftermath after military action....It seemed clear that Bush had made up his mind to take military action, even if the timing was not yet decided. But the case was thin. Saddam was not threatening his neighbours, and his WMD capability was less than that of Libya, North Korea or Iran. We should work up a plan for an ultimatum to Saddam to allow back in the UN weapons inspectors. This would also help with the legal justification for the use of force.

From the ever-excellent Secrecy News (Friday the 13th ed.), which goes on to say: "Coverage of the matter has been sparse in the U.S. The Los Angeles Times reported on it yesterday, more than a week after the story broke in the UK on May 1, and the Washington Post followed today." Second, Seymour Hersh talks about Iraq, My Lai and the President of the United States of America:

But I think what's more important than that is that this guy, this Bush, absolutely believes in what he's doing. He's not like a nervous Richard Nixon, worried about, you know, "They're coming after me," or Lyndon Johnson quitting over Vietnam with great uncertainty about whether he is doing the right thing. This guy is absolutely convinced....I have a friend who is a major player who went to Iraq recently. There's been a series, unreported, a series of missions in Iraq that have all been there to study the war -- where are we? -- and they've all come back pretty negatively. This guy came back and he saw the President months ago. And he said, "Mr. President, we're losing the war in Iraq." And there was a sort of a three-second beat and Bush said, "You mean we're not winning." And this guy said, "Hey, I told him what I had to say. If he wants to turn it the way he wants to, that's the way it goes." You know, so he hears what he hears.

Link by way of Ken MacLeod: " You know how this stuff ends? It ends with your cities in rubble, your capital occupied, and your leaders hanged."

Error in Cygwin after upgrading: The procedure entry point _impure_ptr could not be located in the dynamic link library cygwin1.dll. Solution: As it says here, just reinstall the Cygwin package (and only the Cygwin package).

Someone wrote, like, nine of the firefox extensions I have lusted for all these years. Credit where credit is due.

Had a server up and die on me yesterday at work. What's more, it was the Very Important Server that does almost, but not quite, everything: Samba (only one, natch), NIS master, SMTP/POP/IMAP, CVS/SVN, printing, and since the installation of the disk array, serving quite a few home directories, too. I was answering a user's question -- "Oh, this should be on the wiki..." -- and noticed that the web server wasn't up. Another user poked up his head to ask if CVS had disappeared for a reason. Aw, crap. There were no lights on -- no power, disk or network activity, so I knew it wasn't good. The fans in the front and in the power supply weren't working, so it really wasn't good. Other things plugged into the same power bar were fine, so I tried power-cycling: no response. I unracked it, popped off the lid and watched the fans start briefly then die as I toggled the power switch again. Final verdict: not good. I took it to a better place to crack it open, and grabbed some spare parts: power supply, memory, graphics card. By the time I got everything back there, maybe five minutes had passed since I'd unracked it. And of course it turned back on. I checked the CPU temperature in BIOS: 30C. A quick check of the heatsinks and drives showed they were quite fine, too. I mean, yeah, it had been five minutes, but I'd think there'd be some residual heat I could feel. I was stumped, but decided to swap the power supply anyhow. (If anyone has any other ideas, please let me know.) So naturally, now I'm thinking about what to do about this server to keep this sort of thing from happening again. Here's a short list of the stuff it does:

• NIS master server
• SMTP/POP/IMAP
• CVS/SVN
• Samba PDC
• NFS for internal drives and the drive array

In order: NIS: Throw more slaves at it (though we've got two already, so I suspect that we're fine.) SMTP/POP/IMAP: The poor cousins, at least for now. Am assuming that an outage of SMTP/POP/IMAP that can be fixed in an hour is fine, and a longer outage indicates bigger problems. CVS/SVN: To some extent, just subsets of NFS. At any rate, I'm treating this like mail: a brief outage can be lived with, and a longer outage means I have bigger problems. Samba: A BDC is obviously in order and shouldn't be too difficult (said the guy who's never worked with LDAP before), at least as far as authentication goes. However, fileserving is made stupidly more difficult by the way we're serving home directories to Windows clients: all the home directories are listed as \\VeryImportantServer\foo. The better way to do this would be to run Samba on the other file servers as well (\\SomeSmallerServer\foo). Can't believe this only just occurred to me. NFS: The biggie. Obviously we should be breaking out home directories to some other server, but that just pushes the question over a machine or two: instead of worrying about the Very Important Server that Does Almost Everything, we're worrying about The One With The Files. Since the disk array is connected via SCSI to two machines (of which the VIS is only one), it would be possible, if the VIS was raptured again, to simply fsck the arrays and them export them from the second machine. This takes time, though: close to half an hour to fsck a 1TB drive. (I've never found the settings for newfs that are supposed to make fsck times approach that of a journaled FS; if anyone can fill me in, please let me know.) And there is some provision in amd for failover, but (as I understand it) not much. Another option is using ha+drdb, which looks quite promising. This means moving to Linux, though; I'm not opposed to that, but since I don't have a second drive array around I have no way of testing this, let alone gradually phasing this in. Hm. Any ideas, let me know.

My wife has a simple question: Who Won America's Next Top Model? Seriously! We were out. We have a bet going and everything. First reply with the right answer wins.

I am counting on all y'all.

Finally got my new PVR-500MCE working under Linux with the ivtv drivers. I'd been getting (black, fade to static, fade to black, rinse & repeat), and since a number of other people had reported the same problems I was beginning to think I'd just have to wait for the developers to catch up. But the solution turned out to be pretty simple. In the output from dmesg, I kept getting this line:

May 22 02:22:11 hunsacker ivtv: Encoder Firmware may be buggy, use version 0x02040011

What the hell, maybe the developers know what they're talking about, right? So I visited the wiki page, grabbed the revision they recommended, then ran:

ivtvfwextract.pl /path/to/firmware /tmp/encode.img /tmp/decode.img
mv /lib/modules/ivtv-fw-enc.bin /lib/modules/ivtv-fw-enc.bin-old
cp /tmp/encode.img /lib/modules
ln -s /lib/modules/encode.img  /lib/modules/ivtv-fw-enc.bin
shutdown -h now

I waited a minute to make sure the card would lose the firmware, rebooted, then ran these two bit from my Makefile:

insmod:
     sudo modprobe cx25840 debug=1 no_black_magic=1
        sudo modprobe tuner debug=1
        sudo modprobe wm8775 debug=1
        sudo modprobe tveeprom debug=1
        sudo modprobe ivtv ivtv_debug=1 
        sudo modprobe tda9887 debug=1

test:
       /home/aardvark/bin/ivtv-0.3.3h/utils/ivtvctl --set-input 6 ; \
       for i in `seq 2 99` ; do \
                        ptune.pl --channel $$i --input /dev/video0 --freqtable ntsc-cable --tuner-num 0 ; \
                        cat /dev/video0 > test-channel-$${i}-$${j}-input-6-dev-0-tuner-0.mpg & \
                        sleep 5 && kill $$! ; \
        done 

And sure enough, it worked! Now: on to MythTV!

My, how things change... RFC 528, SOFTWARE CHECKSUMMING IN THE IMP AND NETWORK RELIABILITY, written by John McQuillan, was published in June of 1973. It's a surprisingly readable document that introduces packet checksums on the Internet. From TFRFC:

Our idea of the Network has evolved as the Network itself has grown. Initially, it was thought that the only components in the network design that were prone to errors were the communications circuits, and the modem interfaces in the IMPs are equipped with a CRC checksum to detect "almost all" such errors. The rest of the system, including Host interfaces, IMP processors, memories, and interfaces, were all considered to be error-free. We have had to re-evaluate this position in the light of our experience.

IMP stands for Interface Message Processor, and was what we'd now call a router. Having grown up with TCP/IP (drawn from me ma's teat, bye!), it's hard for me to imagine a protocol without some kind of checksum, or assuming that nearly all of your equipment would be error-free -- but then again, I have the benefit of 30 years of network research. It's fascinating to find out where your assumptions come from. Knowing his audience, the author threw in a good war story:

One of the earliest problems of this kind was discovered in 1971. The Harvard IMP was sometimes crashing in an unknown manner so that all the other IMPs were affected. It was finally determined that its memory was faulty and sometimes the routing messages read out from memory by the modem output interfaces were all zeroes. The adjacent IMPs interpreted such an erroneous message as stating that the Harvard IMP had zero delay to all destinations -- that it was the best route to everywhere! Once this information propagated to the other IMPs, the whole network was in a shambles.

(Lest we think that we've left this sort of thing behind, there are BGP flaps and such to keep us honest.) Tales like this weren't just there to entertain, though; he was anticipating serious objections about whether checksumming was worth it.

On of the major questions about such approaches is their efficiency. We have been able to include the software checksum on all packets without greatly increasing the processing overhead in the IMP. The method described above involves one checksum calculation at each IMP through which a packet travels. We developed a very fast checksum technique, which takes only 2 msec per word.

And in case the breakup of The Beatles, the Nixon presidency and the sight of a man playing golf on the moon at taxpayers' expense was not enough to convince you that, yes, it was a very different time, check out this approach to data collection:

On March 13, a new version of the IMP program was released with software checksum code. In this program, when a packet is found to have an incorrect checksum it is discarded, and a copy of the data is sent to the NCC.

Ah, the things you can do on a research network. And lastly, we have a foreshadowing of RFC 761, the first RFC to describe TCP:

Finally, we are looking into the structure of an optional IMP- Host/Host-IMP checksum to complete Host/Host end-to-end checksum. Under such an arrangement, the IMP and Host could agree to verify the checksums on the messages transferred over the interface between them, and the appropriate signalling mechanisms would be provided to handled errors. With this technique in effect, two Hosts could be certain that their messages were delivered error-free or else they would be notified of an error, and could then retransmit their message if desired.

In Democracy's Shadow: The Secret World of National Security