DSL's down, and I don't care

So as of November, we've got a new place to live now, right in downtown Vancouver. It's back in the West End; I lived there for seven years, and with my (now) wife for two of those. We moved out to the sticks for cheaper rent and a bigger place, but realized we missed downtown: most of our friends are here, and there was really nothing much where we were living. It was (is) a nice place, but a bit of a black hole as far as things to do go.

We were lucky, and found a nice place; the building manager took a shine to us, I guess, and offered us the place as we were looking at it. It costs more but it's about the same size as our current place, and we're happy to be moving back.

'Course, this does bring up the question of Internet access. I'm hosting five websites on Thornhill, the Linux server, and doing DNS for another domain that belongs to a friend of mine. My ISP is not the greatest value; a static IP is currently costing me an extra $80 per month (and the TOS still prohibit servers, although they've yet to enforce it), and I just can't justify that with the extra rent we'll be paying. That means going to DHCP, dynamic DNS, and I don't know what-all.

There are other options, of course. Shaw is the local cable company, and I could always go to ADSL. God knows there's lots of choices there; I used to work for one of them. However, my experience there has made me extremely wary of ADSL in Vancouver.

We resold ADSL service from a company that I don't want to name; let's just say that if you think of the c in E=mc2, you'll think of their name. When we started I was quite impressed: static IP address (usually a 10.something, but public ones were available if you asked) and servers were okay. But then it turned into this absolute nightmare:

• It turned out that they weren't the main movers and shakers for the service. We knew they resold to other companies -- they made no secret of that -- so our assumption (because we were too stupid to ask, that's why) was that they controlled the equipment. But they didn't. How did we find that out? Follow the bouncing ball:

• They originally set up 3Com modems so that they could query them via SNMP for traffic stats, because their business model involved giving away a small amount of bandwidth suitable for most people if you didn't include file-sharing, then banging the people who overshot at $20/GB (at least, that was what we charged).

• Only the 3Coms didn't work as well as planned, and anyway were in the customer's hands so you couldn't really trust them anyhow. So they went to MAC address-based traffic counting. Free bonus: MAC address-based filtering, too -- you had to have a MAC address the router knew about, or you couldn't get out of their private network. Only that bit they didn't tell us about. Yep, they didn't say a thing.

• We found this out because we had people who suddenly couldn't connect. With much prodding we could get the people we knew to try and fix things. Usually, it wouldn't last too long, and we were back to square one. Connectivity was utterly erratic; sometimes it worked, sometimes it didn't, and there wasn't a blessed thing we could do about it except plead with our upstream folks for help. Of course, we were doing all this while pretending to the customer that we had our hands directly on all the misbehaving equipment. This was when we began to suspect that we were in the same naive position as our customers.

• Eventually, we discovered that was, in fact, the case. Every time we called up to beg and plead, we were one of a half-dozen ISPs. And then they would have to go beg and plead for help on our behalf to their shadowy masters. Unless it was after 5pm, of course, which turned out to be quitting time when you're a shadowymaster. Whee!

• We found out that we were supposed to be able to add MAC addresses via a web app (IE only, of course, just to make things even more fun), and then those MAC addresses would be allowed out. This did not work. More prodding told us that they were working on it, and it would work Real Soon Now.

• With even more prodding it came out that the reason this was all happening was because there were problems with the database that held the MAC addresses of the customers. When the database had problems, the routers defaulted to DENY rather than the perhaps more sensible option of just letting the fucking packet through, we'll fix it later.

• Then we found out the database was MS-SQL, and wondered if that might be part of the problem too.

• Then Blaster hit.

• Then many, many months later, we started getting bills, which had not been coming in all this time, from our upstream people. This wasn't just for service for x customers over y months, of course; no, that would be too simple. Instead, this was bandwidth for x customers over y months, including the people who turned on Kazaa then went on vacation for a week. Records were either non- existent or untrustworthy, and neither we nor the customers'd had a way of checking their usage. (That too was coming Real Soon Now.) Of course we had to get some money from the customers -- at $20/GB, remember, and some people were 100 GB over their limit. You try talking to a customer who's just got a bill for $2000 for usage over the last n months, when they were expecting $39.95 plus tax.

Christ almighty, it was such a giant clusterfuck. I began to sing between support calls, to the tune of "Jimmy Crack Corn":

A customer called in to 'fess Our techn'cal service was the best So why was he in such distress? The database dropped his MAC address. Sing it! DSL's down, and I don't care DSL's down, and I don't care DSL's down, and I don't care -- It's Lightspeed's fault again!

Eventually it did begin to work, and from what I understand it does quite well today. But still...shudder. Never again.

They resold their services to a bunch of different local ISPs, most of which I recognised (can't remember who they are now, worse luck), that I'm deathly afraid of getting stuck in the giant sucking wound that is Lightspeed Internet.

There's Telus, of course, but they're liable to be even worse; they've been going through hellacious layoffs in the last year, and horror stories abound about the wretched customer service these days. In fact, they're even being investigated by the CRTC because of the number of complaints. Besides, Telus' service was spotty to begin with; their DHCP servers would go down frequently, and take people's connection with it. (Ha. It got to the point where the only selling point I could repeat in good conscience was the fact that our tech support people were easier to reach than Telus'.)

So...unless I can find an ISP with a penchant for handing out cheap static IP addresses and being generous with traffic, I'll do dynamic DNS. Some day I'll colocate, or get a virtual server. Until then, I'll settle for cheap.

Tags:

Worst._title._ever.

title: Worst. Title. Ever. date: 2004-09-27 09:02:55

Good idea:

The sad thing I've noticed is that some people new to the world of lawyer blogging have never heard of blogs like BeSpacific.com. Instead, their news aggregators may be filled with new blogs, which is not a bad thing in itself, but lack blogs that I consider to be bedrock legal blogs. That bothers me. Maybe it shows that I'm getting older. But Bob is on to something important. So, I'm announcing a new feature of this blog where I'll highlight the core legal blogs that meet my definition of excellence.

Terrible title:

I'm tentatively calling this feature "Essential Blawgs."

My advice: stay away from the fish.

Tags:

Rtfem

title: RTFEM date: 2004-09-25 16:12:38

Okay, this is a tangled tale...

I've got an old install of FreeBSD I decided to bring up to date. Being the funky sort, I decided to do a portupgrade and keep the OS the same. (Security fixins [fixins! yeah!] have been applied, so I'm not too worried about doing make world.) And of course, desktop of the elder gods (...) Gnome was installed -- 2.4. Whee! Dive right in, right? That's why the gods gave us port-upgrade! Wrong. Got this error:

/usr/bin/ld: warning: libintl.so.4, needed by /usr/X11R6/lib/libgconf-2.so, may conflict with libintl.so.6
/usr/bin/ld: warning: libgmodule-2.0.so.200, needed by /usr/X11R6/lib/libgconf-2.so, may conflict with libgmodule-2.0.so.400
/usr/bin/ld: warning: libgobject-2.0.so.200, needed by /usr/X11R6/lib/libgconf-2.so, may conflict with libgobject-2.0.so.400
/usr/bin/ld: warning: libgthread-2.0.so.200, needed by /usr/X11R6/lib/libgconf-2.so, may conflict with libgthread-2.0.so.400
/usr/bin/ld: warning: libglib-2.0.so.200, needed by /usr/X11R6/lib/libgconf-2.so, may conflict with libglib-2.0.so.400
../../libgnomevfs/.libs/libgnomevfs-2.so: undefined reference to `bonobo_poa_get_threaded'
gmake[3]: *** [test-vfolder] Error 1
gmake[3]: Leaving directory `/usr/ports/devel/gnomevfs2/work/gnome-vfs-2.6.1.1/modules/vfolder'
gmake[2]: *** [all-recursive] Error 1
gmake[2]: Leaving directory `/usr/ports/devel/gnomevfs2/work/gnome-vfs-2.6.1.1/modules'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/usr/ports/devel/gnomevfs2/work/gnome-vfs-2.6.1.1'
gmake: *** [all] Error 2 *** Error code 2 Stop in /usr/ports/devel/gnomevfs2.

which lead me to this page, which (sadly) wasn't much help. I figured out that the problem was conflicting versions of glib, but WTF to do about that? After much searching, I began to remember faintly something about a FreeBSD Gnome upgrade page. Sure enough, this was the one I was thinking about. And look at this:

It is not possible to upgrade from GNOME 2.4 to GNOME 2.6 by simply running portupgrade(1). There are new dependencies, and ports will build out-of-order, eventually causing the build to fail. Additionally, GTK+-2 cannot install when there are input methods installed which were linked against older GTK+-2 versions. To work around these problems, and to provide an update mechanism as simple as portupgrade(1), the FreeBSD GNOME team has produced a comprehensive upgrade script. The script can be downloaded from: http://www.FreeBSD.org/gnome/gnome_upgrade.sh Simply download that script, and save it to disk.

Sigh. So I read the bit about how running the script after doing The Thing You Shouldn't Do would probably not cause problems, and decided to plunge ahead. But things ended badly when it came time to upgrade Scrollkeeper: [configure does some stuff...] checking for DocBook XML DTD... configure: error: not found. Make sure you have the DocBook DTD installed and ensure that it is registered in /usr/local/share/xml/catalog.

Wha'? Being me, I ignored the bit about the file it was looking for and dove right into the configure script. By the time I came out the other side, I figured out that it was unable to find /usr/local/share/xml/catalog. Then I re-read the error message. Well, fuck. After some digging around, I found out that this file was provided by sdocbook-xml on the old system -- version 4.1.2.5. The version I was trying to upgrade to ws 4.1.2.5_2. Shouldn't be that big a difference...but it is. The Makefile for the older port has this at the end: post-install: ${MKCATALOG} -q -c ${CATALOG} install sdocbook catalog But the newer port's Makefile has this: post-install: ${XMLCATMGR} -sc ${CATALOG_PORTS_SGML} add CATALOG ${SDOCBOOKDIR}/catalog ${XMLCATMGR} -c ${CATALOG_PORTS_XML} add nextCatalog ${SDOCBOOKDIR}/catalog.xml which, natch, doesn't make the file that scrollkeeper's configure script is looking for. Freshports.org has this note on its sdocbook-xml page:

Switch to using xmlcatmgr from mkcatalog. # Maintainers, please let me know if I break something in your port. Submitted by: hrs

After that is listed a crapload of PRs that the change fixes, or at least addresses. What to do? The thing that every sysadmin does sooner or later: cheat. # ln -s /usr/local/share/xml/catalog.xml /usr/local/share/xml/catalog And bugger me senseless with an iPod if it doesn't work. Who said cheaters never prosper?

Tags:

Ladies_and_gentlemen,_i_give_you_coprolitic

title: Ladies and Gentlemen, I give you Coprolitic date: 2004-09-24 07:06:30

Goddamn, but this is one god-awful case. Any time I'm asked why we use Shuttles, I'm gonna give 'em an earful.

1. The electrical outlet in the back of the case is not part of the power supply; instead, it's connected by a three-inch cable to the actual power supply. There are three bare pieces of metal coming out the back of the outlet, which have three leads with those metal sleeves attached. Think old AT power supply switches.
2. The power supply is held in place by screws that go into the ventilation holes in the power supply case. There's a small chance they might actually be dual-purpose by design, but mostly it looks like they're using very short screws in order to avoid hitting anything inside.
3. The disk basket (what the hell is the right term? the metal bit that holds the CD, hard drive and a floppy) can't be removed (as it can in Shuttles), so installing the motherboard or hard drive is a huge pain in the ass.
4. The disk basket has a bracket that hangs underneath that is meant to hold the hard drive. There is a flange at one end where it slides into place, and at the other there is ONE screw holding it in place. Needless to say, that does not inspire confidence.
5. The disk basket looks like the space for the floppy can hold a hard drive (like Shuttles). That's a lie. Because of the screws holding the front cover in place, it's about a quarter-inch too narrow for a hard drive.
6. The clearance on either side of the hard drive bracket is very narrow, so it's difficult to guess which end is meant to go in first. I tried one side only to find that the electrical connector wouldn't allow the case to close.
7. The disk basket is just a little off-center. (It was the office Apple guy who noticed this.) That means the hard drive must go in so the electrical connector goes on the wider side, right? Right -- but it still doesn't let the case close nicely. It'll do it, but it causes cringing when you think of the electrical connector being squished up like that.
8. Getting the IDE cable to go to the hard drive is a huge pain in the ass. The orientation of the motherboard (VIA 1000, I believe) put the first IDE connector on one side of the case, and the disk basket -- naturally -- put the hard drive connector at the other end. It'll go, but it causes cringing when you think of how much you had to crimp the IDE cable to get it to work.

I may put in more later as therapy brings back more memories. But for the love of everything you hold holy, stay away from this case. That's the Checkercube case, available at http://www.checkercube.com; it sucks ass through a straw. You got that, Google?

Tags:

Rogue_dhcp

title: Rogue DHCP date: 2004-09-21 14:53:54

One of the pieces of equipment at work is an oscilloscope that runs W2K. "WTF?" I hear you say. "Saint Aardvark, WTFSOF?" But it's true. Don't know why, but it's true.

(We have another oscilloscope that came with an unregistered copy of XP, but that's another story.)

Being the aspiring good sysadmin I aspire to be, I bought some cheap cable routers -- you know, the generic DLinkSys jobbies with a built-in firewall. I hooked it up, told people not to hook it up to the network without it, and forgot about it... ...until today when I was working on our firewall and noticed it was blocking broadcasts to 172.16.0.255, UDP port 137.

That's not a netblock we use, so I was a bit surprised. Good ol' tcpdump showed it was anouncing itself as the local master for workgroup INFINEON. Oh shit, it's the oscilloscope. I checked out the lab and, sure enough, the firewall was being used as a quick-n-dirty switch on the firewalled side, and the oscilloscope was plugged in. Fuck!

To make matters worse, a little bit later someone comes up to me and asks if there's anything "funny" with the network. (I love that question. It's so...definite.) Checked it out, and his laptop has grabbed an IP address from the (fortunately, by-now-disconnected) DHCP server that comes with the router. Double fuck!

I ran off to London Drugs to get a switch, and was lucky enough to find a 16-port Linksys. (SMCs are for shit. SMC? Quality? It is to laugh. Linksys switches are giving me trouble too, but at least it's less trouble.) Set up, and everything is working for now. So here's my mistakes:

1. Not making it perfectly clear how to hook up the router correctly, and not making it impossible (or at least painful) to hook it up any other way.
2. Not making it obvious -- written warnings, flashing neon, whatever-- that the router was not a switch.
3. Not having something, somewhere, to at the very least watch for weird IP addresses and report them, or (better yet) to watch for rogue DHCP servers and report them, or (best of all) to watch for and shoot down with lasers any rogue DHCP servers.

There is, of course, the mistake of not having managed switches that would mitigate all of these mistakes, but with luck we'll be getting those shortly.

Tags:

cie.saintaardvarkthecarpeted.com

My wife and I are thinking of moving back downtown soon, so that means housecleaning in preparation. I've been biting the bullet and putting together an email to the local LUG mailing list offering up 7 (!) computers in various states. There's a lot of things started that never got finished.

Cie (named after Cie Baxter, of course) was my first server. It's got a 200MHz Pentium, 48MB of RAM and a 2GB IDE drive. Friends of mine had upgraded, and they asked me if I wanted their old computer. Hey, who's gonna turn down a free computer?

My first computer job was at a small ISP. I quickly wrangled a static IP (still allocated!) from the sysadmin and snaked an ethernet cable over the ceiling tiles from my desk to the server room. It was meant to be an IPv6 tunnel broker, but that quickly fell by the wayside; I got the basic routing sorted out, but then I lost interest when it came time to figure out an authentication scheme. I'm like that: lots of ideas, little follow-through. I've been lucky to stay interested in computers as long as I have.

What it did become was the web, mail and DNS server for my domain, my wife's, and a few friends of mine (though that last one I'm only doing DNS and secondary MX). It ran Slackware 7.0, straight from the CD set I bought at Chapters back when they still had books (and Linux CDs, apparently).

I remember convincing the sysadmin at work that I could handle securing BIND 9 (quietly convinced the entire time I was going to get r00ted within a week), and telling the owner of the company (who is even more flighty than I am) that an IPv6 tunnel broker would provide lots of value to our customers, and amazing the friends who had donated it in the first place that an old computer that would barely run Windows 98 could be of any use to anyone.

Incidentally, I was always a bit amused by the fact that I had an AOpen sticker on my server; I learned to loathe AOpen modems while at Dowco, and I swore I'd never buy anything with their name on it. But hey, since Cie was free, I guess it doesn't count...

I left Dowco but kept Cie, and it sat here by my desk at home, using up my ridiculously expensive static IP address for a long time. It was fine. And then, a while back, I tried to set up a pretty heavy PHP-based CMS for a friend of mine. Cie choked, right away: it took 20 seconds to render a page. That was no good at all.

I asked around, and the consensus was it was the RAM, idiot. That and the noisy PS fan convinced me it was time to upgrade.

My boss was selling his old computer -- a Compaq desktop machine with a 500MHz P3, 64MB of RAM and a SCSI tape drive. I bought a big-ass hard drive (and made sure it was going to be quiet -- best investment ever), put the tape drive in my fileserver, bought a 256MB stick of RAM and sat down to build a new server. I went with Slackware 9, spent some time locking down my firewall and /etc/fstab, got the latest kernel and OpenSSH sources, and installed, compiled, swore, reinstalled, formatted, and reinstalled again. Thus was born Thornhill (named after Lisa Thornhill, of course).

It's only fitting that I always meant to use Cie as a honeypot, but never got around to it.

Tags:

Links for later

• http://xoomer.virgilio.it/flavio.stanchina/debian/fglrx-installer.html
• http://fabrice.bellard.free.fr/qemu/

Tags:

Rss_over_dns:_is_it_good,_or_is_it_whack?

title: RSS over DNS: is it good, or is it whack? date: 2004-09-10 20:16:32

A quick Google turns up SSH over DNS. But after reading this, I'm wondering if anyone has tried RSS over DNS. Bueller? Bueller? (I hesitate to link to this posting, because this guy seems to know what he's doing, and I'm pretty sure I don't. But since it's 8pm on a Friday and I've had my snifter of port, I'm going to risk looking like an ignorant ass.)

Pro: You've got an infrastructure that has caching built in. You control the TTL. There's the TXT record, which has already been hacked to do awful, terrible things. You could compress your feed ('cos everyone knows that bzip2 is a magic bullet), or split it into different host names like the DNSTorrent guy.

Con: Short record limit. Me talking out my ass. Millions of geeks gnashing their teeth at the downfall of the intarweb. (It looks like there's been lots of thought [or at least some] about using BitTorrent, with a general consensus of "no, it won't work".)

Incidentally, the government of Alberta has a crapload of RSS feeds available. Who knew?

Tags:

Libgphoto_< >_database

title: libgphoto <-> database date: 2004-09-09 20:21:18

So I run Ansel as a gallery program for my website. I recommend it: it's simple, does what it needs to, and gets out of the way. (I notice, BTW, that 2.0 has just been released. I'll have to have a look.)

I've customized it a fair bit for my site -- little things like the ability to edit a caption, or to go backward in an album as well as forward. It's nothing big so far, but I've been thinking of releasing the changes as a fork of Ansel. (I did send the author the first batch of changes I made, but they didn't make it to his version [which is fair and his choice].)

I've been working a bit, every now and then, on the ability to make big sweeping changes at once: edit a bunch of photos, move 'em around in an album, that sort of thing. But there's another issue holding me back: getting pictures into Ansel in the first place.

You can upload a picture to Ansel, as you'd expect. But having a digital camera has really increased the volume of pictures my wife and I take, and adding pictures one at a time is a major pain in the ass. It's one thing to do it for the first 50 pictures, but after that it's insane. And adding 50 pictures in one upload is doubly crazy, at least with my ignorance of PHP. So what to do?

We just bought an iBook, and iPhoto is on there, of course. One of the impressive things is that it found the digital camera and grabbed the pictures from it without problems. By contrast, gphoto took some messing -- to be fair, the camera we bought wasn't supported by gphoto when we bought it, but was added shortly thereafter (phew!). Red Hat or gtkam or some combination of the two will occasionally get cranky: drivers don't get loaded, or the pictures won't all be downloaded, or things will just crash. So when iPhoto Just Worked (TM), I assumed that was it for gphoto.

But it wasn't: the way it manages photos pisses off my wife, and she prefers gphoto and Linux (!). So we're back to our original problem: how to add a bunch of photos to the gallery in one go?

I got to thinking about this the other day, and realized the problem was a bit more basic than that; really, it was: how to add a bunch of photos to a database in one go? I started thinking about duplicating iPhoto's functionality with MySQL, then forgot about it. Instead, I decided that what I really wanted was something like an SQL plugin for gtkam. You'd specify a database, connection details, and what table to put it in. Exif info could go in, along with thumbnails. Then, for full geek points, you tunnel the connection over SSL to your server. Huh? Huh? Yeah!

I had a quick look through Google to see if anyone else was doing this...no. Then I braced myself, held my nose and waded into the awful, slow-as-fuck, confusing and generally bad Sourceforge mailing list archive for gphoto. (I've tried, without success, to subscribe to the gphoto mailing list three times. Don't get me started.) Again, nada.

I don't know enough about how libmysqlclient.so (ha!) to know if, say, just using gphoto from the command line to, say, dump to standard out might work; it seems unlikely.

Partly I'm putting this here as a reminder to myself. Partly I'm putting this here in the hopes that someone will call bullshit if it's needed. But I'm also putting it here in the hopes that some actual programmer, of which I am not, will take up the idea. It'd be cool. I'd totally link to your site.

Tags:

Aha!

A while back I set up greylisting on Postfix for my home server. It works well, but I have the same concerns now that I did then. The script (smtpd-policy.pl from the examples section of Postfix' source) feels like a bit of a crock; yes, it's just the example script, but I don't like the Berkeley DB files, and comments in the code like "DO NOT create the greylist database in a file system that can run out of space" make me nervous. It hasn't been a problem -- in, oh, six months of running the file is only up to about 5.5 MB. But still: there's no provision for removing old entries, which means an awful soul-searching battle with the database if you ever need to trim it.

I had a brief look at the script tonight, hoping to find a way to maybe hack in MySQL support, but decided to check with Saint Google first. Sure enough, there's gps, the Greylist Policy Service for Postfix. Uses C++ for speed and MySQL/PostgreSQL for the backend, which is nice. I should be able to hack up a migration script for the old entries (just as soon as I hack up a migration script for all the old journal entries...), and all should be good.

One thing I'm noticing with greylisting, though, is just how many attempts are being made from multiple IP addresses within a short time; one attempt, today, had attempts from four different IP addresses within five minutes, all from the same made-up email address. The original Perl script has the advantage that I can change it easily -- I know Perl, and I'd be pretty much starting from scratch with C++ -- and maybe add the ability to track this sort of thing. It'd be nice to be able to tarpit attempts to do this, say on the third attempt.

Tarpitting...another problem with Linux. The TARPIT module for netfilter has yet to be updated to work with the 2.6 kernel, and I really don't want to switch back to 2.4 just for this. LaBrea is nice, and I'm running a lashed-together natd configuration on my FreeBSD firewall box in conjunction with LaBrea running on my desktop on a second interface. It works, but it doesn't work in the case of a Linux webserver running on its own, outside the main firewall. I'm even less a kernel hacker than I am a C++ programmer, and figuring out the compiling problems and changed skbuff route structures (say) is beyond me. It's things like this that make me want to move to OpenBSD. Yeah, rebuilding a server and learning a new firewall language is a pain in the ass, but at least it's one I can handle.

Tags: postfix spam mysql

Perfect_timing

title: Perfect timing date: 2004-09-02 06:59:49

So back in February (Damn, I've got to get all these old journal entries into WordPress) (although that isn't one I've ported...) I wrote about network problems.

We've got three Linksys Etherfast 4124 switches cascaded together, and an errant 8-port switch elsewhere in the office managed to freeze up all three of them. To track down the source of the problem, I had to do a Binary Level One Elimination Search (rip out half of the cables, see if the problem went away; if so, plug in half the remaining cables and see if it came back; rinse and repeat).

Naturally, all this was complicated by the fact that we're using dumb switches -- no management ability, no console ability, so no way to figure out what might be going on. (The fact these things were freezing up in the first place is yet another complicating factor, but that's another story.)

I did a little reading, asked a few questions (which you folks were kind enough to answer), and put in a request for Catalyst switches. The budget is currently before The Board.

Welp, yesterday it happened again. There was a brief thunderstorm, and at least one power surge knocked a bunch of the smaller (8-port) switches we have deployed at the network edge (ie, desks) for a loop. I was in the middle of trying to figure out the cause of one (small, just a few computers that couldn't connect) network outage when a telephone repair guy showed up on an unrelated service call. When I finished showing him what needed to be done, the whole network was frozen.

The BLOES revealed the problem to be the switch I'd been looking at before, which I suppose I should've suspected. A power cycle seemed to fix the small switches, and was the only thing that unfroze the big switches. Afterward, I went around and checked all the small switches we have, power-cycled the zombies, and made sure everyone was okay. Then I told my boss what had happened and why. I suspect I'll get the switches, plus some UPSs and better surge protectors.

The larger problem is that we don't have nearly the network drops we need. These small switches are everywhere, because the channels in the floor for network cables are nearly full. We're looking for more space -- a move is scheduled by next June, when our current lease expires -- and I've let my boss know that we couldn't possibly have too many network drops. 3 ethernet + 1 phone jack every ten feet would be great, and I think they'd all be used within six months.

In other news, the move went well; tunneling MySQL over SSH worked perfectly. Hurray! Also, I'm in the process of setting up a Debian server for a friend of mine. He's a web designer who's thinking about running his own server. He's also a Mac guy who doesn't have the funds to get an Xserve, so I'm giving him an old computer to learn Linux on: how to install stuff, how to run a nameserver, build a firewall, send mail, yadda yadda. I've come across some good tutorials on setting up a server, but I'm also looking for something that'll tell him why, not just how. If anyone has any suggestions, please let me know.

Tags:

Remember_that_great_swooping_shot_out_of_cam's_mouth_in_ferris_bueller's_day_off?

title: Remember that great swooping shot out of Cam's mouth in Ferris Bueller's Day Off? date: 2004-08-27 14:35:16

From this June posting to the wine-users mailing list:

On Sat, Jun 19, 2004 at 06:54:09PM -0400, eternal wrote:

The cvs sources reserve memory up front. This is incompatible with FreeBSD's mmap address allocation algorithm. The current Wine implementation can't work by design on FreeBSD. mhmm... rather weak, if you ask me... when is this as of? the wine-20040505 port didnt have this issue, but, then again, it had alot of other issues that made it useless..... Some time in May. Check the creation date of wine/libs/wine/mmap.c when the wine_anon_mmap() function was moved out of wine/libs/wine/loader.c. I've asked a question on FreeBSD's arch@ mailing list, but haven't had a reply yet. I'll give it a week or 10 days and if no response by then, I'll email one of the FreeBSD vm developers directly with a cc to the private developers mailing list. I see no reason why the FreeBSD algorithm can't be changed to allow Wine to function the way it is now coded. It is unlikely that a change to the mmap address allocation algorithm will ever make it into the FreeBSD4 tree though. Hopefully by the time a FreeBSD5 stable branch is created. -- John Birrell

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Tags:

File under Golden

So I had a bit of a brainstorm the other day. I've got two servers: Here and There. There's some stuff Here that needs to move There. The problem is that the server Here is in use a fair bit, and part of that use involves INSERTing things in MySQL and then SELECTing them back again. It's a pain to shut down things Here altogether in preparation for moving There, particularly as the move is liable to take, oh, twenty-four hours or so. The database needs to be consistent between the two, but the length of the move makes that impractical unless Special Measures are taken.

Dark server room. Midnight. We see THE SUPERVISOR talking to THE SYSADMIN.

SUPERVISOR: That database needs to be consistent, dammit!

SYSADMIN: (tightly) I can't do that without taking...special measures.

SUPERVISOR grimaces.

SUPERVISOR: Whatever it takes, dammit. I don't want to know.

SYSADMIN: All right, then. I'll do your dirty work.

SYSADMIN turns slowly and walks out the door.

SUPERVISOR: Dammit!

I will conced that's a little dramatic. But what else would you call MILITARY-GRADE ENCRYPTION, i.e. SSH tunnels from Here to There? (It must be military grade; it's developed in Canada.) Okay, so it's not that big a deal for you people what think all the time. But it was pretty clever, I thought, and would ensure that the everything was, like, cool and stuff because -- this is the good part, see -- we would tunnel the MySQL connection from Here to There over SSH! Brilliant! It only needs a short break in the service from Here, then all the database updates that might come from Here go There! Yeah! So I began trying that out today. It's was a bit of a pain to set up. I had to do some funky firewall-fu There to get SSH in in the first place. Then I had to figure out the right syntax for netmasks for hosts.allow (for the record, it's 255.255.255.0, not /24). Then I had to figure out how to get the MySQL client to connect to an arbitrary port. That took a while. I offer you this hard-won piece of knowledge in the spirit of Free Knowledge:

When using the MySQL client, do not confuse the-Hoption (output in HTML, please) with the-hoption (connect to the specified host, please). That's a silly mistake to make.

However, what's not a silly mistake is expecting -h localhost to do the right thing and connect. This is either an omission in the otherwise-excellent MySQL, or else a case of our nameserver not having a record for localhost. I strongly suspect the latter.

That said, it appears to be working: I can now be refused a connection to the MySQL server There from Here. Truly, I am a golden god.

Except maybe when it comes to backups or SCSI or something. I ran into some problems with AMANDA's backups last night. I saw these rather frightening messages this morning in dmesg. After sticking my tongue cutely out the side of my mouth to indicate fierce concentration and colouring in some printed log files in different flourescent colours, I was left with this series of messages:

Aug 23 23:46:57 localhost /kernel: (sa0:ahc0:0:3:0): SCB 0xe - timed out Aug 23 23:46:57 localhost /kernel: >>>>>>>>>>>>>>>>>> Dump Card State Begins < <<<<<<<<<<<<<<<< Aug 23 23:46:57 localhost /kernel: <<<<<<<<<<<<<<<< Dump Card State Ends >>>>>>>>>>>>>>>>>> Aug 23 23:46:58 localhost /kernel: (sa0:ahc0:0:3:0): Queuing a BDR SCB Aug 23 23:46:58 localhost /kernel: (sa0:ahc0:0:3:0): Bus Device Reset Message Sent Aug 23 23:46:59 localhost /kernel: (sa0:ahc0:0:3:0): SCB 0xe - timed out Aug 23 23:46:59 localhost /kernel: >>>>>>>>>>>>>>>>>> Dump Card State Begins < <<<<<<<<<<<<<<<< Aug 23 23:46:59 localhost /kernel: <<<<<<<<<<<<<<<< Dump Card State Ends >>>>>>>>>>>>>>>>>> Aug 23 23:46:59 localhost /kernel: (sa0:ahc0:0:3:0): no longer in timeout, status = 34b Aug 23 23:46:59 localhost /kernel: ahc0: Issued Channel A Bus Reset. 1 SCBs aborted Aug 23 23:46:59 localhost /kernel: (sa0:ahc0:0:3:0): failed to write terminating filemark(s) Aug 23 23:47:59 localhost /kernel: (sa0:ahc0:0:3:0): SCB 0xe - timed out Aug 23 23:47:59 localhost /kernel: >>>>>>>>>>>>>>>>>> Dump Card State Begins < <<<<<<<<<<<<<<<<

...and on it goes.

Saint Google asserts that this is probably a case of SCSI cables not being terminated properly, or getting too close to the power supply. Sure enough, the latter may be a problem. I made what adjustments I could without taking down the server, and we'll see what happens tomorrow. Weird. I am having the strangest sense of deja vu right now looking at that log entry in vi. Huh.

What else? I'm typing this right now at a local coffee shop where I was able to pick up wireless service; unfortunately, the cheap bastards want money. I tried pinging various addresses for a while, thinking about setting up an IP-over-ICMP-or-possibly-over-DNS proxy from my home network, then gave up and turned off the wireless card. It's good to know that it works, and it's good to know that there are places left where you can hear both Lisa Stansfield and Rick Astley in the space of five minutes. And there was much rejoicing.

Cool bit of the day from the PHP docs:

<directory /var/www/html/mydatabase>
    php_value mysql.default_user fred
    php_value mysql.default_password secret
    php_value mysql.default_host server.example.com
</directory>

Graham Rule at ed dot ac dot uk, you rule.

Tags: mysql

Cool

From procinfo(8), part of sysutils:

-Ffile Redirect output to file (usually a tty).

Nice if, for example, you want to run procinfo permanently on a
virtual console or on a terminal, by starting it from init(8) with
a line like: p8:23:respawn:/usr/bin/procinfo -biDn1 -F/dev/tty8 `

At last, a Linux equivalent of systat -vm. Or nearly, anyway.

Tags: windows

My_scrollback_buffer_is_bigger_than_your_scrollback_buffer

title: My scrollback buffer is bigger than your scrollback buffer date: 2004-08-20 23:59:19

There are two big-ass reasons why FTP sucks ass: clear-text passwords and the way it fucks with firewalls. Both are awful hangovers from the early days of the Internet where cute little elves would pop out of your compiler to offer hints on the fun they were having next door.

We laugh now at the pooheads who would telnet to their server, or open up their firewalls a port further than necessary. So why the fuck don't Dreamweaver et. al. have scp plugins? Why are we constantly having to open up an old, insecure protocol for the sake of poorly designed, overpriced software?

Ahem. As you were.

In other news, Knoppix 3.4 will not only boot from a USB CDROM without trouble, it will not hang on autodetecting partitions and writing them to /etc/fstab. Both these steps tripped up 3.3. Whee, what a mad merry-go-round my life is!

Also, here are some stats on kernel compilation times. In the one corner we have a 2.8GHz P4, 512KB cache, 800MHz frontside bus with 1GB of RAM and a 7200 RPM IDE hard drive. In the other corner, we have a EPIA-M MiniITX mobo with 1 GHz Via CPU, 64KB cache, 256MB of RAM, a FSB speed I can't be bothered to look up and a 4400 RPM IDE laptop drive. The time was for "make dep && make bzImage" on version 2.4.26 of the Linux kernel with a pretty random (by which I mean specific to our needs) configuration. Try to guess which is which:

real 1m51.998s user 1m45.920s sys 0m5.120s

real 6m7.849 user 5m24.530 sys 0m25.130

Just for fun, I tried swapping the drives around: the P4 got the laptop drive, and the MiniITX board got the full-on Kevin's mom. Results:

real 2m8.743s user 1m44.840s sys 0m6.160s

real 6m39.898s user 5m25.500s sys 0m25.940s

for the first time, and then:

real 1m54.601s user 1m45.330s sys 0m5.550s
real 5m54.717s user 5m26.410s sys 0m25.690s

after that. The fuck?

Also, have a look at this thread. I think I speak for all of us when I say that Linux will simply not be ready for the desktop until its scrollback buffer behaves like FreeBSD's. After all, the REAL measure of a man's worth is the size of his scrollback buffer. Yeah, baby!

Tags:

Notes_from_today

title: Notes from today date: 2004-08-19 19:39:46

1. DNS vs. access.db: It's a strong man's battle. In the end, though, having an unresolvable domain in the MAIL FROM: address means that Sendmail's DNS checks will trump anything you mighthave put in acces.db. There's always the accept_unresolvable_domains feature, but that's about as ugly a kludge as maintaining your own DNS entry for the domain in question.

2. Connectors connect: If there's a problem with your new iBook's wireless reception, make sure the antenna connector is firmly seated into the Airport card. Still unable to warbus, but I'm blaming my tinfoil hat.

3. You damn betcha I am, ratface: umlazi sounds pretty damned neat indeed. I'm told that User-Mode Linux is, currently, a hack that should be replaced by more machines, but I'm keeping this in my bookmarks file anyway.

And that's my bus stop, folks.

Tags:

NEWS FLASH

When you have:

1. a PHP-enabled Apache web server,
2. with a working MySQL connection,
3. already-working pages in PHP that can connect successfully to the database in question,
4. account details for MySQL, and
5. all the necessary privileges in MySQL and the server

you DO NOT need me to install phpMyAdmin in order to manipulate tables. Nor do you get bonus points for asking me how to connect to MySQL without phpMyAdmin. No, thank you.

Tags: rant mysql

Ld_library_path_vs._bcm4306

title: LDLIBRARYPATH vs. BCM4306 date: 2004-08-17 07:06:50

At the Pacific Slamatarium, SATURDAY! SATURDAY! SATURDAY!

I wrote earlier about a developer who found that ls, among other commands, would dump core when he went to a certain directory. What's more, it only worked for him, and only if he used tcsh -- if he switched to bash, everything was fine.

Well, I was a bit of an idiot for wondering if I should be compiling debug versions of ls. First clue was when he went to another directory nearby, ran ls and got this message: ls: error while loading shared libraries: libc.so.6: ELF file data encoding not little-endian What the...Then I realized that another significant thing about this was what was in the directories he was having problems with: different versions of GCC/glibc/Linux, cross- and native-compiled.

Okay, so somehow ld was looking in the current working directory for libraries to load (ack!). But why? I took a look at his environment and found:

LD_LIBRARY_PATH=:/home/foo/this:/home/foo/that:/usr/local/foo:/usr/local/bar [...]

Sure enough, take out that leading colon at the beginning and everything was fine.

I'm not sure right now if this would be a bug^wfeature of ld or the shell, but it was good to get to the bottom of it.

So the next thing to get working is wireless access. First of all, the Airport Extreme that we bought for the iBook will not do passive mode sniffing/tracking/blogging (still learning all this, so pls. correct errors in terminology); it uses a Broadcom chipset, and Broadcom is not interested in helping the folks at Kismac (thank you, Sam and anonymous stranger. Hm. And the Linksys WMP54GS won't work on my machine for two reasons:

1. It uses the BCM4306 chipset from Broadcom.
2. It needs a PCi2.2 motherboard, and I've got this old Abit BH6 which almost certainly isn't.

Back to the store with the PCI card, and the hunt will continue. I might get the WAP54 for the Linux-running coolness, but we'll have to see.

Tags:

Gloria!

My wife and I kinda made an impulse purchase on the weekend: a new 12" iBook G4. It was weird: I made a joke about buying a laptop. Then I explained that I was only joking, but if we were going to buy one it should be an iBook since I kept hearing how sweet they were. Then we were going to go to Stanley Park, hang out at the beach, but maybe go to London Drugs (I don't know about you Americans, but in Canada we go to the drugstore for everything...car insurance, furniture, computers, you name it. Oh, and occasionally prescriptions) to see what prices were like. Then we were buying one. It all happened so fast.

So far, it's pretty damned impressive. After all the trouble I had to go to get gphoto to work with our digital camera, my wife just plugged it in here and it worked with iPhoto right away. Not only that, but we were looking at a slideshow of the crack-induced photos we'd taken while Fur Elise played in the background. Fucking unreal, man.

It's weird: I do feel a bit like I've made a deal with the devil. I've come to agree more and more with RMS about Free-as-in-Freedom, and here I am with a closed-source OS. Yada-yada-Darwin, what about Aqua? But it's sooooo nice...well, mostly, anyway.

I'm trying to use MacStumbler at the moment to find a wireless network to hook up to, but no luck: it just sits there, looking like it's scanning but with no more feedback than a scrolling bar. Dammit, I thought W2K was the only culprit there...and dammit, if I can't blog from the steps of the Vancouver Art Gallery, this thing is going back to the store. I suspect a problem with MacStumbler, but it's hard to be sure; I managed to find five or six access points at the office with Knoppix and the work laptop, and (apparently) wasn't able to find a thing with MS. I need to find a command-line version.

So far, though, that's my only complaint. Pretty fucking sweet, if you ask me.

Had a problem at work with Debian and VNC: the alt keys wouldn't work, for some reason. This was pretty annoying for the developer who really, really wanted to use Emacs. It took me about an hour of poring through Google -- Jesus Christ, the number of complaints about ALT keys disappearing, and Good God the long uber-thread about the change in keyboard behaviour between Debian versions -- to find the solution: vncserver --compatiblekbd A-ha!

Back to work and still no wireless access. Carousel is a LIE!!!

UPDATE: The VNC trick doesn't work. Details: The developer is running VNCViewer under VNC to connect to an X desktop on a Debian machine. On that machine, he's opening up an xterm and running User-Mode Linux. Alt-equals-meta works for Emacs when run on the Debian machine, but not for Emacs when run in the User-Mode Linux xterm. Fuck. UPDATE: Buddy found the trick: shift-left-click in the xterm to get the menu, then click "Meta sends escape". Double fuck!

Tags: hardware emacs

Stay on target IV...

Getting closer to getting MySQL working. I came across this post today which seemed to be nearly identical to what was happening to me. I followed the suggestion and took out the --enable-static option I'd been putting into configure. Result: much happier, with hardly any crashing at all. Now if I can just get it to find the user.frm table, I'll be a happy monkey. All this to pick up a copy of libmysqlclient.so. I must be on crack.

Tags: mysql