thebulkclub.com

So a while back, Slashdot posted a story about TheBulkClub.com, an online forum for heathen cowfucking spammer scum ("Suppose you were a lying, sociopathic thief. And suppose you were a spammer. But I repeat myself." -- Mark Twain) that, sadly, left its membership list and other goodies exposed.

Being the good citizen that I am, I posted a reply that, I flatter myself, was both informative and helpful: it pointed the way to several mirrors of the information, including one on my own site. Well, what do I receive the other day but this charming email:

Date: Wed, 11 Aug 2004 10:23:03 -0700 (PDT) From: EmailSupplyNET <emailsupplynet@yahoo.com> Subject: Question about website To: aardvark@example.com Hey, I like (part) of your website, http://saintaardvarkthecarpeted.com It's informative. There was something on your site about "thebulkclub.com" Did you create that site for them or something? I run an email list site and am trying to contact them for advertising on their forums/boards... Any ideas/help? Thanks in advance, Thanks, www.EmailSupply.net EmailSupplyNet@Yahoo.Com 877.426.6636 --------------------------------- Do you Yahoo!? Yahoo! Mail is new and improved - Check it out!

It's quite the site. They offer a sample list -- 4MB of email addresses, meant to be a sample of the up to 14 million you can buy. I must warn you, it would be wrong to run this command:

while [ true ] ; do
wget http://www.emailsupply.net/sample.txt -O /dev/null
done

So don't do that. But my question is, what should I do? I'm open to ideas, suggestions, thoughts, plans and dicta.

Tags: spam

_that's__weird

title: That's weird date: 2004-08-13 20:44:23

So it was a busy day at work: I had to do some juggling with home directories on our file server for Windows people, and set up a new Linux server for people running User-Mode Linux.

Which, BTW, rocks...but be sure to read this link. I came across this problem today (freezing at "Initializing stdio console driver", but managed to get around it by installing a new version of uml-utlities. Admittedly, I'm only trying the 2.4 series. But that didn't mean I wasn't able to find a weird thing...

So most of our workstations run FreeBSD. Our main NFS server runs FreeBSD. But we've got a couple workstations running Redhat Linux, and this problem was on one of them. It was very weird: Every time he ran ls in a particular NFS-mounted directory, ls would segfault and dump core. It was just this particular directory. And after some investigation, it turned out to be dependent on being this particular user.

I tried going to that directory. I could run ls just fine. I was running bash, so I tried running /bin/csh (most of the developers here run csh...poor bastards)...everything worked. I tried getting him to run bash; if he ran it in the problematic directory it dumped core, and if I got him to cd somewhere else and then come back and run ls it dumped core. If I sued to root and then to him, it dumped core. I tried, as him, going into another directory, very nearby in the tree with the same number of characters in the path. It was fine.

I got desperate and got him to try rebooting his machine. It still dumped core. WTF?

I'm curious enough at this point that I'm seriously considering digging up the source and compiling a debug version, then running it under GDB. This is all far enough beyond my experience that it's ridiculous. Still, I have to know or it's gonna kill me.

Tags:

Dd_if=_dev_zero_of=_dev_brain

title: dd if=/dev/zero of=/dev/brain date: 2004-08-12 09:36:46

Am I the only one who has to put something like this in their crontab?

# 0 or 7 - Sunday
# 1 - Monday
# 2 - Tuesday
# 3 - Wednesday
# 4 - Thursday
# 5 - Friday
# 6 - Saturday

For some reason, trying to figure this out by hand just makes my brain hurt.

Tags:

The_johnstown_arp_flood

title: The Johnstown Arp Flood date: 2004-08-10 12:45:17

So last night, in the midst of other network problems, I notice lots of messages like this on our FreeBSD machines:

/kernel: arplookup 1.2.3.4 failed: host is not on local network

WTF? But as this was not the network problem I was looking for, I decided I did not need to see its identification and left it.

This morning, an SSH session to a test box became unresponsive after being idle for a few minutes; pings didn't work either. I ran over and checked that no one had disconnected it -- no one had -- and was able to ping my machine and others from it just fine.

As I walked back to my desk, someone came up to me and asked if the network was having problems right now, since he was no longer able to reach the network from his machine. I asked him to give it a try again, and went back to my desk to try getting to the test machine again. It was fine at first, then stopped responding again -- no SSH, no ping. I called the other guy and asked him if his computer was fine -- it was. I tested connectivity from my computer, but everything responded just fine except for the test box.

I walked over again, logged in and tried pinging my desktop machine. It took a good five or six seconds to respond, but then the responses were seen starting at packet 0. What the...I checked out other machines and saw that the arplookup message was turning up again. Time to check it out.

Well, first clue is that the address was almost one I'd assigned to a developer for his User-mode Linux sandbox: 3.4.1.2. I logged in and checked it out, but there was no indication it was using the address -- ifconfig turned up nothing, nor did tcpdump or the arp table. I checked the other machines' arp tables, but they had no entry either.

Then I remembered that this guy had been complaining about intermittent slow network access yesterday and today. At the time I figured it was related to the original problems I'd been checking out, but maybe they weren't. I decided to grit my teeth and talk to him.

First clue: the arp table on his W2K box (poor bastard) had an entry for 1.2.3.4. Aha! I tried running tcpdump on a laptop running FreeBSD hooked up to the same switch, but no luck. I was kind of hoping for a stray arp who-has or something, but I guess not.

Second clue: he mentioned in passing that the test box of his own by his desk, a single-board computer running Linux, was being used to test an ethernet driver that he was developing. He had the kernel set up to ping his User-mode Linux every five seconds. He also had it set up to watch for incoming traffic, swap bytes around, and then send it to his User-mode Linux. Aha!.

We agreed:

• that it was likely he was swapping the wrong bytes in the packets that went out
• that it was probably a bad thing for this to be happening
• in the interests of all concerned he should probably disconnect his test board from the LAN for now
• that he should modify the kernel to make it a bit less chatty

My suspicion at this point is that the bogus traffic was wreaking havoc with ARP tables, possibly those belonging to the (cheap, unmanaged, hopefully soon-to-be-upgraded-to-expensive-Cisco-Catalyst) switches that connect our whole network, or possibly just several Very Important Servers. I realize that it only caused problems with a few machines, not the whole network, but I'm wondering if this might be similar to what happened in February. Gotta say, though...the intense debugging activity almost fulfilled my deep-seated sysadmin fantasy of debugging raw Ethernet frames on the fly, so I'm happy.

Tags:

Incoming!

title: Incoming! date: 30 December 2003 12:00:00 PST

Holy crap. I wrote a few weeks ago about setting up spampot.py on my home machine. Since then I've had probes, but most were not relayed because the probes didn't match what spampot.py was looking for. (Interestingly, I looked at the traffic with tcpdump and found the probes starting conversations with "EHLO"; when spampot.py returned "Command unrecognized", they went away. But then they'd be back in ten minutes, regular as clockwork...)

But in the last few days...well, take a look at the number of connections:

Dec 13 | 2
Dec 14 | 2
Dec 16 | 3
Dec 17 | 3
Dec 18 | 2
Dec 19 | 2
Dec 20 | 2
Dec 21 | 1
Dec 22 | 1
Dec 23 | 1
Dec 24 | 14
Dec 25 | 1
Dec 26 | 4
Dec 27 | 2
Dec 28 | 396
Dec 29 | 2021
Dec 30 | 13671

Going through my mail logs, I can see that one message a day has been relayed (ie, has made spampot.py think it's a probe and has thus been allowed to be sent) since the 22nd. I guess that's what's convinced the spammers it's real.

I think the thing to to at this point is to compile tarpitting into the kernel and really slow 'em down...any thoughts?

Original entry.

Tags:

Windows 2000

A real OS shows messages when it's booting to let you know how far along it's gone. If something goes wrong, you can see where. A real OS doesn't have a fucking marquee of slowly-changing colour that you have to stare at intently from six inches away to see if it's frozen in place and yet another cold reboot needs to happen. Oh, and it logs how the boot went. Every time. Without being asked.

Also, if an OS comes with a package manager -- and a real OS does -- it shows you the fucking version of the software that's installed. A real OS knows that knowing ~FooTastic installed is only half the battle -- knowing that it's the buggy 6.2 build is just as important.

I try to give Windows and Microsoft a chance, I really do. But this is fucking ridiculous.

Original entry

Tags: windows rant packagemanagement

Random hints

growisofs will overwrite a DVD+RW when run non-interactively iff you use the option "-use-the-force-luke". Heh. Read the source for details.

To make a bootable FreeBSD CD, it's not enough to simply dd of=iso and burn that. Do this instead:

    mkisofs -R -b boot/cdboot -no-emul-boot -o foo.iso /tree/to/copy

This hints brought to you by the letter J, the letter O, and the number pi.

Original entry

Tags: bsd

Formmail tarpitting

I've been trying to come up with a way to tarpit formmail spammer probes/attacks, and I haven't had much luck yet. This is an outline of what I've done and what I plan on trying next. If anyone has any thoughts on this, please let me know. In particular, I'm looking for any approaches I've overlooked; I'm sure there's a lot.

Background: Matt's old version of Formmail, up to at least version 1.9, had serious terrible bad vulnerabilities that would let a spammer use it to send any email anywhere, no matter how much you tried to secure it.

At my last job (ISP helpdesk), I get complaints every now and then of spam coming from our mail server; it was almost always spammers using Formmail to do their dirty work. I'd have to track down websites where an old copy of Formmail was lurking, shut it down, and try to clear the mail queue of as much crap as possible. (This got a lot easier once I discovered [ngrep|http://www.packetfactory.net/projects/ngrep/] and had the root password ~SlashdotJournal_29August2003). Eventually I went and replaced all the copies w/the NMS version of Formmail, which did the trick wonderfully. I could drop it in to a website under attack and it would work right away: spam would stop, legitimate requests would still work.

I still get Formmail probes on my website all the time. A while back, I decided to send the spammers something more than just a [404|http://saintaardvarkthecarpeted.com/wheredidthispagego?] page. Using Apache's ~ScriptAliasMatch directive, any request with that matches "/cgi-bin/formmail" (case-insensitive) in the URL gets redirected to my copy of (ta-da!) Formmail Weasel.

Formmail Weasel is a boringly simple Perl script that parses the request made to it, logs everything to a database, and displays an innocuous "thanks for the submission" page (not that the robot ever read it). There's another script that displays the last ten requests in horrible tables. That's it so far.

Once, I got curious and sent off a fake reply to an address mentioned in one of the probes, making it look like a vulnerable Formmail script had been found. (Future plans for Formmail Weasel include the ability to send off these fake replies automatically, and x-ray vision.) Within a week, there were all kinds of attempts to send spam going on -- maybe one a minute or so. After a few weeks of this, the spammer figured out that it wasn't working, and stopped.

That was interesting, and moderately gratifying, but I wanted to cause pain. I want to imagine spammer wails of dismay. Tarpitting immediately leaped to mind. But I can't simply tarpit port 80 and be done with that: I'm still running Apache to serve a few websites, and I don't want to interfere with that. Besides, Formmail probes go by website, not IP addresses, so I need to have www.somethingorother resolve to my server in order to attract scans.

First I decided to try directing Formmail requests to a separate port. Using Apache's ~RedirectMatch directive and a separate ~VirtualHost thingy, I sent all requests for formmail to port 2348 (aka port random) where Formmail Weasel would be listening and Apache would be logging. For good measure, I set up tcpdump too.

My first hope was that the probe robots looking for Formmail scripts would follow the redirect, and I'd be able to capture the traffic on port 2348 w/tcpdump for analysis. ("Lookee here: spammers use SYN packets! Guess we know what to look for now, professor!")

My second hope was that I could provoke an attack by sending off a fake reply, and see whether the attack robots would follow the redirect. Maybe, if I was extraordinarily lucky, I could just tarpit port 2348 and be done with it.

I forgot about it for a week after sending off the decoy email. Today I checked the Apache logs and the tcpdump file: nothing on either one. But when I checked the main logs for my website, there had been half a dozen requests for formmail; the robots simply didn't follow the redirect. I made sure that the redirect was still working, then cried for a bit.

As I see it, this leaves me with a couple options that don't involve deep heavy network hacking:

• Leave Formmail Weasel the way it is: an essentially passive annoyance to spammers.
• Be a little more crafty.

From the requests I've seen, Formmail probes will look for a few common variations on the extension (.pl, .cgi) with some capitalization variations thrown in for good measure (formmail, Formmail, ~FormMail). This gives me a way of distinguishing an attack (an attempt to send spam) from a probe (seeing whether or not there's a script that can be exploited).

Formmail Weasel could designate one of these (let's say ~FormMail.cgi) as one that is the signal of an attack. Probes that came in for other variations would result in an email being sent off to the spammer, but with the attack address in it. In other words, any probes for FormMail.pl, formmail.cgi, or Formmail.cgi would result in an email back to the spammer indicating that ~FormMail.cgi was successful. At that point, the spammer (hopefully) takes the bait and begins the attack.

At this point we can use the [Linux iptables string matching kernel module|http://www.netfilter.org/documentation/pomlist/pom-extra.html#string] to look for packets that have the request in them, and tarpit them. You'd have to be specific about what exactly to look for: something like "GET ~or /cgi-bin/~FormMail.cgi", plus the host/site/whatever directive. But this is a small enough part of the request, and close enough to the beginning, that it should serve as a way of flagging that address as one that should be tarpitted.

Another option, and probably an easier and more effective one, would be to have Formmail Weasel set up separate iptables rules to tarpit the addresses that are part of the attack. You could age them and phase them out after a short/long while. One possible problem with this is that I've seen Formmail attacks that have come from many different IP addresses simultanteously; these usually end up being open proxies. You'd have to take care not to flood your firewall with tarpitting rules.

Original entry

Tags: spam

Hardware demons

Jesus Christ. Every time I mess around with hardware or upgrades, I swear I'll never do it again. Then I forget.

My first computer, bought eight years ago now, was a 486 w/16MB of RAM and some amount of HD space. I installed Slackware on it, got a 33.6 modem, and had email and net access. Then a roommate sold me his old P90. It crashed constantly until I figured out I had set the CPU voltage wrong. It took me a long time to figure that out, and I was nearly ready to hurl the thing out the window.

A few years later I upgraded to my current desktop machine, a 333 Celeron overclocked to 450 MHz. The machine is fine unless I open up the case to add/remove/shift something in it; then it will, for a day, spontaneously reboot. I've checked it for shorts and can't find any. I don't know what I'm missing, but I'm sure it would be obvious to someone else.

And now the latest. My wife bought an iMac from her old work a few years ago, and has had problems w/it since. It just crashes for no good reason. It'll work fine for two weeks, then she can't keep it running for more than an hour. So last week I went out and bought her a fairly skookum machine: Athlon 2600 (I think...details to follow), ECS K7S5A mobo, 60GB HD and 256 MB RAM.

I got it all home and assembled it. The mobo and Red Hat 9 (not my favourite, but great for my wife) called the CPU a 2000 (1.6GHz instead of 2.0), so I looked around and decided a BIOS upgrade would be in order. Did that and promptly lost the back USB -- bad, since her keyboard and mouse are USB. The front ones, hooked up to the pins on the motherboard, still worked. Tried rolling the BIOS back, but nothing: the back, onboard USB just didn't go. Fuck.

So I went out and got some additional USB risers a few days later. I added them; no problem. Then I had to add a connector from the CDROM's audio to the motherboard. I made the mistake of removing one of the USB connectors while the power was still on. Didn't even think; just did it. Now the BIOS freezes at "Checking NVRAM...". Flashed the CMOS half a dozen times, left it off most of the night while we went to see Finding Nemo (not as good as Monsters, Inc., but still well worth it), and no change.

Today I'm going to stop by my new hardware supplier of choice|http://www.ntcw.com/ to pick up a Gigabyte 7VAX. We'll see if I got ripped off on the CPU or what.

Mostly, though, I am not going to fuck with this computer again. I mean it this time.

Original entry

Tags: hardware upgrades linux

Slashdotting your P200 HOWTO

So I'm the Geek You Know for about six friends, and one of them needs a website. I volunteer my li'l server, no problem. But she wants something she can manage herself, and as she's not a geek that means something easy on the eyes and easy to use.

So I start looking at (can't use this phrase w/o gritting my teeth) content management systems, c/o Freshmeat and OpenSourceCMS.com (this site rocks). I've tried out two or three so far, and all have had the same results: my li'l server is dreadfully overworked.

It's a P200, 48MB of RAM, and it's fine at serving up static content: most of my site and my wife's site is just that, so it's not a problem most of the time. But start throwing some MySQL into the picture, and things slow down fast.

I'd settled, sort of, on Back-End as a likely contender; I liked its management pages, can't beat the street cred when CUPE uses it, there's an integrated gallery, and the installation went well. But when I tried it out...holy crap, it was slow: 10 seconds to throw up a page. Admittedly, better than the 30 seconds with some other packages, but still.

I figured it was time to move the database to the faster computer. I've got a Celeron, 450MHz, 384MB RAM, that I use for my desktop. Wasn't doing much besides 87 xterms and setiathome, so I figured out how to move it over there. Still slow. Well, decided to try some benchmarks. That's what separates us from the animals.

ab, against my own (static) site, shows 1000 requests being served, none dropped, concurrency 5, in 26 seconds. Against the Back-End demo site I set up, it timed out. I upped the timeout; same thing. In frustration I set concurrency to 500, set up iostat and top to watch on both the database and the web server (fancy!), and waited.

And waited.

After five minutes, the SSH session showing the stats on the P200 stopped. (Load on the Celeron and its disks barely registered, BTW.) I logged in via the KVM to see what was happening, and the answer is: not much. "eth0: card reports no resources", whole lotta processes being killed due to lack of memory, and fifteen minutes it is still chugging its way back to freedom. (Don't want to reboot and ruin that 60-day uptime...)

So: My questions to all of you are:

1. If it's not the CPU dedicated to the database that's the bottleneck, what is? Is it the PHP processing? Should I be moving the web server to the faster processor?
2. Anyone know a light -weight CMS system? Don't need a lot of bells and whistles; mostly this'll be a way of editing a mostly static site. Comments, polls, user journals...eh.

My thanks in advance for whatever help you can provide.

Original entry.

Tags: slashdot

/dev/console and the MIBs of Heaven

Interesting problem with The Inside Thing at work today. The Inside Thing runs headless -- no video card, no serial port (yet...sigh), connections only over IP. SSH to The Inside Thing has been no problem, and I never thought it would be.

One of the developers today loaded a debugging version of the FreeBSD kernel module he's working on, and found that it really slowed things down: a test script that would complete in a second, using the non-debugging module, would take a minute or more to run; in addition, the whole system would slow down to the point of near-unusability. WTF?

The debugging version does a lot of kernel printfs (I'm not a developer, so forgive me for any imprecision in language here). Logging is done to two places: /var/log/whatever, and over UDP to The Outside Thing, which has its syslog daemon listening to port grep syslog /etc/services. /var, on The Inside Thing, is just this big (32 MB) memory filesystem, so that shouldn't be a problem. And the network connection is gigabit ethernet, so that shouldn't be an issue.

I ran fstat while the program was running; it showed nothing unexpected: files open in /usr (where the program lives), the developer's home directory (NFS), /dev/insidething and /var/log/whatever. But run systat -vm, and hey, what's this: tons of interrupts on sio0.

This didn't work:

rm /dev/cuaa
mknod /dev/cuaa0 c 2 2

So on to less drastic measures.

I tried upping the serial port speed (we'd turned it on, but still haven't got a socket we can hook to yet) from 9600 to 115200 in /etc/ttys, and HUPping init; no change. (Incidentally, to get the serial port working on another FreeBSD machine over a null modem cable, I had to set it to 9600.3wire; strange. Or perhaps not.)

My boss came by at that point, and told me that the kernel printfs were not affected by stuff like getty and init; instead, there was a kernel option or possibly a sysctl that set that. Sure enough, look around and there's machdep.conspeed: 9600. Set it to 115200 and whee, look at things go! The debugging program ran in 30 seconds, which by this point seemed like a definite improvement.

I experimented a bit and found the highest machdep.conspeed could be set to something like 118900. Like before, this was better but by no means great. Then my boss came in again and announced a new sysctl MIB, greater than all the rest. This one was The Light, and the other one only came to announce The Light to the world:

kern.consmute

Set to 1, and all those kernel printfs still get logged to syslog, but never slow down The Inside Thing. I'm assuming that all this was trying to go out over the serial port after FreeBSD detected no video card...but I'm the first to admit that's probably a crack-addled dream.

Original entry.

Tags: slashdot

What's wrong with this code?

In the grand tradition of ryanr's journal, let's see who figures out what's wrong with this Bourne shell script. First prizeis a Cadillac Eldorado. Second prize is a set of steak knives. Third prize is Cowboy Neal fires you.

Background: yesterday at work, my home-grown backup system choked when it tried to burn a 740MB ISO to CD. (Still waiting for a tape drive.) I decided to finally implement the long-delayed exclusion of certain files (.core, .o, etc.). I know, I know, should've put it in from the start, but this was the first time it became an issue.

Anyhow, I was testing to see if my changes worked, and they didn't: the files were not being excluded. What was doubly weird was that I could run, on its own, the command the script was running, and it would work fine: everything that should have been excluded was. I finally boiled it down to this script:

    #!/bin/sh

    TAR_EXCLUDE="--exclude='*.core' --exclude='*.a'
    --exclude='*.o' --ignore-case --exclude='*cache*'"

    # This command works:

    /usr/bin/tar cvj --exclude='*.core' --exclude='*.a'
    --exclude='*.o' --ignore-case --exclude='*cache*' -f
    backup.tar /home/foo

    # And this command doesn't:

    /usr/bin/tar cvj $TAR_EXCLUDE -f backup.tar /home/foo

I tried putting echo behind each command, and I tried putting -x in the shebang; both showed the same output for both commands. (That make sense?)

What did I do wrong?

Original entry.

Tags: slashdot

Super NIS weirdness

So I'm working on The Thing today, and it's decided that the automount daemon needs to be set up on The Inside Things. (The Inside Things are in a separate network, with The Outside Thing acting as a gateway between them and the rest of our internal network. And so everyone knows, The Thing is running FreeBSD.) It's not going to be left like this when The Thing is deployed, but it's handy for right now.

Only The Inside Things are on a separate network -- 10.0.0/24, as opposed to 192.168.0/24 for the rest of our network -- so ypbind isn't working. I'm not too familiar with NIS/NFS, so this is taking me a while to figure out.

Eventually I decide that I need to enable NIS for amd to work, and to do NIS I need to bind to the right server. Well, in the man page for ypbind I see the -S option: bind to a particular server. Should work, right?

So I boot The Inside Thing, and do these commands:

domainname thing ypbind ypset -h localhost -d thing 192.168.0.1

At the same time I'm running tcpdump on The Outside Thing to watch what happens, because these commands aren't working. And I see the weirdest thing: packets going to another, completely foreign IP address, port 111: RPC.

I scratch my head, try again: same thing. Reboot The Inside Thing, try again: same thing. The Inside Thing is running nothing more than NFS and SSH, I'm the only one on it, and still it keeps going to this IP.

I look up the IP address and it belongs to the Washington State Department of Transportation. WTF?

Try it on The Outside Thing -- unnecessary, since it's running amd quite happily, but I want to see what happens. Same thing.

I check the source code for ypset on the off chance that Theo de Raadt (he wrote it) put in some kind of trojan to...I don't know, ask for his driveway in Seattle to be plowed. Nothing -- but then, my rule of thumb has always been "If you're looking at source code, you're in over your head." (True for me, and if the source code is written in anything other than Perl or Bash. Still learning.)

I have no idea what the hell was going on. Anyone?

Flash! Just tried it at home on my FreeBSD gateway: same results. Jesus.

Original entry.

Tags: slashdot

I found it. Do I get a prize?

So I was trying to set up a diskless boot system on FreeBSD at work last week for The Thing. I'd found this article on booting FreeBSD with PXE, and it was nearly all I needed...except that this was about installation, and what I needed was a working system.

So I started fooling with it and trying to figure out how to add things like individual swap files and configuration information -- The Thing is going to have up to ten or so computers booting disklessly -- and then I came across a passing reference to /etc/rc.diskless1 and /etc/rc.diskless2. Bless their pants, the good folks at FreeBSD had already come up with a way of doing all this, and pretty much all I had to do was read those two scripts and /usr/share/examples/diskless/clone_root.

I got the hang of it pretty quickly and (mostly -- I have a habit of not following each and every instruction every single time, which is why Automation Rox) got it working...sort of. Something was going wrong and I couldn't figure out what it was.

See, what happens is you create, on the server, /disklessroot, which has a copy of almost everything the diskless unit is meant to use for a filesystem. /var is a memory filesystem (MFS) populated on boot with canonical files (see /etc/mtree), and /tmp is a symlink to /var/tmp. /etc, in turn, is filled with files specified in /conf on the server: you can set a default, then add stuff specific to particular hosts specified by IP address. And if a file called disklessremount is around in the right place, /etc will be filled first with the base files you'd expect, then overlaid with the stuff in the default section, then the host-specific stuff.

Only that last part wasn't happening. Somehow it'd get to the part where it was meant to copy the base files, then all these errors would pop up. I was convinced I was doing something wrong, but I couldn't figure out what. So, single-user then sh -x /etc/rc.diskless1 2>&1 | less. (God, I love the -x flag for sh/bash. Lovelovelove.) I print out the scripts so I can follow along. I follow along.

And I found a bug!

At one point it checks to see if the diskless_remount file is around; if it is, it creates the MFS, then proceeds to fill it. It uses eval a couple times in its checks, which always throws me; I've never been clear on what eval is for. But I muddled through it, then realized there needed to be a second slash on one line. Without it the MFS creation subroutine doesn't get the proper arguments.

Okay, no big deal; it's not like I found a ticking time bomb in the centre of the earth and had to alert everyone within 24 hours. But I was happy I was able to recognise it. I was all prepared to send in a PR when I found this one. Welp, there goes fame and fortune. :-)

In other news, it looks like FreeBSD does not like USB keyboards; we've got we're using for The Thing, where the PS/2 socket is inaccessible, but FreeBSD immediately panics and dumps registers when it finds it or when we plug it in. I'm going to see about maybe getting a core dump from it using Michael Lucas' excellent instructions.

Original entry.

Tags: slashdot

SSHD on Windows 2000? Surely you jest!

So I'm at my new job (first job as a sysadmin...I'm so happy) and someone asks me if I can get an SSH server working on their machine. No big deal, right? I mean, everyone's running FreeBSD, right? Wrong: the computer this guy's talking about is running Windows 2000.

But I google, and fuck me if I don't come across instructions for running an OpenSSH server on Cygwin. Now don't get me wrong, I'd tried Cygwin before and had been very, very impressed w/the idea of running VI under W2K. But SSH? Cazart!

The instructions above were almost complete. But we've got a domain going (on Samba, natch), and the guy couldn't touch his files while logged in as Administrator. (I know, I know, shouldn't log in as Admin...but somehow it's hard to see that as being as bad as logging in as root...)

So some more google and came up with this. THe important bit is to change the GECOS field as suggested.

But: but but but: the other important bit is to start with a clean cygwin install. Follow the instructions here if need be. I banged my head against this long and painfully until I got it right.

On another note, I'm a Unix guy and I'm quite happy with that. But I've been thrust into W2K admin, and I find myself uncomforatably ignorant. One of the things I've always loved about Linux, FreeBSD et. al. is the sheer embarrassment of resources for the newbie. Where, o where are an equivalent humiliation of knowledge for Windows newbies? There has to be something out there for the technically proficient yet unfamiliar new guy. Bueller? Bueller?

Original entry.

Tags: slashdot

Transcript, 10/100 joy

So it looks like CNN has corrected the omission that I mentioned earlier. I guess it was just a mistake.

In other news, finally upgraded the link between Francisco (NFS server, FreeBSD) and Hardesty (NFS client, RedHat) to 100Mb/s cards. Aw yeah. Why did I leave it this long?

And I watched Logan's Run with my wife t'other night. To my surprise (she's not an SF fan) she liked it. And now I want to run around SkyTrain stations shouting, "Carousel is a LIE! You can LIVE!" Great stress reliever after a bad week at work.

Update March 21 2003: Found this Kuro5hin story on the transcript difference.

Original entry.

Tags: slashdot

Help desk call

Cust. calls in having problems connecting: email works, but he can't go to websites. Check out his settings, everything looks find. So I give him an address to try typing in: http://207.102.64.2. "Type that into the address bar of your web browser, hit enter, and tell me what happens." Naturally, he can't dial in while on the phone w/me, so he hangs up and calls back a few minutes later.

"Didn't work," he said. "It said 'action cancelled'."

"'Action cancelled'?" I said. "That sounds weird. Internet Explorer said that?"

"No, Outlook Express. I don't use Internet Explorer to go to websites."

"I'm sorry?"

"I use Outlook Express."

"To go to web pages?"

"Yes."

Problem solved.

Original entry.

Tags: slashdot

Help desk call

Message left on our voice mail:

"Hi, this is so-and-so. My email isn't working, hasn't been for about 12 hours or so, so something's obviously wrong at your end. I can still get in through your back door, but I don't want to do that. Give me a call, please."

Original message.

Tags: slashdot

Help desk call

I have someone on the phone with me right now complaining about the reception on their TV.

The gentleman went into great detail about the set (one year old Panasonic), what happens to the picture (solid horizontal line, thin; dancing white lines, also thin) and the variation over time (usually works for an hour in the evening, crappy other times). He's blaming it on the internet access built into his building (RJ45 ports in apartments, wireless access from the roof of one of the buildings), which we do helpdesk and provide connectivity for.

Original entry.

Tags: slashdot

Difference in transcript's of Blix' report to the UN

cf: http://www.cnn.com/2003/US/02/14/sprj.irq.un.transcript.1/index.html and

http://www.heraldsun.news.com.au/common/story_page/0,5478,5987651%255E663,00.html"

Notice the absence of this chunk of Blix' report in the CNN transcript:

I trust that the Iraqi side will put together a similar list of names of persons who participated in the unilateral destruction of other proscribed items, notably in the biological field.

The Iraqi side also informed us that the commission, which had been appointed in the wake of our finding 12 empty chemical weapons warheads, had had its mandate expanded to look for any still existing proscribed items. This was welcomed.

A second commission, we learnt, has now been appointed with the task of searching all over Iraq for more documents relevant to the elimination of proscribed items and programmes.

It is headed by the former Minister of Oil, General Amer Rashid, and is to have very extensive powers of search in industry, administration and even private houses.

The two commissions could be useful tools to come up with proscribed items to be destroyed and with new documentary evidence. They evidently need to work fast and effectively to convince us, and the world, that this is a serious effort.

The matter of private interviews was discussed at length during our meeting. The Iraqi side confirmed the commitment, which it made to us on 20 January, to encourage persons asked to accept such interviews, whether in or out of Iraq.

So far, we have only had interviews in Baghdad. A number of persons have declined to be interviewed, unless they were allowed to have an official present or were allowed to tape the interview.

Three persons that had previously refused interviews on UNMOVIC's terms, subsequently accepted such interviews just prior to our talks in Baghdad on 8 and 9 February. These interviews proved informative. No further interviews have since been accepted on our terms. I hope this will change. We feel that interviews conducted without any third party present and without tape recording would provide the greatest credibility.

At the recent meeting in Baghdad, as on several earlier occasions, my colleague Dr. ElBaradei and I have urged the Iraqi side to enact legislation implementing the UN prohibitions regarding weapons of mass destruction. In a letter just received two days ago, we were informed that this process was progressing well and this morning we had a message that legislation has now been adopted by the Iraqi National Assembly in an extraordinary session. This is a positive step.

Mr President, I should like to make some comments on the role of intelligence in connection with inspections in Iraq.

A credible inspection regime requires that Iraq provide full cooperation on "process" -- granting immediate access everywhere to inspectors -- and on substance, providing full declarations supported by relevant information and material.

However, with the closed society in Iraq of today and the history of inspections there, other sources of information, such as defectors and government intelligence agencies are required to aid the inspection process.

I remember how, in 1991, several inspections in Iraq, which were based on information received from a Government, helped to disclose important parts of the nuclear weapons programme.

It was realized that an international organization authorized to perform inspections anywhere on the ground could make good use of information obtained from governments with eyes in the sky, ears in the ether, access to defectors, and both eyes and ears on the market for weapons-related material. It was understood that the information residing in the intelligence services of governments could come to very active use in the international effort to prevent proliferation of weapons of mass destruction.

This remains true and we have by now a good deal of experience in the matter. International organizations need to analyse such information critically and especially benefit when it comes from more than one source.

The intelligence agencies, for their part, must protect their sources and methods.

Those who provide such information must know that it will be kept in strict confidence and be known to very few people. UNMOVIC has achieved good working relations with intelligence agencies and the amount of information provided has been gradually increasing. However, we must recognize that there are limitations and that misinterpretations can occur. Intelligence information has been useful for UNMOVIC.

In one case, it led us to a private home where documents mainly relating to laser enrichment of uranium were found. In other cases, intelligence has led to sites where no proscribed items were found. Even in such cases, however, inspection of these sites were useful in proving the absence of such items and in some cases the presence of other items -- conventional munitions. It showed that conventional arms are being moved around the country and that movements are not necessarily related to weapons of mass destruction.

The presentation of intelligence information by the US Secretary of State suggested that Iraq had prepared for inspections by cleaning up sites and removing evidence of proscribed weapons programmes. I would like to comment only on one case, which we are familiar with, namely, the trucks identified by analysts as being for chemical decontamination at a munitions depot. This was a declared site, and it was certainly one of the sites Iraq would have expected us to inspect.

This is a pretty big chunk. Without it, the dry humour of "Our reservation on this point does not detract from the appreciation of the briefing." just makes no sense -- there's no context to it.

I checked CNN's HTML, and it's not in there; there's no typo keeping it from view. Conspiracy theories are for the weak. Anyone got any other explanations?

Original entry.

Tags: slashdot