18 Dec 2005
Okay, so I've finally got the onboard flash chip detected on this
thing. It took a few things to get this working:
- I had to apply this patch to the MTD code, which apparently never made it to their main tree.
- I had to twiddle the GPIO on this chip just so, which I figured out by disassembling the stock bootloader that comes with this thing.
- I had to take out a bit of code that I'd put in 'cos I thought I was smarter than the kernel folks. That took a while to recognize.
Still can't actually erase the flash, even with the GPIO twiddle,
without causing a kernel panic. And the code that does ttwiddle itself
is ugly, and pretty much stuck in a random place. But progress!
Tags:
nwr04b
06 Dec 2005
A DHCP server at work is listening on multiple interfaces, some of
which have multiple aliases, and so the reply appears to come from one
of those aliases -- not the "main" one that I want. The solution is to
use the option server-identifier.
Tags:
toptip
06 Dec 2005
These guys have reverse-engineered the Broadcom 43xx wireless
chipset and are writing a GPL'd replacement driver. Broadcom are the
kind folks who won't release binary-only drivers, let alone
datasheets, for their stuff, which means the only thing you can do is
run NDISWrapper (Linux) or Project Evil (FreeBSD).
My wife and I share an Apple iBook with an Airport Extreme card
that, natch, won't do passive mode. However, the
reverse-engineered specs are available, which should help this
guy in his efforts to write a new driver. I think I'll drop him
(and the Kismac folks) a line and point it out.
Tags:
02 Dec 2005
Welp, I've got my module working...at least, in the sense that it
makes Baby Linus cry. It's pretty ugly, but it loads and unloads and
PEEKs and POKEs memory the way I want it to, which is all I
need. (Used to have a VIC-20 when I was a kid, where you'd have
to POKE different bits of memory to make it play a tune. Never thought
I'd be duplicating that 20 -- no, closer to 30 -- years later.)
Of course, I'm still having no luck at all actually probing for
flash. What I'm doing should match up with both the datasheets and
what the firmware does, but it's just not working; instead of getting
0xAD back (the manufacturer code) I just see the bit of memory
that's actually there.
I've come up with a few reasons this might be happening:
- The flash datasheet lies. Unlikely, since the firmware seems to follow the same sequence I'm doing.
- The CPU datasheet lies about where the flash memory is. Possible, but it doesn't strike me as all that likely either; what I see at that location (
0x2000.0000 for those of you playing the home game) is firmware that does exactly what you'd expect booting firmware to do: set up memory, clear some registers, print a menu, and generally get the thing ready to go.
- The CPU datasheet lies about what's needed to program the flash. Possible. I know there's at least one write-protect bit; unless I turn it off, any attempt to write to flash results in a protection fault. It's possible there's more.
- Timing problems. Sounds reasonable.
- The process is different once you've booted. Okay, maybe. I haven't looked at the original firmware from the manufacturer (the stuff that comes after the bootloader, I mean); it's possible there's a different sequence that needs to happen.
- I'm not doing what I think I'm doing. I'm thinking about endianness here, which always confuses me. Dunno about this. I've tried disassembling my module, and though I don't understand a lot of what's going on, it still looks pretty familiar and pretty much what I'd expect.
- I've missed something. Bingo, because I definitely don't know what I'm doing.
For example: the flash datasheet says that one of the locations you
POKE to is 0x555 -- yet both the firmware and the AMD-compatible
flash driver in the kernel POKE to 0x554. Why? Obviously it's a
4-aligned (word-aligned?) address, but why does that still work? (For
the firmware, I mean, which can obviously still write to flash.)
Or there's the whole question of bus or map width, which is important
to the Linux drivers. Take this bit, again from the AMD-compatible
driver:
static inline void send_unlock(struct map_info *map, unsigned long base)
{
wide_write(map, (CMD_UNLOCK_DATA_1 < < 16) | CMD_UNLOCK_DATA_1,
base + (map->buswidth * ADDR_UNLOCK_1));
wide_write(map, (CMD_UNLOCK_DATA_2 < < 16) | CMD_UNLOCK_DATA_2,
base + (map->buswidth * ADDR_UNLOCK_2));
}
Dead simple routine -- but why is ADDR_UNLOCK being multiplied by
the buswidth? According to the (suspect) driver in Codeman's kernel
for the CX84200 flash, the buswidth was 2 -- which means that instead
of writing to (say) 0x554, it would write to 0xAA8. Wha'?
I may have to break down and set up a JTAG interface on this
thing. I've been avoiding it 'til now out of stubbornness and lack of
soldering skill, but it may be the only way to figure out what the
hell is going on.
Tags:
nwr04b
02 Dec 2005
title: Wow III
date: 2005-12-02 07:33:43
These guys are writing their own GPL'd firmware for Prism54-based
wireless cards. The crazy part is that these things have embedded ARM9
cores running at 30MHz...so not only does some of their reverse
engineering look familiar to me (except that, you know, they're
talented :-) but you have the prospect of a wireless USB card running
Linux. Crazy.
(I know, there's probably a lot of good reasons why you couldn't
actually run Linux on the thing. But still!)
Tags:
30 Nov 2005
...when the passing of a fire truck in the middle of the night means
you obsess for half an hour about getting backup tapes out of your
apartment.
Tags:
backups
29 Nov 2005
Over the last few weeks, I've been slowly picking away at the original
firmware for the NWR04B. This is the stuff that lets me upload Linux
to the router, and then run it (as opposed to the various firmwares
from different companies that do useful stuff like firewalling, web
control panels and so on). This is the second time I've taken on this
beast, and like the first time it's because I can't get the damned
thing to write to flash from Linux.
Writing to flash should be simple. I've got a datasheet for the chip
that's on the router, and it lays out pretty explicitly exactly what
you need to do to erase a sector, program a random location, find out
what model chip you have, and so on. I keep watching what the kernel
does by throwing printfs everywhere, but I still get no response from
the flash. So I decided to see how the firmware does it.
The first time I looked at it I was just confused. There were at
least half a dozen places in the firmware where I could find the
magic numbers used to send the flash into command mode, where you
could query or program it. The two numbers were right next to each
other, and it was obviously no coincidence. But trying to read what
the damned thing was doing just made my head hurt. All these
registers, changing all the time, and no hint of what they were when
you got here....
I tried, once, following from the very beginning, and keeping track of
what register held what value. That didn't last very long. Then I
thought of whipping up a quick Perl script to do the same...you know,
writing an ARM simulator. That didn't last very long either (though
for the sort of very basic functionality I was after, it might not
have been that hard).
After that, I gave up and upgraded the kernel. At the very least, I
figured the new MTD framework would probably make the problem pretty
generic; if I was very lucky, it might even have drivers already in
place. Instead, though, I kept having the same problem: the numbers
looked good, but it just didn't respond the way I thought it
should. The framework was there to query the thing in three different
ways, but none of them did what I wanted.
So...back to the firmware, and this time I'm beginning to understand
it a little better. For example: there's a big series of tables at the
end of the program. I only noticed 'em this time, but they're pretty
fundamental to what I'm after. There's a list of sectors you can
erase, and then addresses for routines to erase a sector, fill a
region with bytes, program it, and check some details on the chip
itself.
And the half-dozen places where you can find the command numbers are
there because there are half a dozen erase-a-sector routines -- which,
in turn, is because this firmware supports six chips from four
manufacturers: SST, AMD, Atmel and Hynix. It's the Hynix chip that
I've got, so that means I can focus on that.
Since I can see the addresses of all these routines, I'm confident now
that I can pick out where a subroutine starts and ends, and how to
pick out the registers used to carry arguments. With that down, I can
pick out other routines and see what calls them, and I can pay less
attention to keeping track of all the registers at all times.
Now I need to look at the driver in the kernel, and compare it to what
I can see in the original firmware. The firmware's check for the chip
I've got seems to match what the kernel does, so at least that part's
good.
Part of the problem is having to recompile and reload the kernel each
time I want to check the driver, and the fact that the probing is not
under my control; I'm still trying to wrap my head around the driver
initialization sequence in the kernel. I'm thinking of writing a toy
kernel module to do the writes I want, since it looks like the MTD
drivers can't be compiled as modules on my own. This would save me
having to reboot with a new image all the time.
Ah, well...at least this is all fun. I'm still having the time of my
life here. :-)
Tags:
nwr04b
25 Nov 2005
Okay, number one: I love Sysinternals. Not enough that the guy takes
on Sony...no, he's gotta make Windows utilities that MS should've made
and releases 'em gratis. Not only that, they're text-friendly: Regmon
saves log files in fucking text, the way God intended. Sweet.
Number two: Weird-ass problem with Office 2003 on this one
computer. Check out the result of a simple browse through the user's
home directory on a Samba server:
104 HKCR\CLSID\{0AFACED1-E828-11D1-9187-B532F1E9575D}\ShellFolder\CallForAttributes
104 HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\NonEnum\{0AFACED1-E828-11D1-9187-B532F1E9575D}
158 HKCR\CLSID\{0AFACED1-E828-11D1-9187-B532F1E9575D}\InProcServer32\(Default)
158 HKCR\CLSID\{0AFACED1-E828-11D1-9187-B532F1E9575D}\InProcServer32\LoadWithoutCOM
433 HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\User
463 HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints\D\Version
3952 HKLM\Software\Policies\Microsoft\System\DNSclient\PrimaryDnsSuffix
3952 HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Domain
3952 HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Hostname
3956 HKLM\System\CurrentControlSet\Control\ComputerName\ActiveComputerName\ComputerName
Four thousand checksk of PrimaryDnsSuffix, Domain, Hostname and ComputerName. WTF is going on?
Tags:
windows
21 Nov 2005
For the second time in two weeks, an executable (!) Windows virus
inside a .zip file has made it past ClamAV because signatures had not
yet made it to the database. (What I mean is, the file extension alone
was enough to show it was an executable.) The first time it happened,
I came up with a very quick and dirty patch for ClamAV; now I'm
applying it.
I think it should work...all it does is use ClamAV's own code for
detecting a Windows executable and return a bit early when scanning a
file. I wanted to use ClamAV rather than MIMEDefang because Clam has
code built-in for safely scanning ZIP files, and I wanted to avoid
having to code something like that on my own in MDF. But if someone
out there knows a good way of doing this, please let me know; Google
didn't turn up anything I was happy with.
I've not yet submitted this to the ClamAV mailing list, but I'm sure
that there are many good reasons it should be laughed out of town. If
this is useful to you, great, but I promise nothing. I do NOT
recommend applying it. It will probably break everything, refuse to
run, and end up hiding WMD in your bedroom. It's your responsibility
to make sure it works for you. But hey, if you like playing with
patches to important software by people who don't know what they're
doing, enjoy! (Just to be explicit: released under the GPL.)
(BTW, the patch is suitable for dropping into FreeBSD's ports at
/usr/ports/security/clamav/files. At least, it works for me...)
UPDATE: The patch now covers clamd, not just clamscan, and also
updates the man pages. Whee!
Tags:
windows
20 Nov 2005
While setting up Gentoo as a domU in Xen, I ran into this error while booting:
* Mounting /dev for udev ... [ ok ]
* Configuring system to use udev ...
* Populating /dev with device nodes ...[ oops ]
* The "tar" command failed with error:
rd/c5d22p2: Cannot mknod: No space left on device
tar: rd/c5d22p3: Cannot mknod: No space left on device
tar: rd/c5d22p4: Cannot mknod: No space left on device
tar: rd/c5d22p5: Cannot mknod: No space left on device
tar: rd/c5d22p6: Cannot mknod: No space left on device
tar: rd/c5d22p7: Cannot mknod: No space left on device
...
Gentoo was given 32MB of memory by Xen (not very much, but I'm just
fooling around right now). The fix is to put these two entries into
/etc/conf.d/rc:
RC_DEVICE_TARBALL="yes"
RC_DEVICES=static
Gotta say, I'm pretty impressed with Xen so far. I managed to get it
set up within QEMU as described here. The speed was pretty
intolerable on my desktop (500MHz P3) but is quite decent on the
otherwise-mostly-idle 2.4GHz P4. I'm currently emerging Apache 2 and
wondering about whether I should give SELinux a try. Fun fun fun...
Tags:
15 Nov 2005
Xen will not work on an AMD K6. Sigh. I've got a broken-down
PII, another broken-down Celeron II, and my desktop (currently
doubling as my firewall). The PII doesn't work, I don't hold out much
hope for the Celeron, and I really don't want to muck about with my
desktop machine right now. Hm.
I'm hoping to remake my webserver using Xen. Alioth has given me
some great info, and I'm ready to give it a try. But I need
something to practice on first...either that or I practice on the
current webserver by running Xen with its current kernel as dom0,
set up a XenU instance, get that right, migrate everything there, then
lock down dom0. Which leaves me with the old, unmaintainable instance
in dom0. Ack. Maybe I'll give QEMU+Xen a try.
Tags:
xen
15 Nov 2005
title: I'm number one!
date: 2005-11-15 10:34:43
At least, when you Google™ for dd if=/dev/zero. Ha!
Also, check out mrtg-select, just released under the
GPL. Moderately helpful software that lets you display a subset of
MRTG graphs based on keyword and timespan.
Tags:
14 Nov 2005
Would you like me to pick a character for you? [YES/NO]
YES
You are the sysadmin. A user comes to you and asks for a package of
Matplotlib for python 2.2.
SEARCH GOOGLE
Google says that packages are available for Debian at the Matplotlib
homepage, in the packages section.
GO MATPLOTLIB PACKAGES
You see packages for Python 2.3 and a link to the mailing lists.
GO MATPLOTLIB HOMEPAGE
You see source distributions for Matplotlib, including the
latest version.
DOWNLOAD MATPLOTLIB
The Matplotlib tarball is in your home directory now.
UNTAR MATPLOTLIB
Okay.
COMPILE MATPLOTLIB
Things proceed well until you see the following
error:
``
building 'matplotlib._nc_cntr' extension
gcc -DNDEBUG -g -O3 -fno-strict-aliasing -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -I/usr/include -I/usr/include/python2.2 -c src/_nc_cntr.c -o build/temp.linux-i686-2.2/_nc_cntr.o -DNUMERIC=1
src/_nc_cntr.c:1718: error: syntax error before "init_nc_cntr"
src/_nc_cntr.c:1719: warning: return type defaults toint'
src/nccntr.c: In function init_nc_cntr':
src/_nc_cntr.c:1723: warning:return' with no value, in function returning non-void
src/nccntr.c:1729: warning: `return' with no value, in function returning non-void
error: command 'gcc' failed with exit status 1
VI SRC/_NC_CNTR.C
There is no such file.
VI src/_nc_cntr.c
There is no such file.
COMPILE MATPLOTLIB
Things proceed well un
CTRL-Z
The process has been suspended.
VI SRC/_NC_CNTR.C
There is nothing obviously wrong with this file.
GO MATPLOTLIB HOMEPAGE
You see packages for Python 2.3 and a link to the mailing lists.
GO MATPLOTLIB MAILINGLISTS
They're hosted on Sourceforge. You see lists for users and
developers.
GO LIST USERS
Sourceforge times out.
GO LIST DEVEL
Sourceforge times out.
GO LIST DEVEL
You see the search page, but without any stylesheet.
SEARCH _NC_CNTR.C
Sourceforge times out.
GO GMANE
You see the following message: "Welcome to GMANE, the NNTP-friendly
mailing list archive! You can search all your favourite mailing lists
here."
SEARCH _NC_CNTR.C
You see the following search results:
1. RH 3.2/Python 2.2 build problems (gmane.comp.python.matplotlib.general)
2. Re: (Matplotlib-users) Installation issue (gmane.comp.python.matplotlib.general)
3. Numarray problems on AIX (gmane.comp.python.numeric.general)
GO 1
You see the following post to gmane.comp.python.matplotlib.general:
I'm having problems building matplotlib on Redhat 3.2 with python
2.2. It complains about src/_nc_cntr.c. Does anyone know what's
happening? You notice a cute flag indicating the country of origin of
the email.
SEARCH REPLY
You can't do that on Gmane. You notice a cute flag indicating the
country of origin of the email.
SEARCH THREAD NEXT
You can't do that on Gmane. You notice a cute flag indicating the
country of origin of the email.
SEARCH THREAD LIST
You can't do that on Gmane. You notice a cute flag indicating the
country of origin of the email.
SEARCH SUBJECT LIST
You can't do that on Gmane. You notice a cute flag indicating the
country of origin of the email.
EDIT URL
You open the URL in vi.
EX:s/articles.gmane.org/blog.gmane.org/wq
You get the latest posts in gmane.comp.python.matplotlib.general and
lose the search result you had.
GO BACK
Okay.
EDIT URL
You open the URL in vi.
EX:s/articles.gmane.org/comments.gmane.org/wq
You see the same post with a different stylesheet. You notice a cute
flag indicating the country of origin of the email.
SEARCH REPLY
Ah, there it is! You see the following reply:
> My guess is that this is because cntr.c is using a python2.3 only
> macro, and we haven't sufficiently tested with python2.2. Try
> replacing all occurrences of MODINIT_FUNC with extern "C"
> DL_EXPORT(void).
You notice a cute flag indicating the country of origin of the email.
SEARCH REPLY
There doesn't seem to be one. The user comes back and asks if the package is done yet.
Tags:
rant
13 Nov 2005
It's been a couple weeks now since I upgraded to the 2.4.31 kernel,
and I'm still trying to get write access to the flash memory from
Linux. This is turning out to be a real pain. The whole point of
upgrading kernels was so that I could use the up-to-date version of
the MTD drivers, backported by the uClinux folks. This has helped, but
I'm not there yet.
The MTD drivers attempt to probe the chip to see what it is, who made
it and what it can do. To do so, it writes a few values to special
locations, then reads back from another special location. There are a
couple standards for this sort of thing (CFI, JEDEC), and the
datasheet I've got for this flash chips says it supports those.
It also turns out that this is what the MTD drivers call an
AMD-compatible flash device -- the commands to unlock a sector, say,
or to spit out a device number, match those from AMD for some of their
flash chips. So that's at least three different sorts of drivers to
use, and three different ways of saying "Are you this kind of device?"
According to the datasheet I've got, all of this should work: the CFI
probes, the JEDEC probes, the AMD stuff. The CPU datasheet says the
flash is mapped to 0x2000.0000 after boot, and all the debugging
whatnot I've thrown into the drivers say that's where they're writing
to. Yet all I get back is raw memory. It matches what I read from the
flash memory under the old kernel, and it matches what you'd expect
from the bootloader on this thing -- set up some registers, reset
devices, print the menu then jump to a loaded flash image.
I'm unsure what's going on here. The location of flash doesn't seem to
be wrong. If the flash chip datasheet is wrong, I've got some fairly
big problems, I think. But I can't figure out why I can't get the
answers I'm expecting.
Tags:
nwr04b
10 Nov 2005
Like unto sand in the cornhole:
"Solution". Last seen in a SuSE install disk in place of the word "program" or "application". I swear to God, the next person who uses this word at me in this fashion will have their souls fried in a delicate cornstarch batter.
Autoyast. 'Cos, you know, why in God's name would you take a hint from Kickstart and create a config file to match the installation that just finished? You know, so you can install by hand once and then use that as a template for everything that follows?
Autoyast, part two. Some sort of checking would be nice...like, say, "Ha ha, you thought we would generate all the Grub entries that weren't specified? The ones that are necessary to boot? SUCKER!"
Debian. Why the fuck does it keep ignoring my MIME options and attempting to open, automatically, attachments of all sorts from Mutt? If I want this sort of stupid, insecure behaviour, I'll go back to Outlook Express.
Debian, part two. And why are there eleventeen goddamned user accounts, very obviously meant for system maintenance, with valid goddamned shells?
Tags:
rant
06 Nov 2005
Took a while, but I managed to get uClinux version 2.4.31 compiled and
working on the router. I may release another firmware package, or I
may wait until the mtd stuff is working. Looks like the newer drivers
may handle the flash chip without needing a special driver... Here's
where I am now:
Linux version 2.4.31-uc0 (aardvark@rearden) (gcc version 2.95.3 20010315 (release)) #13 Sun Nov 6 13:10:29 PST 2005
Processor: Conexant CX84200 revision 1
Architecture: cx84200
Reserving page zero for vector table
hm, page 00000000 reserved twice.
hm, page 00001000 reserved twice.
hm, page 00002000 reserved twice.
hm, page 00003000 reserved twice.
hm, page 00004000 reserved twice.
hm, page 00005000 reserved twice.
hm, page 00006000 reserved twice.
hm, page 00007000 reserved twice.
On node 0 totalpages: 2048
zone(0): 0 pages.
zone(1): 2048 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/nfs nfsroot=192.168.23.254:/home/aardvark/nwr04b/nfsroot ip=192.168.23.12:192.168.23.254:::testf
Calibrating delay loop... 6.32 BogoMIPS
Memory: 8MB = 8MB total
Memory: 6480KB available (1108K code, 193K data, 52K init)
Dentry cache hash table entries: 1024 (order: 1, 8192 bytes)
Inode cache hash table entries: 512 (order: 0, 4096 bytes)
Mount cache hash table entries: 512 (order: 0, 4096 bytes)
Buffer cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 2048 (order: 1, 8192 bytes)
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
JFFS2 version 2.1. (C) 2001 Red Hat, Inc., designed by Axis Communications AB.
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
eth0: a0:98:76:54:32:10
physmap flash device: 8000000 at 20000000
FIXME: Made it here: 140 in physmap.c
FIXME: Made it here: 55 in cfi_probe.c
Unhandled fault: vector exception (0) at 0x1
fault-common.c(97): start_code=0x1198, start_stack=0xe3a02322)
Internal error: Oops: 0
I'm tracking down the source of that error, which happens when
physmap_write16 is called from cfi_probe_chip. I'm wondering right
now if it's because I said in the config file that the bus width is 16
bits; based on some earlier notes, I think it might be 32
bits. We'll have to see.
Tags:
nwr04b
04 Nov 2005
title: DomainKeys and SpamAssassin
date: 2005-11-04 06:19:28
Got this error when testing SpamAssassin 3.1.0 on a FreeBSD machine:
spamd[96933]: rules: failed to run DK_POLICY_SIGNALL test, skipping:
spamd[96933]: _(Can't locate object method "header" via package "Mail::DomainKeys::Message" at /usr/local/lib/perl5/site_perl/5.8.7/Mail /SpamAssassin/Plugin/DomainKeys.pm line 213. )
Thanks to this post, I found this patch from this bug,
and things seem to be working now.
Tags:
02 Nov 2005
Upgrading SpamAssassin at work; we're using 2.63, and they're up to,
what, 3.1.0 now? The upgrade itself was relatively painless, but for
complicated reasons it was integrated with Mimedefang, and I didn't
like that. MDF is great, but:
- It takes out the SA score header. This can be corrected, but
- it turns the SA score into a number, rather than a series of asterisks, which makes it difficult to filter with a regex, or with Outlook. (I have SA set conservatively, but the header makes it easy to filter more aggressively if that's what you want.
- Finally, MDF puts the SA report into the message as an attachment. Admittedly, it's a plain-text attachment, but that doesn't console the Outlook users who are worried (and rightly so) about clicking on attachments.
Hm. Will have to figure out a way around that; maybe just run
spamc/spamd like I currently do.
I've also got word that, due to some old prototype equipment no longer
being needed, I will have three new boxes to play with. Woohoo! I'm
already planning the DRBD fileserver.
Finally, I managed to get the new version of uClinux to compile and
run on the NWR04B. Sweet...except that I didn't check out a particular
tag, and I'm having to guess at the date when I did check out my tree,
which makes it difficult to say exactly what I've got. Currently
checking out with the date set to when I think I grabbed it, then may
upgrade/downgrade to the latest tag (currently 2.4.31).
Tags:
spam
nwr04b
02 Nov 2005
First, Netdisco just fucking ROX. A while back I was looking for
a way of telling what machines were hooked up to the ProCurve switches
scattered around our network. I'd written a half-assed program to do
just that and was glumly contemplating how much work it'd be to do it
right when I came across Netdisco. It's a pain to install, but good
Gopod, it works wonders.
However: Sourceforge irritates me beyond reason. Its mailing list
searchability is terrible, its thread display is worse, and it keeps
timing out when I try to open a bunch of mailing list links (like, 10)
in different tabs. ARGHHH.
Tags:
rant