Tag: slashdot
So Monday afternoon the new mail server started to slow down. I'd been
getting worried over the weekend about the fact that the MRTG graphs
were showing big timeouts, and was thinking that if I had to do it
over again I'd not use vinum so much: the disk access was just really
slow. (True story: during the install I'd noticed this, and tried
untarring a file (600kb or so). It took 17 seconds, nearly all disk
access. Tried it on my box at home for comparison (450MHz Celeron,
random cheap disk drives) and it took 6 seconds. Hrm...) On Monday it
got to the point where it was refusing connections...arghh. So I
swapped in the old co-front-end server (no filtering) that I'd taken
off to let the new one handle the load, stopped Sendmail, tried not to
panic and thought about what to do. Talked it over w/co-workers/people
who know better, and came to the conclusion that the box really did
need to be rebuilt.
(ObDisclaimer: I really, really wish I was certain I know what I'm
talking about here.)
Vinum was set up like this: three separate partions on each of four
drives made up three separate vinum devices (one for /tmp, one for
/var, and one for /usr). I've been told since that you can split up
one big vinum device into diff. partitions/slices, so that was a
waste. As well, the four drives were all on two controllers (did I
mention they're IDE?). Plus, in the vinum conf. file I just put
everything in order: /dev/ad0s1, /dev/ad1s1, /dev/ad2s1,
/dev/ad3s1. Again, I've been told since that that means vinum will end
up writing to them in that order, which slows down writes even more.
Decided that we didn't really need to worry about keeping the spam we
caught in the event of a disk failure, but the mail queue would be a
nice thing to have (even though, at any given moment, it's 99%
mutli-thousand returns to spammer mail servers that aren't accepting
any mail...gotta do something about that). So I set up /usr, / and
/var to live on the onboard Promise mirror array; 40GB total. One
drive each went on each of the two vanilla IDE controllers left; one
is for /var/procmail, and one for /var/procmail/spamassassin. It seems
like such a waste to have 40GB for each of those directories (!), but
I can't do much about that now.
Stayed late on Monday to set it up. Took about four hours, what with
figuring out why I wasn't able to mount partitions via NFS
(whoops-a-daisy). If I were to do it again, I could probably do it in
two if everything went right. Put it back on the front end about
quarter to ten, went home and slept, then took the other, older mail
server off the front end the next day. Disk access is much better now,
and hopefully it'll stay that way.
God it was fun, though. This is exactly the sort of thing I want to
do, and while the mistakes are painful the learning experience is
great. (Part of the reason I'm putting all this up here is so that it
might be useful to someone else, a la the excellent FreeBSD Diary
page. The other part, of course, is sheer ego. And for prospective
employers to find...shudder. :-))
So to learn: more about vinum. More about Sendmail: got the O'Reilly
book at work; made it through all the LHS/RHS chapters only to get to
where it said "But of course no one uses that. Now we'll talk about
.mc files." Arghh! Plus, started reading about m4 and nearly threw the
book across the room in disgust. I'm sure there is a Very Good Reason
for yet another scripting language, but I'll be damned if I can figure
out what it might be at this point in my young life. (Given the age of
Sendmail, I'm guessing it was easier to write one rather than wait for
perl/python/Turing machines to be invented.) God, m4 is hairy. I know,
I know, plug away at it, but still.
Back on topic: more about NFS. I figured out how to install a kernel
from a read-only NFS-mounted partition, but I've yet to figure out how
to install ports from a read-only NFS-mounted /usr/ports. (Li'l help?)
Other news: bought a 486 (SX!) laptop on eBay for $21 US. As it turns
out, should be easy enough to install Linux or FreeBSD on here,
which'll mean a cool li'l email/diagnostic toy^Hol (you know, for all
those times I debugging raw Ethernet frames onsite). Should be in next
week or so.
Original entry.
