18 Apr 2010
I'm at work today. There was a scheduled power outage in the building
that holds our server room. It was set for 7am to 11am; I got in at
6am 'cos I'm a keener and wanted to make sure I had lots of time to
swap to the backup website.
Power came back on at 10:30 or so. 10:45 I wandered over to the
building to see if that was it; didn't want to rush anyone, but
thought it'd be worth asking. There was no one there. Sweet, thinks
I, let's head down and turn on some machines. (Most are Sun machines
and therefore work just fine remotely; some are older IBM machines,
where I haven't figured out how to do IPMI over the network, and some
are Dell machines where, whee! who knows how it'll work today?)
Turns out A/C's still off. Call the university folks; turns out
someone's still working on it. But they're off for lunch, so no idea
just yet how long that'll be. I'll be calling back in an hour to
see if we have any idea. If they're working on it 'cos something went
wrong, well, that's life. If they're working on it 'cos they had
scheduled something, then I'm irritated I wasn't told beforehand. Oh
well, we'll see what happens. (Update: Our rooftop A/C failed to
come on after the power came back on. A full investigation, with
twelve helicopters full of determined journalist-engineers, will be
launched tomorrow. THIS GOES ALL THE WAY TO THE TOP, PEOPLE.)
In other news, it wasn't a complete waste of time today:
I found the Python LDAP module and used it to add a bunch of
AutoFS entries to the tree. The simplebrowser.py script included in
the examples is quite nice. (Though it does make me wonder why I
haven't started using one of the billion-and-three LDAP browser
tools instead of a) complaining about phpLDAPadmin or b) trying to
remember the syntax for ldapsearch).
I got to wear my safety shoes.
Downloading Rocks to try installing it on three old servers
haning around.
Discovered that a workstation's hard drive is failing; fortunately
it's not needed right away.
And, erm, that's it.
Loadsoflinks:
And just one more thing: my younger son turns two on Tuesday. We had an
early party:
Tags:
waitinfortheman
python
dell
12 Apr 2010
Wednesday, 5am: 2yr-old son wakes up and starts to shout. I go into
his room to pick him up and step into a pool of cold vomit. He seems
fine.
Thursday, 5:15am: 2yr-old son wakes up and starts to shout. I go into
his room to pick him up and step into a pool of cold vomit. I stay
home a few hours to clean up the carpet. He seems fine.
Friday, 6:10am: 2yr-old son wakes up and starts to shout. I go into
his room to pick him up and turn on the light. I avoid stepping into
a pool of cold vomit. 4yr-old son wakes up and starts to throw up; he
throws up every half hour 'til noon. I stay home to take care of the
kids. We put a towel on the carpet by the head of 2yr-old son's crib.
4yr-old son grey-green most of the day but picks up after nap time.
2yr-old son seems fine. We watch fourteen hours of television.
Saturday, 2am: 2yr-old son wakes up and starts to shout. Clara goes
down and finds him in a pool of vomit within the crib. She sleeps in
the rocking chair with him 'til 7am. 4yr-old son much better ("I feel
MUCH better today!" he says). 2yr-old son whiny and miserable. Clara
and I feel like shit and are whiny and miserable. Clara's parents
take 4yr-old son for the day. I drive us all to the doctor, worried
about how many things are going on right now (night vomits + bug +
teething + what the hell else?); dr tells us almost certainly the same
bug, but manifesting itself differently in all of us. Clara drives us
home while I slump against the car window, cursing the sunlight.
2yr-old son refuses to nap so I hold him in a rocking chair for two
hours while reading James Herriot. We watch sixteen hours of
television and eat plain crackers dipped in filtered water.
Sunday: 2yr-old son feeling better. He has not thrown up in the
night. 4yr-old son feeling even better than yesterday. Clara and I
still feel like shit. We take the kids to the park in the morning.
2yr-old son naps on his own. Clara and I gradually improve through
the day. We eat plain crackers with salt in the morning, plain
crackers with peanut butter in the afternoon, then four pounds of
leftover mac and cheese at 8pm. We watch "Office Space" and laugh
weakly.
Monday, 4:45am: 2yr-old son wakes up and starts to shout but agrees to
go back to sleep. 4yr-old son does the same at 5:45am.
Tags:
geekdad
31 Mar 2010
Okay, last micro-update today I swear: you can query Wikipedia over
DNS. Here's the description, a short presentation on how
it's done, and a bash function. Now to see if I can get
something hacked up in Emacs.
Tags:
dns
emacs
31 Mar 2010
The bge driver for OpenBSD says that the Broadcom BCM5700 series
of interfaces has two MIPS R4000 cpus. And you can run
Linux on an R4000, or NetBSD.
Must...stop...recursion...
Tags:
hardware
networking
31 Mar 2010
Holy crap. PDF supports multimedia, javascript, and launching
arbitrary programs. Haven't checked the standard (warning:
PDF) yet to see if they support the evil bit.
Tags:
security
26 Mar 2010
Removing the bookmark that shows you how to crimp your own RJ45 cables.
Tags:
23 Mar 2010
I think I've finally figured out what's going on with my bacula-sd
hangs. At the risk of repeating myself, this post is adapted from
the bug report I've just filed.
Here's the situation: with the latest Bacula release (5.0.1), I
regularly see bacula-sd hang when running jobs; it happens often and
at seemingly random times; and when this happens, I see two bacula
processes, a parent and a child. (Since bacula is multi-threaded, I
usually just see one storage daemon process.) This came to my
attention when I came back from vacation to find a week's worth of
backups stalled (sigh).
When bacula-sd hangs, the traceback of the relevant thread in the
parent process's looks like this:
Thread 10 (Thread 0x466c0940 (LWP 12926)):
#0 0x00000035aa4c5f3b in read () from /lib64/libc.so.6
#1 0x00000035aa46cc07 in _IO_new_file_underflow (fp=<value optimized out>) at fileops.c:590
#2 0x00000035aa46d5ce in _IO_default_uflow (fp=<value optimized out>) at genops.c:435
#3 0x00000035aa468e8b in _IO_getc (fp=<value optimized out>) at getc.c:41
#4 0x00002b76479565c0 in bfgets (s=0x466bf710 "", size=<value optimized out>, fd=0x60080a0) at bsys.c:617
#5 0x000000000040d432 in release_device (dcr=0x60988b8) at acquire.c:533
[snip]
Here, bacula's storage daemon has just finished running a job, and
before it releases the tape drive to someone else runs the "Alert"
command. This is specified in the config file for the storage daemon,
and is meant to see if the drive has, say, run out of magnetism during
the last job. Here's the source code in stored/acquire.c:
alert = get_pool_memory(PM_FNAME);
alert = edit_device_codes(dcr, alert, dcr->device->alert_command, "");
bpipe = open_bpipe(alert, 0, "r");
if (bpipe) {
while (fgets(line, sizeof(line), bpipe->rfd)) { /* AardvarkNote: This is where the parent hangs */
Jmsg(jcr, M_ALERT, 0, _("Alert: %s"), line);
}
status = close_bpipe(bpipe);
}
Meanwhile, the child process stack looks like this:
Thread 1 (Thread 0x466c0940 (LWP 13000)):
#0 0x00000035aa4df9ee in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x00000035aa4d06a5 in _L_lock_1206 () from /lib64/libc.so.6
#2 0x00000035aa4d05be in closelog () at ../misc/syslog.c:419
#3 0x00002b764795cd35 in open_bpipe (prog=<value optimized out>, wait=0, mode=<value optimized out>) at bpipe.c:138
#4 0x000000000040d3f1 in release_device (dcr=0x60988b8) at acquire.c:531
[snip]
open_bpipe() can be found in lib/bpipe.c; it's a routine for forking
a child process (FORESHADOWING: not another thread!) and sets up a
pipe between parent and child. The relevant bits look like this:
/* Start worker process */
switch (bpipe->worker_pid = fork()) {
[snip]
case 0: /* child */
if (mode_write) {
close(writep[1]);
dup2(writep[0], 0); /* Dup our write to his stdin */
}
if (mode_read) {
close(readp[0]); /* Close unused child fds */
dup2(readp[1], 1); /* dup our read to his stdout */
dup2(readp[1], 2); /* and his stderr */
}
/* AardvarkNote: This is where the child hangs: */
closelog(); /* close syslog if open */
for (i=3; i<=32; i++) { /* close any open file descriptors */
close(i);
}
execvp(bargv[0], bargv); /* call the program */
closelog() itself is simple: it "closes the current Syslog
connection, if there is one." But running strace on the child
process just shows a lot of futex calls...nothing very useful there
at all. So what the hell is going on, and why it hanging at
closelog()?
Some background info on threads: In Linux at least, they're user-land
things, and the kernel doesn't know about them. To the kernel, it's
just another process with a PID. Implementing threads is left as an
exercise to the reader...or in this case, to glibc and NPTL.
Since threads are part of the same process, they share memory. (A
fork(), by contrast, copies over parent memory to the child -- and
then the child has its own copy of everything, separate from the
parent.) glibc/NPTL implements locks for certain things to make sure
that one thread doesn't stomp all over another thread's memory
willy-nilly. And those locks are done with futexes, which are
provided by the kernel...which explains why it would show up in
strace, which tracks system calls.
Why is this relevant? Because in the glibc code, closelog() looks
like this:
void
closelog ()
{
/* Protect against multiple users and cancellation. */
__libc_cleanup_push (cancel_handler, NULL);
__libc_lock_lock (syslog_lock); /* AardvarkNote: This is where things hang */
closelog_internal ();
LogTag = NULL;
LogType = SOCK_DGRAM; /* this is the default */
/* Free the lock. */
__libc_cleanup_pop (1);
}
That __libc_lock_lock (syslog_lock) call is there to prevent two
threads trying to mess with the syslog file handle at one time. Sure
enough, the info for frame #2 of the child process shows that the
process is trying to get the syslog_lock mutex:
#2 0x00000035aa4d05be in closelog () at ../misc/syslog.c:419
419 __libc_lock_lock (syslog_lock);
ignore1 = <value optimized out>
ignore2 = <value optimized out>
ignore3 = <value optimized out>
As noted in the mailing list discussion, closelog() should be a noop
if there's no descriptor open to syslog. However, in my case there
is such a file descriptor, because I've got bacula-sd configured to
log to syslog.
Well, as the Bible notes, mixing fork() and threading is
problematic:
There are at least two serious problems with the semantics of fork() in
a multi-threaded program. One problem has to do with state (for
example, memory) covered by mutexes. Consider the case where one
thread has a mutex locked and the state covered by that mutex is
inconsistent while another thread calls fork(). In the child, the
mutex is in the locked state (locked by a nonexistent thread and thus
can never be unlocked). Having the child simply reinitialize the mutex
is unsatisfactory since this approach does not resolve the question
about how to correct or otherwise deal with the inconsistent state in
the child.
And hey, doesn't that sound familiar?
Now that I had an idea of what was going on, I was able to find a
number of similar problems that people have encountered:
This post describes a multi-threaded app that hung when a signal was
received during a syslog() call
This thread from the libc-alpha mailing list, describes a
similar problem:
> The particular problem I'm seeing is with syslog_lock.
>
> If another thread is writing to syslog when the fork happens,
> the syslog_lock mutex is cloned in a locked state.
This Debian bug describes a multithreaded app deadlocking when
syslog() is called after a fork()
The Bible describes the POSIX-ish way around this, the pthread_atfork()
call.
This blog entry has an overview of the problem with threads and
fork(), and of pthread_atfork()
So: what seems to be happening is that, when the stars align, the
child process is being created at the same moment that the parent
process is logging to syslog. The child is created with a locked
syslog_lock mutex, but without the thread that had been holding
it...and thus, without anything that can release it. The child
blocks waiting for the mutex, the parent blocks on the child, and
backup jobs halt (well, at least the spooling of jobs to tape) until I
kill the child manually.
This was complicated to find for a number of reasons:
My gut feeling (see also: handwavy assertion) is that logging to
syslog is relatively unusual, which would explain why this problem
has taken a while to surface.
It's a race condition that's exacerbated by having multiple jobs
running at once; I'd only recently implemented this in my Bacula
setup.
btraceback, a shell wrapper around gdb that comes with
Bacula, is meant to run the debugger on a hung Bacula process. I
used it to get the tracebacks shown above. It's great if you don't
know what you're doing with gdb (and I certainly don't!). But it
has a number of problems.
- In my setup, bacula-sd ran as root; however, for reasons I don't
fully understand,
btraceback is usually called by the unprivileged
bacula user. That meant I had to modify the script to run gdb with
sudo, and change sudoers to allow bacula to run "sudo gdb" without a
password. Otherwise, the dump was useless -- particularly
frustrating before I'd figured out how to duplicate the problem.
btraceback can be triggered by sending a fatal signal to the
bacula-sd process (say, "kill -6"). That's great when you notice a
hang and can run the script. But it won't trace child processes --
which was where the interesting things were -- and it was a while
before it occured to me to do that manually.
Bacula has an --enable-lockmgr option that's meant to catch
deadlocks, kill itself and run btraceback. However, it's not
enabled by default in the SRPMs I was using to build bacula, and in
any case it watches for deadlocks on Bacula's own locks -- not
external locks like syslog_lock.
So what to do?
For now, I'm removing the log-to-syslog option from bacula-sd.conf.
When I do that, I see no problems at all running jobs.
On the programming side -- and keep in mind I'm no programmer -- it
looks like there are a number of options here:
Don't call closelog() after fork() in open_bpipe(). This may
leave an open descriptor to syslog, which may be a problem. (Or maybe
it's just ugly. Don't know.)
Don't fork() a child process when running the alert command, but
start another thread instead. I have no idea why a fork() is
preferred over a new thread in this scenario, but I presume there's a
good reason.
Use pthread_atfork() to set things up for a fork(). That's
what The Bible says to do, but I don't know what Bacula would
actually need to do with it in order to make this deadlock go away.
Good lord, I'm closing in on 1700 words here. If you've stuck with me
this long, I hope it was interesting; once I got over the panic it was
a fascinating problem, and it's taught me a lot about processes,
threads and deadlocks. And how can that be a bad thing?
Tags:
backups
debugging
18 Mar 2010
This is not quite me but a) I did have a phall, once, long
ago, and b) he really does look like me. (Or I look like
him...what are the rules of precedence in these things?)
I have been very, very busy over the last couple of days but I am
finally starting to see a way through it. I'm hoping that Bacula
problems are behind me for now.
Update: Nope, locked up two minutes after I wrote that. But I got
the backtrace I needed!
Tags:
meta
16 Mar 2010
I've been off on vacation for a week while my parents have been
visiting. It's been fun, relaxing, and generally a good time. But
today it was back to work.
I found out that backups have not been running since the day after I
went on vacation. There were something like 650 jobs in Bacula
stacked up, waiting to run; the storage daemon was stuck trying to do
something, and nothing was getting done.
Near as I can tell, the storage daemon was stuck in a deadlock. I've
got some backtraces, I've posted to the devel mailing list, and it
looks there's a bug, recently fixed, that addresses the problem. Inna
meantime I've put in Nagios monitoring for the size of the run queue,
and I'm figuring out how to watch for the extra bacula-sd process that
showed up.
Tonight, at least, the same problem happened again. This is good,
because now I have a chance to repeat it. Got another backtrace,
killed the job, and things are moving on. Once things are finished
here, I think I'm going to try running that job again and seeing if I
can trigger the problem.
Sigh. This was going to be a good day and a good post. But it's late
and I've got a lot to do.
Tags:
bacula
backups
debugging
04 Mar 2010
Did you know that Postfix's cleanup daemon consolidates messages
going to the same envelope recipient? I did not.
Tags:
postfix
01 Mar 2010
Sometimes random bits of my personality make themselves clear to me.
Example: today, a good part of my day was spent trying to figure out
exactly what a particular server did. I knew it ran a program, and
what the program was called, but that was it; it had been shut down
and sent over to me for installation in $WORK's server room. I had
to:
- reset the password
- get access to the ILOM
- change pam.d/* (
authconfig! didn't know about that!)
- start the service
- figure out the licensing
- track down why it wasn't starting at boot time
All of which reminded me of a quote from the biography of Isaac Newton
right now (which I can't track down, worse luck)...something about
how, without understanding, someone whose tools break will be unable
to figure out what's wrong. Much more succinct than that in the
original. And that, in turn, reminded me of something I saw on a
mailing list recently (or a website, or...) about how a sysadmin is
someone who asks why something works, not just how to make it so.
Example: I came across this article, shortly after "Aurora Attack
-- Resistance Is Futile, Pretty Much" on the howling void, and
that made me realize that I'd been thinking in the panicky sort of way
outlined. And that reminded me of the time I learned about The
Rapture as a kid and went home and told everyone about it. I was
told something, it was compelling, and I took it in wholesale, like
snorting cocaine, and ran to tell everyone about it. And that made
me realize that I've done no checking at all on that Wired article,
none, but it's compelling and here I am telling everyone about it.
And I was relating this all to my wife and she said yeah, you dive in,
but it's a process. You spent the next n years of your life
learning about American fundamentalist Christianity and Bible history
and you didn't just hide under the bed scared forever.
Random note: Ken MacLeod's The Execution Channel just got
scarier. (It's a process, remember?)
'Nother random note: Glad to see Planet Sysadmin is back up; I
missed it.
Tags:
25 Feb 2010
Ugh, late night...which these days, means anything past 9:30pm. Two
machines down at work with what I think are unrelated problems.
First one appears to have had OOM-killer run repeatedly and leave the
ethernet driver in a bad state; I know, but the OOM-killer kept
killing things until we got this bug.
Second one appears to have crashed and/or rebooted, but the hardware
clock got reset to December 2001 in the process -- which meant that
when it tried to contact the LDAP servers, none of their certificates
were valid yet.
Again, ugh. But I did come across this helpful addition to my
toolkit:
openssl s_client -CAfile /path/to/CA_cert -connect host:port
which, I just realized, I've rediscovered, along with having the
same fucking problem again.
And did I mention I'm up at 5am tomorrow to move some equipment around
at work? Ah well, I have safety boots now. I'll be suitably rewarded
in Valhalla.
Tags:
debugging
25 Feb 2010
Just saw a Microsoft wireless keyboard and mouse, packaged together,
for $320. For that much money, I want Darwinian evolution.
Tags:
25 Feb 2010
I mentioned that I've been having problems with Bacula recently.
These have been aggravated by the fact that the trigger seems to be a
job that takes 53 hours to finish.
Well, I think I've got a handle on one part of the problem. See,
when Bacula is doing this big job, other jobs stack up behind it --
despite having two tape drives, and two separate pools of tapes, and
concurrent jobs set up, the daily jobs don't finish. The director
says this:
9279 Full BackupCatalog.2010-02-20_21.10.00_10 is waiting for higher priority jobs to finish
9496 Full BackupCatalog.2010-02-23_21.10.00_13 is waiting execution
9498 Full bigass_server-d_drive.2010-02-24_03.05.01_15 is running
9520 Increme little_server-var.2010-02-24_21.05.00_38 is waiting on Storage tape
9521 Increme little_server-opt.2010-02-24_21.05.00_39 is waiting on max Storage jobs
but storage says this:
Running Jobs:
Writing: Full Backup job bigass_server-d_drive JobId=9498
Volume="000031"
pool="Monthly" device="Drive-0" (/dev/nst1)
spooling=1 despooling=0 despool_wait=0
Files=708,555 Bytes=1,052,080,331,191 Bytes/sec=11,195,559
FDReadSeqNo=22,294,829 in_msg=20170567 out_msg=5 fd=16
Writing: Incremental Backup job little_server-var JobId=9508 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0)
spooling=0 despooling=0 despool_wait=1
Files=156 Bytes=3,403,527,093 Bytes/sec=72,415
FDReadSeqNo=53,041 in_msg=52667 out_msg=9 fd=9
Writing: Incremental Backup job little_server-etc JobId=9519 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0)
spooling=0 despooling=0 despool_wait=0
Files=9 Bytes=183,606 Bytes/sec=3
FDReadSeqNo=72 in_msg=50 out_msg=9 fd=10
Writing: Incremental Backup job other_little_server-etc JobId=9522 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0)
spooling=0 despooling=0 despool_wait=1
Files=5 Bytes=182,029 Bytes/sec=3
FDReadSeqNo=45 in_msg=32 out_msg=9 fd=19
Writing: Incremental Backup job other_little_server-var JobId=9525 Volume="000017"
pool="Daily" device="Drive-1" (/dev/nst0)
spooling=0 despooling=0 despool_wait=0
Files=0 Bytes=0 Bytes/sec=0
FDSocket closed
Out of desperation I tried running "unmount" for the drive holding the
daily tape, thinking that might reset things somehow...but the console
just sat there, and never returned a prompt or an error message.
Meanwhile, storage was logging this:
cbs-01-sd: dircmd.c:218-0 <dird: unmount SL-500 drive=1
cbs-01-sd: dircmd.c:232-0 Do command: unmount
cbs-01-sd: dircmd.c:596-0 Try changer device Drive-0
cbs-01-sd: dircmd.c:617-0 Device SL-500 drive wrong: want=1 got=0 skipping
cbs-01-sd: dircmd.c:596-0 Try changer device Drive-1
cbs-01-sd: dircmd.c:612-0 Found changer device Drive-1
cbs-01-sd: dircmd.c:625-0 Found device Drive-1
cbs-01-sd: block.c:133-0 Returning new block=39cee10
cbs-01-sd: acquire.c:647-0 JobId=0 enter attach_dcr_to_dev
...and then just hung there. "Aha, race condition!" I thought, and
sure enough a bit of searching found this commit in November:
"Fix SD DCR race condition that causes seg faults". No, I don't have
a segfault, but the commit touches the last routine I see logged
(along with a buncha others).
This commit is in the 5.0.1 release; I wasn't planning to upgrade to
this just yet, but I think I may have to. But I'm going on vacation
week after next, and I'm reluctant to do this right before I'm away
for a week. What to do, what to do...
Tags:
backups
bug
debugging
upgrades
23 Feb 2010
Backups: Bacula has been giving me problems the last week or so. I've got this
file server I'm trying to back up; it's got a 2TB partition, and I've
been naively trying to just grab it all in one go. Partly that's
because it hasn't been backed up before, and I figured this'd be the
quickest, simplest way to get going.
What's happened is that after slurping 2 TB over a 100 Mbit connection
(no, there's no way to make that quicker), which takes 53 hours, the
writing to tape fails for reasons I've yet to figure out. Bacula
doesn't say "Oh, the first bit worked so I can just grab that next
time...." (To be fair, that's probably a much harder problem than I
imagine.) And in the meantime, despite having two drives and two
pools of tapes, backups for other stuff pile up behind this big
backup and then don't work: they get put on spool space, but then
despooling to tape fails.
Contact manglement: I've been looking for a contact management program for $WORK.
Requirements:
- database (not LDAP), so I can do funky joins and such
- web-based
- FIAF
- members can belong to more than one group
- extensible (ideally w/o my intervention) without making me feel like
I've returned to maintaining the database frontend at my last job
This turns out to be surprisingly hard to find, and not just because
Freshmeat's interface is terrible. Applications appear to fall
into n categories:
- Incomplete and not updated since 2004
- Complete but full of bletcherous hacks like direct SQL manipulation
of the table (making future modification hard), or assuming that a
person belongs to one-and-only-one group; often intended for
personal use, not business, as evidenced by space for a birthday and
contact fields for 30 different IM platforms
- SugarCRM
- Replacements for Access/Filemaker
So now I'm trying to decide between using Dadabik, which'll let
me make a frontend w/o much work as long as I can come up with a
schema, or modifying one of the complete-but-bletcherous apps and
getting a prettier page. (I'm always paranoid about people refusing
to use a web-based tool because it isn't pretty enough; I don't know
how to make it prettier and it's not something I personally care about
enough to do something about, so I'm caught between don't care and
don't know how to fix it if I do care. As a result I panic.)
Family: Son #2 went to the hospital Sunday night with his mom; he's fine,
but I was up 'til they got back at midnight. Still got up at 5:30am
as usual, thinking I'd catch up last night. Then Son #1 had a bad
nightmare last night and it took a while to get him calmed down.
Spent a couple hours after that staring at the ceiling, trying to get
myself calmed down. Still up at 5:30am as usual.
Dentist: Root canal didn't work. My former dentist, who is the second most
graceless dentist I've ever seen, couldn't get through and referred me
to an endodontist (someone who does root canals; thank you,
Wikipedia). My appointment for them is on April 1st.
And that is that.
Tags:
backups
geekdad
18 Feb 2010
Sorry -- importing some old entries from my Slashdot journal, and I
forgot the date from one of 'em...which made it look like it was 2002
all over again.
Got a root canal today. Wish me luck.
Tags:
meta
16 Feb 2010
Remember: when adding access to your Fishworks/Unified Storage System
7310, LDAP entries must include objectClass: shadowAccount. That
took me a while to track down.
Tags:
debugging
16 Feb 2010
I've set up ipv6 again on my home server; a reboot + doing everything
by hand + not writring it down means a) I'm a baaaad sysadmin and b)
had to wait 'til now to find the time to get it going again.
I'm really curious to know what IPv6 connectivity is available at
UBC. Must ask mailing list...
Tags:
ipv6
meta
16 Feb 2010
Not to turn this blog into just a collection of links, but Bunnie
Huang has a fascinating couple of entries up on MicroSD cards. The
first is a bit of info on how they're packaged; the second
details how he came across some poorly made cards and what that
reveals about the economics of MicroSD manufacturing. Makes me wonder
what kind of ghost parts might be in my server room...
Tags:
15 Feb 2010
Valerie Aurora always makes for interesting reading.
This entry is no exception:
If you spend all day with your co-workers, socialize only with
your co-workers, and then come home and eat dinner with -- you
guessed it -- your co-worker, you might go several years
without hearing the words, "Run Solaris on my desktop? Are you
f-ing kidding me?"
Schwartz's "the financial crisis did it" explanation for Sun's
demise is a symptom of an inbred company culture in which
employees at all levels voluntarily isolated themselves from
the larger Silicon Valley culture. Tech journalists write
incessantly about the exchange of expertise and best practice
between companies as a major driver of the Bay area's
success. But you have to actually talk to your competition to
do that -- over a beer, or maybe a pillow.
Tags:
solaris
