The Life of a Sysadmin

Carousel is a lie!

Entries from November 2006.

The universe occasionally says "Fuck you"
2006-11-10 12:09:26

If your machine has hard drives that are, in theory, removeable because they have a front catch, but in practice require you to open up the case to disconnect the SCSI and power cables, that's not a server.

If your machine's CD drive fails and it takes you fifteen minutes of searching to find the unlabelled holes on the bottom of the case that allow access to the screws that are attached to the bottom of the drive so that you can actually remove the drive, that's not a server.

For $399 US, thank you, for the Academic edition of MathMagic, I expect better goddamned installation instructions than this:

  • Windows - MathMagic Pro Edition Full installer with some old versions of fonts and Plugin (Please run this full installer first.) [ a url ] - MathMagic Pro Edition v3.5 (application only. The latest version.) (Please use this v3.5 application, instead of v3.0 installer by the installer, after moving it into /Program Files/MathMagic Pro/ folder) [ another url ] - new CS & CS2 plugin (Please use this Plug-in, instead of the Installer installed one. Copy it into InDesign's Plug-ins/Equations folder.) [ whee! lookit alla urls! ] - new fonts for PDF embedding (If you want to embed MathMagic fonts in your PDF documents on Windows, please download the new MathMagic TrueType font set, and replace the preinstalled ones(remove the old MathMagic fonts from Windows Fonts directory and copy these new fonts into Windows Fonts folder). [ sale! sale on urls! ]

The email goes on to suggest that installation instructions can be found on their website (but neglect to mention that it only covers the Mac version), or "in User Guide documents that you can find after installation." What a crock.

No tags
8 o'clock, the lights were on at Shea
2006-11-11 21:52:43

Woot! I managed to install OpenBSD 4.0 on my work laptop this afternoon while Arlo slept in my arms. Not only that, it automagically set up X and I figured out wireless + OpenVPN. Woot! Firefox is running, I've got Mozex and Adblock going...the only thing left is to figure out how to get IceWM to start up automagically.

Tags: geekdad, openbsd.
Bones of an Idol
2006-11-17 21:50:30

Thursday: Go to The Other University to do some prep for the move coming up next week. Check in with their computer store (where you pretty much have to buy things) to see how the order on the console server is going. The guy behind the counter looks up the order, frowns, and tells me that it seems their supplier does not have one in any of their three Canadian warehouses. Okay, so how long will it take to get one in? He looks at me earnestly and says that, sometimes, they never come in. I ask at what point I can count on the supplier a) giving up and b) informing me of that fact. He frowns again, and suggests that I check back in a couple weeks (four weeks after I've placed the order) just to be safe.

Friday: Get email from contractor/university liason for new building to say that network and electrical connections will not be ready in time because the requests were received so very late. While The Other Guy was supposed to get them in long ago, I should've been on top of this.

Monday, a stat in Canada: Go to the old building to do a serverectomy on a soon-to-be-formerly shared rack. The Other Guy mentions that the new server room has water on the floor. I go over to look, and it's a rapidly evaporating puddle, irregular in shape and maybe two metres across at its widest. I can't figure out where it's coming from. Turns out there's some other stuff that should become formerly shared as well, so I spend time poring over Sun Enterprise 1 workstations (which I like) and old inkjet cartridges for printers that may no longer be around (which I don't like). Ask The Other Guy, who's been involved with the move a lot longer than I have, what electrical connections he's asked for him and for me (long story) in the new building. He says that he gave them the model number of the Sun rack he's got (which has built-in, and very nice, PDUs) and asked them to figure out what he needs.

Tuesday: Moving day. As expected, network and electrical are not present; we've got 2 x 15A 120V circuits. Also, the leak is back, and we can see that it's coming from a small leak in the concrete roof. I move my rack into another room; The Other Guy spreads a blanket over his rack. The liason promises us that the contractors are on the job to fix the roof. The network connections (two fiber, two Cat5) get terminated, so I call the local network folks to get that taken care of. The university wireless network is not present in the new building.

Wednesday: The contractors show up to start fixing the leak. The network connections have been set up. The contractors have put in a big tube of plastic sheeting, taped to the roof at one end and a 40-gallon recycling barrel at the other. The Other Guy decides things are good enough and starts setting up his rack; I elect to hold off another day.

Thursday: The contractors say the roof is fixed, so I move the rack in and start hooking things up. The new OpenBSD firewall comes up nicely -- thank you, pf developers -- as does the main Sun server. Next up is the SunRays in the lab, only they're not. I take my laptop in and try to verify connectivity. I can't. The Other Guys suggests that the VLANs on my new switch are the problem and suggests just simplifying things. I do and keep testing. Traffic from the laptop's RFC 1918 address just never makes it to the server. In a fit of desperation I try using an address in our routable subnet, and it works. This takes me until 8pm to figure out. I email various bosses explaining how far I've got, and the campus network folks to ask if they're filtering this subnet in some way. (This isn't completely out of the question; this place has a reputation for a pretty locked-down network.)

Friday: I buttonhole the guy at the campus network office and ask him about this. He considers this and realizes that while he's forgotten to unblock DHCP (told you it was pretty locked down), the other behaviour I'm seeing can be explained if I've somehow got my interfaces crossed. I'm doubtful but give it a try, which is a good thing because suddenly everything works. I don't understand it or what I did wrong, but assume that I was simply too tired the previous night and thank him profusely for taking the time to talk to me. I am now where I should have been twenty hours before. Mighty battles emerge with Sun's DHCP and Sunray servers. In the end, I have to delete the Sunray configuration, delete all DHCP configurations, and then add the Sunray configuration back. This works, which annoys me; why are there all these opaque configurations around? Not a single plain-text file in sight. I manage to get a printer working, then another. DHCP is modified so that laptops work as well. I call it a night and head home.

Tags: fail, hardware, network.
OpenBSD netboot problems - unknown error code 72
2006-11-18 21:50:28

(Note: edited to actually be correct this time. :-)

While trying to get a Sparc machine to boot disklessly so I could install OpenBSD on it, I kept getting these errors:

Boot: bsd.rd
Automatic network cable selection succeeded : Using TP Ethernet Interface 
Using BOOTPARAMS protocol: ip address: 192.168.23.25, hostname: roark
root addr=192.168.23.10 path=/home/aardvark/openbsd-sparc64/chroot
open /sbus@1f,0/ledma@e,8400010/le@e,8c00000/bsd.rd: Unknown error: code 72

tcpdump showed that the machine was trying to contact the NFS server (192.168.23.10) by udp on port 0; the server kept responding with an ICMP port unreachable error. Googling turned up one other person back in '99 (!) who had the same problem, but no fix.

What was weird was that this had worked during an earlier install -- only the running of MAKEDEV hadn't completed (don't ask), so I didn't have /dev/console when I booted up, which meant no nothing once it tried to mount the root directory.

I started looking at the traffic in greater detail, and saw that the packet to port 0 was, according to Ethereal^WWireshark, a nicely formed NFS call trying to get the filehandle for the kernel (/bsd). Well, what would make it send it there? After all, mountd was listening on the same port it'd been contacted on a moment ago...

Looking at the call to portmap on port 111, I saw that the client was asking for the port for nfsd, but was being told that there was no such thing -- that the port number was zero. What the...I checked rpcinfo -p and saw that, yep, there was no nfsd...and then realized my mistake: mountd only deals with mount requests; it's nfsd that actually reads/writes files, gives information about their size and modification times, and so on. I'd been starting the NFS stuff by hand since this was a one-off, and had totally forgotten to start nfsd. I did so, and suddenly all went well. PEBCAK.

No tags
Stupid, uninformative errors
2006-11-21 15:41:24

Spent half the day trying to figure out why a Sun Directory Server had suddenly lost its ability to replicate over SSL. The logs said:

[21/Nov/2006:00:01:00 -0800] - INFORMATION - NSMMReplicationPlugin - conn=-1 op=-1 msgId=-1 -  Replication over SSL FAILED as SSL is not enabled. Check that the attribute nsslapd-security in cn=config is on.
[21/Nov/2006:00:01:00 -0800] - ERROR<8318> - Repl. Transport  - conn=-1 op=-1 msgId=-1 -  [S] Bind failed with response: SSL configuration error  (808).
[21/Nov/2006:00:01:00 -0800] - ERROR<8221> - Incremental Protocol - conn=-1 op=-1 msgId=-1 -  Failed and requires administrator action [ldap.example.com:636]
[21/Nov/2006:00:01:00 -0800] - ERROR<8221> - Incremental Protocol - conn=-1 op=-1 msgId=-1 -  Failed and requires administrator action [ldap.example.com:636]

Google turned up nada. In the end, it turned out that the last time the directory server had been started, the security token had not been provided. Restarted the server, typed in the token on standard input, and replication works again.

Yes, this is a job for expect -- but this approach has failed for coworkers in the past. I'll have to look into it.

No tags
Canada Post Sucks (UPDATED)
2006-11-25 10:18:15

Update Jan 7, 2007: After sending a snail-mail copy, slightly edited, to Canada Post, I got a letter back from their Customer Service department. It reads in part:

At the outset, I would like to offer our sincere apologies for the service failure. Our delivery personnel are expected to always issue a delivery notice if no one is at the residence to accept an item. Furthermore, a final delivery notice should be mailed from the postal outlet five days later if the item has not been claimed. I am sorry that proper procedures were not followed with your parcel. I have asked the Supervisor at [your local delivery centre] to review this incident with our delivery person. As well, I have brought the failure to issue a final notice to the attention of our Retail Business Manager who oversees the [local retail outlet where the parcel was held]. The [local delivery centre] is the pickup location for all postal items for addresses near you. (You could not find an outlet for the ID 0000268453 because the number belongs to the [local delivery centre]; items are never carded to delivery depots. The message they inputted -- "Item has been transferred tothe Post Office for pick up as a carded item" -- refers to the action they took rather than to their location.) The enclosed information from our website shows how to locate postal outlets, although if proper procedures had been followed, your delivery notice would have informed you of the location. I regret the lack of timely assistance on our part to help locate your parcel and the obvious inconvenience you experienced. If you pre-paid the shipping charges for the returned parcel, please send me the invoice and I will arrange for a refund.

So they did screw up and I got angry; they're sorry and they're making up for it, and I'm satisfied. I don't like deleting old posts, but if I did I'd delete this one. As it is, I'm closing comments. This was an angry post, I'm getting angry comments, and I don't want that. My fault.

The original post follows:

SERVICE, At Mailposte.ca wrote on November 24, 2006:
&gt; Hello Hugh Brown,
&gt; 
&gt; Thank you for your message to Canada Post.
&gt; 
&gt; This message is simply to confirm that our delivery confirmation system now 
&gt; shows this item as being successfully delivered on 11/07/2006.
&gt; 
&gt; Regards,
&gt; 
&gt; Julie Pich?
&gt; Customer Service

Thanks very much for your response.  

The web tracking page for this parcel now says that, as of 11:08
(presumably AM, no idea which time zone) on November 24th, "Item was
unclaimed by Receiver and will be returned to Sender".  It also says
that delivery was attempted, not completed, on November 7th.  As
Canada Post's website says that the information available there is the
same as is available to its customer service representatives, I'm
unsure why I'm being told now that delivery has succeeded.

I never received the first notice.  I did not receive any subsequent
notice.  The link on the tracking page that was meant to show me where
I could pick up the item simply said, "No outlets found for outletID:
0000268453".

I will be contacting the shipper to let them know that, when the
parcel finally makes it back to them, they should ship it to me via
courier: FedEx or UPS, but not Canada Post or Purolator.

I am extremely disappointed with Canada Post.  I never received notice
telling me that a parcel was available for pickup; I only found out
that delivery had been attempted by checking the website.  The
location of the retail outlet where my parcel sat could not be found
on Canada Post's website.  After waiting for a second notice to be put
in my mailbox, I contacted Canada Post via the website and asked
specifically about these problems.  The response, which came 7 days
later, had no response to these questions.  The information it
contained was wrong.  I am now told that my package is being returned
to the sender as it is unclaimed.

I will not use Canada Post in the future for any important delivery.
(Exceptions may be made for bill payments, and other items for which I
want plausible deniability.)  I will strongly recommend to friends and
family that Canada Post be avoided for any important deliveries.

Thank you again for your time.

Yours sincerely,
Hugh Brown

&gt; 
&gt; ----- Original Message -----
&gt; From:         aardvark@vcn.bc.ca 
&gt; Sent:         November 17, 2006 02:04
&gt; To:           SERVICE, At Mailposte.ca
&gt; Subject:      Delivery Notice Card
;&gt; 
&gt; 11/17/2006
&gt; Name:Hugh Brown
&gt; Comments:According to the tracking number I've been given (9184 1445 5359
&gt; 2002), a notice was left November 7th.  I never received this notice.  It
&gt; is now November 16th and I have not received a second notice.  As well,
&gt; the info on Canada Post's website for this tracking number says that the
&gt; parcel is being held at a New Westminster post office, yet the link that
&gt; is supposed to give directions to this post office says  No outlets found
&gt; for outletID: 0000268453 .  Where should I pick up this parcel?
&gt; Customer #:
&gt; Company Name:
&gt; Mailing Address: [snip]
&gt; City:New Westminster
&gt; Province/State:BC
&gt; Country:Canada
&gt; Postal/Zip Code:V3L 5V8
&gt; Email:aardvark@vcn.bc.ca
&gt; Telephone Number: [snip]
&gt; Fax:
&gt; Issue Type:No card received
&gt; Notice Type Received:No notice
&gt; Notice No.:
&gt; Delivery Agent on Card:
&gt; Name on Card:Hugh Brown
&gt; Is this your regular postal outlet:
&gt; Date Card Received:-1 -1, -1
&gt; Date of Last Pickup Attempt:-1 -1, -1
&gt; Date on Card:-1 -1, -1
&gt; Item Type:
&gt; Item Serial # on Card:
&gt; Item Available:
&gt; How is your mail delivered:Apartment
&gt; Time of Delivery Attempt:
&gt; Name of Postal Outlet on Card:
&gt; Address of Postal Outlet on Card:
&gt; Amount Due:
&gt; Item held at Post Office at request of sender:
&gt; Please specify if there is a mark in the box - Other:
&gt; 
&gt; 
No tags

RSS Feed