Serial console FAIL (somewhere...)

This is irritating...

We've got four new Dell R410 servers at work. Natch, I want 'em working with serial consoles so I don't have to sit in the server room. Three of them worked; the fourth did not, despite having identical BIOS/Grub settings.

The symptom was quite maddening: After getting past the various BIOS checks, the Grub menu would not appear unless you sat there and typed something. After that, you'd get the usual Grub entries and could boot as usual. If you did not hit a key, the machine would just hang -- no response to keypresses at all, and you'd have to power cycle.

I spent a stupid amount of time comparing BIOS and Grub settings but was unable to find anything different. Finally today I typed "grub console timeout serial dell" into Google and found this bug in Launchpad, with this comment as the last one:

Having the same hanging issue at the Grub 1.5 stage on brand new R200 Dell servers running OpenSuse 10.3. The terminal timeout is set to 10 and we get 10 press any key to continue messages and then a full system hang requiring a hard reboot.

If we do press any key on a connected console (using Dell's Serial Over Lan) or locally before then end of the timeout then it boots fine so seems to be a bug in continuing at the end of the wait time.

Removing the terminal line from /boot/grub/menu.1st seems to fix the issue on our servers. The console in this case is sent by BMC to both the local screen and the remote console with no timeout so works a treat. This may only work with Dell's BMC/SOL but thought I'd mention it in case anyone else has spent a day getting frustrated with this like we have.

This worked a treat, with the added bit of weirdness that I had two "terminal" lines:

terminal --timeout=2 serial console
serial --unit=0 --speed=9600
default=0
timeout=5
serial --unit=1 --speed=115200
terminal --timeout=5 serial console

and now I have one:

terminal --timeout=2 serial console
serial --unit=0 --speed=9600
default=0
timeout=5
serial --unit=1 --speed=115200
# terminal --timeout=5 serial console

Yes, I know that's redundant, but again: it worked on the other three machines.

I don't know if this is a problem with Grub, with Dell's firmware or something else, but Gott in himmell I hate bugs like this.