Snmp_heartbreak

title: SNMP Heartbreak date: 2006-05-19 05:44:40

I'm still trying to get Heartbeat all working on the two file servers at work. The bit that's getting me down is STONITH -- in particular, the apcmastersnmp plugin.

For some reason, it just will not send out the SNMP request saying "reboot that there outlet". It's not very specific about why, either. The weird thing (well, one of a few) is that running the stonith command will send the request (once you figure out the goddamned syntax for the config file...Christ on a crutch, the documentation is poor), but the hearbeat process itself, which just calls the library directly rather than using the stonith command, does not.

strace shows that heartbeat forks off a child to send the request. That child then goes about closing all its file descriptors, then trying to sendto(2) on a file descriptor (socket descriptor?) that's one of the bunch it just closed. We get EBADF, then it logs the failure.

(This is a little further than I was getting, BTW; it turns out to be essential to put the MIB file for the PDU into /usr/share/snmp/mibs. I didn't think about that, but it makes sense.)

So I've compiled a debug version of heartbeat (Debian rocks: DEB_BUILD_OPTIONS=debug,nostrip dpkg-buildpackage -rfakeroot and away you go), and it turns out to be snmp_synch_response that's failing. Of course, that's in the NET-SNMP library, so now I'm preparing to compile a debug version of that and see what's going on.

I'm of two minds on this. Failover would really be a good thing, and I can't do it w/o STONITH. And I hate like hell to just give up and say, "Oh, it's too hard for me." OTOH, this is just taking so damned long, and it is an older version (though it is the one in stable). I may take a look at the 2.0 series and see how that works...just hope I don't have to throw away all this work. <grumble / >