And my conscience has it stripped down to science
Why does everything displease me?
Still, I'm trying...

"Christmas with Jesus", Josh Rouse

* The easiest and best by far is for the app to do it.  It knows its
  state intimately and is in the best position to do this.  However,
  the app needs to support this.  Not necessary to have it explicitly
  save the process (as in, kernel-resident memory image, registers,
  etc); if it can look at logs or something and say "Oh, I'm 3/4
  done", then that's good too.

* The Condor scheduler supports this, *but* you have to do this by
  linking in its special libraries when you compile your program.  And
  none of the big vendors do this (Matlab, Mathematica, etc).

* BLCR: "It's 90% working, but the 10% will kill you." Segfaults,
  restarts only work 2/3 of the time, etc.  Open-source project from a
  federal lab and until very recently not funded -- so the response to
  "There's this bug..." was "Yeah, we're not funded. Can't do nothing
  for you." Funding has been obtained recently, so keep your fingers
  crossed.

One admin had problems with his nodes:  random slowdowns, not caused
by cstates or the other usual suspects.  It's a BIOS problem of some
sort and they're working it out with the vendor, but in the meantime
the only way around it is to pull the affected node and let the power
drain completely.  This was pointed out by a user ("Hey, why is my job
suddenly taking so long?") who was clever enough to write a
dirt-simple 10 million iteration for-loop that very, very obviously
took a lot longer on the affected node than the others.  At this point
I asked if people were doing regular benchmarking on their clusters to
pick up problems like this.  Answer: no.  They'll do benchmarking on
their cluster when it's stood up so they have something to compare it
to later, but users will unfailingly tell them if something's slow.

I asked about HPL; my impression when setting up the cluster was, yes,
benchmark your own stuff, but benchmark HPL too 'cos that's what you
do with a cluster.  This brought up a host of problems for me, like
compiling it and figuring out the best parameters for it.  Answers:

* Yes, HPL is a bear.  Oak Ridge: "We've got someone for that and
  that's all he does."  (Response: "That's your answer for everything
  at Oak Ridge.")

* Fiddle with the params P, Q and N, and leave the rest alone.  You
  can predict the FLOPS you should get on your hardware, and if you
  get 90% or so within that you're fine.

* HPL is not that relevant for most people, and if you tune your
  cluster for linear algebra (which is what HPL does) you may get
  crappy performance on your real work.

* You can benchmark it if you want (and download Intel's binary if you
  do; FIXME: add link), but it's probably better and easier to stick
  to your own apps.

Random:

* There's a significant number of clusters that expose interactive
  sessions to users via qlogin; that had not occurred to me.

* Recommended tools:
  * ubmod: accounting graphs
  * Healthcheck scripts (Werewolf)
  * stress: cluster stress test tool
  * munin: to collect arbitrary info from a machine
  * collectl: good for ie millisecond resolution of traffic spikes

* "So if a box gets knocked over -- and this is just anecdotal -- my
  experience is that the user that logs back in first is the one who
  caused it."

* A lot of the discussion was prompted by questions like "Is anyone
  else doing X?" or "How many people here are doing Y?"  Very helpful.

* If you have to return warranty-covered disks to the vendor but you
  really don't want the data to go, see if they'll accept the metal
  cover of the disk.  You get to keep the spinning rust.

* A lot of talk about OOM-killing in the bad old days ("I can't tell
  you how many times it took out init.").  One guy insisted it's a lot
  better now (3.x series).

* "The question of changing schedulers comes up in my group every six
  months."

* "What are you doing for log analysis?" "We log to /dev/null."
  (laughter) "No, really, we send syslog to /dev/null."

* Splunk is eye-wateringly expensive: 1.5 TB data/day =~ $1-2 million
  annual license.

* On how much disk space Oak Ridge has:  "It's...I dunno, 12 or 13 PB?
  It's 33 tons of disks, that's what I remember."

* Cheap and cheerful NFS:  OpenSolaris or FreeBSD running ZFS. For
  extra points, use an Aztec Zeus for a ZIL: a battery-backed 8GB
  DIMM that dumps to a compact flash card if the power goes out.

* Some people monitor not just for overutilization, but for
  underutilization: it's a chance for user education ("You're paying
  for my time and the hardware; let me help you get the best value for
  that").  For Oak Ridge, though, there's less pressure for that:
  scientists get billed no matter what.

* "We used to blame the network when there were problems.  Now their
  app relies on SQL Server and we blame that."

* Sweeping for expired data is important.  If it's scratch, then
  *treat* it as such: negotiate expiry dates and sweep regularly.

* Celebrity resemblances: Michael Moore and the guy from Dead Poet's
  Society/The Good Wife.  (Those are two different sysadmins, btw.)

* Asked about my .TK file problem; no insight.  Take it to the lists.
  (Don't think I've written about this, and I should.)

* On why one lab couldn't get Vendor X to supply DKMS kernel modules
  for their hardware:  "We're three orders of magnitude away from
  their biggest customer.  We have *no* influence."

* Another vote for SoftwareCarpentry.org as a way to get people up to
  speed on Linux.

* A lot of people encountered problems upgrading to Torque 4.x and
  rolled back to 2.5.  "The source code is disgusting.  Have you ever
  looked at it?  There's 15 years of cruft in there. The devs
  acknowledged the problem and announced they were going to be taking
  steps to fix things. One step: they're migrating to C++.
  [Kif sigh]"

* "Has anyone here used Moab Web Services? It's as scary as it sounds.
  Tomcat...yeah, I'll stop there." "You've turned the web into RPC. Again."

* "We don't have regulatory issues, but we do have a
  physicist/geologist issue."

* 1/3 of the Top 500 use SLURM as a scheduler.  Slurm's srun =~
  Torque's pdbsh; I have the impression it does not use MPI (well,
  okay, neither does Torque, but a lot of people use Torque + mpirun),
  but I really need to do more reading.

* lmod (FIXME: add link) is a Environment Modules-compatible (works
  with old module files) replacement that fixes some problems with old
  EM, actively developed, written in lua.

* People have had lots of bad experiences with external Fermi GPU
  boxes from Dell, particularly when attached to non-Dell equipment.

* Puppet has git hooks that let you pull out a particular branch on a node.

And finally:

Q: How do you know you're with a Scary Viking Sysadmin?

A: They ask for Thor's Skullsplitter Mead at the Google Bof.