Today I was out with the kids, and we came across a place selling
fresh-baked pretzels. "Mmm, those smell GOOD," said Eli. "Want to
try making some when we get home?" I asked. "Sure!" he said, sounding
surprised that such a thing was even possible. One quick check of
"More Food that Really Schmecks" later, and we were off to the races.
LISA again! This is the fifth? (Washington, Baltimore, San Diego, San
Jose, Seattle, and now Boston) sixth! that I've been to. Saturday's
flight in was fairly uneventful, except a) it didn't bother my
sciatica too much, so yey and b) I forgot my coat on the plane, and it
doesn't look like Air Canada has a working system to take "Hey, did
you see a coat?" calls.
Fortunately I can count on the kindness of Andy Seely, who brought an
extra coat and loaned it to me. For his kindness I have given him a
"Taggart Transcontinental" t-shirt, and let him buy me supper. I'm
nothing if not generous.
Sunday I spent the entire day at the Google SRE tutorial, which was
very, very cool; a big part of it was an exercise to architect a
system that would read and join logfiles. It took a long time to wrap
my head around how everyone was thinking about this, but writing down
the moving parts made it all a lot clearer. In the end, my team's
proposal approximated the final example config presented by Google, so
that was good. Final sol'n, BTW, used 101 machines. The math all
worked out, but it still made my jaw drop. When I asked the
presenters about this, they grinned. "We've forgotten how to count
small," one of them said.
Today was spent in "Everything you ever wanted to know about operating
systems but were afraid to ask", aka "Caskey's Brain Dump". It was a
pretty awesome talk, covering everything from silicon through
filesystems. Well worth it; I'd love a recording of it, since the
slides simply don't do it justice.
If you voted for Hillary Clinton this week, you likely feel
despondent, confused and unable to reconcile how the country elected
Donald J. Trump. “Don’t people see how dangerous this man is?”
Clinton supporters asked. “Our entire way of life is at stake.”
I get it. I opposed Mr. Trump, too. But this is how nearly half the
country felt eight years ago. It does not matter if we do not
understand one another’s feelings. What matters is that we at least
How do we stop the cycle?
Tuesday night, as it became apparent that Mr. Trump would win, I saw
myself as others may see me. Pundits were beside themselves talking
about sexism, “whitelash” and bigotry. I read three articles
comparing him to Hitler. I understand what they meant. But just as
President Obama was not a Manchurian candidate, Mr. Trump is not
Hitler. The seeds of 1933 may have been planted, but they can grow
only through our hate and divisiveness.
I don’t question your right and reasons to feel fear. But don’t fear
Donald Trump the way I feared Barack Obama. I read a perfect
election summation: The people who were against Mr. Trump took him
literally but not seriously. His supporters took him seriously but
not literally. It is the same pattern of 2000 and 2008. We heard
President Obama was coming for our church and our guns. We were
mocked. We thought those who laughed were lying or stupid. Yet, I
still go to church, sometimes with a gun.
Well, that happened. Tuesday night I went to bed hoping for better
news. I woke up at 2.30am, unable to sleep; after a while I gave up
and came down. Still no good news. President Trump it is, help us
But. This morning there was an ISS flyover, and a rare semi-clear sky
to see it. So I fired up the ISS HDD viewer on my laptop, went
outside, and watched the sun rise from the ISS while I watched it fly
overhead, a wonder heading for the dawn.
Today's title from the subject line of some spam I just got. ("a
spam"? "a spammy email"? just "spam"?)
Mystery flu-like illness continues, or at least its fallout; I've
had lower back pain for the last ~ 4 weeks. Doctor says removing
spine is "not an option" but I've done some Googling and
$WORK continues apace. After taking a week of Python training, we're
using Go for a new tool we're building. Haven't got a good sense
for what it's like just yet, but so far I don't seem to be making a
mess of things.
Tried out drone.io at $WORK yesterday and holy god, is it
good. Auth with our internal Github, then activate repos, and boom!
it runs tests on every new commit on any branch, watches for PRs,
the whole nine yards. When I think of the amount of work we had to
do to get Jenkins to do this, it's insane. Plus the whole
fire-up-sibling-docker-containers-for-tests thing is very, very
Sportsball has started up again with a vengeance: practices on
Monday and Wednesday, games on Fridays and Saturdays. Somebody stop
I've registered for LISA 16, woot! This will be my fifth --
wait, sixth? -- LISA, ten years after my first time attending.
Not sure who's gonna be the theme band this year -- I've done New
Pornographers, Josh Rouse, Soul Coughing and Sloan. And since he's
co-chair this year, it seems like a good time to pull out that
picture of Matt Simmons (@standaloneSA) as a PHP dev:
I have spent this weekend debugging shit on my home server: I've
managed to break my IPv6 tunnel and Docker networking, and for some
reason /etc/resolv.conf was emptied I don't even know why. Last
weekend I spent far too much time debugging problems with DKIM and SPF
and breaking my wife's email, and that was with only one domain; I've
still got another to go through. I am thoroughly sick and tired of
I have always thought it important to run your own server (buy me a
beer some time to get the reasons), but I am done. Done, I say. I am
ready, at this point, to throw money at someone or something to just
make this go away. I still want my own SSH server at home -- that's
too much to give up -- and I still like checking my mail with Mutt.
But web hosting, docker networking, metrics, monitoring, DNS, backups,
calendaring -- gahhhhhhhhhhhhhh.
Probably, I am just going to put all this away for now, leave the
unpaid sysadmin work for another time.
The recent Lawfare Podcast episode "Disrupting ISIS Recruitment
Online" makes fascinating listening. It's a recording of a panel
discussion consisting of two Google-affiliated companies that do
targeted advertising aimed at, well, disrupting ISIS online
recruitment, and the US Undersecretary of State for Public Diplomacy
and Public Affairs.
It is, at first listen, profoundly weird to hear the jargon of online
advertising applied to propaganda. (It's propaganda I agree with, but
propaganda nonetheless.) But then I realized where I'd come across
the idea before: Robert A. Heinlein's "If This Goes On --".
Here's a quote:
'I'm in the Psych & Propaganda Bureau,' he told me, 'under Colonel
Novak. Just now I'm writing a series of oh-so-respectful articles
about the private life of the Prophet and his acolytes and attending
priests, how many servants they have, how much it costs to run the
Palace, all about the fancy ceremonies and rituals, and such
junk. All of it perfectly true, of course, and told with unctuous
approval. But I lay it on a shade too thick. The emphasis is on the
jewels and the solid gold trappings and how much it all costs, and
keep telling the yokels what a privilege it is for them to be
permitted to pay for such frippery and how flattered they should
feel that God's representative on earth lets them take care of him.'
'I guess I don't get it,' I said, frowning. 'People like that
circusy stuff. Look at the way the tourists to New Jerusalem
scramble for tickets to a Temple ceremony.'
'Sure, sure-but we don't peddle this stuff to people on a holiday to
New Jerusalem; we syndicate it to little local papers in poor
farming communities in the Mississippi Valley, and in the Deep
South, and in the back country of New England. That is to say, we
spread it among some of the poorest and most puritanical elements of
the population, people who are emotionally convinced that poverty
and virtue are the same thing. It grates on their nerves; in time it
should soften them up and make doubters of them.'
'Do you seriously expect to start a rebellion with picayune stuff
'It's not picayune stuff, because it acts directly on their
emotions, below the logical level. You can sway a thousand men by
appealing to their prejudices quicker than you can convince one man
by logic.. You can sway a thousand men by appealing to their
prejudices quicker than you can convince one man by logic. It
doesn't have to be a prejudice about an important matter
either. Johnnie, you savvy how to use connotation indices, don't
'Well, yes and no. I know what they are; they are supposed to
measure the emotional effects of words.'
'That's true, as far as it goes. But the index of a word isn't fixed
like the twelve inches in a foot; it is a complex variable function
depending on context, age and sex and occupation of the listener,
the locale and a dozen other things. An index is a particular
solution of the variable that tells you whether a particular word is
used in a particular fashion to a particular reader or type of
reader will affect that person favorably, unfavorably, or simply
leave him cold. Given proper measurements of the group addressed it
can be as mathematically exact as any branch of engineering. We
never have all the data we need so it remains an art-but a very
precise art, especially as we employ "feedback" through field
sampling. Each article I do is a little more annoying than the
last-and the reader never knows why.'
I'll leave my ambivalence about Lawfare for another day. For now:
the podcast makes fascinating listening, and if you haven't read "If
This Goes On--", I highly recommend it.
This week I've been taking Python3 training at work: 4 days of
staying at home and concentrating on Python. The result? 4 days to
work on Python, sharpening my skills, and that's a good thing. The
lecture was not that hot, but what was useful was having the
exercises in front of me, waiting to be done and no distractions to
keep me from them. And after all that, the biggest difference I
notice between Python 2.7 and Python 3 is print "foo" vs
print("foo"). (Which shows you how much Python I know. But
still.) I finished the exercises a few hours early, so I spent the
time trying to solve the coding challenge we give new people at
OpenDNS. (I didn't get that one; instead, I got the "this machine
is borked in 12 different ways, please solve it" challenge.) This
has been a wonderful way to stretch my brain, and work on something
very very different from what I do every day. I wish work had the
same sort of course for Ruby and Go.
There are three people I know that are, or have been, close to death
in the last year. One had a double mastectomy last fall when she
discovered she had breast cancer. Another fell down confused one day
earlier this year and discovered she had stage 4 brain cancer. And
the third got taken to hospital a few weeks ago because, it turns out,
she's alcoholic and has pretty much destroyed her liver.
One is getting better; her hair is growing out after chemo and
radiation, she's playing music again (she's incredibly talented), and
seems to be nearly endlessly positive (at least around me). Another
is taking things day by day, travelling when she can, trying to eat
(her sense of taste has been destroyed by the radiation treatment),
hanging out with her grandchildren. And the third has been in detox
for a few weeks now, and has a long road in front of her if she's
One was a close friend, then we lost touch, and now we make a point of
seeing each other regularly; it's not as often as I'd like because we
each have our own commitments, but it's wonderful to talk to her again
after so long because she's funny and talented and just a righteous
pleasure to be around. Another lives far away, and for a long time
has been someone I knew about rather than knew; she's a good person,
but our lives are separate, joined only by the people we have in
common. And the third was always a wonderful, funny person to talk to
at the social occasions we ended up at together, and I loved her
writing when she kept a blog, but I could in no way say I knew her.
One my wife and I have been able to help, at least in a material
way. Another, my wife and I have helped someone else be able to travel
to see her. And the third...well, the whole problem has only just
emerged, and we have no idea what to do, or what will help, or the
prospects of her being around long enough to help.
I've come to learn the way I react to things like this: shock and
numbness for an hour or two, then being surprised when I burst into
tears, then long weeks of worrying. I've begun to be wary of hearing
about someone I haven't thought of in a while, because this is when
things and people fall apart and some days it feels like the news is
never good. And I've started to think about why we go to funerals,
and the way grief and mourning and remembering are built somehow into
our DNA, our shared heritage with all the other animals that cherish
and love and mourn and, in their turn, die.
Tonight I went out to the local park with the scope. I had a bit
better list than last time and stuck to it, and ad the end of the
night I was able to shut down & be home in 10 minutes. Not bad at
So: Quick look at Saturn to start with, before it set beneath the
trees. Very nice.
Managed to split Double-Double in Lyra, but I had to use the 6mm
Radian to do it. South pair easier to split than the north pair.
Followed ISS with the scope (17mm, 71x) and man, that was neat.
Omicron Draco: double star, yellow and green/blue. Not my thing,
double stars, but I do like the ones that resemble Earth & the sun
(blue + yellow). Colours on this one were more subtle, though.
M56: Faint. No sign of resolution.
NGC 6939: OC in Cepheus. Took a while to track this down, as it was
a lot fainter than I expected. Got a sketch.
NGC 6543, the Cats-Eye Nebula: Saw this straight off, an obviously
non-stellar object. Faint blue. Neat.
M92: Aw! Looks like photo of a spiral galaxy. Liked it better
than M13, which I looked at next.
NGC 6229: GC in Hercules. Faint like a Q-Tip, and no resulution at
NGC 6709: OC in Lyra. Nice! Big, sparse, and kind of reminds m of
a fish shape. Mentioned in "Annals of the Deep Sky", which I'm enjoying.
Packed it in at 12.15am. Overall, a lot better than last time.
The cloud sensor is behaving badly of late, refusing to post
updates. I suspect it's overheating, but it's hard to get data when
the thing is across town. I had set up an SSH tunnel back to home
so I could connect to it, but did the classic fail when I tried to
convert it from running in tmux to running under supervisord and it
didn't work and now I can't get back in. Waiting to get back to my
inlaws' so I can debug it more properly.
But while it was working, I got it logging to InfluxDB (running on
my home server over) over said SSH tunnel. Pretty sweet! And it
was not hard to import all the previous stats I had as well.
And so but InfluxDb was running in a Docker container on the home
machine, along with Grafana. Which of course led to running
Telegraf on a lot of things, and collectd where I couldn't (hello
OpenWRT). Which now OMG the stats. And the annotations.
But holy crap, Docker and IPv6 is a giant whirlwind of
Because the kids' old laptop was, frankly, shitty, we got them a new
cheap laptop and I did some complicated surgery resulting in two
swapped hard drives and a new install of Debian 8 on Zombie, my home
server, which is now managed nearly entirely by Chef. This was a
lot of trouble -- setting up everything under a medium-weight config
mgt system like Chef, I mean -- but I think it was worth it.
I've got the first breadboarded version of an Arduino weather
station going, currently logging stats to InfluxDB. It is (/me
checks Grafana) (/me checks on Arduino serial port for transmitter
because receiver serial port acting badly) 67% humidity, 19.1 deg C
and 1011.5 hPa and falling.
Today was another visit to my in-laws for lunch, and we got the cloud
sensor fully deployed for the first time! Check it out:
The enclosure is a plastic container, one of a set of 6 I got from London
Drugs for $15. It's got clamps to keep the lid on tight, and a rubber
gasket to keep water out. I cracked the lid of the first one I worked
when I drilled a hole without proper support, but my father-in-law has
a drill press (one of the many advantages of marrying well) and we
were able to get a second container set up.
We drilled a hole in the top for the sensor can to poke through and
sealed it with silicone. I thought that would hold it in place (ie,
that it would act as a glue as well) but no dice. Instead, we placed
the first container -- which is smaller and was designed to nest
inside -- in there as well. It sat under the sensor and kept it in
place. It took a little bit of arranging to route all the USB cables,
but we got it sorted out in the end.
My father-in-law crafted a shelf for the container out of some cedar
planking and some steel bits he had lying around. We screwed that to
the patio, then the container to the shelf. Plug in the Raspberry Pi
to a nearby outlet, and boom -- we had data!
...until yesterday at noon, that is; that's when the last update was
posted. (I'm starting to regret not having set up an SSH tunnel back
home.) I'm not sure at this point what happened, but overheating
seems a reasonable guess. We made some holes on the bottom of the
container for ventilation, but it's a concern. I think the Pi itself
should be good for longer periods -- I had it running outside my house
here for days in a row, just not inside the container. I'll have to
power cycle it and see what happens.
On the data front, I realized a while back that the CSC is only
updated twice a day...which means downloading every hour is overkill.
The script still runs every few hours, but now it checks whether the
newly-downloaded chart's checksum is the same as the
previously-downloaded example; if it is, it throws it away. And I've
got the analysis script set up to convert the colours of individual
squares in the CSC into cloud coverage predictions. Next step is to
figure out how to store this info so I can work on it. All in all,
not a bad bit of progress.
Today we visited my in-laws for our usual Sunday lunch, and I took
along the cloud sensor to demonstrate for my father-in-law. He's a
retired millwright with a strong sense of curiosity, so he enjoyed
seeing it a lot. We set it up by a gazebo (?) he has in his back
yard, with the Pi hanging from the wall and the sensor clipped to the
After some clouds rolled out, it got seriously sunny. The data
For the record, the ambient temperature is not really to be trusted.
It kept reading in the mid-thirties, but a nearby thermometer showed
nothing higher than 25 all day. And when the sensor is deployed here,
it seems to register about a 10 degree higher temp for the sky;
doubtless it's the combination of the wide FOV (90 degrees) and the
narrow slice of the sky it can see from my front porch.
The day before I'd bought a plastic container to use as an enclosure,
so we kicked around ideas about how to make it work. I'd picked it
because the clipping lid seemed like it would keep out the weather
quite well, especially since the lid completely overhangs the
container -- but I'd forgotten to think about shedding water. The lid
is recessed maybe half a centimetre, and any rain would just pool in
there -- not what I want. We agreed that the enclosure definitely
needs at least a flat roof, and ideally something rounded that would
let rain roll off. A plastic bowl with a flat bottom would do the
So what worked well?
Wireless worked without a hitch -- very happy with this, as I wasn't
sure it would reach from inside to the back yard
Data logging worked too, including status updates when rebooting
Good to kick around ideas
Got script working to analyze the CSC and turn it into hour-by-hour
predictions; some initial ideas about how to store that data, and
how to visualize that and logged temperature data
Committed API key to git without noticing; had to generate a new one
Desolder header pins so eval board can be mounted with the sensor
poking through the lid of whatever
The MLX90614 eval board arrived at last. I managed to get the header
pins soldered on without melting much, and hooked up to the Raspberry
Pi. The default firmware logs the temperature in Fahrenheit to the
serial port every tenth of a second, and I was able to read it out
with screen without a problem.
Next up was adjusting the firmware slightly to slow it down a bit,
write out ambient temperature as well, and to switch to Celsius. (Cue
feeling bad about not using Kelvin.) That took all of ten minutes, so
huzzah for that.
Finally, I set up an account at ThingSpeak.com (free as in beer,
plus free as in freedom software, plus export of data whenever
you like), and set up a stupid simple script to send data both to a
CSV file and to, you know, the cloud. The result? SWEET, SWEET
Clockwise from top left, that's: sky temperature (sensor pointing
up), ambient temperature, and the delta (ambient - sky). It's
interesting to see how smooth the ambient temp is compared to the
Oh, and here's a shot of how it looks sitting on top of the BBQ
Not at all weatherproof, of course, so that comes next. With the way
I soldered the header pins on (all pointing up), that's going to be a
challenge; I may de-solder them and have the top as bare as possible.
That would let me (say) drill a hole in a Tupperware container, push
the sensor through, then epoxy around it for waterproofing. As for
the Pi...not too sure. I may try to mount it out of the weather, then
run the sensor out under the sky.
I've ordered the MLX90614 eval board plus a Redboard from Sparkfun
(can't have too many Arduinos!) (arriving shortly)
I've done some initial playing around with the Arduinos and the
accelerometer included in the Sunfounder kit, as a standin for the
MLX90614. It amazes me how easy it is to get started with all of
I've grabbed one of the Raspberry Pis around the house and got it
logging data over the USB serial port from the Arduino
Even got WiFi working on the Pi; it's an older one that doesn't have
it built-in, but I've got an extra Asus N13 lying around that seems
to work well
I've modified ttylog to include the date in its output; may not
seem like much, but I'm no C programmer
Rough plan right now is:
Have the Arduino log once a second to the serial port; that's far
too much data, but at least it'll be easy to see if it's working
Log that with ttylog; it'll be running under supervisor, and will
log for an hour before exiting (and starting up again)
Once an hour, assuming WiFi works in its final location, try to rsync
the log files home; if not working, pick up the log files manually
Put it all in a waterproof case of some sort, and find a place to
plug it in at my in-laws (they've got a nice big yard)
Yesterday at work, I set up the live cast of the CRS-8 launch on the
big TVs in the kitchen:
There were a lot of people watching with me. The footage from the
rocket was absolutely amazing...but of course, the landing was
spectacular. I found myself swearing with amazement, over and over
again, I'm sure much to the amusement of my coworkers.
I'm fortunate enough that there are a string of evening ISS flyovers
happening for me right now, and Heavens Above has tracking info for
the Dragon as well. Sure enough, there was one tonight just before
I went out to a local baseball field with 10x50 binoculars; nothing at
all resembling a dark sky, but of course it was enough to catch the
ISS as it rose. At first I thought it might be the Dragon -- it was
faint and looked smaller through the binos than I've seen it
previously. Of course, most times I see it much higher in the sky
(and thus closer, larger & brighter). As it rose, it became apparent
that it was indeed the ISS. It went slowly through the Pleiades (now
that was pretty!) and right by Menkalinan (Beta Aurigae). But where
I stopped following the ISS with the binos and let it rise out of view
-- and just a few seconds later, following almost exactly on the same
track, a dim satellite came into view. Dragon! It was maybe two
degrees further from the ISS than would fit in the FOV of the binos.
I followed it along until nearly the zenith, then tried to catch it
with the naked eye. Sure enough, there it was -- maybe 4th mag or so,
much dimmer than the ISS but still visible. Although it was harder to
see this way, it was so much more wonderful -- it was so obviously in
pursuit of the ISS. It's amazing to see a supply ship on its way to a
space station. I like living in the future.
So after doing some digging around, I think there's a simpler approach
than using Peltier coolers, and that's using an IR temperature
sensor. This guy has built his own using this approach, though
he's using Arduino controllers to read them. That led me down the
Arduino path, and after a lot of reading I think I've got an approach
that might work.
Sparkfun sells the MLX90614 temperature sensor in a couple of
different formats: bare sensor, or on an eval board. After reading
that tutorial, my understanding is:
I can connect the evaluation board to an FTDI cable/breakout board,
and hook that up via USB to the Pi. The default sketch in the
evaluation board will give me temperatures in Fahrenheit once per
second over a serial port. Later, I can change the sketch by using
the Arduino IDE. Pro: Quick to start, USB is dirt simple, and I
don't need a RedBoard or similar. Cons: Not as flexible as it would
be if connected to RedBoard, since that would give it lots of
expansion possibilities (humidity sensor, motion-activated
potato cannon, etc.)
I can connect the evaluation board to the Pi via I2C. Can still
reprogram the sketch later. Pro: Not really sure. Cons: Have to
build my own I2C connector....not that hard, from what I can
see, but I'm a newbie.
I can get the bare sensor (no eval board) and hook it up to the Pi
via I2C. Pro: Unsure. Cons: Much more fiddly than anything I've
I can jump right in to Arduino and get an Inventor's Kit. I
can use the bare sensor (as shown in the tutorial -- start on
breadboard, package it up somehow when I'm confident it's working),
or the eval board (doing something like [this example4), using I2C
in both cases. Pro: Lots of room for expansion, Arduinos are fun,
etc. Cons: Will take me a while to get up to speed.
Assuming I've got all that right...my inclination is to start with the
FTDI breakout board and USB; that'll make the learning curve easier,
and I can get the Inventor's kit later on.
I've asked on the SparkFun forum whether I've got all this
right...time will tell. But getting quickly started with the USB
seems like a good way to start.
For a while now I've been wondering idly how I could measure
cloudiness. My goal is to both track how cloudy it is now (and over
time), and to compare actual cloudiness with predictions from
A few days ago I came across an approach that I think might work.
This person measured the current coming from a Peltier cooler
when exposed to the night sky. The difference in temperature between
the ground-facing side (warm) and the sky-facing side (cold) varied
depending on whether it was cloudy (less difference in temp == less
current) or not (greater difference in temp == more current). It
occurred to me that I could use a Raspberry Pi I've got lying around
to take that same approach.
Since then I've been browsing around, and here's what I've found: