Algorithm for implementing a dead man's switch.

Mar 04 2018

So, you're probably wondering why I'm posting this, because it's a bit off of my usual fare.  The reason is I think it would be useful to make available a fairly simple algorithm for implementing a general purpose dead man's switch in whatever language you want, which is to say a DMS that could conceivably do just about anything if it activated.

But what's a dead man's switch?  Ultimately, it's a mechanism that has to be manually engaged at all times if you want something to happen, and if that switch turns off for some reason, something else happens (like a failsafe).  A good example of this is the bar on the handle of a power lawnmower you have to hold down so it'll move while the engine's running.  If you let go of the bar the engine keeps running but the lawnmower doesn't keep rolling forward.  Another example can be found in locomotives; the conductor has to hold down a switch or lever so the engine will pull the train, and if that lever is ever let go (say the engineer has a heart attack or is otherwise incapacitated) the throttle closes and the train will grind to a halt.  More along the lines of what I'll be talking about are the watchdogs found in industrial controllers and realtime operating systems.  While running normally a software process inside the device flips a bit somehow - say, writing a 0 into a certain device node.  If the underlying hardware ever finds that the bit didn't get flipped within a certain period of time it reacts somehow to fix things (for example, it might reboot in an attempt to un-stick the gizmo).

Making offline backups of a Linux machine using Backblaze.

Jan 14 2018

As frequent readers may or may not remember, I rebuilt my primary server last year, and in the process set up a fairly hefty RAID-5 array (24 terabytes) to store data.  As one might reasonably expect, backing all of that stuff up is fairly difficult.  I'd need to buy enough external hard drives to fit a copy of everything on there, plus extra space to store incremental backups for some length of time.  Another problem is that both Leandra and the backup drives would be in the same place at the same time, so if anything happened at the house I'd not only not have access to Leandra anymore, but there's an excellent chance that the backups would be wrecked, leaving me doubly screwed.

Here are the requirements I had for making offsite backups:

  • Backups of Leandra had to be offsite, i.e., not in the same state, ideally not on the same coast.
  • Reasonably low cost.  I ran the numbers on a couple of providers and paying a couple of hundred dollars a month to back up one server was just too expensive.
  • Linux friendly.
  • My data gets encrypted with a key only I know before it gets sent to the backup provider.
  • A number of different backup applications had to support the provider, in case one was no longer supported.
  • Easy to restore data from backup.

After a week or two of research and experimentation, as well as pinging various people to get their informed opinions, I decided to go with Backblaze as my offsite backup provider, and Duplicity as my backup software.  Here's how I went about it, as well as a few gotchas I ran into along the way.

Quick and dirty copies of website with wget.

Jan 14 2018

Let's say there's a website that you want to make a local mirror of.  This means that you can refer to it offline, and you can make offline backups of it for archival.  Let's further state that you have access to some server someplace with enough disk space to hold the copy, and that you can start a task, disconnect, and let it run to completion some time later, with GNU Screen for example.  Let's further state that you want the local copy of the site to not be broken when you load it in a browser; all the links should work, all the images should load, and so forth.  One of the quickest and easiest ways to do this is with the wget utility.

Automating deployment of Let's Encrypt certificates.

Jan 06 2018

A couple of weeks back, somebody I know asked me how I went about deploying SSL certificates from the Let's Encrypt project across all of my stuff.  Without going into too much detail about what SSL and TLS are (but here's a good introduction to them), the Let's Encrypt project will issue SSL certificates to anyone who wants one, provided that they can prove somehow that they control what they're cutting a certificate for.  You can't use Let's Encrypt to generate a certificate for google.com because they'd try to communicate with the server (there isn't any such thing but bear with me) google.com to verify the request, not be able to, and error out.  The actual process is complex and kind of involved (it's crypto so this isn't surprising) but the nice thing is that there are a couple of software packages out there that automate practically everything so all you have to do is run a handful of commands (which you can then copy into a shell script to automate the process) and then turn it into a cron job.  The software I use on my systems is called Acme Tiny, and here's what I did to set everything up...

Quick and easy SSH key installation.

Dec 27 2017

I know I haven't posted much this month.  The holiday season is in full effect and life, as I'm sure you know, has been crazy.  I wanted to take the time to throw a quick tip up that I just found out about which, if nothing else, will make it easier to get up and running on a Raspberry Pi that you've received as a gift.  Here's the situation:

You have a new account on a machine that you want to SSH into easily.  So, you want to quickly and easily transfer over one or more of your SSH public keys to make it easier to log in automatically, and maybe make running Ansible a bit faster.  Now, you could do it manually (which I did for many, many years) but you'll probably mess it up at least once if you're anything like me.  Or, you could use the ssh-copy-id utility (which comes for free with SSH) to do it for you.  Assuming that you already have SSH authentication keys this is all you have to do:

[drwho@windbringer ~]$ ssh-copy-id -i .ssh/id_ecdsa.pub pi@jukebox
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_ecdsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out
    any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now
    it is to install the new keys

pi@jukebox's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'pi@jukebox'"
and check to make sure that only the key(s) you wanted were added.

Now let's try to log into the new machine:

[drwho@windbringer ~]$ ssh pi@jukebox
Linux jukebox 4.9.70-v7+ #1068 SMP Mon Dec 18 22:12:55 GMT 2017 armv7l

The programs included with the Debian GNU/Linux system are free software;

# I didn't have to enter a password because my SSH pubkey authenticated me
# automatically.
pi@jukebox:~ $ cat .ssh/authorized_keys
ecdsa-sha2-nistp521 AAAAE....

You can run this command again and again with a different pubkey, and it'll append it to the appropriate file on the other machine (~/.ssh/authorized_keys).  And there you have it; your SSH pubkey has been installed all in one go.  I wish I'd known about this particular trick... fifteen years ago?

Administering servers over Tor using Ansible.

Dec 02 2017

Difficulty rating: 8.  Highly specific use case, highly specific setup, assumes that you know what these tools are already.

Let's assume that you have a couple of servers that you can SSH into over Tor as hidden services.

Let's assume that your management workstation has SSH, the Tor Browser Bundle and Ansible installed.  Ansible does all over its work over an SSH connection, so there's no agent to install on any of your servers.

Let's assume that you only use SSH public key authentication to log into those servers.  Password authentication is disabled with the directive PasswordAuthentication no in the /etc/ssh/sshd_config file.

Let's assume that you have sudo installed on all of those servers, and at least one account can use sudo without needing to supply a password.  Kind of dodgy, kind of risky, mitigated by only being able to log in with the matching public key.  That seems to be the devopsy way to do stuff these days.

Problem: How to use Ansible to log into and run commands on those servers over the Tor network?