Restarting a Screen session without manual intervention.

Jun 11, 2017

To keep the complexity of parts of my exocortex down I've opted to not separate everything into larger chunks using popular technologies these days, such as Linux containers (though I did Dockerize the XMPP bridge as an experiment) because there are already quite a few moving parts, and increasing complexity does not make for a more secure or stable system.  However, this brings up a valid and important question, which is "How do you restart everything if you have to reboot a server for some reason?"

A valid question indeed.  Servers need to be rebooted periodically to apply patches, upgrade kernels, and generally to blow the cruft out of the memory field.  Traditionally, there are all sorts of hoops and gymnastics one can go through with traditional initscripts but for home-grown and third party stuff it's difficult to run things from initscripts in such a way that they don't have elevated privileges for security reasons.  The hands-on way of doing it is to run a GNU Screen session when you log in and start everything up (or reconnect to one if it's already running).  This process, also, can be automated to run when a system reboots.  Here's how:

Notes on using the Kryoflux DiskTool utility to make archival images of floppy disks.

May 28, 2017

Some time ago, I found myself using a Kryoflux interface and a couple of old floppy drives that had been kicking around in my workshop for a while to rip disk images of a colleague's floppy disk collection.  It took me a day or two of screwing around to figure out how to use the Kryoflux's software to make it do what I wanted.  Of course, I took notes along the way so that I would have something to refer back to later.  Recently, I decided that it would probably be helpful to people if I put those notes online for everyone to use.  So, here they are.

Website file integrity monitoring on the cheap.

May 28, 2017

A persistent risk of websites is the possibility of somebody finding a vulnerability in the CMS and backdooring the code so that commands and code can be executed remotely.  At the very least it means that somebody can poke around in the directory structure of the site without being noticed.  At worst it would seem that the sky's the limit.  In the past, I've seen cocktails of browser exploits injected remotely into the site's theme that try to pop everybody who visits the site, but that is by no means the nastiest thing that somebody could do.  This begs the question, how would you detect such a thing happening to your site?

I'll leave the question of logfile monitoring aside, because that is a hosting situation-dependent thing and everybody has their own opinions.  What I wanted to discuss was the possibility of monitoring the state of every file of a website to detect unauthorized tampering.  There are solutions out there, to be sure - the venerable Tripwire, the open source AIDE, and auditd (which I do not recommend - you'd need to write your own analysis software for its logs to determine what files, if any, have been edited.  Plus it'll kill a busy server faster than drilling holes in a car battery.)  If you're in a shared hosting situation like I am, your options are pretty limited because you're not going to have the access necessary to install new packages, and you might not be able to compile and install anything to your home directory.  However, you can still put something together that isn't perfect but is fairly functional and will get the job done, within certain limits.  Here's how I did it:

Most file monitoring systems store cryptographic hashes of the files they're set to watch over.  Periodically, the files in question are re-hashed and the outputs are compared.  If the resulting hashes of one or more files are different from the ones in the database, the files have changed somehow and should be manually inspected.  The process that runs the comparisons is scheduled to run automatically, while generation of the initial database is normally a manual process.  What I did was use command line utilities to walk through every file of my website, generate a SHA-1 hash (I know, SHA-1 is considered harmful these days; my threat model does not include someone spending large amounts of computing time to construct a boobytrapped index.php file with the same SHA-1 hash as the existing one; in addition, I want to be a good customer and not crush the server my site is hosted on several times a day when the checks run), and store the hashes in a file in my home directory.

Getting stuck upgrading Bolt and what to do about it.

May 01, 2017

UPDATE - 20170512 - More SQL surgery.

So, as you've no doubt noticed I've been running the Bolt CMS to power my website for a while now.  I've also mentioned once or twice that I've found it to be something of a finicky beast and doing anything major to it can be something of an adventure.  I tried to upgrade my site last week (tonight, by the datestamp on this post) and had to restore from backup yet again because something went sideways.  That something was the upgrade process going wrong and throwing an exception because of something in the cache directory, where Bolt temporarily stores HTML files rendered from templates used to make pages that your web browser displays.

As it turned out, the upgrade process was choking on the old cache directories created and used by v2.x of the Bolt CMS.  Here is the upgrade process that I used:

  • BACK UP YOUR SITE.
  • Log into your web hosting provider's server via SSH.
  • Download the latest version of the flat file structure build of Bolt.
  • If you didn't back up your website, BACK UP YOUR WEB SITE.
  • cd ~/my.website.here
  • If you didn't back up your website and things go pear-shaped, it's your fault.  Don't say I didn't warn you.
  • Uncompress the new version of Bolt you just downloaded: tar xvfz ~/bolt-latest-flat-structure.tar.gz --strip-components=1
  • Try running the upgrade: php app/nut setup:sync
  • If it throws an exception on you, erase the entire on-disk cache.  Don't worry, it'll be rebuilt as people visit your site: rm -rf app/cache/*
  • Try running the upgrade again: php app/nut setup:sync
  • It should complete successfully.  If it doesn't you may need to do the following two things before re-running the upgrade command again:
    • mkdir -p app/cache/production/data/
    • chmod -R 0775 app/cache/
    • If you still have problems, jump into the Bolt CMS Slack chat and politely ask good questions: https://boltcms.slack.com/
  • If the command finishes normally, try opening the frontpage of your website.  It should be up and running.
  • If you can see the frontpage of your website, try logging in.  You should be able to.
  • Try making a test post with a new entry.  Be sure to test saving the post partway through.  You do save your work every few minutes, don't you?
  • Success.

Special thanks to Bob and thisiseduardo in the Bolt CMS Slack chat for their assistance and hand-holding while I stumbled around trying to make this hapen.

Gargantuan file servers and tiny operating systems.

Apr 29, 2017

We seem to have reached a unique point in history: Available to your average home user are gargantuan amounts of disk space (8 terabyte hard drives are a thing, and the prices are rapidly coming down to widespread affordability) and enough processing power is available for the palm of your hand that makes the computational power that put the human race on the moon compare in the same was that a grain of sand does to a beach.  For most people, it's the latest phone upgrade or more space for your media box.  For others, though, it poses an unusual challenge: How to make the best use of the hardware without wasting it needlessly.  By this, I mean how one might build a server that doesn't result in wasted hard drive space, wasted SATA ports on the mainboard, or having enough room to put all of that lovely (and by "lovely" I really mean "utterly disorganized") data that accumulates without even trying.  I mentioned last year that I rebuilt Leandra (specs in here) so I could work on some machine learning and search engine projects.  What I didn't mention was that I had some design constraints that I had to follow so that I could get the most out of her.

To get the best use possible out of all of those hard drives I had to figure out how to structure the RAID, where to put the guts of the Arch Linux install, and most importantly figure out how to set everything up so that if Leandra did blow a hard drive the entire system wouldn't be hosed.  If I partitioned all of the drives as described here and used one as the /boot and / partitions, and RAIDed the rest, if the first drive blew I'd be out an entire operating system.  Also, gauging the size of the / partition can be tricky; I like to keep my system installs as small as possible and add only packages that I absolutely need (and ruthlessly purge the ones that I don't use anymore).  20 gigs is way too big (currently, Leandra's OS install is 2.9 gigabytes after nearly a year of experimenting with this and that) but it would leave room to grow.

Decisions, decisions.

So, what did I finally decide on?

OpenVPN, easy configuration, and that damned ta.key file.

Apr 15, 2017

Now that ISPs not selling information about what you do and what you browse on the Net is pretty much gone, a lot of people are looking into using VPNs - virtual private networks - to add a layer of protection to their everyday activities.  Most of the time there are two big use cases for VPNs: Needing to use them for work, and using them to gain access to Netflix content that isn't licensed where you live.  Now they may as well be a part of everyday carry.

So: Brass tacks.  Here's a quick way to set up your own VPN server, as well as a solution to a problem that frustrated me until very recently.  For starters, unless you're an experienced sysadmin don't try to freestyle the setup.  There is an excellent script on Github called openvpn-install that will do all of the work for you (including adding and deleting users) in less than a minute.  Use it to do the work for you.  Please.  Also, if you build an OpenVPN server, consider going in with a couple of friends on the cost.

Chances are you're running either Windows or Mac OSX (Linux and BSD users, you know what to do) so you'll need an OpenVPN client on the users' end.  This means that you want to run either the Windows version of the OpenVPN client or an OSX client like Tunnelblick.  However, these clients assume that you're just loading an all-in-one configuration file, called an .ovpn file.  If you've never done it before they're remarkably tricky to build but they're basically a copy of the OpenVPN client.conf with all of the crypto keys embedded in special stanzas.  It took me a lot of fumbling and searching but I eventually figured out how to reliably make them.  To save you some time here's a copy of the one I use with all the unique stuff removed from it.  If you open it in a text editor you'll notice a couple of things: First, the very first non-commented line says that it's for the client and not the server.  Second, I have it configured to use TCP and not UDP.  This is so that you don't have to reconfigure the firewall you're behind to get your traffic through.  Keep it simple, trust me on this.  Third, the ca, cert, and key directives are commented out because those keys are embedded at the end of the file.  Fourth, I have tls-auth enabled so that all traffic your server will handle is authenticated for better security.

If you freestyle (that is, build by hand) your OpenVPN server, you'll need to keep in mind the following things: