Using Ansible to restart a bunch of services running under systemd in --user mode.

Dec 02 2019

Let's say that you have a bunch of servers that you admin en masse using Ansible.  You have all of them listed and organized in your /etc/ansible/hosts file.  Let's say that each server is running a system service (like my Systembot) running under systemd in --user mode.  (Yes, I'm going to use my exocortex-halo/ repository for this, because I just worked out a good way to keep everything up to date and want to share the technique for everyone new to Ansible.  Pay it forward, you know?)  You want to use Ansible to update your copy of Systembot across everything so you don't have to SSH into every box and git pull the repo to get the updates.  A possible Ansible playbook to install the updates might look something like this:

Challenge accepted: Archiving a Mastodon account with Huginn

Nov 17 2019

Last weekend I was running short of stuff to hack around on and lamented this fact on the Fediverse.  I was summarily challenged to find a way to archive posts to the Fediverse in an open, easy to understand data format that was easy to index, and did not use any third party services (like IFTTT or Zapier).  I thought about it a bit and came up with a reasonably simple solution that uses three Huginn agents to collect, process, and write out posts as individual JSON documents to the same box I run that part of my exocortex on.  This is going to go deep geek below the cut so if it's not your cup of tea, feel free to move on to an earlier post.

Experimenting with btrfs in production.

Oct 19 2019

EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space.  Also did a couple of experiments with rebalancing btrfs and monitored how long it took.

A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up.  I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places.  The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy.  There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis.  There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed).  This was a problem that required fairly major surgery to fix, on both hardware and software.

By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra.  This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration.  This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error.  If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).

Using Huginn to get today's weather report.

Aug 03 2019

A common task that people using Huginn set up as their "Hello, world!" project is getting the daily weather report because it's practical, easy, and fairly well documented.  However, the existing example is somewhat obsolete because it references the Weather Underground API that no longer exists, having been sunset at the end of 2018.  Recently, the Weather Underground code in the Huginn Weather Agent was taken out because it's no longer usable.  But, other options exist.  The US National Weather Service has a free to use API that we can use with Huginn with a little extra work.  Here's what we have to do:

  • Get the GPS coordinates for the place we want weather reports for.
  • Use the GPS coordinates to get data out of the NWS API.
  • Build a weather report message.
  • E-mail it.

As happens sometimes, the admins of the NWS API have imposed an additional constraint upon users accessing their data: They ask that the user agent string of whatever software you use be unique, and ideally include an e-mail address they can contact you through in case something goes amiss.  This isn't a big deal.

This tutorial assumes that you've worked with Huginn a bit in the past, but if you haven't I strongly suggest that you read my earlier posts to familiarize yourself.

Okay.  Let's get started.

An annoying problem solved: Accessing JSON documents with an API.

Jul 28 2019

I spend a lot of time digging around in other people's data.  If I'm not hunting for anything in particular then it's a bit of a crapshoot, to be honest, if only because you never know what you're in for.  You can pretty much take it to the bank that if you didn't assemble it yourself, you can't count on it being complete, well formed, or anything approximating the output of a human being (it usually came out of a database, but I think you see what I'm getting at).  Sometimes, if I'm really lucky I'll just get hold of a JSON dump of the database, which to be fair is better than nothing when there isn't even an API to use.  From time to time I'll make an attempt at fitting the data into a database of some kind, sometimes MySQL, sometimes SQLite, or occasionally an API layer like Sandman2.  This is all well and good, but it winds up being more of an adventure than I'm looking for.  I'd much rather be Indiana Jones prowling around in the temple than Rambo going through a preparation montage because Indy was actually getting stuff done.

Wow, this article went a little off the rails.  I was never good at writing intros to new code... anyway.

Got some new hardware installed.

Jul 20 2019

For a couple of years now, I've had my eye on the community of people who've had RFID or NFC chips implanted somewhere in their bodies, usually in the back of the hand.  If you've ever used a badge to unlock a door at work or tapped your phone on a point-of-sale terminal to buy something, you've used one of these two technologies in your everyday life to do something useful.  What I've wanted to do for a while was use an implanted chip as a second authentication factor to my servers for better security.  As for why I couldn't just use something like a key fob or a card or something.. there were a bunch of reasons, most of them having to do with only being able to find what I needed in bulk or the cost being too high because the equipment was aimed at corporate IT departments, where they have a need to crank out a couple of dozen ID badges an hour.  There is also the fact that, while I've been curious about various forms of body modification over the years I never really got into into them for a couple of reasons that probably aren't terribly interesting so why not do something useful?

Due to the fact that not everybody is going to be okay with me talking about an elective medical procedure, I'm going to put the rest of this article after the fold.

Sometimes the old ways may be best.

Feb 02 2019

A couple of weeks back, I found myself in a discussion with a couple of friends about searching on the Internet and how easy it is to get caught up in a filter bubble and not realize it.  To put not too fine a point on it, because the big search engines (Google, Bing, and so forth) profile users individually and tailor search results to analyses of their search histories (and other personal data they have access to), it's very easy to forget that there are other things out there that you don't know about for the simple reason that they don't show stuff outside of that profile they've built up.  If you're a hardcore code hacker you might find it very difficult to find poetry or the name of a television show you saw once unless you take fairly drastic action.  The up-side of this profiling is that, inside of your statistical profile search results are great.  You can find what you need, when you need it.  But outside of that?  Good luck.

The point of the discussion was that there were ways that we could escape this filter bubble through application of self-hosted software and a little cooperation.

Ironically, searching through my conversation history I can't seem to find the thread in question so I'm relying entirely upon on-board storage (as it were).  So, go ahead and laugh while I geek out.  First, a little bit of Internet history.

Organizing a data hoard with YaCy.

Feb 02 2019

 It should come as little surprise to anyone out there that I have a bit of a problem with hoarding data.  Books, music, and of course files of all kinds that I download and read or use in a project for something.  Legal briefs, research papers (arXiv is the bane of my existence), stuff people ask me to review, the odd Humble Bundle... So much so that a scant few years ago I rebuilt Leandra to better handle the volume of data in my library.  However, it's taken me this long to both figure out and get around to making it easier to find anything in all that mess.  If I can't find it, I can't do anything with it, or even figure out what I do or don't have.  I also don't often have console access so it's not as if I can SSH in and grep for what I need.  I use Nginx as a web server on Leandra so actually getting access to files when I need them is trivial.