It's going on summer in the Bay Area, which means that it's warming up a bit both outside and inside (because air conditioning is Not A Thing out here). That, coupled with the not inconsiderable research infrastructure I have at home has left me wondering and worrying about just how hot my office gets during the day while I'm working. Now, I could just put a simple little thermometer on my shelf (and I did) but my concerns are a bit bigger than that. What happens if my office temperature reaches a critical point and servers start melting down on me? I've dealt with heat damage in the past and don't particularly care to shell out a grand or so to replace parts that flatlined because I was away from the house and couldn't respond in time. That, and the fact that I need to keep my mind busy while I'm stuck in quarantine if I'm going to be honest, are the reason why I built yet another weird-assed exocortex project: A relatively simple hardware monitor connected to a Raspberry Pi, and a bot that listens for commands and responds with what it can detect of the local temperature or humidity when I send it a message.
Longtime readers have no doubt observed that I plug a lot weird shit into my exocortex - from bookmark managers to card catalogues to just about anything that has an API. Sometimes this is fairly straightforward; if it's on the public Net I can get to it (processing that data is a separate issue, of course). But what about the stuff I have around the lab? I'm always messing with new toys that are network connected and occasionally useful. The question is, how do I get it out of the lab and out to my exocortex? Sometimes I write bots to do that for me, but that can be kind of clunky because a lot of stuff doesn't necessarily need user interaction. I could always poke some holes in my firewall, lock them to a specific IP address, and set static addresses on my gadgets. However, out of necessity I've got several layers of firewalls at home and making chains of port forwards work is a huge pain in the ass. I don't recommend it. "So, why not a VPN?" you're probably asking.
I'd been considering VPNs as a solution. For a while I considered the possibility of setting up OpenVPN on a few of my devices-that-are-actually-computers and connecting them to my exocortex as a VPN concentrator. However, I kept running into problems with trying to make just a single network port available over an OpenVPN connection. I never managed to figure it out. Then part of me stumbled across a package called Nebula, originally developed by Slack for doing just what I wanted to do: Make one port inside available to another server in a secure way. Plus, at the same time it networks all of the servers its running on together. Here's how I set it up.
Not too long ago I was noodling over a problem: I wanted to break up the scheduling queues in Huginn to make my fleets of agents a little more efficient when the execute. The best way I could think of was to make some of the schedules stochastic - periodically have an agent roll some dice and depending on what comes up decide whether or not to trigger the agents downstream. So, of course I started looking for a random number generator that would basically roll 1d10. However, the Liquid templating language that Huginn uses internally doesn't have any function to do this and I didn't feel like bodging one together even though it would probably work well enough for my purposes. So, off to my local Searx instance to see what I could scare up.
It was then that discovered (or re-discovered, maybe) some interesting things about the meta-search engine in question.
Throughout this series I've shown you how to set up a Matrix server and client using Synapse and Riot, and make it much more robust as a service by integrating a database server and a mechanism for making VoIP more reliable. Now we'll wrap it up by doing something neat, building a simple agent network in Huginn to post what I'm listening to into a Matrix Room. I have an account on libre.fm that my media players log to which we'll be using as our data source. Of course, this is only a demonstration of the basic technique, you can, in theory plug whatever you want into a Matrix server because the API was designed to be extensible.
We're going to assume that you've already set up a Matrix server and have an account on it, and that you have access to a working Huginn install.
Let's say that you have a bunch of servers that you admin en masse using Ansible. You have all of them listed and organized in your /etc/ansible/hosts file. Let's say that each server is running a system service (like my Systembot) running under systemd in --user mode. (Yes, I'm going to use my exocortex-halo/ repository for this, because I just worked out a good way to keep everything up to date and want to share the technique for everyone new to Ansible. Pay it forward, you know?) You want to use Ansible to update your copy of Systembot across everything so you don't have to SSH into every box and git pull the repo to get the updates. A possible Ansible playbook to install the updates might look something like this:
Remember when I got an authentication chip implanted last summer? Here are the pictures I took before and after the procedure, and in case you're feeling brave here's the video footage. (20191230 - Also uploaded to my Peertube account.)
Last weekend I was running short of stuff to hack around on and lamented this fact on the Fediverse. I was summarily challenged to find a way to archive posts to the Fediverse in an open, easy to understand data format that was easy to index, and did not use any third party services (like IFTTT or Zapier). I thought about it a bit and came up with a reasonably simple solution that uses three Huginn agents to collect, process, and write out posts as individual JSON documents to the same box I run that part of my exocortex on. This is going to go deep geek below the cut so if it's not your cup of tea, feel free to move on to an earlier post.
EDIT - 20200311 @ 1859 UTC-7 - Added how to replace a dead hard drive in a btrfs pool.
EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space. Also did a couple of experiments with rebalancing btrfs and monitored how long it took.
A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up. I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places. The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy. There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis. There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed). This was a problem that required fairly major surgery to fix, on both hardware and software.
By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra. This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration. This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error. If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).
A common task that people using Huginn set up as their "Hello, world!" project is getting the daily weather report because it's practical, easy, and fairly well documented. However, the existing example is somewhat obsolete because it references the Weather Underground API that no longer exists, having been sunset at the end of 2018. Recently, the Weather Underground code in the Huginn Weather Agent was taken out because it's no longer usable. But, other options exist. The US National Weather Service has a free to use API that we can use with Huginn with a little extra work. Here's what we have to do:
- Get the GPS coordinates for the place we want weather reports for.
- Use the GPS coordinates to get data out of the NWS API.
- Build a weather report message.
- E-mail it.
As happens sometimes, the admins of the NWS API have imposed an additional constraint upon users accessing their data: They ask that the user agent string of whatever software you use be unique, and ideally include an e-mail address they can contact you through in case something goes amiss. This isn't a big deal.
This tutorial assumes that you've worked with Huginn a bit in the past, but if you haven't I strongly suggest that you read my earlier posts to familiarize yourself.
Okay. Let's get started.
I spend a lot of time digging around in other people's data. If I'm not hunting for anything in particular then it's a bit of a crapshoot, to be honest, if only because you never know what you're in for. You can pretty much take it to the bank that if you didn't assemble it yourself, you can't count on it being complete, well formed, or anything approximating the output of a human being (it usually came out of a database, but I think you see what I'm getting at). Sometimes, if I'm really lucky I'll just get hold of a JSON dump of the database, which to be fair is better than nothing when there isn't even an API to use. From time to time I'll make an attempt at fitting the data into a database of some kind, sometimes MySQL, sometimes SQLite, or occasionally an API layer like Sandman2. This is all well and good, but it winds up being more of an adventure than I'm looking for. I'd much rather be Indiana Jones prowling around in the temple than Rambo going through a preparation montage because Indy was actually getting stuff done.
Wow, this article went a little off the rails. I was never good at writing intros to new code... anyway.