Last summer my day job sent me down to San Diego, CA to attend the Linux Security Summit and report back. Unfortunately just about all of the content there intersected in no way, shape, or form with anything we're working on so it was largely a dog wash. I probably won't attend again because, balancing the cost against the information gotten it just wasn't worth it. I did, however, take a couple of engineers from Oracle for their first good sushi dinner ever, took an amphibious boat tour of San Diego Bay, and hiked along the waterfront for a couple of hours.
If you were part of the hacker scene in the 1980's or 90's (or you played a certain tradition in Mage: The Ascension around that time) you undoubtedly have come across the weird, wonderful, bewildering, and occasionally insightful antics of The Cult of the Dead Cow, a crew of hackers originally based out of Texas who were well known for their periodic text file releases. What isn't well known until very recently is that many cDc alumni have gone on to do great things, from starting one of the first security companies to ascending to C-level status at some well known megacorps to overseeing government security initiatives. Earlier this year one Joseph Menn (nice guy, by the bye) wrote a book about them which is extensively researched and fun to read in general. Menn's book tour happened to bring him to the Internet Archive along with some cDc alumni.
Quite late, I know. It's been a busy year.
Let's say that you have a bunch of servers that you admin en masse using Ansible. You have all of them listed and organized in your /etc/ansible/hosts file. Let's say that each server is running a system service (like my Systembot) running under systemd in --user mode. (Yes, I'm going to use my exocortex-halo/ repository for this, because I just worked out a good way to keep everything up to date and want to share the technique for everyone new to Ansible. Pay it forward, you know?) You want to use Ansible to update your copy of Systembot across everything so you don't have to SSH into every box and git pull the repo to get the updates. A possible Ansible playbook to install the updates might look something like this:Click for the rest of the article...
Remember when I got an authentication chip implanted last summer? Here are the pictures I took before and after the procedure, and in case you're feeling brave here's the video footage. (20191230 - Also uploaded to my Peertube account.)
This is the initial announcement of a new project pwnagotchi-bt-power (mirrored at Gitlab), a short utility written in Python which will cleanly shut down a Pwnagotchi from an Android phone over Bluetooth using the Bluedot app.
Share and enjoy!
Last weekend I was running short of stuff to hack around on and lamented this fact on the Fediverse. I was summarily challenged to find a way to archive posts to the Fediverse in an open, easy to understand data format that was easy to index, and did not use any third party services (like IFTTT or Zapier). I thought about it a bit and came up with a reasonably simple solution that uses three Huginn agents to collect, process, and write out posts as individual JSON documents to the same box I run that part of my exocortex on. This is going to go deep geek below the cut so if it's not your cup of tea, feel free to move on to an earlier post.Click for the rest of the article...
entropic debugging - noun phrase - The phenomenon in which one can spend weeks on end debugging something using a multitude of techniques, give up in frustration and/or disgust for a couple of days, come back to the project and discover that somehow the bugs have magickally fixed themselves (as verified by diffs and file hashes if one cares to check). The phenomenon is so named due to the second law of thermodynamics, which states that entropy can never decrease, only increase in an isolated system. In other words, as entropy increases overall in the universe it somehow wiped out the bugs in question. See also kinetic pattern baldness.
wires - noun - Person to person backchannels.
"I had to pull some wires to get that expense report fixed before the boss saw it."
EDIT - 20200311 @ 1859 UTC-7 - Added how to replace a dead hard drive in a btrfs pool.
EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space. Also did a couple of experiments with rebalancing btrfs and monitored how long it took.
A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up. I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places. The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy. There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis. There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed). This was a problem that required fairly major surgery to fix, on both hardware and software.
By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra. This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration. This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error. If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).Click for the rest of the article...