EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space. Also did a couple of experiments with rebalancing btrfs and monitored how long it took.
A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up. I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places. The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy. There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis. There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed). This was a problem that required fairly major surgery to fix, on both hardware and software.
By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra. This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration. This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error. If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).
UPDATED: 18 March 2019 - External display adapters that actually work with this model (and Arch Linux) added.
For various reasons, I found that I had a need to upgrade Windbringer's hardware very recently. This might be the first time that a catastrophic failure of some kind was not involved, so it's kind of a weird feeling to have two laptops side by side, one in process and one to do research as snags cropped up. This time around I bought a Dell XPS 15 Touch (9570) - I was expecting things to be substantially the same, but this did not seem to be the case. Some things that I found myself ignoring because I had no use for them aren't in this newer model, and some things have changed as technology has advanced rather a lot in the last five years.
As before, first I'll post the hardware specs, and then follow up with everything I had to tinker with to get working as well as how I went about it. As usual, I went with 64-bit Arch Linux (2019.02 installation build).
The Magick Poke - noun - When you touch a failing appliance, light bulb, or other gizmo in the just the right way as you're replacing it, and it spontaneously starts working again. This usually saves it from the trashcan or dumpster. Comes from the POKE command in Commodore BASIC which could let you do some pretty strange things by putting just the right value into just the right memory location, usually by fat-fingering a value.
If you've known me for any length of time, chances are you've heard about my fascination with telephones and some of the weird stuff that you sometimes find if you misdial once in a while. Sweep tones, ringbacks, ANACs, and more unusual things. However, it's rare that some of those weird things happen to ring me up.
A couple of weeks ago I started getting phone calls at all hours of the day; not terribly unusual in itself, save that every time I pick up I hear a prompt to leave a voicemail ("Press one to leave a voicemail.") Ordinarily you're supposed to get those when you try to call somebody and they don't answer, not the other way around. The few occasions when the call hasn't come at 0-dark-thirty and when I haven't been at work occupied by other things I've experimentally tapped '1' expecting to be prompted to do as suggested. Instead of an empty voicemail box I heard the recitation of what sounds like operational statistics of machinery of some kind by a clearly mechanical voice, probably a relatively cheap speech synthesis system. "Pump one has an emergency. Pump pressure is low. Pump two has an emergency. Pump pressure is low. Pump three is offline. Pump four is offline. Well one is offline. Well two is offline."
My first hypothesis was that the automated pumping system at a gas station somewhere was misconfigured and accidentally dialed my line. That seems like a flimsy hypothesis because I doubt that a monitoring system would be configured to dial a phone number in an entirely different state; it seems more likely that it would be configured to call an office in the same region or state. Also, such a monitoring system should not initiate a call with a prompt to press a key on the phone to leave a voicemail. During some experimentation, I was able to get the phantom caller to recite some figures, speak some of the specifics of the system events in halting, mechanical English, but little else. The user interface seems very limited, just a handful of menus and two or three options in each. A little detective work shows that all of the numbers are in or around Columbus, Indiana. After considering it a little more, I wonder if they're automated alarm systems of some kind because not all of them recite pump status but slightly different things. It seems certain that they're not configured properly.
While I do not advocate calling up any of these numbers and messing with them, because I have no idea what is actually on the other end, if you happen to occasionally get similar phone calls and are scratching your head over them I hope this article will shed a little light on the situation. Beneath the cut are a few of the numbers and transcriptions of some of the voice prompts to hopefully serve as search engine fodder. If worse comes to worse, you can always block the numbers and be done with it, especially if they have the propensity to catch you at bad times.
Oh, and if you happen to be the admin of one or more of these systems? Please double-check their configs. I think you've got a couple of bugs that need fixed. Somebody who really needs to get those alerts isn't.
There is a graphic novel that is near and dear to my hearts by Warren Ellis called Planetary, the tagline of which is "It's a strange world. Let's keep it that way." This first article immediately made me go back and reread that graphic novel...
The field of deep learning has been around for just a short period of time insofar as computer science is concerned. To put it in a nutshell deep learning systems are software systems which attempt to model highly complex datasets in abstract ways using multiple layers of other machine learning and nonlinear processing algorithms stacked on top of one another, the output of one feeding the input of another. People are using them for all sorts of wild stuff these days, from sifting vast databases of unstructured data for novel patterns to new and creative ways to game the stock, bond, and currency markets. Or, if you're Terence Broad of London, accidentally get DMCA takedown requests.
Broad is working on his master's degree in Creative Computing, and as part of that work developed a deep learning system which he trained on lots of video footage to see if it became a more effective encoder by letting it teach itself how to watch video, in essence. It's not an obvious thing but representing video as data ("encoding") is a wild, hairy, scary field... there are dozens of algorithms for doing so and even more container formats for combining audio, video, and other kinds of data into a single file suitable for storage and playback. Broad built his deep learning construct to figure out more efficient ways of representing the same data in files all by itself, without human signal processing experts intervening. He then ran the movie Bladerunner through his construct, dumped its memory and uploaded it to video sharing site Vimeo. What happened shortly thereafter was that one of Warner Brothers' copyright infringement detection bots mistook the video output by Broad's deep learning construct by dumping its memory for a direct rip of the movie because the output of his deep learning system was so accurate and sent an automatic takedown request to the site because it couldn't tell the difference from the original. One of the videos in the article is a short side-by-side comparison of the original footage to the construct's memory. There are differences, to be sure - some of the sequences are flickering, rippling blotches of color that are recognizable if you look back at the original every few seconds, but other sequences are as a good a replica as I've ever seen. Some of the details are gone, some of the movement's gone, but a surprising amount of detail remains. If you grew up watching nth-generation bootlegs of the Fox edit of Bladerunner where the color's all messed up, you know what I'm talking about.