Experimenting with btrfs in production.

Oct 19 2019

EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space.  Also did a couple of experiments with rebalancing btrfs and monitored how long it took.

A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up.  I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places.  The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy.  There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis.  There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed).  This was a problem that required fairly major surgery to fix, on both hardware and software.

By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra.  This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration.  This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error.  If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).

Click for the rest of the article...

Echoes of popular culture and open source.

Oct 03 2019

(Note: This post is well beyond the seven year limit for spoilers.  If you haven't seen 2001 or 2010 by now, I can't help you.)

Many years ago, as a loomling, one of my very first memories was of seeing the movie 2010: The Year We Make Contact on cable.  That the first 'real' record I ever listened to was the soundtrack to that movie should come as no surprise, but that's not really relevant.  I was quite young so I didn't get most of it, but I remembered enough about it that it gave me some interesting questions (so I thought; I was six, okay?) to ask at the library later.  The thing that struck me the most about the movies was, unsurprisingly, the monolith.  The universal alien device, which manipulated proto-hominids on Earth by teaching them how to hunt, gather, and make war, as well as making unspecified changes to their evolutionary path; which served as a monitoring outpost; which implemented the endpoints of a vast interstellar (intergalactic? interdimensional?) wormhole network; which turned a gas giant into a miniature star.  If you like, the monolith was a universal key to unlock the mysteries of the universe and inspire growth and change.

Many, many years later I was a computer geek in my late teens, just dumb enough to think I knew the right questions to ask, just smart enough to know that I didn't know nearly as much as I should.  I knew that college was coming up one way or another and I'd have to get my ducks in a row to do work there and hopefully get some research done.  I also knew that it wasn't going to be easy.  I'd just graduated from a hotwired Atari microcomputer with a modem to a modest PC clone, a 386 cobbled together out of hand-me-down components, stuff I'd scavenged out of dumpsters, and the odd weekend trip to the computer show.  I knew that there was this thing called Ethernet, and the college I was going to had just started rolling out connections of same to dorm rooms, and it was a pre-req for a comp.sci major.  I also knew that I needed an OS that could connect to the Net somehow, but I didn't have the connections to get my hands on the new hotness back then, nor did Leandra have the specs to run it if I did.

Click for the rest of the article...

Pen testing vs security assessment.

Sep 29 2019

A couple of weeks back while traveling I had an opportunity to spend some time with an old colleague from my penetration testing days.  Once upon a time we used to spend much of our time on the road, living out of suitcases, probably giving the TSA fits and generally living la vida Sneakers.  I'm out of that particular game these days because it's just not my bag anymore.  The colleague in question is more or less on the management side of things at that particular company.  Contrary to what one might reasonably assume, however, we didn't spend a whole lot of time reminiscing about the good old days, nor did we complain about all those kids on our respective lawns.  What we did do was have a conversation that I've been ruminating on since I got home.

A lot of business entities ask and pay for penetration tests - a team of relatively tame hackers goes to town on their infrastructure with little to no insider knowledge to see what they can get into (within certain limits, usually) and the client uses the results as their roadmap to figure out what they need to fix.  To a certain extent, this makes sense - sometimes the stuff that's broken doesn't make its presence known until somebody stumbles across it and gives it the business.  But... the way these things usually go is, the client fixes everything the red team tore through like a thermite lance through a baby's crib and that's about it.  They usually don't touch anything else, even to see how it stood up to second- and third- order effects.  And this is a pretty serious problem, as evidenced by the overall state of information security in the last quarter century.

Click for the rest of the article...

Neologism: The Paperless Office

Oct 15 2019

The Paperless Office - proper noun phrase - When the only reason your workplace seems to use no actual paper on a day to day basis is because the printer is always inoperable when someone needs to use it the most.  This leads to everyone giving up on the printer entirely.

Please Try This At Home: Dr. Mixael Laufer

Sep 28 2019

In September of 2019 a conference called Please Try This At Home was held in Pittsburgh, PA.  One of the talks was given by Dr. Mixael Laufer on the topic of how to acquire pharmaceuticals such as mifepristone (local mirror) and misoprostol (local mirror) for emergency personal use.  I spoke with Dr. Laufer and the person who made this recording, and they both agreed to let me post it for download and archival as long as I sent them the links to it.  So, here it is.

Neologism: Basketball mode

Aug 31 2019

basketball mode - noun phrase - When a service or application crashes and restarts itself over and over, i.e., bouncing like a basketball every few seconds.  Considered an outage.

Summer vacation is rapidly coming to an end.

Aug 31 2019

It seems as if another summer is rapidly coming to an end.  The neighbors' kids are now back in school, school buses are now picking their way down the streets, and due to Burning Man coming up it's now possible to eat in a real restaurant in the Bay Area for the next couple of days.  I've been pretty quiet lately, not because I've been spending any amount of time offline but because I've been spending more time doing stuff and just not writing it up.  I've been tinkering with Systembot lately, adding functionality that I really have a need for at home, namely, remotely monitoring a wireless access point running OpenWRT in the same way that I watch the rest of my stuff.  Due to the extreme system constraints on your average high-end wireless access point (2 CPUs, 128 megs of storage, 512 megs of RAM) it's not feasible to install Python and a Halo checkout, so I had to figure out how to get the system stats I need remotely.  What I wound up doing was standing up another copy of the standard OpenWRT web server daemon and writing a bunch of tiny CGI scripts which run local commands and return the information to Systembot for processing and analysis.  It wound up being a fun exercise in working with tight constraints, though I think there are still some bugs to be shaken out.

Click for the rest of the article...

Using Huginn to get today's weather report.

Aug 03 2019

A common task that people using Huginn set up as their "Hello, world!" project is getting the daily weather report because it's practical, easy, and fairly well documented.  However, the existing example is somewhat obsolete because it references the Weather Underground API that no longer exists, having been sunset at the end of 2018.  Recently, the Weather Underground code in the Huginn Weather Agent was taken out because it's no longer usable.  But, other options exist.  The US National Weather Service has a free to use API that we can use with Huginn with a little extra work.  Here's what we have to do:

  • Get the GPS coordinates for the place we want weather reports for.
  • Use the GPS coordinates to get data out of the NWS API.
  • Build a weather report message.
  • E-mail it.

As happens sometimes, the admins of the NWS API have imposed an additional constraint upon users accessing their data: They ask that the user agent string of whatever software you use be unique, and ideally include an e-mail address they can contact you through in case something goes amiss.  This isn't a big deal.

This tutorial assumes that you've worked with Huginn a bit in the past, but if you haven't I strongly suggest that you read my earlier posts to familiarize yourself.

Okay.  Let's get started.

Click for the rest of the article...