Experimenting with btrfs in production.

Oct 19 2019

EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space.  Also did a couple of experiments with rebalancing btrfs and monitored how long it took.

A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up.  I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places.  The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy.  There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis.  There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed).  This was a problem that required fairly major surgery to fix, on both hardware and software.

By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra.  This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration.  This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error.  If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).

Organizing a data hoard with YaCy.

Feb 02 2019

 It should come as little surprise to anyone out there that I have a bit of a problem with hoarding data.  Books, music, and of course files of all kinds that I download and read or use in a project for something.  Legal briefs, research papers (arXiv is the bane of my existence), stuff people ask me to review, the odd Humble Bundle... So much so that a scant few years ago I rebuilt Leandra to better handle the volume of data in my library.  However, it's taken me this long to both figure out and get around to making it easier to find anything in all that mess.  If I can't find it, I can't do anything with it, or even figure out what I do or don't have.  I also don't often have console access so it's not as if I can SSH in and grep for what I need.  I use Nginx as a web server on Leandra so actually getting access to files when I need them is trivial.

Life and times.

Oct 14 2018

Long time readers are probably wondering where I've been lately.  The answer is kind of long and is worth a post all on its own.  The short version of the story is, work's been eating me alive lately.  This is our busiest time of year and it's been all hands on deck for a couple of weeks now.  In point of fact, last week was our quarterly all-hands meeting, where everybody on my team was flown into town for a solid week of meetings.  All day, every day.  Most of my visible activity lately took the form of parts of my exocortex running on automatic with some hit-and-run posting while waiting for the coffee maker at work to top me up in between meetings.

This also means that I haven't had a whole lot of patience for interacting with people.  Not in the sense that people can feel frustrated with other people or their actions, but in the sense that interacting with people in a meaningful way - having a real conversation - takes more compute cycles than I have available right now.  After fourteen hours in a conference room with 40 other people, not only am I out of social, but I'm mentally exhausted.