Experimenting with btrfs in production.

Oct 19 2019

EDIT - 20191104 @ 2057 UTC-7 - Figured out how long it takes to scrub 40TB of disk space.  Also did a couple of experiments with rebalancing btrfs and monitored how long it took.

A couple of weeks ago while working on Leandra I started feeling more and more dissatisfied with how I had her storage array set up.  I had a bunch of 4TB hard drives inside her chassis glued together with Linux's mdadm subsystem into what amounts to a mother-huge hard drive (a RAID-5 array with a hotspare in case one blew out), and LVM on top of that which let me pretend that I was partitioning that mother-huge hard drive so I could mount large-ish pieces of it in different places.  The thing is, while you can technically resize those virtual partitions (logical volumes) to reallocate space, it's not exactly easy.  There's a lot of fiddly stuff that you have to do (resize the file system, resize the logical volume to match, grow the logical volume that needs space, grow the filesystem that needs space, make sure that you actually have enough space) and it gets annoying in a crisis.  There was a second concern, which was figuring out which drive was the one that blew out when none of them were labelled or even had indicators of any kind that showed which drive was doing something (like throwing errors because it had crashed).  This was a problem that required fairly major surgery to fix, on both hardware and software.

By the bye, the purpose of this post isn't to show off how clever I am or brag about Leandra.  This is one part the kind of tutorial I wish I'd had when I was first starting out, and I hope that it helps somebody wrap their mind around some of the more obscure aspects of system administration.  This post is also one part cheatsheet, both for me and for anyone out there in a similar situation who needs to get something fixed in a hurry, without a whole lot of trial and error.  If deep geek porn isn't your thing, feel free to close the tab; I don't mind (but keep it in mind if you know anyone who might need it later).

Genetic jiggery-pokery.

Jul 04 2016

It's long been known that DNA encodes information in a four-bit pattern which can be read and processed like any other bitstream. Four different nucleotides, paired two by two, arranged in one of two configurations side by side by side in a long string of letters, many times longer than the size of the cell containing the full DNA strand. Every cell in every single lifeform contains the same DNA sequence, regardless of what the cell actually does. So how, many have asked, does a cell know if it should help produce hair, or skin, or pigments, or something else? As it turns out, there is more than one layer of information encoding at work in DNA - the way in which DNA is folded in three dimensions also encodes information used by the cell. Inside of every cell the DNA is tightly wound around a cluster of eight proteins called histones, which provide a superstructure to support the two meter long molecule. The question then becomes, how are the specific parts of the DNA molecule directly involved in what a given cell does, called nucleosomes kept accessible to the rest of the cellular machinery? Hypotheses to this effect have been going around since the 1980's but only recently has computational simulation been feasiable to put them to the test. As it turns out, the loops, twists, bends, curves, and folds that DNA undergoes around the histone octomers keep keep those functional nucleosomes exposed so that they can be acted upon. The simulations randomly pushed, pulled, prodded, and twisted virtual DNA strands to see what would happen, and they noted that nucleosomal configurations were in fact impacted. Those simulation results were then verified through laboratory observation of two species of common yeasts. It was also confirmed that point mutations can also influence the folding of DNA, which can result in changes in the frequency of synthesis of proteins due to change in accessibility of those nucleosomes. The entire (highly technical) paper (it gave me a headache on the first readthrough, okay?) is available in its entirity on PLOS ONE under a Creative Commons By Attribution v4.0 International license.