A couple of weeks ago, one of my co-workers mentioned in passing that he'd surprised himself by adding an SSD (solid state drive) to his file server at home. To recap a bit, Leandra, my primary server at home has a sizable RAID-5 array storing all of my data. However, one of the tradeoffs is that stuff recently written to the array is a little slow to be read back. It's really not noticeable unless you're logged in and running commands, and even then the lag is something like one or two seconds. Noticeable but not actually problematic. At any rate, I'd been wanting to do some tinkering lately and had an Amazon order planned because I wanted to do some electronic work on my warwalking rig so I figured that, depending on the cost, I might add an SDD to my order. Much to my surprise, a 120 gigabyte SSD is incredibly cheap, I paid a hair under $20us for a Kingston A400. Emminently affordable.
Long time readers are probably wondering where I've been lately. The answer is kind of long and is worth a post all on its own. The short version of the story is, work's been eating me alive lately. This is our busiest time of year and it's been all hands on deck for a couple of weeks now. In point of fact, last week was our quarterly all-hands meeting, where everybody on my team was flown into town for a solid week of meetings. All day, every day. Most of my visible activity lately took the form of parts of my exocortex running on automatic with some hit-and-run posting while waiting for the coffee maker at work to top me up in between meetings.
This also means that I haven't had a whole lot of patience for interacting with people. Not in the sense that people can feel frustrated with other people or their actions, but in the sense that interacting with people in a meaningful way - having a real conversation - takes more compute cycles than I have available right now. After fourteen hours in a conference room with 40 other people, not only am I out of social, but I'm mentally exhausted.
We seem to have reached a unique point in history: Available to your average home user are gargantuan amounts of disk space (8 terabyte hard drives are a thing, and the prices are rapidly coming down to widespread affordability) and enough processing power is available for the palm of your hand that makes the computational power that put the human race on the moon compare in the same was that a grain of sand does to a beach. For most people, it's the latest phone upgrade or more space for your media box. For others, though, it poses an unusual challenge: How to make the best use of the hardware without wasting it needlessly. By this, I mean how one might build a server that doesn't result in wasted hard drive space, wasted SATA ports on the mainboard, or having enough room to put all of that lovely (and by "lovely" I really mean "utterly disorganized") data that accumulates without even trying. I mentioned last year that I rebuilt Leandra (specs in here) so I could work on some machine learning and search engine projects. What I didn't mention was that I had some design constraints that I had to follow so that I could get the most out of her.
To get the best use possible out of all of those hard drives I had to figure out how to structure the RAID, where to put the guts of the Arch Linux install, and most importantly figure out how to set everything up so that if Leandra did blow a hard drive the entire system wouldn't be hosed. If I partitioned all of the drives as described here and used one as the /boot and / partitions, and RAIDed the rest, if the first drive blew I'd be out an entire operating system. Also, gauging the size of the / partition can be tricky; I like to keep my system installs as small as possible and add only packages that I absolutely need (and ruthlessly purge the ones that I don't use anymore). 20 gigs is way too big (currently, Leandra's OS install is 2.9 gigabytes after nearly a year of experimenting with this and that) but it would leave room to grow.
So, what did I finally decide on?
Long, long time readers of my blog might remember Leandra, the server that I've had running in my lab in one configuration or another since high school (10th grade, in point of fact). She's been through many different incarnations and has run pretty much every x86 CPU ever made since the 80386. She's also run most of the major distributions of Linux out there, starting with Slackware and most recently running Arch Linux (all of the packages of Gentoo with none of the spending hours compiling everything under the sun or fighting with USE flags). It's also possible to get a full Linux install going with only the packages you need in a relatively small amount of disk space; my multimedia machine, for example, is only 2.7 gigabytes in size and Leandra as she stands right now has a relatively svelte 1.1 gigabytes of systemware. However, Arch Linux was an early adopter of something called systemd, which aims to be a complete replacement of the traditional UNIX-like init system that tries to manage dependencies of services, parallelize startup and shutdown of system features, automatically start and stop stuff, replace text-based system logs with a binary database, and all sorts of bleeding edge stuff like that.
Some people love systemd. Some people hate systemd. Personally, I think it is what my besainted grandmother would say, enough to piss off the Pope. That's not really what I'm writing about, though. What I'm writing about is a problem I ran into getting Leandra back up and running after building a fairly sizeable RAID array with logical volumes built on top of it.