Gargantuan file servers and tiny operating systems.

Apr 29, 2017

We seem to have reached a unique point in history: Available to your average home user are gargantuan amounts of disk space (8 terabyte hard drives are a thing, and the prices are rapidly coming down to widespread affordability) and enough processing power is available for the palm of your hand that makes the computational power that put the human race on the moon compare in the same was that a grain of sand does to a beach.  For most people, it's the latest phone upgrade or more space for your media box.  For others, though, it poses an unusual challenge: How to make the best use of the hardware without wasting it needlessly.  By this, I mean how one might build a server that doesn't result in wasted hard drive space, wasted SATA ports on the mainboard, or having enough room to put all of that lovely (and by "lovely" I really mean "utterly disorganized") data that accumulates without even trying.  I mentioned last year that I rebuilt Leandra (specs in here) so I could work on some machine learning and search engine projects.  What I didn't mention was that I had some design constraints that I had to follow so that I could get the most out of her.

To get the best use possible out of all of those hard drives I had to figure out how to structure the RAID, where to put the guts of the Arch Linux install, and most importantly figure out how to set everything up so that if Leandra did blow a hard drive the entire system wouldn't be hosed.  If I partitioned all of the drives as described here and used one as the /boot and / partitions, and RAIDed the rest, if the first drive blew I'd be out an entire operating system.  Also, gauging the size of the / partition can be tricky; I like to keep my system installs as small as possible and add only packages that I absolutely need (and ruthlessly purge the ones that I don't use anymore).  20 gigs is way too big (currently, Leandra's OS install is 2.9 gigabytes after nearly a year of experimenting with this and that) but it would leave room to grow.

Decisions, decisions.

So, what did I finally decide on?

Arch Linux, systemd, and RAID.

May 13, 2016

Long, long time readers of my blog might remember Leandra, the server that I've had running in my lab in one configuration or another since high school (10th grade, in point of fact). She's been through many different incarnations and has run pretty much every x86 CPU ever made since the 80386. She's also run most of the major distributions of Linux out there, starting with Slackware and most recently running Arch Linux (all of the packages of Gentoo with none of the spending hours compiling everything under the sun or fighting with USE flags). It's also possible to get a full Linux install going with only the packages you need in a relatively small amount of disk space; my multimedia machine, for example, is only 2.7 gigabytes in size and Leandra as she stands right now has a relatively svelte 1.1 gigabytes of systemware. However, Arch Linux was an early adopter of something called systemd, which aims to be a complete replacement of the traditional UNIX-like init system that tries to manage dependencies of services, parallelize startup and shutdown of system features, automatically start and stop stuff, replace text-based system logs with a binary database, and all sorts of bleeding edge stuff like that.

Some people love systemd. Some people hate systemd. Personally, I think it is what my besainted grandmother would say, enough to piss off the Pope. That's not really what I'm writing about, though. What I'm writing about is a problem I ran into getting Leandra back up and running after building a fairly sizeable RAID array with logical volumes built on top of it.