Life and times.
Long time readers are probably wondering where I've been lately. The answer is kind of long and is worth a post all on its own. The short version of the story is, work's been eating me alive lately. This is our busiest time of year and it's been all hands on deck for a couple of weeks now. In point of fact, last week was our quarterly all-hands meeting, where everybody on my team was flown into town for a solid week of meetings. All day, every day. Most of my visible activity lately took the form of parts of my exocortex running on automatic with some hit-and-run posting while waiting for the coffee maker at work to top me up in between meetings.
This also means that I haven't had a whole lot of patience for interacting with people. Not in the sense that people can feel frustrated with other people or their actions, but in the sense that interacting with people in a meaningful way - having a real conversation - takes more compute cycles than I have available right now. After fourteen hours in a conference room with 40 other people, not only am I out of social, but I'm mentally exhausted.
When you factor in everything happening in the United States right now, from insurance coverage for pre-existing conditions on the chopping block again to Brownshirt (local mirror) like shenanagains (local mirror) taking place around the country (local mirror), quite frankly the last thing I feel like doing is talking to anyone right now. I'm making as much time as I can reasonably get away with to take care of my mental health because I can't afford to go to pieces these days. Those of you who know me well, I can hear you chuckling already at the mention of "my" and "mental health."
Okay. Here's the random shit that I've been doing rather than chatting. Buckle up.
During a bout of insomnia last week I decided to kill a couple of hours by running down something I've been curious about for years: A certain novel that was, depending on your perspective eerily prophetic in some important ways. By this I mean the book Abduction by one Harrison James, which is considered by some in the lunatic fringe to reflect many of the details of the Patty Hearst affair in 1974 (link anonymized) years before it actually happened. Always one to explore something weird, I tasked a couple of bots with finding me a copy of said text, an effort which took a couple of realtime years because there aren't many copies of the book floating around for sale and, as far as I can tell, the Internet Archive doesn't have one. It appears to be long out of print and the folks who manage to find it tend to hang onto their copies due to its relative rarity. Anyway, after having a copy on my bookshelf for a couple of days I decided to read it while having nothing better to do.
It's been said that the Symbionese Liberation Army may have read the book and decided to act it out, in the process somehow getting infected by the early 70's revolutionary memeplex written into it. if this is indeed the case (and I don't think it is), the situation is far more fucked up than the mere suggestion. Speaking as someone who is something of a conisseur of bad movies and a long-time mstie this is an impressively mediocre (at best) book. To put it another way, it's probably the most boring porn I've ever read. Even without speedreading I tore through it in about two hours and not only did I not feel the slightest urge to go back to bed (because I was neither tired nor bored), but I suddenly felt as if I understood how the people who write the advertising blurbs for porn must feel by the end of the workday. If the SLA thought that the contents of this book would be a nifty thing to act out they had bigger problems than a lack of something to do with their lives. I mean... Plan 9 from Outer Space was more interesting than Abduction, and that is so bad a movie that MST3k Live couldn't make it funny.
The universe has a bizarre sense of humor. Chalk it up to the surrealist current.
Last weekend, in a fit of needing to get stuff done to clean both house and head, I did a little work on Leandra to mitigate a wildly optimistic design decision made during construction. What I did was I split the RAID array up such that the /home and /var filesystems were six terabytes in size each. At the time this seemed like a reasonable thing to do given the amount of data I thought I'd be working with but in practice all of my data winds up in my home directory and /var isn't used nearly as much. I started running low on disk space so the time had come to reshuffle things a bit, which thankfully due to RAID and Logical Volume Management didn't require a full rebuild. What it did require, however, was rebooting into rescue mode because systemd doesn't have an idea of single user mode. I don't think I need to talk about how much I hate systemd, the rest of the Linux community has done a fine job of that for me. Moving on.
This meant that I triggered a full reboot of Leandra and, at the syslinux boot screen add the word "rescue" to the end of the boot command. One root password later I was at a maintenance shell figuring out what the hell I had to do to resize the virtual filesystems. What I wound up doing (and here's where things turn into an IT procedural) was firing up the RAID-5 manually:
[root@leandra ~]# mdadm --assemble --scan
That done, I then had to activate the physical volumes, the volume group, and the logical volumes so they'd be accessible. LVM has physical volumes (hard drives that it knows it can use), volume groups (collections of physical volumes part of the same array), and logical volumes (things that pretend to be partitions but are actually arbitrarily reconfigurable chunks of volume groups). systemd's rescue mode did not make it straightforward to do this, Arch Linux's documentation to the contrary.
[root@leandra ~]# pvscan
[root@leandra ~]# vgchange -ay
Now here's where things go screwy: Rather than the /dev/leandra block devices I was expecting to work with I was presented with the following:
[root@leandra ~]# ls -alF /dev/dm-*
brw-rw---- 1 root disk 254, 0 Oct 13 22:04 /dev/dm-0
brw-rw---- 1 root disk 254, 1 Oct 13 22:04 /dev/dm-1
brw-rw---- 1 root disk 254, 2 Oct 13 22:04 /dev/dm-2
brw-rw---- 1 root disk 254, 3 Oct 13 22:04 /dev/dm-3
Four Linux device mapper devices, none with obvious names. Thankfully, I put filesystem labels on everything so it was fairly easy to suss out which of the device nodes I needed to tinker with. As long as I have a device node of the right type I can do whatever I want with it, friendly names or not. So, let's peek at the filesystem labels:
[root@leandra ~]# for i in /dev/dm-?; do
> echo -n "$i - "
> e2label $i
done
/dev/dm-0 - srv
/dev/dm-1 - opt
/dev/dm-2 - var
/dev/dm-3 - home
Perfect. I needed to work with /dev/dm-2 and /dev/dm-3. To free up disk space I needed to shrink /dev/dm-2 down to a more reasonable size before I could do anything else.
[root@leandra ~]# e2fsck -f /dev/dm-2
[root@leandra ~]# resize2fs -p /dev/dm-2 1536G
[root@leandra ~]# lvresize -L 1536G /dev/leandra/var
All told, it took maybe 45 minutes to resize /dev/leandra/var down to 1.5 terabytes in size (from 6.0 terabytes). While I could probably have taken it smaller I can always do that later if I need to. Once that part of the process was done I had to then add the newly freed-up space to /dev/leandra/home, like so:
[root@leandra ~]# lvresize -l +100%FREE /dev/leandra/home
[root@leandra ~]# e2fsck -f /dev/dm-3
[root@leandra ~]# resize2fs /dev/dm-3
This part of the process took significantly less time, maybe ten minutes in all. End result:
[root@leandra ~]# df -h /home /var
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/leandra-home 11T 3.5T 7.1T 33% /home
/dev/mapper/leandra-var 1.5T 36G 1.4T 3% /var
Much better.
I don't really have anything else to yammer on about right now. I made a fine art of doing nothing last weekend: Sleeping in, getting out for a walk now and then, eating when I was hungry, reading books and writing when I wasn't.
Be careful out there, everybody. These are dangerous times.