Situation report, or, "What blew up this time?"

Aug 15, 2007

Between getting back home from a field assignment late on Friday night, recovering from two weeks on the road eating way too much takeout, and stuff happening at home, I haven't had much time to do anything in the way of writing. I can honestly say that it hasn't been a boring couple of weeks, but there's a lot to be said for sitting at home engaging in a high impact workout (read, my glutius maximii striking the couch at -9.8 m/s^2 once a night for five nights) to unwind.

On Monday night after work and dinner with Lyssa and Laurelinde I trucked out to Seele's apartment in Maryland for her birthday party, and spent most of the evening with people that I haven't seen in far too long, save for the occasional glimpse on IRC when I have time to see who's about. Much of the night was spent lounging around watching people play deceptively tricky games on the Nintendo Wii, one of the Wario game collections that I don't remember the name of. I have to admit, I can see why people like the Wii so much - the graphics were done in the cartoon style of the mid 80's that I happen to be fond of, which is really hard to do on a computer, and they are a lot of fun. There was also a strong undercurrent of surreality in some of the games, to the extent that I was waiting to hear a Voltaire song or two, which sadly didn't happen. Later in the evening we cut Seele's ice cream cake, which turned out to be wonderful. It's hard to find good ice cream down here...

It's probably the subliminal messages embedded in the Wii's games kicking in after two days.

All in all, I had a good time with everyone on Monday night. I got home around 2300 EST5EDT but made the mistake of staying up until 0100 local time checking my e-mail (work-related and not) and talking to Lyssa and Laurelinde to catch up. I slept only fitfully that night, and got up shortly before 0600 local time to get ready for work. In hindsight this was probably a bad idea because I was still dragging from the week before... Unbeknownst to me, Akara (the VA.net mail server) has been having drive problems - while her drives are mirrored, one of them has sector errors that were corrupting one of the file systems and apparently causing some e-mail to bounce (because there was still a slow yet steady current of incoming e-mail). 'lex Pendragon contacted me yesterday and mentioned that a message he'd sent bounced, which sent me off to investigate. Thinking that it was an antivirus software update I'd applied I remotely rebooted Akara... who didn't come back up.

As it turned out, she'd almost fully rebooted, but when it came to mounting the /var file system on the array, it failed catastrophically, which caused the boot process to hang and wait for the root password to be entered, so that manual maintenance could take place. I don't have serial consoles on any of my servers (though I'm seriously considering rigging one up from Leandra) so I didn't know this. After getting home after work I took at look at what was going on and triggered a manual file system reconstruction, a procedure that took better than three hours all told. Thankfully, Akara came back up normally around 2100 EST5EDT last night, and spent the rest of the night (and a good part of the morning hours) churning through the backlog of messages.

Following the adventures in system administration (there's a title for a webcomic if I ever saw one) last night, I collapsed into bed and slept clear until 0630 EST5EDT this morning. During dinner with Jason and Lyssa (between getting home and Akara coming back up) I... wasn't really there, I suppose. I don't remember much of last night at all. My screwed up sleep schedule and exhaustion finally caught up to me and while I drove everyone around safely, there wasn't much else of my central nervous system operating normally. I remember watching Akara come back up and then going to bed, and that's really about it.

Because of this, I'm seriously considering setting up a Google Apps account for the Network and letting Google handle e-mail, antivirus, and spam filtering for everyone. It's at the point where I'm not sure that I really have the time, energy, or inclination to maintain a full-scale SMTP server in my lab. Sure, you'd lose the encrypted file systems, let's be clear on this, but that's what PGP and GnuPG are for. Every account would have 2 gigs-plus of space and up to 100 accounts could be set up on a particular domain. On top of this, you'd have a webmail system (Gmail, really) in addition to POP access (no idea why Google hasn't set up a proper IMAP service yet). If that's not enough (you know who you are), I could spring for the enterprise package (10 gigs of space for every user account of a hosted domain) but then I'll start asking for donations to defray the cost ($50us/year).

Akara's hanging in there, but remember that one of two mirrored drives is failing. Slowly, surely, but failing. She's going to go down again in the future, and I don't know when (nor do I have the compute cycles right now to make an educated guess). I can't guarantee that there will be no data loss, either. We were damned lucky this time.

So, that's where things stand right now. I open the floor for comments, questions, and suggestions on the e-mail front.