« Death and distruction… | Home | .plan file updated. »

Let this be a lesson to everyone...

Tuesday 20 March 2007 at 2:50 pm
Always double-check what machine you're working on. And always, always make sure your backups are good!

In the state of Alaska, a sysadmin at the department of revenue accidentally reformatted a hard drive that contained information pertaining to a oil fund account worth roughly $38bus.

Yep - billions of US dollars.

For an encore, rather than restore the data for the account from a backup hard drive, he then reformatted the backup drive on top of that - he probably hit the up arrow, changed the device name, and touched it off again. By this time, any sysadmin worth the name would be sweating bullets and praying to any pantheon that was on call that evening.

Then, as common sense would dictate, he went to the backup tapes to try to restore the last known good copy of of the account data.... and the backup tapes were bad. Such is the stuff of nightmares that sometimes result in the dreamer never awakening again.

Ultimately, they had to call in contractors to key in by hand 800,000 pages of hardcopy to reconstruct the account data.

Tags: , , , , , ,
AddThis Social Bookmark Button
Fight Spam! Click Here!

Trackback link:

Please enable javascript to generate a trackback url

two comments recorded.

You know, the guy who erased the data fucked up.

But the guy that does the backups – whose job is to insure those backups are working – really fucked up. I’d forgive the guy that erased the data. I’d can the backup operator. Possibly using a pressure canner….

Hasufin - 20 03 07 - 20:57 - Reply to comment?

He fucked up pretty badly. When he hosed the backup drive, that would have been grounds for dismissal at a lot of shops.

The backup operator definitely should have been fired. Every training class, every book, and indeed the documentation for each and every backup system out there tells you how to run the verification procedure, that backups should be verified, and to run test restores of at least one backup a week to make sure that they took. It’s not some big secret, and it’s the sort of thing that can always be used to justify taking extra time at work to push something else back.

The Doctor (URL) - 21 03 07 - 12:41 - Reply to comment?


  
Remember personal info?

/ Textile
  (Register your username / Log in)

Notify:
Hide email:

Small print: All html tags except <b> and <i> will be removed from your comment. You can make links by just typing the url or mail-address.