Let this be a lesson to everyone...

20 March 2007

Always double-check what machine you're working on. And always, always make sure your backups are good!

In the state of Alaska, a sysadmin at the department of revenue accidentally reformatted a hard drive that contained information pertaining to a oil fund account worth roughly $38bus.

Yep - billions of US dollars.

For an encore, rather than restore the data for the account from a backup hard drive, he then reformatted the backup drive on top of that - he probably hit the up arrow, changed the device name, and touched it off again. By this time, any sysadmin worth the name would be sweating bullets and praying to any pantheon that was on call that evening.

Then, as common sense would dictate, he went to the backup tapes to try to restore the last known good copy of of the account data.... and the backup tapes were bad. Such is the stuff of nightmares that sometimes result in the dreamer never awakening again.

Ultimately, they had to call in contractors to key in by hand 800,000 pages of hardcopy to reconstruct the account data.