Antarctica Starts Here.

EDITED: Added Creative Commons license block. Other content remains the same.

Long ago, in the days of DOS and OS/2, deleting a file meant that it was gone for good. How file systems worked was a mystery to just about everybody, and so we were told to back up our data often lest a mistake or drive crash wipe out something important, leaving us up a certain body of water sans propulsion. Years passed, as they are wont to do, and someone discovered that data didn't really evaporate when it was deleted, it was just renamed in such a way that it couldn't be seen anymore. This discovery lead to the creation of many data recovery tools for DOS (for OS/2 was on the way out), and many a user who'd just deleted the wrong file heaved a sigh of relief. For people concerned with deleted data being extracted from their systems without their knowledge, such as law enforcement, intelligence and defense agencies of many countries around the world, and people who risk persecution or execution for their work, brows began to sweat and many sleepless nights were had.

As alluded to before, just because you delete something doesn't mean that the data winks out of existence. Most every file system does basically the same things to keep track of files and free disk space when files are deleted: a table of some kind is kept which maps blocks of disk space to names of directories and files. Block 123387 of the first hard drive holds part of ntoskrnl.exe, block 98926127 contains /usr/local/src/gtkwifi-1.10/README, and so forth. When a file is deleted the entry in the file systems' tables is updated to reflect the fact that it's slot is now available for re-use, as are the blocks on disk associated with that table entry. Moreover, the first block of a file points to the second, which points to the third and so forth, making a chain of disk blocks for that file. So long as those disk blocks aren't overwritten by parts of another file, that data might still be extracted by someone who gets hold of your computer (or just the hard drive) for a while. While many of the data forensics packages are too expensive for your average user to get hold of, there are some good ones out there that are open source (like Autopsy and ddrescue (which was designed with damaged drives more than forensic analysis in mind)) or at least highly affordable (like FTK Imager from AccessData). There are two ways that one can dig deleted data out of a drive of some kind. The first is to traverse the data structures that the file system uses to keep track of disk blocks, directories, and files (I gave a thumbmail description of these a few paragraphs ago) and look for the telltale signs of file deletion. A list of all of the recently deallocated disk blocks is presented to the user along with a list of possible filenames that they could correspond to. An option is given to either un-delete the files (which you're probably familiar with) or extract them to another location (the forensic technique). However, there is no guarantee that the contents of the file will be uncorrupted, or even usable because deleting a file returns all of the blocks associated with a file to free status (effectively unlinking all of them) and some of them may have been allocated to other files since that time. The second way is to analyze each and every block on the drive and look at the pointers to other disk blocks, in effect re-assembling the chains without going through the file system. While this takes longer and requires much more expertise (as well as specialized software), this is also the generally accepted Way To Go About Things as far as the data forensics community is concerned. This is also the method by which it is sometimes possible to extract data thought erased a very long time ago.

In the mid-1990's, a researcher named Peter Gutmann wrote a famous paper about recovering deleted and overwritten data from hard drives by highly funded and motivated parties (read: intelligence agencies) using a technique called magnetic force microscopy. This method, if implemented, would require a great deal of expertise, a large amount of highly specialized equipment, and an unknown period of time to reconstruct individual bits on the platters of a hard drive by measuring the minute residual magnetic field of a particular bit on the drive which remains after it's been changed through the re-use of a particular disk block. If you do any research at all into secure data deletion, you're going to come across this paper and if you sit down and read it all the way through, it paints the matter at hand in a way described by a good friend of mine as "Rocks fall, you all die."

Things aren't quite as simple or as bad as they seem. First of all, that paper was written with two particular hard drive technologies, MFM and RLL data encoding in mind. Depending on how long you've been using computers, you will probably recall working with MFM or RLL hard drives on an IBM PC, and you'll also probably fondly recall the family-friendly names used for same, "boat anchors." Those drives aren't used anymore, and in fact they aren't even manufactured anymore. You'll even have a hard time finding them at Goodwill Computers, swap meets, or on eBay. Hard drive technology has changed a great deal since Gutmann's paper was written. Secondly, the attacks that Gutmann describes in his paper assume an entity with the expertise, hardware, and funding popularly attributed to the National Security Agency (folklore says that No Such Agency has acres and acres of Cray supercomputers in its basement for cracking codes; doing a little research about the practical requirements of such a setup, however, would tell you that their even having two Crays at Fort Meade would be highly improbable due to the power and cooling infrastructure required to keep the Crays from melting down). Unless you're way the hell out there, chances are there is nothing at all you do that would warrant anyone going to such lengths to recover your data, should they get hold of your computer. Don't bother rigging up a thermite charge or a .45 round to destroy your hard drives should someone ask to see your computer.

Now that we've got that out of the way, the United States government is acutely aware of the security risk posed by the disposal of data storage media without sanitizing it first. In 1991, the National Computer Security Center worked up an unclassified document for the DoD, thoughtfully titled A Guide to Understanding Data Remanence in Automated Information Systems (version 2) (NCSC-TG-025, known to the infosec community as the Green Book). While I recommend that you sit down and read this document (downloadable from the above link or requestable from the government as a galley edition) at least once to understand the specifics, what it boils down to is what kinds of data storage media there are and how best to render them unreadable. It's actually not too dry as government documentation goes.

So, what, as someone who is reasonably concerned about information security and personal privacy, do you do? How far should you go to keep people from piecing together your deleted files?

The answer is, you don't have to go very far. It doesn't take much to render a file, or even an entire hard drive irrecoverable.

The simplest way is to physically destroy the drive - if you trash the platters inside the drive, all the king's horses and all the king's men will never be able to put your information back together again. Take a screwdriver and remove the screws or bolts from the metal cover of the hard drive (on the side opposite the circuit board) to expose the drive heads and spindle. Put on safety glasses and take a large hammer to the platters a few times. Sometimes the platters are made of a fragile material (I've been told ceramic but I don't actually know what it is) which shatters when struck (which is why I recommend eye protection), but most of the time they're made of aluminum. I usually take a dozen whacks at the drive with a ball-peen hammer just to be safe. If a hard drive isn't going to be used ever again (it conked out or it's too old or small to get any real use out of) then this is your best bet.

Physical destruction is the only method to render optical media unreadable. There are shredders on the consumer market that can reduce a CD to sparkly confetti and grinders that remove the reflective and data-bearing layers of optical disks from the label side. You can also use a sturdy pair of scissors to cut optical disks into tiny pieces or a blowtorch to melt them (but do this outside because the smoke is toxic). The most fun way of trashing an optical disk, however, is to put it in the microwave (ideally someone else's) for a few seconds which ruins the data-bearing and aluminum layers of the disk and leaves a pretty fractal pattern visible through the polycarbon.

Magnetic media can also be decomissioned through a process called degaussing. Put simply, this means getting your hands on a big-ass electromagnet (sold for just this purpose) and running the storage media over the business end for a while. The idea is that the microscopic magnetized regions on the disks which represent the ones and zeroes are all re-aligned according to the magnetic field the degausser generates, erasing the data in the process. It's rumored that the more powerful units can actually warp the platters inside the drive but I've never seen evidence of this so take such stories with a grain of salt.

If you plan on re-using the drive, your options are limited to wiping the contents of the disk or the individual file with special software. Wiping a hard drive is often referred to as decomissioning or flattening the drive, and can be done any number of ways. Probably the simplest involves attaching the drive to a Linux, BSD, or Mac OSX machine and running the command (as the root user) dd if=/dev/random of=/dev/drive_to_wipe bs=4k, which copies random (pseudo-random, actually) junk out of the kerenl's generator and writes it to the drive, overwriting the file system and data as it goes. Be warned that this can take a very long time depending on the size and speed of the drive, and using specialized software is more advantageous. You could use the bcwipe utility from Jetico to accomplish the same thing (bcwipe -b -m 1 /dev/drive_to_wipe); I find that it works a bit faster than the dd method.

If you need to wipe a number of hard drives in one machine or a number of systems simultaneously I cannot recommend DBAN highly enough. DBAN may be downloaded as an .iso image (for burning to CDs or DVDs) or a Windows installer (for floppy disks or USB drives). After preparing your media of choice, boot the target machine with it and follow the instructions on the screen (there aren't many). DBAN will then cycle through all of the hard drives on the machine and overwrite them with random data, thus turning your data into unusable garbage.

Darik's Boot and Nuke: when you absolutely, positively must trash every last machine in the room. I've been using it to decomission machines since 2001 and I carry a credit card CD-ROM with DBAN on it as part of my normal field kit.

Solid-state storage is an entirely different matter. Sure, it's cheap (and getting cheaper all the time) and rapidly growing in capacity, but there are a few things that you need to keep in mind if you want to use it securely. Firstly, flash storage devices actually hold a bit more than they're rated for due to the fact that each data cell has to be blanked before it can be rewritten; this burns out the data cells eventually (on average, after a half-million writes). Hard drives are constructed with extra sectors which the controller circuitry maps over failed disk sectors for much the same reason. To prevent the storage cells from giving out too early the controller circuitry built into the device implements wear leveling, which spreads writes out over the least-recently written to cells in the device. While this means that your USB key will probably be good for a couple of years of casual use, it also means that there could be bits and pieces of files spread all through the storage cells which could be extracted by a forensic investigator. The DoD's recognized method of sanitizing USB keys typically involves a hammer. One thing that you can do, however, is overwrite the file and then overwrite all of the unused space on the flash drive, but doing so too often will shorten the effective lifespan of your drive.

These are all scorched-earth solutions, however. Chances are you aren't interested in erasing and rebuilding your system every time you want to erase some files, so instead you'll have to securely overwrite and delete files to be rid of them. If you read the Gutmann paper, he originally suggested overwriting files 38 times with a series of special patterns which are thought to make it next to impossible for Them to recover erased data. However, a few years later he published a follow-up to his paper in which he said:

In the time since this paper was published, some people have treated the 38-pass overwrite technique described in it more as a kind of voodoo incantation to banish evil spirits than the result of a technical analysis of drive encoding techniques. If you're using a drive which uses encoding technology X, you only need to perform the passes specific to X, and you never need to perform all 38 passes. For any modern PRML/EPRML drive, a few passes of random scrubbing is the best you can do. As the paper says, "A good scrubbing with random data will do about as well as can be expected". This was true in 1996, and is still true now.

bcwipe -f -r -md -w /tmp/
bcwipe -f -r -md -w /usr/tmp/
bcwipe -f -r -md -w /var/tmp/
for i in /home/ ; do bcwipe -f -r -md -w $i/tmp/ ; done