How to: Data recovery

[Adunaic]

After having the joys of trying to recover data from a drive that has been formatted and is now failing I thought I would document what I did and provide a guide for those searching how to do this, especially on fedora.

To set the scene of what is in this guide I will first summarise my situation.

• One laptop that had windows install.
• Windows stopped booting and so Linux was installed (it was intended that the windows partition would be shrunk but this failed for some reason).
• After some months of use the disk began to fail.
• After mounting the disk on a livecd the only needed files, photos could not be found on the windows partition.

The first thing to know is that data is not really deleted from a hard drive, it is only ever deleted from the partition. As such unless the area of the hard drive that the data was on has had new data written to it, the original is still there.

Even when a drive is re-formatted/partitioned the data can still be there (which is why a hard drive should never be discarded without wiping in completely). Even installing a new OS on top of an old one will leave some of the data recoverable.

A word of warning, once it has been decided that data has been lost and needs to be recovered then you first need to power down the drive/machine; You can then start the machine again booting from a live cd or, if it is not an OS drive mount the drive as read only.

I would always suggest that regardless of whether data has been accidentally deleted, a partition corrupted or the disk is dying you should always make an image of the drive and work from that. In the case of a dying disk this is a necessity, although for the other two you can get away with working with the disk directly, but its always nice to test with an image first, especially trying to reconstruct a partition.

Imaging a drive/filesystem (assumed damaged)

There are a few options for imaging a drive, all of these are variations or improvements on dd.

• dd - simple low-level copying
• dd_rescue - an evolution of dd designed to handle bad sectors.
• GNU ddrescue - disk imaging tool that "copies data from one file or block device to another, trying hard to rescue data in case of read errors."

dd is a tool designed for low-level copying of raw data; as such it may be used to copy one device to another. One flaw with it is that it cannot handle bad blocks of data, to get around this two other programs were created, dd_rescue and GNU ddrescue. (GNU) ddrescue is the superior and faster of the two programs, thus I will only focus on using ddrescue.

Warning - dd is also known as "death and destruction" since if you use it incorrectly it can wipe your hard drive and leave the data unrecoverable.

ddrescue

To install ddrescue issue the following command from the terminal.

 su -c 'yum install ddrescue'

ddrescue is used with the following parameters

 ddrescue [options] in_device out_device [logfile]

whilst a log file is optional it is highly recommended that you use one, without it you will have to start from scratch if you run out of space or the process is killed. If the device has bad sectors then a logfile is non-optional.

In this guide it is assumed that the disk /dev/sda has bad sectors and needs imaging. We shall also assume that it is going to be imaged to a folder called 'recovery' on a USB hard drive that is mounted to '/media/usbdrive'. Do not forget that the drive or device you are imaging the disk to must be bigger then the original device.

Data recovery is divided into two parts, the main reason this is split into two parts is because if the drive is failing we wish to minimise use of the drive. This is achieved by only copying the parts of the drive that are healthy first, and then going back to copy the faulty parts of the drive ( a process that requires intensive disk usage). The end result is maximising the amount of data copied before the drive dies.

This is achieved by running ddrescue with the '-n' flag, this tells ddrescue not to split or retry the damaged sections of the disk.

 su ddrescue -n /dev/sda /media/usbdrive/recovery/sda.img /media/usbdrive/recovery/sda.log

Once this has finished we can go back and try to recover the data in blocks with bad sectors. Here we use two flags, '-d' to access the disk directly and bypass the kernel cache, and '-r', the maximum number of retries.

Note that the output file given and log file should be the same as in the previous command as ddrescue will read the locations of the bad blocks from the logfile and append its new copies to the image.

 su ddrescue -d -r3 /dev/sda /media/usbdrive/recovery/sda.img /media/usbdrive/recovery/sda.log

This command can be run again with more retries but 3 is the recommended.

Extracting files from the recovered image.

There are many options for extracting files from a disk. We shall only discuss a few here.

• foremost - This recovers files based on their headers and footers and is filesystem independent.
• Scalpel - This is a fast file carver similar to foremost.
• Photorec - Another filesystem independent recovery program, despite the name it does much more than recover photos.

In my experience not all of these programs will find the same files, so there is some use to trying them all. Scalpel and foremost will also be able to find fragments of files. Scalpel was by far the fastest, but did not retrieve the meta-data for jpegs so was not as useful to me (see 'Cleaning up')

Foremost

Foremost can recover files from an image or a device, and the syntax for both is the same. The files will be recovered to a directory that you specify so it is best to create one.

Foremost is not installed by default so we install it by doing:

 su -c 'yum install foremost'

Then create a drive to recover the files too.

 mkdir /media/usbdrive/recovery/foremost

whilst there are several options, only two are really need, '-i' for the input device/image and '-o' for the output directory. You can also tell foremost which files to recover. In my case, I only needed jpg so the flag was added for that (-t jpg).

foremost comes preconfigured to recover the following types of files: jpg,gif,png,bmp,avi,exe,mpg,mp4,wav,riff,mat,wmv,m ov,pdf,ole (This will grab any file using the OLE file structure. This includes PowerPoint, Word, Excel, Access, and StarWriter),doc,zip

To use foremost do:

 su foremost -t jpg -i /media/usbdrive/recovery/sda.img -o /media/usbdrive/recovery/foremost

Scalpel

Scalpel is a fast file carver. It works much like foremost by looking at file headers and matching them to a list of known types. By default in fedora all file types are enabled. It is best to edit this to just the types you want. This can be done by commenting out the appropriate lines in /etc/scalpel/scalpel.conf

To install it type:

 su -c 'yum install scalpel'

Scalpel takes the input device/image as an argument and the output directory as an option. So first we create a directory to dump the output to:

 mkdir /media/usbdrive/recovery/scalpel

and then set it to work

 su scalpel /media/usbdrive/recovery/sda.img -o /media/usbdrive/recovery/scalpel

[SIZE]Photorec[/SIZE]

The final program is photorec. It was designed to recover photos from digital cameras, but has been extended to cover hard disks and many more file types. It is part of the testdisk package so it installed by doing

 su -c 'yum install testdisk'

Photorec is initiated from the command line but it is interactive and will prompt you with what to do. Start it by doing:

 photorec /media/usbdrive/recovery/sda.img

For the time being I shall skip providing instructions for photorec as I think it is clear, but if people want instructions I can add them.

[Adunaic]

Cleaning up

These three tools will recover any jpegs that they come across, this includes a lot of thumbnails from internet browsing (facebook amazon etc.) In my case I had about 3000 photos on the drive but several hundred thousand jpegs were recovered. With such a large number of files you cannot (realistically) go through them one by one and delete the ones you do not want.

The recommended way appears to be to use 'find' to locate only jpegs above 1Mb in size. This is fine but in my case photos were taken on an older camera that did not create 1Mb jpegs or on a new one that compressed many to below 1Mb in size. As such I tried two other methods to filter the recovered size.

• by resolution
• by make

This is where I ran into problems with scalpel, filtering by these methods requires accessing the meta-data of the images and scalpel did not seem to retrieve this with the image.

To access the meta-data I used identify command. This needed ImageMagick which can be installed by doing.

 su -c 'yum install ImageMagick'

To sort by resolution I wrote a script that would extract the first part of the resolution and copy the file to a new location if it was above a certain size (in this case 800). The script was saved in my home area and was:

 
#!/bin/bash                                                      
#filterByResolution.sh 
j="`identify -verbose $1 | grep 'geometry'`"
k="${j:17:4}"
l=`echo -e "$k" | grep "[^0-9]" > /dev/null;echo $?`
if [ $l -gt 0 ]
then 
    if [ $k -gt 800 ]
    then
	cp $1 large/.
    fi
else 
    k="${j:17:3}"
    l=`echo -e "$k" | grep "[^0-9]" > /dev/null;echo $?`
    if [ $l -gt 0 ]
    then
	if [ ${j:17:3} -gt 800 ]
	then
	    cp $1 large/.
	fi
    fi
fi

This code will copy a file to a sub-directory called large if its resolution is greater than 800. Since it takes a single file as an argument I ran it in a find command. This was ran in the directory of the recovered files, in this case for foremost.

cd /media/usbdrive/recovery/foremost
mkdir large
find . -iname '*.jpg' -exec /<path_to_script>/filterByResolution.sh {} \;

I felt that filtering by resolution was hard as many non-relevant images passed this and some images were ignored as they were too small. I then remembered that all of my photos were always taken using a fujifilm camera and card so I filtered by make instead

This was once again done using a simple script like above, in this case the script was

#!/bin/bash                                                      
#filterByMake.sh
j="`identify -verbose $1 | grep 'exif:Make: FUJIFILM'`"
if [ "$j" != "" ]
then 
    cp $1 large/.
fi

and ran it by doing

cd /media/usbdrive/recovery/foremost
mkdir large
find . -iname '*.jpg' -exec /<path_to_script>/filterByMake.sh {} \;

The final note is that file names were not recovered with the files, as such I wrote a script to rename all files to the data and time they were taken to help sort through them.

#!/bin/bash
#renameToDateAndTime.sh
newName="`identify -verbose $1 | grep 'exif:DateTimeOriginal:'`"
mv $1 ${newName:27:10}_${newName:38:8}.jpg

This is once again run from the directory containing the recovered (and filtered)

cd /media/usbdrive/recovery/foremost/large
find . -iname '*.jpg' -exec /<path_to_script>/renameToDateAndTime.sh {} \;

Whilst I am not an expert on this I will attempt to answer any questions that people have. I am also open to comments on how to improve any of the scripts or methods in this guide.

[dd_wizard]

Very nice, I'd like to nominate this for a Sticky.

dd_wizard

[sea]

Great job done!
Thank you for this contribution.

sea

[Adunaic]

Originally Posted by dd_wizard

Very nice, I'd like to nominate this for a Sticky.

dd_wizard

Cheers; I shall try and update it with a few more things (once I have tried those methods and when I have some time)

[salemeni]

goog guide, great job

comparing objects

[anshumandhuliya]

I had 175 gb vFat partition which i mistakenly formatted to the ext format. And i only wrote nearly 300 mb of new data. i tried the "foremost" package and extracted all the possible files from the 175 gb partition. i faced the following problems

1. the recovered jpg files are 6 lakh!! (dont know how!) but now there is no file browser that can open the folder contents.
2. Same file is being recovered twice, and thrice. (May be that contributed a little to the 6 lakh images)
4. The recovered videos of .mov ext just wont play!!!
3. all the previous folder hierarchy is now lost.

the first one is making the extraction useless! it all took a lot of time more than 3 hours. external hard disk belongs to my friend that i have to return today. Mainly jpg photos and recorded videos are important for me in the partition. What to do?

[DBelton]

Very nice guide, Aduniac. Well written and easy to understand.

Just one thing to add, and it's really something that should be a no-brainer but I have seen people try and recover data and not follow this simple rule.

Always recover to a drive other than the one you are pulling the data off of

Edit:
I just noticed the date on the original post.. Why didn't I see this sooner? It slipped right by me without being noticed. I guess I am just getting old and missing the important things.

[Adunaic]

Originally Posted by anshumandhuliya

I had 175 gb vFat partition which i mistakenly formatted to the ext format. And i only wrote nearly 300 mb of new data. i tried the "foremost" package and extracted all the possible files from the 175 gb partition. i faced the following problems

1. the recovered jpg files are 6 lakh!! (dont know how!) but now there is no file browser that can open the folder contents.
2. Same file is being recovered twice, and thrice. (May be that contributed a little to the 6 lakh images)
4. The recovered videos of .mov ext just wont play!!!
3. all the previous folder hierarchy is now lost.

the first one is making the extraction useless! it all took a lot of time more than 3 hours. external hard disk belongs to my friend that i have to return today. Mainly jpg photos and recorded videos are important for me in the partition. What to do?

Sorry, I have only just noticed comments on this thread and I appologise for not responding faster. (My excuse is that I was on honeymoon when the comments were posted and I must have missed them when I got back. Very sorry.)

1. I am not sure what you mean by "lakh".
2. Yes, the same files can be recovered many times. I am not sure why, usually I see this with previews that have been cached somewhere else on the drive.
4. Foremost is very good at recovering files. However sometimes the whole file cannot be recoverd as parts of the file have been overwritten or the drive is corrupted where they are stored. This is most obvious in images where there will be black boxes where parts of the image were lost.
I suspect this has happened with your .mov files - as they are a lot more complicated than an image it maybe that the loss of a small part of them leaves them unusable. Have you tried vlc? It is sometimes good with partially corrupted files.
3. Foremost (and most of the others) only recover file information. It will not retrieve the directory structure.

---------- Post added at 07:33 PM ---------- Previous post was at 07:31 PM ----------

Originally Posted by DBelton

Very nice guide, Aduniac. Well written and easy to understand.

Just one thing to add, and it's really something that should be a no-brainer but I have seen people try and recover data and not follow this simple rule.

Always recover to a drive other than the one you are pulling the data off of

Edit:
I just noticed the date on the original post.. Why didn't I see this sooner? It slipped right by me without being noticed. I guess I am just getting old and missing the important things.

Yes you are correct, I probably should make that clearer. I will have a look through the whole thing and make sure it is correct at a later date - however I am very busy at the minute, part of the reason I missed these comments I suspect is because I am very busy so I rarly get a chance to have a look at the forum of late.

Thanks for the input and compliments.

[nonamedotc]

Excellent post, Adunaic. I help out some friends with data backup/recovery every now and then and this guide will come in very handy! Very well organized and clear.

Originally Posted by Adunaic

Sorry, I have only just noticed comments on this thread and I appologise for not responding faster. (My excuse is that I was on honeymoon when the comments were posted and I must have missed them when I got back. Very sorry.)

1. I am not sure what you mean by "lakh".

[ ... ]

1 lakh = one hundred thousand (100,000).

[dd_wizard]

Congratulations! That's the best reason I've ever heard for a late reply. I hope you had fun, and enjoy all the advantages of wedded bliss!

dd_wizard

[Adunaic]

Originally Posted by dd_wizard

Congratulations! That's the best reason I've ever heard for a late reply. I hope you had fun, and enjoy all the advantages of wedded bliss!

dd_wizard

Thank you. It was great.