Fedora Linux Support Community & Resources Center

Go Back   FedoraForum.org > Fedora Resources > Guides & Solutions (No Questions)
FedoraForum Search

Forgot Password? Join Us!

Guides & Solutions (No Questions) Post your guides here (No links to Blogs accepted). You can also append your comments/questions to a guide, but don't start a new thread to ask a question. Use another forum for that.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 2nd August 2011, 09:33 PM
Adunaic Offline
Registered User
 
Join Date: Mar 2009
Location: Lancaster, UK
Posts: 922
linuxfedorafirefox
How to: Data recovery

After having the joys of trying to recover data from a drive that has been formatted and is now failing I thought I would document what I did and provide a guide for those searching how to do this, especially on fedora.

To set the scene of what is in this guide I will first summarise my situation.
  • One laptop that had windows install.
  • Windows stopped booting and so Linux was installed (it was intended that the windows partition would be shrunk but this failed for some reason).
  • After some months of use the disk began to fail.
  • After mounting the disk on a livecd the only needed files, photos could not be found on the windows partition.

The first thing to know is that data is not really deleted from a hard drive, it is only ever deleted from the partition. As such unless the area of the hard drive that the data was on has had new data written to it, the original is still there.

Even when a drive is re-formatted/partitioned the data can still be there (which is why a hard drive should never be discarded without wiping in completely). Even installing a new OS on top of an old one will leave some of the data recoverable.

A word of warning, once it has been decided that data has been lost and needs to be recovered then you first need to power down the drive/machine; You can then start the machine again booting from a live cd or, if it is not an OS drive mount the drive as read only.

I would always suggest that regardless of whether data has been accidentally deleted, a partition corrupted or the disk is dying you should always make an image of the drive and work from that. In the case of a dying disk this is a necessity, although for the other two you can get away with working with the disk directly, but its always nice to test with an image first, especially trying to reconstruct a partition.

Imaging a drive/filesystem (assumed damaged)

There are a few options for imaging a drive, all of these are variations or improvements on dd.
  • dd - simple low-level copying
  • dd_rescue - an evolution of dd designed to handle bad sectors.
  • GNU ddrescue - disk imaging tool that "copies data from one file or block device to another, trying hard to rescue data in case of read errors."

dd is a tool designed for low-level copying of raw data; as such it may be used to copy one device to another. One flaw with it is that it cannot handle bad blocks of data, to get around this two other programs were created, dd_rescue and GNU ddrescue. (GNU) ddrescue is the superior and faster of the two programs, thus I will only focus on using ddrescue.

Warning - dd is also known as "death and destruction" since if you use it incorrectly it can wipe your hard drive and leave the data unrecoverable.

ddrescue

To install ddrescue issue the following command from the terminal.

Code:
 su -c 'yum install ddrescue'
ddrescue is used with the following parameters

Code:
 ddrescue [options] in_device out_device [logfile]
whilst a log file is optional it is highly recommended that you use one, without it you will have to start from scratch if you run out of space or the process is killed. If the device has bad sectors then a logfile is non-optional.

In this guide it is assumed that the disk /dev/sda has bad sectors and needs imaging. We shall also assume that it is going to be imaged to a folder called 'recovery' on a USB hard drive that is mounted to '/media/usbdrive'. Do not forget that the drive or device you are imaging the disk to must be bigger then the original device.

Data recovery is divided into two parts, the main reason this is split into two parts is because if the drive is failing we wish to minimise use of the drive. This is achieved by only copying the parts of the drive that are healthy first, and then going back to copy the faulty parts of the drive ( a process that requires intensive disk usage). The end result is maximising the amount of data copied before the drive dies.

This is achieved by running ddrescue with the '-n' flag, this tells ddrescue not to split or retry the damaged sections of the disk.

Code:
 su ddrescue -n /dev/sda /media/usbdrive/recovery/sda.img /media/usbdrive/recovery/sda.log
Once this has finished we can go back and try to recover the data in blocks with bad sectors. Here we use two flags, '-d' to access the disk directly and bypass the kernel cache, and '-r', the maximum number of retries.

Note that the output file given and log file should be the same as in the previous command as ddrescue will read the locations of the bad blocks from the logfile and append its new copies to the image.

Code:
 su ddrescue -d -r3 /dev/sda /media/usbdrive/recovery/sda.img /media/usbdrive/recovery/sda.log
This command can be run again with more retries but 3 is the recommended.

Extracting files from the recovered image.

There are many options for extracting files from a disk. We shall only discuss a few here.
  • foremost - This recovers files based on their headers and footers and is filesystem independent.
  • Scalpel - This is a fast file carver similar to foremost.
  • Photorec - Another filesystem independent recovery program, despite the name it does much more than recover photos.

In my experience not all of these programs will find the same files, so there is some use to trying them all. Scalpel and foremost will also be able to find fragments of files. Scalpel was by far the fastest, but did not retrieve the meta-data for jpegs so was not as useful to me (see 'Cleaning up')

Foremost

Foremost can recover files from an image or a device, and the syntax for both is the same. The files will be recovered to a directory that you specify so it is best to create one.

Foremost is not installed by default so we install it by doing:

Code:
 su -c 'yum install foremost'
Then create a drive to recover the files too.

Code:
 mkdir /media/usbdrive/recovery/foremost
whilst there are several options, only two are really need, '-i' for the input device/image and '-o' for the output directory. You can also tell foremost which files to recover. In my case, I only needed jpg so the flag was added for that (-t jpg).

foremost comes preconfigured to recover the following types of files: jpg,gif,png,bmp,avi,exe,mpg,mp4,wav,riff,mat,wmv,m ov,pdf,ole (This will grab any file using the OLE file structure. This includes PowerPoint, Word, Excel, Access, and StarWriter),doc,zip

To use foremost do:

Code:
 su foremost -t jpg -i /media/usbdrive/recovery/sda.img -o /media/usbdrive/recovery/foremost


Scalpel

Scalpel is a fast file carver. It works much like foremost by looking at file headers and matching them to a list of known types. By default in fedora all file types are enabled. It is best to edit this to just the types you want. This can be done by commenting out the appropriate lines in /etc/scalpel/scalpel.conf

To install it type:

Code:
 su -c 'yum install scalpel'
Scalpel takes the input device/image as an argument and the output directory as an option. So first we create a directory to dump the output to:

Code:
 mkdir /media/usbdrive/recovery/scalpel
and then set it to work

Code:
 su scalpel /media/usbdrive/recovery/sda.img -o /media/usbdrive/recovery/scalpel
[SIZE]Photorec[/SIZE]

The final program is photorec. It was designed to recover photos from digital cameras, but has been extended to cover hard disks and many more file types. It is part of the testdisk package so it installed by doing

Code:
 su -c 'yum install testdisk'
Photorec is initiated from the command line but it is interactive and will prompt you with what to do. Start it by doing:

Code:
 photorec /media/usbdrive/recovery/sda.img
For the time being I shall skip providing instructions for photorec as I think it is clear, but if people want instructions I can add them.
Reply With Quote
  #2  
Old 2nd August 2011, 09:34 PM
Adunaic Offline
Registered User
 
Join Date: Mar 2009
Location: Lancaster, UK
Posts: 922
linuxfedorafirefox
Re: How to: Data recovery

Cleaning up

These three tools will recover any jpegs that they come across, this includes a lot of thumbnails from internet browsing (facebook amazon etc.) In my case I had about 3000 photos on the drive but several hundred thousand jpegs were recovered. With such a large number of files you cannot (realistically) go through them one by one and delete the ones you do not want.

The recommended way appears to be to use 'find' to locate only jpegs above 1Mb in size. This is fine but in my case photos were taken on an older camera that did not create 1Mb jpegs or on a new one that compressed many to below 1Mb in size. As such I tried two other methods to filter the recovered size.
  • by resolution
  • by make

This is where I ran into problems with scalpel, filtering by these methods requires accessing the meta-data of the images and scalpel did not seem to retrieve this with the image.

To access the meta-data I used identify command. This needed ImageMagick which can be installed by doing.

Code:
 su -c 'yum install ImageMagick'
To sort by resolution I wrote a script that would extract the first part of the resolution and copy the file to a new location if it was above a certain size (in this case 800). The script was saved in my home area and was:

Code:
 
#!/bin/bash                                                      
#filterByResolution.sh 
j="`identify -verbose $1 | grep 'geometry'`"
k="${j:17:4}"
l=`echo -e "$k" | grep "[^0-9]" > /dev/null;echo $?`
if [ $l -gt 0 ]
then 
    if [ $k -gt 800 ]
    then
	cp $1 large/.
    fi
else 
    k="${j:17:3}"
    l=`echo -e "$k" | grep "[^0-9]" > /dev/null;echo $?`
    if [ $l -gt 0 ]
    then
	if [ ${j:17:3} -gt 800 ]
	then
	    cp $1 large/.
	fi
    fi
fi
This code will copy a file to a sub-directory called large if its resolution is greater than 800. Since it takes a single file as an argument I ran it in a find command. This was ran in the directory of the recovered files, in this case for foremost.

Code:
cd /media/usbdrive/recovery/foremost
mkdir large
find . -iname '*.jpg' -exec /<path_to_script>/filterByResolution.sh {} \;
I felt that filtering by resolution was hard as many non-relevant images passed this and some images were ignored as they were too small. I then remembered that all of my photos were always taken using a fujifilm camera and card so I filtered by make instead

This was once again done using a simple script like above, in this case the script was

Code:
#!/bin/bash                                                      
#filterByMake.sh
j="`identify -verbose $1 | grep 'exif:Make: FUJIFILM'`"
if [ "$j" != "" ]
then 
    cp $1 large/.
fi
and ran it by doing

Code:
cd /media/usbdrive/recovery/foremost
mkdir large
find . -iname '*.jpg' -exec /<path_to_script>/filterByMake.sh {} \;
The final note is that file names were not recovered with the files, as such I wrote a script to rename all files to the data and time they were taken to help sort through them.

Code:
#!/bin/bash
#renameToDateAndTime.sh
newName="`identify -verbose $1 | grep 'exif:DateTimeOriginal:'`"
mv $1 ${newName:27:10}_${newName:38:8}.jpg
This is once again run from the directory containing the recovered (and filtered)

Code:
cd /media/usbdrive/recovery/foremost/large
find . -iname '*.jpg' -exec /<path_to_script>/renameToDateAndTime.sh {} \;
Whilst I am not an expert on this I will attempt to answer any questions that people have. I am also open to comments on how to improve any of the scripts or methods in this guide.
Reply With Quote
  #3  
Old 2nd August 2011, 09:44 PM
dd_wizard Offline
Registered User
 
Join Date: Sep 2009
Posts: 1,481
linuxfirefox
Re: How to: Data recovery

Very nice, I'd like to nominate this for a Sticky.

dd_wizard
Reply With Quote
  #4  
Old 2nd August 2011, 10:00 PM
sea Online
"Shells" (of a sub world)
 
Join Date: May 2011
Location: Confoederatio Helvetica (Swissh)
Age: 34
Posts: 3,697
linuxfirefox
Re: How to: Data recovery

Great job done!
Thank you for this contribution.

sea
__________________
Laptop: Toshiba satellite p50-a-11 CPU: Intel i7 8*2400 MHz GPU: GeForce GT 745M RAM: 8192 MB Distro: Fedora (Rawhide) DE: Awesome
Guide: Windows 8+ & Fedora 20+ Dualboot
Repo (sea-devel) :: Text User Interface (TUI), Video Handler Script (VHS), efi-helper, grub2-circled-nasa-themes, Essentials (blame, p, rmtf, rnd, siggen, screenshot...), (BETA) Dev-Scripts, (BETA) Fedora Account Scripts (FAS)
Reply With Quote
  #5  
Old 3rd August 2011, 10:17 AM
Adunaic Offline
Registered User
 
Join Date: Mar 2009
Location: Lancaster, UK
Posts: 922
linuxredhatfirefox
Re: How to: Data recovery

Quote:
Originally Posted by dd_wizard View Post
Very nice, I'd like to nominate this for a Sticky.

dd_wizard
Cheers; I shall try and update it with a few more things (once I have tried those methods and when I have some time)
Reply With Quote
  #6  
Old 20th August 2011, 11:49 AM
salemeni Offline
Registered User
 
Join Date: Aug 2011
Posts: 64
linuxchrome
Re: How to: Data recovery

goog guide, great job

comparing objects

Last edited by salemeni; 6th December 2011 at 09:33 AM.
Reply With Quote
  #7  
Old 29th March 2012, 01:44 PM
anshumandhuliya Offline
Registered User
 
Join Date: Sep 2010
Posts: 7
linuxfirefox
Re: How to: Data recovery

I had 175 gb vFat partition which i mistakenly formatted to the ext format. And i only wrote nearly 300 mb of new data. i tried the "foremost" package and extracted all the possible files from the 175 gb partition. i faced the following problems

1. the recovered jpg files are 6 lakh!! (dont know how!) but now there is no file browser that can open the folder contents.
2. Same file is being recovered twice, and thrice. (May be that contributed a little to the 6 lakh images)
4. The recovered videos of .mov ext just wont play!!!
3. all the previous folder hierarchy is now lost.

the first one is making the extraction useless! it all took a lot of time more than 3 hours. external hard disk belongs to my friend that i have to return today. Mainly jpg photos and recorded videos are important for me in the partition. What to do?
Reply With Quote
  #8  
Old 29th March 2012, 01:50 PM
DBelton Offline
Administrator
 
Join Date: Aug 2009
Posts: 7,334
linuxfirefox
Re: How to: Data recovery

Very nice guide, Aduniac. Well written and easy to understand.

Just one thing to add, and it's really something that should be a no-brainer but I have seen people try and recover data and not follow this simple rule.

Always recover to a drive other than the one you are pulling the data off of

Edit:
I just noticed the date on the original post.. Why didn't I see this sooner? It slipped right by me without being noticed. I guess I am just getting old and missing the important things.

Last edited by DBelton; 29th March 2012 at 01:55 PM.
Reply With Quote
  #9  
Old 16th June 2012, 07:33 PM
Adunaic Offline
Registered User
 
Join Date: Mar 2009
Location: Lancaster, UK
Posts: 922
linuxfirefox
Re: How to: Data recovery

Quote:
Originally Posted by anshumandhuliya View Post
I had 175 gb vFat partition which i mistakenly formatted to the ext format. And i only wrote nearly 300 mb of new data. i tried the "foremost" package and extracted all the possible files from the 175 gb partition. i faced the following problems

1. the recovered jpg files are 6 lakh!! (dont know how!) but now there is no file browser that can open the folder contents.
2. Same file is being recovered twice, and thrice. (May be that contributed a little to the 6 lakh images)
4. The recovered videos of .mov ext just wont play!!!
3. all the previous folder hierarchy is now lost.

the first one is making the extraction useless! it all took a lot of time more than 3 hours. external hard disk belongs to my friend that i have to return today. Mainly jpg photos and recorded videos are important for me in the partition. What to do?
Sorry, I have only just noticed comments on this thread and I appologise for not responding faster. (My excuse is that I was on honeymoon when the comments were posted and I must have missed them when I got back. Very sorry.)

1. I am not sure what you mean by "lakh".
2. Yes, the same files can be recovered many times. I am not sure why, usually I see this with previews that have been cached somewhere else on the drive.
4. Foremost is very good at recovering files. However sometimes the whole file cannot be recoverd as parts of the file have been overwritten or the drive is corrupted where they are stored. This is most obvious in images where there will be black boxes where parts of the image were lost.
I suspect this has happened with your .mov files - as they are a lot more complicated than an image it maybe that the loss of a small part of them leaves them unusable. Have you tried vlc? It is sometimes good with partially corrupted files.
3. Foremost (and most of the others) only recover file information. It will not retrieve the directory structure.

---------- Post added at 07:33 PM ---------- Previous post was at 07:31 PM ----------

Quote:
Originally Posted by DBelton View Post
Very nice guide, Aduniac. Well written and easy to understand.

Just one thing to add, and it's really something that should be a no-brainer but I have seen people try and recover data and not follow this simple rule.

Always recover to a drive other than the one you are pulling the data off of

Edit:
I just noticed the date on the original post.. Why didn't I see this sooner? It slipped right by me without being noticed. I guess I am just getting old and missing the important things.

Yes you are correct, I probably should make that clearer. I will have a look through the whole thing and make sure it is correct at a later date - however I am very busy at the minute, part of the reason I missed these comments I suspect is because I am very busy so I rarly get a chance to have a look at the forum of late.

Thanks for the input and compliments.
Reply With Quote
  #10  
Old 16th June 2012, 08:45 PM
nonamedotc Offline
Community Manager
 
Join Date: Mar 2011
Location: USA
Posts: 3,988
linuxfirefox
Re: How to: Data recovery

Excellent post, Adunaic. I help out some friends with data backup/recovery every now and then and this guide will come in very handy! Very well organized and clear.

Quote:
Originally Posted by Adunaic View Post
Sorry, I have only just noticed comments on this thread and I appologise for not responding faster. (My excuse is that I was on honeymoon when the comments were posted and I must have missed them when I got back. Very sorry.)

1. I am not sure what you mean by "lakh".

[ ... ]
1 lakh = one hundred thousand (100,000).
Reply With Quote
  #11  
Old 16th June 2012, 09:22 PM
dd_wizard Offline
Registered User
 
Join Date: Sep 2009
Posts: 1,481
linuxfirefox
Re: How to: Data recovery

Congratulations! That's the best reason I've ever heard for a late reply. I hope you had fun, and enjoy all the advantages of wedded bliss!

dd_wizard
Reply With Quote
  #12  
Old 16th June 2012, 09:45 PM
Adunaic Offline
Registered User
 
Join Date: Mar 2009
Location: Lancaster, UK
Posts: 922
linuxfirefox
Re: How to: Data recovery

Quote:
Originally Posted by dd_wizard View Post
Congratulations! That's the best reason I've ever heard for a late reply. I hope you had fun, and enjoy all the advantages of wedded bliss!

dd_wizard
Thank you. It was great.
Reply With Quote
Reply

Tags
data, data loss, data recovery, recovery

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Data recovery Nemros Using Fedora 6 29th November 2008 12:01 PM
Data Recovery makia0pt Using Fedora 2 21st June 2008 12:37 AM
Data recovery help Firewing1 Hardware & Laptops 5 11th May 2007 07:08 AM
Data recovery help rbLinuxnub Using Fedora 4 27th March 2007 12:50 PM


Current GMT-time: 13:31 (Monday, 22-12-2014)

TopSubscribe to XML RSS for all Threads in all ForumsFedoraForumDotOrg Archive
logo

All trademarks, and forum posts in this site are property of their respective owner(s).
FedoraForum.org is privately owned and is not directly sponsored by the Fedora Project or Red Hat, Inc.

Privacy Policy | Term of Use | Posting Guidelines | Archive | Contact Us | Founding Members

Powered by vBulletin® Copyright ©2000 - 2012, vBulletin Solutions, Inc.

FedoraForum is Powered by RedHat
...Private Tattoo Photos on Instagram - Hemmings Instagram Photos - Balneario Camboriu Photos - Seoul Garden Sun Plaza Instagram Photos