Fedora Linux Support Community & Resources Center
  #1  
Old 15th November 2007, 05:24 PM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
Angry SHA1 sum always changing - data/fs corruption or issue with kernel?

On x86_64 versions of both Fedora 7 & 8 I have been getting different results from sha1sum for files over 1GB (at least that's when I think it starts). Below is an illustration of the problem:

[Fedora-8-dvd-x86_64]$ sha1sum Fedora-8-x86_64-DVD.iso*; sha1sum Fedora-8-x86_64-DVD.iso*
4552df6b2d6d7cb53a9758e3f38cb06e66578d40 Fedora-8-x86_64-DVD.iso
8b59f879baa3dbb5c8526fab1709a5d19596e0c9 Fedora-8-x86_64-DVD.iso2
922d2ceb5874159438c23fdb2672fcda5aea2728 Fedora-8-x86_64-DVD.iso
e90e59b921ecbecb98cbe03dcaae4eb72f135354 Fedora-8-x86_64-DVD.iso2


[Fedora-8-dvd-x86_64]$ sha1sum Fedora-8-x86_64-DVD.iso*; sha1sum Fedora-8-x86_64-DVD.iso*
b088ced4daee50550357efffad7250d26bdb1075 Fedora-8-x86_64-DVD.iso
23c397b13698fae3ee9e2194f73ccb5db4bfbb8c Fedora-8-x86_64-DVD.iso2
28bc453fb87a1d11ebd30bccfe8263f7dae06d71 Fedora-8-x86_64-DVD.iso
a5232063bfb2342315afef51aa9eb552c187f5a3 Fedora-8-x86_64-DVD.iso2


Fedora-8-x86_64-DVD.iso2 is merely a copy of Fedora-8-x86_64-DVD.iso as it seem that running the same command consecutively resulted in the immediate return of the second call with the result of the previous...

However it illustrates the point quite well was in every single instance, there is no one result that matches any other! This was first discovered on my old install of F7 then duplicated on my fresh install of F8 (on a new harddrive). I have experienced this on both ext3 and xfs file systems.

Some system related information that may help:
AMD Athlon(tm) 64 X2 Dual Core Processor 4600+
2.98GB of Physical RAM
ATA WDC WD3200SB-01K Harddrive (305245Mb capacity)

[Fedora-8-dvd-x86_64]$ uname -a
Linux localhost.localdomain 2.6.23.1-49.fc8 #1 SMP Thu Nov 8 22:14:09 EST 2007 x86_64 x86_64 x86_64 GNU/Linux


[Fedora-8-dvd-x86_64]$ smartctl -H /dev/sda
smartctl version 5.37 [x86_64-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


It is the most infuriating problem and having been googling for quite some time, I given up and finally decided to ask! It has meant that I have spent hours over bradband using bit torrent in an effort to download installation isos for Fedora which alway drop back to 98% download to 100% recheck and then drop back to 98% endlessly.

The F8 install was eventually done over FTP, hoping that it would have resolved the issue...

Any ideas on where to look and what can be done? I tried the kernel options apic=off noapic nacpi acpi=off & mem=2G, but nothing makes the slightest bit of difference. There are no errors from dmesg either.

Last edited by Andrew F; 15th November 2007 at 10:11 PM. Reason: ext3 and xfs file systems gave same results now mentioned
Reply With Quote
  #2  
Old 16th November 2007, 09:50 PM
robatino Offline
Registered User
 
Join Date: Feb 2004
Posts: 849
First of all, stop downloading. You probably have some sort of hardware problem and need to fix that first. The first thing that jumps to mind is RAM. Try installing the memtest86+ package, then run 'memtest-setup' as root to add the entry to your GRUB menu (as explained by "rpm -qi memtest86+") and run it for a while. If memory is the problem you should start seeing errors fairly quickly.
Reply With Quote
  #3  
Old 16th November 2007, 10:29 PM
universe_r9 Offline
Registered User
 
Join Date: Nov 2007
Posts: 8
is it real???
__________________
I hope you know, I hope you know
That this has nothing to do with you

It's time to be a Big John now
And Big John don't cry
Reply With Quote
  #4  
Old 16th November 2007, 11:04 PM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
Thanks for your reply robatino.

Quote:
Originally Posted by robatino
First of all, stop downloading.
I did, when trying to find convenient methods of swift installation without the net could not be made to work, until I sort out my issue...

Quote:
Originally Posted by robatino
You probably have some sort of hardware problem and need to fix that first. The first thing that jumps to mind is RAM. Try installing the memtest86+ package, then run 'memtest-setup' as root to add the entry to your GRUB menu (as explained by "rpm -qi memtest86+") and run it for a while. If memory is the problem you should start seeing errors fairly quickly.
I'll give that a try tonight, although I believe that I ran that from a rescueCD for about an hour or 2, before I decided to try something else and no errors were highlighted.

Interestingly however I have run the liveCD from RAM, using yum to update the packages...everything seemed to work fine (so much was in the fs that I had only about 500Mb of RAM free) - but I don't have enough RAM to download the DVD iso to the fs and run the OS, whilst running from RAM...
Reply With Quote
  #5  
Old 16th November 2007, 11:08 PM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
Quote:
Originally Posted by universe_r9
is it real???
Sorry. Is what real? If you are referring to my issue, then, yes it is...and quite frustrating I might add.
Reply With Quote
  #6  
Old 16th November 2007, 11:11 PM
robatino Offline
Registered User
 
Join Date: Feb 2004
Posts: 849
If you already ran it from the rescue CD, and saw no errors over an hour, that's probably not it, since your memory would have to be having errors at least once a minute to be causing errors in a command that takes that long to run. Have you tried doing a diff between one of these files and a copy of it?
Reply With Quote
  #7  
Old 16th November 2007, 11:54 PM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
I am not entirely sure what you mean, but since Fedora-8-x86_64-DVD.iso2 is a copy of Fedora-8-x86_64-DVD.iso, I performed a diff and got the following:

[Fedora-8-dvd-x86_64]$ diff Fedora-8-x86_64-DVD.iso Fedora-8-x86_64-DVD.iso2
Binary files Fedora-8-x86_64-DVD.iso and Fedora-8-x86_64-DVD.iso2 differ


If that is not what you meant, please clarify an I will attempt what you intended. Otherwise, have you any other ideas of how the problem could be tracked down?

I'm beginning to think that I have hit a rather obscure bug in the kernel or the interface to the IDE drives...
Reply With Quote
  #8  
Old 16th November 2007, 11:58 PM
robatino Offline
Registered User
 
Join Date: Feb 2004
Posts: 849
That's what I meant. You could see what the exact differences are by using the cmp command (see the man page) to see how many differences there are, whether they start at a certain point in the files, etc.
Reply With Quote
  #9  
Old 17th November 2007, 12:24 AM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
I have tried the cmp command, and the listing was quite extensive, so I saved the 1st 8 values and repeated the steps. As you can see below, the values change from run to run.

[Fedora-8-dvd-x86_64]$ cmp -l Fedora-8-x86_64-DVD.iso Fedora-8-x86_64-DVD.iso2
3121359 50 40
3372997 225 25
5443791 75 65
29726661 362 162
29982401 204 4
34439388 32 232
35540831 77 277
36454239 237 37


[Fedora-8-dvd-x86_64]$ cmp -l Fedora-8-x86_64-DVD.iso Fedora-8-x86_64-DVD.iso2
3121359 50 40
3372997 225 25
3920079 72 62
4804289 264 64
5443791 75 65
6586575 56 46
7741647 104 114
9511119 117 107


It's almost as if the files are being changed each time! This never occurs with small files.
Reply With Quote
  #10  
Old 17th November 2007, 12:35 AM
robatino Offline
Registered User
 
Join Date: Feb 2004
Posts: 849
I notice that some of the changes are repeated, for example

3121359 50 40
3372997 225 25
5443791 75 65

but most aren't. Also, most of these differences are in 1 or 2 bits, for example

75 (octal) = 00111101 (binary), 65 (octal) = 00110101 (binary).

Do you, or have you ever, had any other OS installed on this box, or any other versions of Linux? If this is really an OS bug it's quite severe, but I'm still betting it's hardware-related.

Edit: Actually, I take it back. A hardware issue this bad would probably cause more serious problems.

Last edited by robatino; 17th November 2007 at 12:48 AM.
Reply With Quote
  #11  
Old 17th November 2007, 12:50 AM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
This machine has been evolving over the years. The latest upgrade was the 64 bit processor - my first. Before that the system had a dual boot with windows on my old processor and since upgrading I was unable to use the old windows so I wiped it for an extra partition. I have only had fedora on the new processor.
Reply With Quote
  #12  
Old 17th November 2007, 01:22 AM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
I have tried to perform the sha1sums from a game CD, to see what would happen, but I'm not quite sure where this leaves me...

[Fedora-8-dvd-x86_64]$ dd if=/dev/dvd bs=2048 count=344526 | sha1sum
344526+0 records in
344526+0 records out
88ce6cca3fc52437545f5f6259f425ec2ab08def -
705589248 bytes (706 MB) copied, 164.727 s, 4.3 MB/s

[Fedora-8-dvd-x86_64]$ dd if=/dev/dvd bs=2048 count=344526 | sha1sum
344526+0 records in
344526+0 records out
379d7cd06023cb72fa9df188cbe1972c75b86988 -
705589248 bytes (706 MB) copied, 164.993 s, 4.3 MB/s


Even that is giving me different sums...

However if I cat the cd to the HD and then perform sha1 sums, I get constant values:

[Fedora-8-dvd-x86_64]$ ll -h tmp.iso Fedora-8-x86_64-rescuecd.iso
-rw-r--r-- 1 root root 104M 2007-11-13 10:27 Fedora-8-x86_64-rescuecd.iso
-rw-r--r-- 1 root root 674M 2007-11-17 01:06 tmp.iso

[Fedora-8-dvd-x86_64]$ sha1sum tmp.iso Fedora-8-x86_64-rescuecd.iso tmp.iso Fedora-8-x86_64-rescuecd.iso
e85d07adb94aee2959b1e1922bb24237454e4bbd tmp.iso
0788cd030347d2121b572f0cf35bb7626dd9b5e5 Fedora-8-x86_64-rescuecd.iso
e85d07adb94aee2959b1e1922bb24237454e4bbd tmp.iso
0788cd030347d2121b572f0cf35bb7626dd9b5e5 Fedora-8-x86_64-rescuecd.iso


Where tmp.iso wat generated by:
cat /dev/dvd > tmp.iso

I then copied tmp.iso to tmp.iso2 and performed a diff. There was no difference!
Reply With Quote
  #13  
Old 17th November 2007, 01:33 AM
robatino Offline
Registered User
 
Join Date: Feb 2004
Posts: 849
As an aside, the easiest way to read an ISO image off a CD or DVD is to use the rawread script at the following link. You can put it in ~/bin and make it executable. It automatically uses the isoinfo command to determine the ISO size and then run the corresponding dd command.

http://www.troubleshooters.com/linux/coasterless.htm

I've noticed that occasionally, to get the right checksum from a disc, I have to reload it - leaving the drive door open and letting the dd or rawread command close it automatically always works for me. Try doing that with both reads.
Reply With Quote
  #14  
Old 17th November 2007, 02:47 AM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
I am aware of that script, but I am performing the steps manually so that I can remember it - my brain is somewhat like a sieve!

I tried your suggestion of not closing the drive tray, but still got differing results:

[Fedora-8-dvd-x86_64]$ dd if=/dev/dvd bs=2048 count=344526 | sha1sum
344526+0 records in
344526+0 records out
2183833bf4785a40c255ae73ce13aa01112779b9 -
705589248 bytes (706 MB) copied, 161.416 s, 4.4 MB/s

[Fedora-8-dvd-x86_64]$ dd if=/dev/dvd bs=2048 count=344526 | sha1sum
344526+0 records in
344526+0 records out
be0b757593de7c4f60a4009079244643e94026a7 -
705589248 bytes (706 MB) copied, 161.919 s, 4.4 MB/s
Reply With Quote
  #15  
Old 17th November 2007, 09:55 AM
Andrew F Offline
Registered User
 
Join Date: Nov 2007
Posts: 12
I ran the memtest last night and after 7 hours no errors had been detected - does the process ever stop!?
Reply With Quote
Reply

Tags
changing, sha1, sum

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Help for a SHA1 function error!!! abhishek2301 Programming & Packaging 1 14th April 2009 01:52 AM
Help with sha1 alfick3 Using Fedora 4 2nd June 2007 03:24 AM
sha1 digest openssl vs php caste Using Fedora 0 27th October 2005 10:13 PM
sha1 checksum does not match jinux Installation, Upgrades and Live Media 21 2nd August 2005 05:57 PM
checkmedia or sha1 which to believe mohelgamal Installation, Upgrades and Live Media 3 15th June 2005 11:27 PM


Current GMT-time: 15:53 (Sunday, 20-04-2014)

TopSubscribe to XML RSS for all Threads in all ForumsFedoraForumDotOrg Archive
logo

All trademarks, and forum posts in this site are property of their respective owner(s).
FedoraForum.org is privately owned and is not directly sponsored by the Fedora Project or Red Hat, Inc.

Privacy Policy | Term of Use | Posting Guidelines | Archive | Contact Us | Founding Members

Powered by vBulletin® Copyright ©2000 - 2012, vBulletin Solutions, Inc.

FedoraForum is Powered by RedHat