F12 boot fails - short block read

[burnie]

Hello - My F12 installation, which was installed on an HP tx2500z laptop and has been operating relatively well since January, now fails to boot after a frozen screen precipitated a forced shutdown. On reboot, I was dropped into a shell with the error message "Error reading block 1546 (Attempt to read block from file system resulted in a short read) while getting next inode from scan." recommending to run fsck manually.

During the fsck run, after a substantial number of Pass 1 inode count corrections, the "Error reading block" message reappeared, with the question "Ignore error<y>" to which I answered "y".

fsck responded with "Force rewrite<y>". Since I did not know what would be written here, I said "no".

I then forced shutdown, rebooted into Windows, and searched for further guidance. One suggestion was to run fsck -y on the affected partition. My laptop drive has four partitions, the first two being allocated to Windows and an HP supplied repair partition. sda3 is a tiny boot partition (100 MB), and sda4/5 is the LVM remainder.

fsck -y /dev/sda3 told me the boot partition was fine.
fsck -y /dev/sda4 stopped immediately and asked "Could this be a zero-length file partition?". Given the ext3 structure, I suppose the correct answer is yes.
fsck -y /dev/sda5 told me fsck.LVM2_Member: not found.

Following this, I then attempted again to reboot, and ran fsck without options. Once again, during Pass 1, Block 1546 exhibited the short read error, again asking about a rewrite - which I once again answered no.

Pass 1 completed with another substantial number of inode count corrections, and Pass 2 started - immediately generating the message:

'passwd' in /etc/pam.d (65556) has deleted/unused node 7831. Clear<y>

After some hesitancy, I said "yes", and of course a second similar message appeared. I was brave enough to answer "yes" six more times, with the first two referring to /etc/pam.d inodes, and the next four referring to /etc/security/console.apps inodes.

The eighth message, also similar, referred to inodes elsewhere - but by now I did not dare to continue and forced shutdown.

Is this repairable, and if so, how?

[CSchwangler]

What error messages are you getting during reboot after your did fsck and attempted to repair the disk? The same as before the fsck?

[burnie]

What error messages are you getting during reboot after your did fsck and attempted to repair the disk? The same as before the fsck?

Seem to be the same errors. Running dd if=/dev/sda5 | hexdump -C scans the entire partition without aborting. I think this means the disk HW is ok, so a file necessary to boot must have become corrupted (correct?). NOT CORRECT -- see later post.

Also, both of the directories from which I had cleared deleted/unused inodes contain only text files, and using the F12 install disk in rescue mode I find that they read cleanly and their contents seem appropriate, although I'm unsure how to verify that they are what they should be.

[jpollard]

that may depend on what disk you have - for instance, if it is one of the new 4K
sector disk, a short read may be expected when the disk/BIOS/ whatever doesn't
expect large blocks - a read of a 512 byte record MAY come up short due to
configuration errors. This would be a really new disk though... and things have not
finished settling out about the 4K sectors...

This is really speculative on my part, so don't take it as gospel.

[burnie]

Rebooted again -- this time escaped to text earlier in the boot sequence. New messages appeared:

Exception fmask 0x0
Failed command READ FPDMA QUEUED
cmd 60/08:00:0d:28:ae/01:00:0d:00:40 tag 0 ncq 135168 in
res 41/40:00:62:28:ae/30:00:0d:00:40 Emask 0x409 (media error) <f>

status: { DRDYERR }
error: { UNC }

end_request: I/O Error, dev sda, sector 229517410
<= here appeared the originally reported Block 1546 short read error =>
Enter root password for maintenance
(or Control-D to continue): <= here I entered the root password =>
<= the following two bash messages had appeared before, but were not noted =>
bash: cannot set terminal process group (-1): inappropriate ioctl for device
bash: no job control in this shell
(Repair filesystem) 1#

---------- Post added at 10:53 AM CDT ---------- Previous post was at 10:41 AM CDT ----------

that may depend on what disk you have - for instance, if it is one of the new 4K sector disk, a short read may be expected when the disk/BIOS/ whatever doesn't
expect large blocks - a read of a 512 byte record MAY come up short due to
configuration errors. This would be a really new disk though... and things have not
finished settling out about the 4K sectors...

This is really speculative on my part, so don't take it as gospel.

I had not failed an fsck several months ago in this manner using the same disk, FWIW.

I do appreciate your speculative support and welcome your assistance, jp

---------- Post added at 11:17 AM CDT ---------- Previous post was at 10:53 AM CDT ----------

Running dd if=/dev/sda5 | hexdump -C scans the entire partition without aborting.

This is not correct. I miscalculated the scanned amount; dd stopped (at block 0x0063a000) after about 1.5% of the partition had been dumped.

[jpollard]

This looks like a failing disk.

Depending on the size of the failure and on available storage on another disk there
are a few things you can try -

Well, one I know of - use dd to copy the disk filesystem to a file:

dd conv=noerror if=/dev/sda5 of=file.iso bs=512

This should copy the disk, substituting a null block for the sectors with errors. then
try to fsck the file. Once this completes, you can mount the saved file via loopback
to retrieve data.

Also, once the disk has been copied it MAY be possible to do a low level format.
This doesn't always work, but sometimes it can clean up an error caused by weak
formatting on old disks that have not had a head crash. This WILL erase any data
on the disk.

[burnie]

This looks like a failing disk.

Yes - what I had hoped to do was map out the failing block and retain the disk.

Depending on the size of the failure and on available storage on another disk there
are a few things you can try -

Well, one I know of - use dd to copy the disk filesystem to a file:

dd conv=noerror if=/dev/sda5 of=file.iso bs=512

Thanks for this suggestion - it looks like a good path.

Also, once the disk has been copied it MAY be possible to do a low level format.
This doesn't always work, but sometimes it can clean up an error caused by weak
formatting on old disks that have not had a head crash. This WILL erase any data
on the disk.

This, I especially like. Thanks much, JP.

[gid99]

Another person is experiencing something like this...
http://forums.fedoraforum.org/showthread.php?t=246886

---------- Post added at 12:59 PM CDT ---------- Previous post was at 12:58 PM CDT ----------

and troubleshooting it as you are

[burnie]

Another person is experiencing something like this...
http://forums.fedoraforum.org/showthread.php?t=246886

---------- Post added at 12:59 PM CDT ---------- Previous post was at 12:58 PM CDT ----------

and troubleshooting it as you are

Not really the same problem, I think. I've a hard disk failure; no trouble mounting the drive it just doesn't read (or, possibly, write) properly.

---------- Post added at 05:43 PM CDT ---------- Previous post was at 03:13 PM CDT ----------

This looks like a failing disk.
dd conv=noerror if=/dev/sda5 of=file.iso bs=512

The command I actually used was

dd conv=noerror if=/dev/sda5 of=/mnt/newdev/sysimage.iso bs=512

The command identified three bad locations (sectors?): 12752, 2566000, and 71637552. At each failing location, there were eight successive attempts, each taking 12-25 seconds, to read the bad sector. Since I did not expect this, I tried a couple of less useful approaches. The entire process took about 2-1/2 hours.

Seems likely that a 133GB partition with three unreadable sectors is likely recoverable. I'll post later how that all works out.

Thanks to all who helped.

---------- Post added at 09:37 PM CDT ---------- Previous post was at 05:43 PM CDT ----------

OK, I'm still stuck. I did not mention up to now that the partition at issue is my LVM partition, and I'm completely clueless about how to mount it. I found that one needs to use losetup, but there seem to be more caveats in its usage than I have so far been able to grok.

If you'll kindly bear with me and this long post, here are data that appear relevant.

My partition table looks like this (please pardon the misalignment):
Device Boot Start End Blocks Id System
/dev/sda1 1 13 102400 7 HPFS/NTFS
Partition 1 does not end on cylinder boundary.
/dev/sda2 13 14260 114439168 7 HPFS/NTFS
/dev/sda3 * 14260 14286 204800 83 Linux
/dev/sda4 14287 30401 129443737+ 5 Extended
/dev/sda5 14287 30401 129442816 8e Linux LVM

I had formatted my 500GB drive as a single Linux partition, then executed the dd command.
Mounting the drive using

mount /dev/sdc1 /mnt/western

works fine, but

mount -o loop -t ext4 /mnt/western/sysimage.iso /mnt/newiso

clued me in by telling me the iso was not ext4.

An additional obvious complexity is that I'm running off the F12 install disk in rescue mode, and my mount table looks like this:
rootfs on / type rootfs (rw,relatime)
/proc on /proc type proc (rw,relatime)
/dev on /dev type tmpfs (rw,seclabel,relatime)
/dev/pts on /dev/pts type devpts (rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000)
/sys on /sys type sysfs (rw,relatime)
none on /tmp type tmpfs (rw,seclabel,relatime)
/dev/loop0 on /mnt/runtime type squashfs (ro,relatime)
/selinux on /selinux type selinuxfs (rw,relatime)
/dev/mapper/VolGroup-lv_root on /mnt/sysimage type ext4 (rw,seclabel,relatime,barrier=1,data=ordered)
/dev/sda3 on /mnt/sysimage/boot type ext4 (rw,seclabel,relatime,barrier=1,data=ordered)
/dev on /mnt/sysimage/dev type tmpfs (rw,seclabel,relatime)
/dev/devpts on /mnt/sysimage/dev/pts type devpts (rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000)
/dev/tmpfs on /mnt/sysimage/dev/shm type tmpfs (rw,seclabel,relatime)
/dev/proc on /mnt/sysimage/proc type proc (rw,relatime)
/dev/sysfs on /mnt/sysimage/sys type sysfs (rw,relatime)
/selinux on /mnt/sysimage/selinux type selinuxfs (rw,relatime)
/dev/sdc1 on /mnt/western type ext4 (rw,seclabel,relatime,barrier=1,data=ordered)

[gid99]

burnie,

The other person couldn't mount the drive either, and their drive may be corrupted as well. In their case, it was probably hosting the / file system and therefore it won't boot properly.

[jpollard]

OK, I'm still stuck. I did not mention up to now that the partition at issue is my LVM partition, and I'm completely clueless about how to mount it. I found that one needs to use losetup, but there seem to be more caveats in its usage than I have so far been able to grok.

I didn't think of that possibility... you would have to dd the lvm disk, not the raw
partition. This will skip the LVM header to get to the filesystem header.

Another possibility is to dd starting after the header, but I don't know how large that
is - 4k? 512 bytes? It would be easier to just dd the lvm device.