LogVol Unexpected Inconsistency ERROR

[Kommissar]

So today I ran Synaptic to get the current batch of updates. One of the updates was a new Kernel for Fedora systems, yay. I reboot so I can use the new kernel (and possibly uninstall the old one) but an error occurs.

As the machine is starting up it does it's usual detecting hardware and setting hostname. After that, it goes to check the root filesystem and bombs with this error:

Checking root filesystem
/dev/VolGroup00/LogVol00: UNEXPECTED INCONSISTENCY; RUN 
fsck MANUALLY.

*** An error occured during the filesystem check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
*** {Some stuff about disabling SELinux here}

So I log into the shell as root and run fsck with the option to check for bad blocks and answer yes to all questions about relocating sectors. It goes through and claims to fix some orphaned sectors and nodes, relocates some sectors that seem to have to references (or something along those lines) and some other things which I did not catch completely. It seems to loop and start itsself again, so after the second time it did this I interrupted it and rebooted. Alas, I have the same error and do not know how to fix this. Please help me.

Thank you in advance.

[SlowJet]

The disk drive may be bad.
It can not be fixed if it is bad.
Download or use the disk MFG diagnostic software to boot up and run the tests on the disk.
If it says it is good then you can use it again for a re-install.
Otherwise, go to the disk MFG site and determine if the SERIAL Number of your disk can be RMA'ed for a new disk.

SJ

[Kommissar]

Originally Posted by SlowJet

The disk drive may be bad.
It can not be fixed if it is bad.
Download or use the disk MFG diagnostic software to boot up and run the tests on the disk.
If it says it is good then you can use it again for a re-install.
Otherwise, go to the disk MFG site and determine if the SERIAL Number of your disk can be RMA'ed for a new disk.

SJ

I think what happened was that it had only read access to the disk. I rebooted using the Fedora CDROM and booted to the rescue kernel and ran e2fsck -fy /dev/VolGroup00/LogVol00 to try and fix it again. As of now it's still going. I'll post the results. I did note that it deleted some nodes that referenced some bad blocks.

If this doesn't work is there anything else that I can do? I really do NOT want to reinstall. Can I backup? How do you backup a linux file system and retain all the SELinux attributes. On a side note, is SELinux even worth running?

[Fmac]

If you get any progress on this, I'd rather like to know - I'm in the same place. Left my box fscking at home - it's probably been going for nearly 10 hours, and last I checked, it seemed to be giving the same messages (mostly relocating data from nodes that seem to have been double-booked) in an endless loop.

[SlowJet]

I would say the RO was the problem.
You did the correct thing to check for bad sectors.

The only backup I know of is the most current cp using the -P and other options, but I haven't the facts as to weather the SELinux attributes are part of that program as a backup.

The other is dump. Again not enough info to relate to SELinux and LVM. (Nobody talks much about the truth, yea know. )

1. Check the disk with the MFG diagnositc test. (It won't hurt and then you will know the status of the disk. Because a bad disk isn't worth fooling around with.)

2. Try to CP or Dump ecah File System with all the settings you can figure out.

3. Most logical, out of all the install of FC3, only a small part of it is your data and the configuration settings.
Backup your data, write down your settings in a fiile and store off line.
If FC3 requires reinstalling it is not so bad, just time. 100 millions Windows people do it every 6 months. It's a piece of cake.

SJ

The proper backup for LVM is a snapshot copy. This is not implimented in LVM2 yet.

[SlowJet]

And, YES, SELinux is a must for more reasons than you can shake a stick at.

SJ

[Kommissar]

Originally Posted by Fmac

If you get any progress on this, I'd rather like to know - I'm in the same place. Left my box fscking at home - it's probably been going for nearly 10 hours, and last I checked, it seemed to be giving the same messages (mostly relocating data from nodes that seem to have been double-booked) in an endless loop.

Well, I solved the endless loop (I think) by using the linux install cd and booting to rescue mode to run e2fsck. But now it seems like this is taking forever. After it removed a node with too many bad references it has just been scanning (I'm assuming, i have no output, but the HDD light is flashing) for the past 5 hours or so.

How long should e2fsck take to scan around 130 gigs in a LVM?

[Fmac]

Over here, I've been doing much the same . . . Running e2fsck from the boot cd fixed a few squawks, but not all. Note that I only have about 40 gigs to scan, also LVM, and it doesn't take more than five minutes for it to go through. Working time may be cut down by the fact that it's not actually performing repairs, but just returning error messages.

So, what I get from e2fsck is a few hundred permutations on the following:

Group xxx's inode table at xxxxxxx conflicts with some other fs block. Relocate? (yes)

For each one of these prompts to to which I respond yes, it eventually returns the following:

Error allocating x contiguous block(s) in block group xxx for inode table: Could not allocate block in ext2 filesystem.

After all of these, it fly through the rest of the passes with no problem, then give me my prompt back with a stern warning that the filesystem still contains errors.

[Kommissar]

FMac... I'd say we're screwed. But I'll leave it to the experts to confirm that.

[Iram Hernandez]

I'm having the same issues. I tried adding a second hard disk to my volume group. I ran the following commands

fdisk /dev/hdb
n (new)
p (primary)
1 (first partition)
t (type)
8e (lvm)
w (write to disk)

pvcreate /dev/hdb1

vgextend VolGroup00 /dev/hdb1

lvextend /dev/VolGroup00/LogVol00 /dev/hdb1

#rebooted with rescue cdrom

lvm vgchange -a y Volume00

lvm lvchange -a y /dev/VolGroup00/LogVol00

e2fsck -f /dev/VolGroup00/LogVol00 (competed without errors)

resize2fs -f /dev/VolGroup00/LogVol00 (added the -f because it was complaing that I should run e2fsck -f and I had already done that)

e2fsck -f /dev/VolGroup00/LogVol00

This is where I started getting the errors described above. I can access my files from the rescue disk, but I can not boot because of LVM inconsistancy errors.

[Iram Hernandez]

OK, I fought my way out of this one. After doing some research I found out that the resize2fs that comes on the stock fedora core 3 rescue disk is hosed. I think it is version 1.35. Version 1.36 addresses some of the problems caused by the broken version. What I wound up doing was burning a fedora core 4 test 3 rescue disk which comes with the correct version of the e2fs tools. I would have prefered to create a fedora core 3 rescue disk with the updated tools but I couldn't figure out how to do that in a timely fashion.

Once I was booted from the fedora core 4 test 2 rescue disk I ran the following commands:

#debugfs -w /dev/VolGroup00/LogVol00 -R "features ^resize_inode"
#e2fsck

e2fsck was able to find the errors and fix them. I was once again able to boot my system.

The project page e2fsprogs at http://sourceforge.net/project/shown...ase_id=296242a suggested that I run ext2prepare after my file system was fixed in order to reenable online file system resizing (which is what I should have probably done to begin with). I'm a bit hesitant to try this at this point after all the nightmares. Does anybody have experience with this procedure?

[zimmertiger]

Hello,

Originally Posted by Iram Hernandez

Does anybody have experience with this procedure?

First , a big Thank you to Iram for this solution - it helped me to repair my filesystem.
What has happened:
I had set up a FC3 box with software RAID, 3 ATA-disks , a RAID1 for "/" and "/boot" , a RAID5 with LVM for the data ...
After a few problems to set up LVM with the anaconda-GUI and a few boot's later i configured the LVM by command line and every thing works fine. To try it out , i resized one of my logical volumes as descibed in the "LVM HOWTO"
Later i copied some data on the volume , spent a few hours with "yum" and run a update via yum after my local repositority works fine.
Reboot , BANG ! Incosistent Filesystem on the resized logical volume : "Group 125's inode table ...'

Running Iram's commands using a FC4 rescue disk works fine !