LVM screwed up: block count exceeds size of device

[daytooner]

I have a system with a logical volume comprising several disks. All was working fine, until I tried to add another drive to the LV.

I went through the standard procedure: created the physical volume for the new drive; added that PV to the existing VG (that was checked and was clean and could mount prior to this); then tried to add the new PV to an existing LV. All went fine until it tried to resize the (ext4) filesystem to add the new drive space.

But that failed. And now when I try to mount the old LV, I get this in dmesg:

EXT4-fs: bad geometry: block count 102432768 exceeds size of device (86773760 blocks)

If I run e2fsck, I get this message:

e2fsck 1.41.4 (27-Jan-2009)
The filesystem size (according to the superblock) is 102432768 blocks
The physical size of the device is 86773760 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort<y>?

I've removed the new drive from the VolGroup as well as wiping out the PV label.

But still no luck. I can't (or REALLY REALLY DON'T) want to lose the data from the exiisting filesystem.

So what do I do now?

TIA

ken

[daytooner]

Basic scenario:

I had an (ext4) LV with several drives (PVs). All was working well, until I tried to add another drive to the LV.

Partway through, the file system resizing crashed. Now when I try to check the file system with e2fsck, I get a bad superblock error. dmesg reports: EXT4-fs: bad geometry: block count 102432768 exceeds size of device (86773760 blocks).

I tried running e2fsck with alternate superblocks, but I get either the bad superblock error, or
/dev/VolGroup3W/LogVol3W: recovering journal
e2fsck: unable to set superblock flags on /dev/VolGroup3W/LogVol3W

(and nothing in dmesg).

Is there any way to correct this? This seems like a fairly simple fix - although I don't know all of the complexities of the ext2/3/4 filesystems :-(. I am assuming/hoping that the original data -on the LV before the resizing - is still intact.

PLEASE PLEASE PLEASE!!! Someone help!

TIA

ken

[leigh123linux]

Threads merged, please don't double post again.

[SlowJet]

"then tried to add the new PV to an existing LV."

man lvextend step missing?

lvm pvcreate /dev/sdn
lvm vgextend volGroup3W /dev/sdn
lvm lvextend --extents %PVS /dev/mapper/VolGroup3W-LogVol3W /dev/sdn
resize2fs -P /dev/mapper/VolGroup3W-LogVol3W

It would help if you supplied some details about the disks
lvm lvs
lvm vgdisplay
lvm lvdisplay
df -l
fdisk -l

SJ

[daytooner]

Originally Posted by SlowJet

"then tried to add the new PV to an existing LV."

man lvextend step missing?

lvm pvcreate /dev/sdn
lvm vgextend volGroup3W /dev/sdn
lvm lvextend --extents %PVS /dev/mapper/VolGroup3W-LogVol3W /dev/sdn
resize2fs -P /dev/mapper/VolGroup3W-LogVol3W

It would help if you supplied some details about the disks
lvm lvs
lvm vgdisplay
lvm lvdisplay
df -l
fdisk -l

SJ

I did all of those steps (sorry if I originally left one out). The problem occurred during the resize2fs step.

The original LV was 331.02GB (LogVol3W). The disk I was trying to add is 111.79GB (sdg / dm-2) The difference between the total and free is a snapshot I did (LV 3WTMP).

Look at the e2fsck output below. The lower superblocks (apparently written out before the crash) see the difference in sizes. The others - at least where the others should be - don't seem to have been written out.

All I want right now is to get the orignal LogVol3W back. How can I do that?

TIA

ken

PS: sorry for the multiple posts

lvm> pvscan
Found duplicate PV ZTecEOM94ZD8YtnZo0rsDtKoSj5mMFPe: using /dev/dm-9 not /dev/dm-6
PV /dev/dm-3 VG VolGroup3W lvm2 [70.15 GB / 0 free]
PV /dev/dm-5 VG VolGroup3W lvm2 [93.16 GB / 0 free]
PV /dev/dm-9 VG VolGroup3W lvm2 [93.16 GB / 0 free]
PV /dev/sdd1 VG VolGroup3W lvm2 [74.55 GB / 0 free]
PV /dev/dm-2 VG VolGroup3W lvm2 [111.79 GB / 101.44 GB free]
PV /dev/sda2 VG vg_elmer lvm2 [28.44 GB / 0 free]
Total: 6 [471.24 GB] / in use: 6 [471.24 GB] / in no VG: 0 [0 ]
lvm> lvs
Found duplicate PV ZTecEOM94ZD8YtnZo0rsDtKoSj5mMFPe: using /dev/dm-9 not /dev/dm-6
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
3WTMP VolGroup3W -wi-a- 10.35G
LogVol3W VolGroup3W -wi-a- 331.02G
lv_root vg_elmer -wi-ao 24.52G
lv_swap vg_elmer -wi-ao 3.92G

(LogVol3W is the LV I am having trouble with; 3WTMP is actually a snapshot of it)

lvm> vgdisplay
--- Volume group ---
VG Name vg_elmer
System ID
Format lvm2
------------snip out root volume info---------------------

--- Volume group ---
VG Name VolGroup3W
System ID
Format lvm2
Metadata Areas 5
Metadata Sequence No 10
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 0
Max PV 0
Cur PV 5
Act PV 5
VG Size 442.80 GB
PE Size 4.00 MB
Total PE 113358
Alloc PE / Size 87389 / 341.36 GB
Free PE / Size 25969 / 101.44 GB
VG UUID 2CXUUY-j6nu-07qP-kKd1-ZkBJ-hNwf-iunkwK

lvm> lvdisplay
--- Logical volume ---
LV Name /dev/vg_elmer/lv_root
VG Name vg_elmer
---------snip out lv_root info--------------------

--- Logical volume ---
LV Name /dev/vg_elmer/lv_swap
VG Name vg_elmer

---------snip out lv_swap info--------------------

--- Logical volume ---
LV Name /dev/VolGroup3W/LogVol3W
VG Name VolGroup3W
LV UUID eBQgqM-RHra-ruaK-CcEt-stnX-U9yR-Uz4UH5
LV Write Access read/write
LV Status available
# open 0
LV Size 331.02 GB
Current LE 84740
Segments 4
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:7

--- Logical volume ---
LV Name /dev/VolGroup3W/3WTMP
VG Name VolGroup3W
---------snip out snapshot info--------------------


[root@Elmer ~]# df -l
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_elmer-lv_root
25302908 2465668 21551916 11% /
/dev/sda1 198337 21780 166317 12% /boot
tmpfs 1036592 0 1036592 0% /dev/shm
[root@Elmer ~]# fdisk -l

Disk /dev/sda: 30.7 GB, 30750031872 bytes
255 heads, 63 sectors/track, 3738 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xd23e7832

Device Boot Start End Blocks Id System
--------------snip out system disk partitions-----------------

Disk /dev/dm-0: 26.3 GB, 26323451904 bytes
255 heads, 63 sectors/track, 3200 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 4211 MB, 4211081216 bytes
255 heads, 63 sectors/track, 511 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-1 doesn't contain a valid partition table

Disk /dev/sdb: 100.0 GB, 100030242816 bytes
64 heads, 32 sectors/track, 95396 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk identifier: 0x00059c90

Device Boot Start End Blocks Id System

Disk /dev/sdc: 100.0 GB, 100030242816 bytes
255 heads, 63 sectors/track, 12161 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00008166

Device Boot Start End Blocks Id System
/dev/sdc1 1 12161 97683201 83 Linux

Disk /dev/sdd: 80.0 GB, 80054059008 bytes
255 heads, 63 sectors/track, 9732 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000f3f53

Device Boot Start End Blocks Id System
/dev/sdd1 1 9732 78172258+ 83 Linux

Disk /dev/sdf: 75.3 GB, 75329372160 bytes
255 heads, 63 sectors/track, 9158 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00056df3

Device Boot Start End Blocks Id System
/dev/sdf1 1 9158 73561603+ 83 Linux

Disk /dev/sde: 75.3 GB, 75329372160 bytes
64 heads, 32 sectors/track, 71839 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/sde doesn't contain a valid partition table

Disk /dev/sdg: 120.0 GB, 120034123776 bytes
64 heads, 32 sectors/track, 114473 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/sdg doesn't contain a valid partition table

Disk /dev/dm-2: 120.0 GB, 120034091520 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-2 doesn't contain a valid partition table

Disk /dev/dm-3: 75.3 GB, 75329339904 bytes
255 heads, 63 sectors/track, 9158 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-3 doesn't contain a valid partition table

Disk /dev/dm-4: 100.0 GB, 100030152704 bytes
255 heads, 63 sectors/track, 12161 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00008166

Device Boot Start End Blocks Id System
/dev/dm-4p1 1 12161 97683201 83 Linux

Disk /dev/dm-5: 100.0 GB, 100030152704 bytes
255 heads, 63 sectors/track, 12161 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00059c90

Device Boot Start End Blocks Id System

Disk /dev/dm-6: 100.0 GB, 100027597824 bytes
255 heads, 63 sectors/track, 12160 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-6 doesn't contain a valid partition table

Disk /dev/dm-7: 355.4 GB, 355425320960 bytes
255 heads, 63 sectors/track, 43211 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-7 doesn't contain a valid partition table

Disk /dev/dm-8: 11.1 GB, 11110711296 bytes
255 heads, 63 sectors/track, 1350 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-8 doesn't contain a valid partition table

Disk /dev/dm-9: 100.0 GB, 100027597824 bytes
255 heads, 63 sectors/track, 12160 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/dm-9 doesn't contain a valid partition table


[[root@Elmer ~]# e2fsck /dev/mapper/VolGroup3W-LogVol3W
e2fsck 1.41.4 (27-Jan-2009)
The filesystem size (according to the superblock) is 102432768 blocks
The physical size of the device is 86773760 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort<y>? y

[root@Elmer ~]# e2fsck -b 11239424 /dev/mapper/VolGroup3W-LogVol3W
e2fsck 1.41.4 (27-Jan-2009)
/dev/mapper/VolGroup3W-LogVol3W: recovering journal
e2fsck: unable to set superblock flags on /dev/mapper/VolGroup3W-LogVol3W

[SlowJet]

ok, fisrt, I don't like this message. /dev/mapper/VolGroup3W-LogVol3W: recovering journal
Why is it in recovery when you are only resizing the end?

Second, why such a big number for supperblock? -b 11239424
Just use the next one on the first disk.

"The lower superblocks (apparently written out before the crash) see the difference in sizes. The others - at least where the others should be - don't seem to have been written out.

All I want right now is to get the orignal LogVol3W back. How can I do that?"

Are you confused here? The superblock doesn't need to be on the bad disk.

But first make sure they are really superblocks for a 4096 blocksize.
e2fsck -B 4096 -n /dev/mapper/VolGroup3W-LogVol3W #note the man page for the -n option uses the wrong case of B, use -B for blocksize, -b for superblock location.

Once you get a new superblock location id'ed
e2fsck -b newsmaller# -B 4096 /dev/mapper/VolGroup3W-LogVol3W

If that doesn't work, I'm thinking it is the partition table. (Which make more sense because there was nothing wrong with your ext f/s before you started.

To back out, just reduce the LV back to the last PE of the previous disk before the new one was lvextend.
Then remove the pv from the VG with vgreduce
lvm vgreduce VolGroup3W /dev/sdg1

SJ

[daytooner]

Originally Posted by SlowJet

ok, fisrt, I don't like this message. /dev/mapper/VolGroup3W-LogVol3W: recovering journal
Why is it in recovery when you are only resizing the end?

Second, why such a big number for supperblock? -b 11239424
Just use the next one on the first disk.

I think the recovery and the big number for the superblock has to do with the fact that the resize2fs crashed before it was finished. Just IMHO.

Are you confused here? The superblock doesn't need to be on the bad disk.

No, I'm not confused. I just used that superblock number as an example of one that would have been in the new disk, and was reported to be a backup superblock.

To back out, just reduce the LV back to the last PE of the previous disk before the new one was lvextend.
Then remove the pv from the VG with vgreduce
lvm vgreduce VolGroup3W /dev/sdg1

SJ

The LV was already at the same size (according to the LVM) as it was before I did the lvextend, so lvreduce wouldn't do anything.

I did remove the PV from the VG successfully.

But, still, e2fsck thinks that the LV is the larger size. As does mount.

BTW: I get the same results (as below) without the -B 4096.

[root@Elmer ~]# e2fsck -n -B 4096 /dev/mapper/VolGroup3W-LogVol3W
e2fsck 1.41.4 (27-Jan-2009)

The filesystem size (according to the superblock) is 102432768 blocks
The physical size of the device is 86773760 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? no

Group descriptor 0 checksum is invalid. Fix? no

Group descriptor 1 checksum is invalid. Fix? no

-----------------------snip----------------------------------
Group descriptor 2647 checksum is invalid. Fix? no

Group descriptor 2648 checksum is invalid. Fix? no

/dev/mapper/VolGroup3W-LogVol3W contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Error reading block 86802432 (Invalid argument) while getting next inode from scan. Ignore error? no

Error while scanning inodes (21700608): Can't read next inode
e2fsck: aborted

[root@Elmer ~]# vgdisplay

--- Volume group ---
VG Name VolGroup3W
System ID
Format lvm2
Metadata Areas 4
Metadata Sequence No 12
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 4
Act PV 4
VG Size 331.02 GB
PE Size 4.00 MB
Total PE 84740
Alloc PE / Size 84740 / 331.02 GB
Free PE / Size 0 / 0
VG UUID 2CXUUY-j6nu-07qP-kKd1-ZkBJ-hNwf-iunkwK

[root@Elmer ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LogVol3W VolGroup3W -wi-a- 331.02G
lv_root vg_elmer -wi-ao 24.52G
lv_swap vg_elmer -wi-ao 3.92G

From dmesg:

EXT4-fs: bad geometry: block count 102432768 exceeds size of device (86773760 blocks)

So what do I do now?

TIA

ken

[SlowJet]

I don't know.
It is an ext4 or hardware or bug.
Adding another disk to VG would not cause ext f/s problems back on disk 1.

Are you fully updated for e2fs-progs and LVM2 and mapper, and kernel with the latest bits?

Make a post on fedora-list see if anyone can help.
https://www.redhat.com/mailman/listinfo/fedora-list

SJ

[daytooner]

Praise the gods of Linux! I repent! I am reborn a true believer!

Somehow, magically, everything got right with this VG/LV. How I don't really know, but I am not pushing my luck any farther.

What I did:

1) removed the new PV from the VG (VolGroup3W)
2) restored an archived version of the metadata for VolGroup3W (vgcfgrestore), one that was before I had added the new PV.
3) added the new disk back into the VolGroup3W, then extended LogVol3W to the new max that included the new PV. Magically that ran 100% successful.
4) Ran resize2fs on LogVol3W - but it first complained and told me to run e2fsck first. With fingers crossed, holding my breath, praying to whatever deity would listen, I did that. And, yea, verily, it ran through cleanly!!!
5) Ran resize2fs again, and yes, repentant sinner that I am, that too was successful.

Just to make sure of everything, I re-ran e2fsck (with -f) and it completed successfully (all clean). I then mounted the newly expanded LogVol3W, and yes, everything was once again right in the world. The previous data was still there, and valid, and the size of the new filesystem was correct (the original size + the new disk size).

Thank you SlowJet, and any others that helped.

ken

PS: I am a bit reluctant to mention this, but do so only as a warning to others: I originally tried to add the new disk to the LV using system-config-lvm. It had worked for me in the past, but this time it just screwed things up royally. Maybe there is a bug in it (although all it really does is spawn the lvm commands, and displays a graphic representation of the lvm), but I won't take any chances any more.

[SlowJet]

The system-config-lvm most likely can not handle the /dev/sdg case but only resize lv on a givven pv.


Sj