Broken LVM configuration?

[rbroberts]

I have a Fedora 12 x86 server with RAID1+LVM acting as a file server. It was running with two degraded RAID1 devices (one drive dead/missing); that is, two of the physical volumes were RAID1 devices with only one drive. I added a new pair of drives, created a single partition on each drive, and made a new RAID1 device which I then added as a physical volume to the volume group. I then migrated the the data off one old physical volume to the new one. Once that was complete, I removed the old physical volume from the group an and added the underlying drive to the other degraded RAID1 device.

At some point in the process, I screwed up. I've been trying to remember exactly where things went wrong. I believe that I may have deactivated the one RAID1 device and added it's underlying drive to the other RAID1 device without doing pvremove. A bit later, I know I screwed up when I incorrectly marked the original good drive as faulty. So I had one "new" drive that was still being synced and the original good drive marked faulty. Based on something I read elsewhere, I concluded that this was not truly fatal and that a reboot would cause the array to be reassembled correctly. So I rebooted.

At that point, the system would reboot and try to start up all of the old RAID1 devices, including the one that no longer existed. That included another one that I didn't mention, but which was not part of the volume group. However, after starting up all those, it would then try to start the volumes find that part of the volumes were missing and then reboot.

I rebooted from the Fedora 12 DVD in recovery mode. That booted perfectly seeing only the correct RAID1 devices and the correct LVM information. Everything mounted correctly except for the /boot partition (which is another RAID1 device). I let the system sit overnight in this mode while the arrays finished syncing. I thought I would be good at that point, but it still won't boot.

After staring at the messages for a while, I concluded I needed to rebuild the initramfs. So, I booted from the DVD, rebuilt the initramfs and rebooted. I no longer get messages about missing parts of the LVM, only the RAID1 devices that are actually being used (and in /etc/mdadm.conf) are started. However, after appearing to have started everything, including the volumes, I get a brief message saying it can't mount them and it immediately reboots.

My data is there, and I believe the on-disk info is correct so the DVD can find it all and mount it, but something is still messed up when I boot from the drives. I'm guessing at this point that dracut is using some configuration information which is not consistent with the on-disk info, but I'm not sure what it is.

I'm wondering if I need to go through the procedure on the Linux Journal website here http://www.linuxjournal.com/article/8874?page=0,0 or if there is something else to get this working.

TIA,

roland

[stevea]

Well that's a problem.

Yes, I think you need to go through the rebuild described in the article. As you will have learned rebuilding the Linux software raid mdadm stuff isn't completely trivial. Of course there is the basic question ... I assume you built an LVM on top of the RAID1 and not the other way around - both are possible.

I'm an advocate of the Linux SW raid - it works very well and is portable (if your mobo breaks then you can move your raid to another - unlike some BIOS-fake raids. Rebuilding after a failure isn't pretty .. could use a nicer tool.

FWIW - if you really want LVM, then examine the LVM mirroring (like RAID1) and striping (like RAID0), These have very similar performance to the mdadm versions and the reconstruction after failure is a lot easier IMO. So you can add two PVs from different volumes to the LVM and tell LVM to creat an LV that stripes across the pair.

-S

[rbroberts]

Yes, I built the LVM on top of RAID. The RAID1 devices are the physical volumes. Long ago, I just used RAID1. A few years back, I layered LVM on top of the RAID1 devices, and then added striping for performance. I have three pairs of drives, each pair a RAID1 device. The "small" pair is used as the physical volume for all of the system partitions as one volume group. The other two are the physical volumes for the other volume group and those logical volumes are all striped. I've done it this way for a few years now and been very happy with the results.

It's worked very well in all that time and that includes handling failed devices and migrating to new physical volumes. This wasn't the first time I had to do this exercise (more like the 3rd or 4th), just the first time I messed up so badly :-(

Thanks.

[stevea]

Sorry to hear that. Recovery is among the worst headaches. Now with 6 disks spinning LVM is an excellent fit for your config. What I hate to see are noobs with a single spindle using the F' default LVM and having difficulty.

Best wishes

[scott32746]

Hello,
In mirror/raid 1 setup you should only need to recreate your raid partition on your new drive add it back to failed raid
# mdadm /dev/md2 --add /dev/sdd1
Then update /etc/mdadm.conf
# cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf_orig
edit /etc/mdadm/mdadm.conf remove all ARRAY info
# mdadm --detail --scan >> /etc/mdadm/mdadm.conf

As the array rebuilds it is going to restore LVM in the new drive.

Now if the drive is not completely failed then you do need to remove it from the array
# /sbin/mdadm /dev/md2 -f /dev/hdd1

you can watch the rebuilt with
# watch -n0 cat /proc/mdstat

http://www.gagme.com/greg/linux/raid-lvm.php

[rbroberts]

Okay, all fixed :-)

I have learned far more than I really had planned about LVM. I'm not even quite sure how launching from the DVD managed to rebuild everything since the information dumped from the two physical volumes was not consistent. It took a while to match everything against what pvs and lvs were telling me, but other than that, the procedure at Linux Journal worked fine.

I'm a very happy camper. Now to figure out why my bacula director decided to stop talking to my clients (or vice versa)....

---------- Post added at 11:32 PM CST ---------- Previous post was at 11:22 PM CST ----------

Originally Posted by scott32746

Hello,
In mirror/raid 1 setup you should only need to recreate your raid partition on your new drive add it back to failed raid

Thanks, but actually none of that was relevant. The RAID1 arrays were just fine, as I said (perhaps not so clearly) in my first message. Somehow, the LVM information was not consistent on the two LVM physical volumes. Booting from the recovery DVD somehow was getting the correct info, booting from the hard drive was not. I didn't have to do anything other than boot from DVD to have the RAID arrays come up correctly and resync. Then I did the usual

mdadm -Esc partitions >> /etc/mdadm.conf

followed by an edit cleanup and rerunning dracut to create an updated initramfs.
I'm not completely clear how the LVM info in the physical volumes ended up different, but assume it was something I did out-of-sequence in the migration to a new PV.

In any event, the Linux Journal article info on recovering the RAID arrays was not relevant to my problem; I didn't need to recover the RAID arrays. But the info on recovering the LVM info was invaluable.

It would be nice if lvs and pvs would spit out values with the same labels as get used in the LVM config files /etc/lvm/backup/<volume-name>. That would have made it a lot faster to compare and get them synched back.

For good measure, I reran dracut after running vgcfgrestore just to be safe. At this point, I'm fully back up and have renamed my RAID arrays back to what they were supposed to be.

I never figured out how to slow down the automatic reboot. In order to capture the error messages, I ended up grabbing a digital camera and taking a screen shot before it could reboot