[SOLVED] raid1 changes corrupting pre-existing array

davewithheld · #1 4th July 2012, 07:17 PM

I always overlap my upgrades by doing a complete new install on a spare 50 gb partition, work out the problems while in dual-boot configuration, then switch over to the new OS when it's ready. I have a raid1 array with Linux raid autodetect (type fd) partitions on two drives. When I tried to go from FC13 to F16, the raid1 array created with and used by FC13 got corrupted by F16, which detected it, sort of, and degraded it by removing one of the disks. When I rebooted in FC13, it found it degraded but used the WRONG DISK!. It was days later that I noticed changes I had made while in FC13 were gone. Switching back and fourth between the two was playing havoc with my existing FC13 OS. I restored from a backup, rebuilt the array, and dumped F16. I am now faced with similar problems in F17 and need to get it resolved. Can someone enlighten me as to what has changed and why the two may be incompatible? The two OS's order the drives differently, but that shouldn't make a difference, should it?

Below is the output from fdisk, gdisk and mdstat on FC13 (I'm reluctant to reboot in F17 again), which reminds me: can I add raid=noautodetect to the "linux" lines in F17's /boot/grub2/grub.cfg while booted in FC13 to keep it from trying to assemble (and corrupting) the array on boot?

etc> fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xf081f081

Device Boot Start End Blocks Id System
/dev/sda1 * 1 24316 195318238+ 7 HPFS/NTFS
/dev/sda2 24317 30844 52436160 fd Linux raid autodetect
/dev/sda3 30845 121601 729005602+ 83 Linux

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003b679

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 6529 52444161 83 Linux
/dev/sdb2 6530 60801 435939840 83 Linux

Disk /dev/sdc: 300.1 GB, 300067970560 bytes
255 heads, 63 sectors/track, 36481 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000dfa62

Device Boot Start End Blocks Id System
/dev/sdc1 * 1 6529 52435968 83 Linux
/dev/sdc3 29500 36162 53520547+ fd Linux raid autodetect
/dev/sdc4 36163 36481 2562367 82 Linux swap / Solaris

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x469d60df

Device Boot Start End Blocks Id System
/dev/sdd1 * 1 121601 976760001 83 Linux

Disk /dev/md0: 53.0 GB, 53011808256 bytes
2 heads, 4 sectors/track, 12942336 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table
etc> gdisk -l /dev/sda
GPT fdisk (gdisk) version 0.7.1

Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present

************************************************** *************
Found invalid GPT and valid MBR; converting MBR to GPT format.
************************************************** *************

Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 8AC82E8D-F644-4C8A-B5F1-EC7C7BBA9FAB
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 8-sector boundaries
Total free space is 5099 sectors (2.5 MiB)

Number Start (sector) End (sector) Size Code Name
1 63 390636539 186.3 GiB 0700 Linux/Windows data
2 390636540 495508859 50.0 GiB FD00 Linux RAID
3 495508860 1953520064 695.2 GiB 0700 Linux/Windows data
etc> gdisk -l /dev/sdb
GPT fdisk (gdisk) version 0.7.1

Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present

************************************************** *************
Found invalid GPT and valid MBR; converting MBR to GPT format.
************************************************** *************

Disk /dev/sdb: 976773168 sectors, 465.8 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): CFE84E90-CA00-4D56-A787-29A3DBA276B3
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 976773134
Partitions will be aligned on 8-sector boundaries
Total free space is 5099 sectors (2.5 MiB)

Number Start (sector) End (sector) Size Code Name
1 63 104888384 50.0 GiB 0700 Linux/Windows data
2 104888385 976768064 415.7 GiB 0700 Linux/Windows data
etc> gdisk -l /dev/sdc
GPT fdisk (gdisk) version 0.7.1

Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present

************************************************** *************
Found invalid GPT and valid MBR; converting MBR to GPT format.
************************************************** *************

Disk /dev/sdc: 586070255 sectors, 279.5 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 42EC4170-28F3-4264-9174-163599BBABFC
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 586070221
Partitions will be aligned on 8-sector boundaries
Total free space is 369032423 sectors (176.0 GiB)

Number Start (sector) End (sector) Size Code Name
1 2048 104873983 50.0 GiB 0700 Linux/Windows data
3 473901435 580942529 51.0 GiB FD00 Linux RAID
4 580942531 586067264 2.4 GiB 8200 Linux swap

etc> cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc3[0] sda2[1]
51769344 blocks [2/2] [UU]

unused devices: <none>

Help would be MUCH appreciated.

Dave W.

***Gareth Jones*** · #2 5th July 2012, 01:37 AM

Out of interest, what kernel version are you using in F16? There was another report of a this for F16 recently, and I had a similar problem with F17 recur enough times that I filed a bug about it.

See http://forums.fedoraforum.org/showthread.php?t=281388, http://forums.fedoraforum.org/showthread.php?t=281211, and https://bugzilla.redhat.com/show_bug.cgi?id=832616.

davewithheld · #3 6th July 2012, 04:38 AM

Don't have F16 any more, gave up, and don't know what kernel it was when I dumped it. Now I'm trying to get F17 to stop breaking the array before I switch over. I'm guessing if I just fix the array from F17 and don't boot in FC13 it will stop breaking; that it's some incompatibility between the old and the new raid1 driver, but I won't be ready to do that for some time, and I really need to be able to switch back and forth. Have you had trouble with F17, alone, or are you battling a similar back-and-forth problem?

***Gareth Jones*** · #4 6th July 2012, 06:44 PM

No, its F17 on its own on my system. The problem recurred after being fixed until I gave up on mdraid and switched to Btrfs-RAID instead (which is working fine, but I wouldn't recommend playing with an experimental FS unless you're into that sort of thing). The first occurrence may have coincided with the 3.4 kernel update, although I can't prove that, hence asking what kernel F16 was on. (F16 always worked fine for me, but I switched to F17 quite soon after it was released and F16 still gets updates.)

davewithheld · #5 8th July 2012, 05:40 AM

I haven't tried to induce it yet, to prove it, but I'm beginning to think it's every time I upgrade the kernel, it corrupts the mdraid array. I'm betting it's a bug (or new feature) in mdadm that's triggered in a package install post-op. I'll post back here if I find any related bugs.

davewithheld · #6 12th July 2012, 02:45 PM

Just to let anyone who might stumble on this thread know, I haven't abandonded it. My F17 install is coming along nicely, and I have switched to having it be the default on boot. What I did to keep it from breaking the raid array was rename mdadm, which was sending mail to root every time it removed a (perfectly good) device from the array. I still haven't figured out why or determined if it hapens on kernel upgrades, but I will have to figure that out, soon, as I will be switching the / to raid1. Before I do that, I'll remove both partitions from the old array and change the partition type on one of them from "fd Linux raid autodetect" to "83 Linux". This will give me access to all the old stuff and put the new OS version on raid1. This is the method I have used for upgrading for over 10 years and has served me very well. I hope the new raid1 "features" wont keep it from working this time.

davewithheld · #7 25th August 2012, 04:56 AM

I think I found the answer to why this was happening. Lots of good info here: https://bugzilla.redhat.com/show_bug.cgi?id=606481. I'm marking this as solved.