Unable to replace drive in software RAID-5 array (Fedora 14)

Tamater · #1 18th November 2011, 06:32 PM

I'm new to managing a RAID-5 array here at work. This is now the second drive that has failed on us out of a batch of 4 in less than a year and we're quite unhappy about it (looking at you, WD). I can't seem to get the new drive to be accepted into the RAID.

Here's my setup:

OS: Fedora 14
Drives: 1 System drive, 3 exisiting 2TB drives in RAID-5 (of the original 4) All SATA

So, I removed the old failing drive (it has been sent off to WD) and inserted my new drive.

Upon boot, I loaded up palimpsest and clicked on the RAID listed on the left:

State: Not running, partially assembled
Components: 4
in the red "Volume" bar graphic it says "RAID Array is not running" however, the "Stop RAID Array" button is enabled. It also says "/dev/md127" in the title bar.

When I click Edit Components, and Add Spare, it allows me to add the new drive.
Then, back on the main palimpsest screen, it still shows the RAID as not running....I click Start Array and gives the following error:

Error assembling array: mdadm exited with exit code 1: mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.

This error occurs whether I add the new drive or not.

[aside] Is this normal that it automounts the RAID to md127 on boot, but tries to be md0 after stopping then restarting?

Also, when I do a /sbin/mdadm --detail /dev/md127, I get the following:

[root@spectrum ~]# /sbin/mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Mon May 9 11:46:07 2011
Raid Level : raid5
Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
Raid Devices : 4
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Tue Nov 15 15:55:43 2011
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : :RAID
UUID : 68fba474:da41eab6:7071f9fc:baa56434
Events : 56793

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 1 1 active sync /dev/sda1
2 0 0 2 removed
3 0 0 3 removed

It's as if it is saying there are only 2 drives connected when I definitely have the original 3 (plus the new one).

How do I get this sorted out? I am not familiar with using mdadm or any other text-based admin of RAID..... These drives contain important medical research data backups and I absolutely must get it back. (some stuff on it has no other backups I've now realized)

Thanks in advance,

Matt

leigh123linux · #2 18th November 2011, 07:31 PM

Post

Code:

mdadm --detail --scan >> /etc/mdadm.conf

And

Code:

fdisk -l

Also list the dev names for the new and missing drives

Tamater · #3 18th November 2011, 07:33 PM

Quote:

Originally Posted by leigh123linux

Post

Code:

mdadm --detail --scan >> /etc/mdadm.conf

And

Code:

fdisk -l

Also list the dev names for the new and missing drives

When I do that command, it does nothing but go back to the prompt.

Here is my /etc/mdadm.conf file:

# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 metadata=1.2 name=:RAID UUID=68fba474:da41eab6:7071f9fc:baa56434
ARRAY /dev/md0 metadata=1.2 name=:RAID UUID=68fba474:da41eab6:7071f9fc:baa56434

leigh123linux · #4 18th November 2011, 07:40 PM

Quote:

Originally Posted by Tamater

When I do that command, it does nothing but go back to the prompt.

Here is my /etc/mdadm.conf file:

# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md0 metadata=1.2 name=:RAID UUID=68fba474:da41eab6:7071f9fc:baa56434
ARRAY /dev/md0 metadata=1.2 name=:RAID UUID=68fba474:da41eab6:7071f9fc:baa56434

Ok post

Code:

mdadm --detail /dev/md0

Tamater · #5 18th November 2011, 07:42 PM

/sbin/mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Mon May 9 11:46:07 2011
Raid Level : raid5
Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
Raid Devices : 4
Total Devices : 2
Persistence : Superblock is persistent

Update Time : Tue Nov 15 15:55:43 2011
State : active, FAILED, Not Started
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : :RAID
UUID : 68fba474:da41eab6:7071f9fc:baa56434
Events : 56793

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 1 1 active sync /dev/sda1
2 0 0 2 removed
3 0 0 3 removed

leigh123linux · #6 18th November 2011, 07:50 PM

Ok add the missing drives using (change/dev/sdxx to match the missing drives)

add the first drive

Code:

mdadm --add /dev/md0 /dev/sdxx

check it's added with

Code:

 mdadm --detail /dev/md0

then add the second drive

Code:

mdadm --add /dev/md0 /dev/sdxx

check it again

Code:

mdadm --detail /dev/md0

Tamater · #7 18th November 2011, 07:53 PM

Quote:

Originally Posted by leigh123linux

Ok add the missing drives using (change/dev/sdxx to match the missing drives)

add the first drive

Code:

mdadm --add /dev/md0 /dev/sdxx

check it's added with

Code:

 mdadm --detail /dev/md0

then add the second drive

Code:

mdadm --add /dev/md0 /dev/sdxx

check it again

Code:

mdadm --detail /dev/md0

what do you mean by first/second drive? there are 4. 3 that should have the degraded RAID on them and a new empty one. in that --detail, it only shows 2 of the 3 RAID drives being connected.....so which order do you mean?

leigh123linux · #8 18th November 2011, 07:58 PM

Quote:

Originally Posted by Tamater

what do you mean by first/second drive? there are 4. 3 that should have the degraded RAID on them and a new empty one. in that --detail, it only shows 2 of the 3 RAID drives being connected.....so which order do you mean?

Post

Code:

su
fdisk -l

The array only has two members currently, whats the dev address for the third drive (missing drive) and the dev address for the new drive?
It doesn't matter which order they are added in.

Tamater · #9 18th November 2011, 08:15 PM

Ok, I understand. I won't be back at that server today, but I will do that on Monday and post my results.

Thanks for your help.

leigh123linux · #10 18th November 2011, 08:19 PM

Quote:

Originally Posted by Tamater

Ok, I understand. I won't be back at that server today, but I will do that on Monday and post my results.

Thanks for your help.

You also need to prepare the new drive before adding it. i.e partition and add raid flag (see step #1)

http://www.jamierf.co.uk/2009/11/04/...n-ubuntu-9-10/

Tamater · #11 23rd November 2011, 08:04 PM

Hi guys, I hope you're still with me. Finally found some time to get back to this problem. Here's an update.

I tried adding the 3rd drive to the array (as mentioned above) using /sbin/mdadm --add /dev/md0 /dev/sda1

Result:

Code:

[root@spectrum ~]# /sbin/mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon May  9 11:46:07 2011
     Raid Level : raid5
  Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Wed Nov 23 14:52:18 2011
          State : active, FAILED, Not Started
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : :RAID
           UUID : 68fba474:da41eab6:7071f9fc:baa56434
         Events : 56794

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       0        0        2      removed
       3       0        0        3      removed

       4       8        1        -      spare   /dev/sda1

Now it has that one as a spare? How do I make it RaidDevice 2 or 3 as it should be?

Actually, now when I try to add it is saying:

Code:

[root@spectrum ~]# /sbin/mdadm --add /dev/md0 /dev/sda1
mdadm: add new device failed for /dev/sda1 as 4: Invalid argument

This is obviously a much bigger problem than adding the new drive.

I should also mention that these 4 drives were moved from an older computer into a much better giant case with better CPUs recently. So, it was moved into a new installation of Fedora. However, it mounted and worked just fine in this new system. It was at this point that I noticed one of the drives had a whole bunch of bad sectors and I sent it back to WD. Would this move have anything to do with it? I could try putting the 3 drives back into the old system to see if that works...

Thanks in advance,

Matt

leigh123linux · #12 23rd November 2011, 09:05 PM

Try

Code:

su
mdadm --grow /dev/md0 --raid-devices=3
mdadm /dev/md0 --assemble -u  68fba474:da41eab6:7071f9fc:baa56434

Tamater · #13 24th November 2011, 06:46 PM

Result:

Code:

[root@spectrum ~]# mdadm --grow /dev/md0 --raid-devices=3
mdadm: /dev/md0: Cannot get array details from sysfs

The assemble line doesn't do anything.....

---------- Post added at 01:43 PM ---------- Previous post was at 01:28 PM ----------

I just don't understand what happened between me taking out the old error-ridden drive and putting in this new drive to make me lose yet another drive from the original 3?? How is this even possible? This is the second time I have lost a RAID. Makes me wonder what RAID is even good for since both times a drive has failed, I have lost the whole thing.

Any more suggestions? (thanks for your help so far)

---------- Post added at 01:46 PM ---------- Previous post was at 01:43 PM ----------

This has turned from Unable to replace drive into lost 2nd drive in 4-drive RAID-5 array, inexplicably....

Why is it just adding it as a spare when it is clearly NOT a spare, but a part of the original array?

leigh123linux · #14 24th November 2011, 06:51 PM

Post

Code:

mount -i |grep sysfs

Tamater · #15 24th November 2011, 06:54 PM

Code:

[root@spectrum ~]# mount -i |grep sysfs
sysfs on /sys type sysfs (rw)
[root@spectrum ~]# mdadm --grow /dev/md0 --raid-devices=3
mdadm: /dev/md0: Cannot get array details from sysfs

---------- Post added at 01:54 PM ---------- Previous post was at 01:53 PM ----------

Still getting this:

Code:

[root@spectrum ~]# /sbin/mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon May  9 11:46:07 2011
     Raid Level : raid5
  Used Dev Size : 1953510400 (1863.01 GiB 2000.39 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Thu Nov 24 13:29:45 2011
          State : active, FAILED, Not Started
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

           Name : :RAID
           UUID : 68fba474:da41eab6:7071f9fc:baa56434
         Events : 56796

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       0        0        2      removed
       3       0        0        3      removed

       4       8        1        -      spare   /dev/sda1

sda1 is listed as a spare still and it is NOT a spare, but one of the original 3.