e2fsck on a large volume taking days to run

[warnockm]

Even your "maintenance mode" has job control, doesn't it?

wow. how have i never used this feature? I moved it to the background. I found that the system is using 13GB of swap space from the 750GB available. I ran the hdparm command on the drives and found them all reading at the same speed, around 1200mb/s (cached) 100mb/s (buffered).

I read that you can run testdisk while fsck is running (its just read only at this point). I shows my directories and files on the broken filesystem, but the directory that has 90% of the files shows up AS A FILE! The directory flag at the beginning of the list doesn't say "d" for this directory. Isn't that kind of weird? The rest appear fine and have contents.

[george_toolan]

There you go. Thrashing would explain the slowliness.

Your file system is messed up and that's why e2fsck needs a lot of memory.

Have you checked the status of your raid with your raid admin tools?

[warnockm]

There you go. Thrashing would explain the slowliness.

Your file system is messed up and that's why e2fsck needs a lot of memory.

Just curious what you mean by thrashing? Is that based on the memory usage? I mean, is that good that its doing so much work to fix it?

Have you checked the status of your raid with your raid admin tools?

the RAID array is fine. as i said i had 2 redundant hard drives in this array. I can't seem to get one to come up, but the other 5 drives are running fine. The admin tools state that the array is running, but does note the one drive is missing. This means the integrity of the array is fine; this is just a filesystem problem.

as always, i really appreciate everyone's input!

[kaefert]

@warnockm:
I'm curious, could you tell us how your story ended? did e2fsck finish? did you save your data with testdisk?

[warnockm]

I let it run literally for weeks. I had a UPS running in case the power went out. In the end the computer locked up (not sure why) and then i gave up. I rebuilt the system and array and this time made 4 partitions on the 8TB array so that if fsck had to run again they would be more manageable file systems sizes.

[kaefert]

Oh my... I hope I have better luck.. Was the data recoverable afterwards, and what tools did you use for it?

If you're interested in my story, I've documented it here:
http://serverfault.com/questions/446...enough-memory-
(in my case its an external USB 3tb Disk with a single ext4 partition)

[george_toolan]

e2fsck can display a progress bar if you call it with the -C parameter:

       -C fd  This option causes e2fsck to write completion information to the
              specified file descriptor so that the progress of the filesystem
              check  can  be monitored.  This option is typically used by pro‐
              grams which are running e2fsck.  If the file  descriptor  number
              is  negative, then absolute value of the file descriptor will be
              used, and the progress information will be suppressed initially.
              It  can later be enabled by sending the e2fsck process a SIGUSR1
              signal.  If the file descriptor  specified  is  0,  e2fsck  will
              print  a  completion  bar  as  it goes about its business.  This
              requires that e2fsck is running on a video console or terminal.

Later you can enable or disable this feature by sending signals to the e2fsck process with the kill command:

SIGNALS
       The following signals have the following effect when sent to e2fsck.

       SIGUSR1
              This signal causes e2fsck to start displaying a  completion  bar
              or  emitting  progress  information.   (See discussion of the -C
              option.)

       SIGUSR2
              This signal causes e2fsck to stop displaying a completion bar or
              emitting progress information.

[jpollard]

I let it run literally for weeks. I had a UPS running in case the power went out. In the end the computer locked up (not sure why) and then i gave up. I rebuilt the system and array and this time made 4 partitions on the 8TB array so that if fsck had to run again they would be more manageable file systems sizes.

The lockup likely was memory related. More swap would have allowed it to run longer.

As I recall, fsck has to rebuild the directory tree - and that rebuild is done in memory. When it runs out the system deadlocks.

[george_toolan]

Swapping to a HDD is extremely slow and he already added 750 GiB ;-)

[jpollard]

Yes, but it still locked up.

We don't really know how large (by large, how many files to scan). Up to a couple of million shouldn't be a problem. 10-15 should start seeing a slowdown. 50 million should take quite a while.

One UNIX based system modified their fsck to not use memory, but a memory mapped disk file just to avoid such deadlocks (I believe it was Solaris, with a 15TB filesystem with more than 60 million files).

One other problem is with lost files (the directory they were in was no longer a directory...) and it is possible something funky happened trying to link thousands of files into lost+found.