SSD Problem - Cannot Access lost+found

Ridd1ck · #1 2nd December 2011, 01:52 PM

Hello,
I Have Fedora 15 64bit and in this server i run mysql cluster data nodes.
After Force shutdown of my server because of power lose of the data center,
i have no acces to lost+found folder.
What can i do now?

I use :

Code:

ls -lh

And the output is:

Code:

total 8.0K
drwxr-xr-x 2 root  root  4.0K Dec  2 03:03 backup
d????????? ? ?     ?        ?            ? lost+found
drwxr-xr-x 4 mysql mysql 4.0K Nov 30 14:32 mysql-cluster

And when i try:

Code:

cd lost+found

The output is

Code:

bash: cd: lost+found: Input/output error

The strange think is that this SSD have double size than the other SSD in the other same Server that i have who running the same things and have the same data.

When i try at the server with the problem:

Code:

df -h

The output is:

Code:

Filesystem            Size  Used Avail Use% Mounted on
rootfs                 66G  2.4G   60G   4% /
udev                   12G     0   12G   0% /dev
tmpfs                  12G     0   12G   0% /dev/shm
tmpfs                  12G  600K   12G   1% /run
/dev/sda1              66G  2.4G   60G   4% /
tmpfs                  12G     0   12G   0% /sys/fs/cgroup
tmpfs                  12G     0   12G   0% /media
/dev/sdb1             272G  103G  156G  40% /ssd

In the healthy server is

Code:

Filesystem            Size  Used Avail Use% Mounted on
rootfs                 66G  2.6G   60G   5% /
udev                   12G     0   12G   0% /dev
tmpfs                  12G     0   12G   0% /dev/shm
tmpfs                  12G  620K   12G   1% /run
/dev/sda1              66G  2.6G   60G   5% /
tmpfs                  12G     0   12G   0% /sys/fs/cgroup
tmpfs                  12G     0   12G   0% /media
/dev/sdb1             272G   54G  205G  21% /ssd

Can you help me with this?

Thanks.

jpollard · #2 2nd December 2011, 02:17 PM

What filesystem is on the ssd?

This can make a significant difference as to how to approach the problem.

A journaled filesystem (ext3/4) then running fsck on the filesystem SHOULD correct the problem. An ext2 based system may have more problems, as you may need multiple passes.

If I remember right, the repair can re-create a lost+found directory.

In your case, it sounds like the lost+found directory itself got corrupted.

Normally, that shouldn't happen (unless it is a sign that the ssd itself is beginning to fail) as the lost+found directory is not referenced by anything while in normal use. This can happen if a power failure hit the device during an fsck run.

Ridd1ck · #3 2nd December 2011, 03:38 PM

The file system is ext2.
This is very strange because after the power lose
The cluster cannot starts with some errors.
And when i use the --initial command for the cluster process (this recreates the filesystem of the cluster) didn't reduce the ssd size.

What can i do know?
Is difficult to me to try something because the cluster right at this time in ssd and is in productive mode.

Thanks.

jpollard · #4 2nd December 2011, 04:27 PM

First, the filesystem has to be repaired.

The cluster very likely cannot start until it is fixed. It is also possible that other damage to the ssd has occurred, but until the filesystem is repaired there won't be much else that will work.

I'm actually surprised that the filesystem would even mount.

BTW, I hope you do have a backup of any needed files on the filesystem. The worst case recovery is to recreate the filesystem, which would leave it empty, but functional. Then restore any needed files.

What is the ssd providing to mysql? just cache? if so, then it should be able to be erased without damage to the database.

PabloTwo · #5 2nd December 2011, 04:38 PM

How about just trying,

Code:

chmod 0700 lost+found

jpollard · #6 2nd December 2011, 06:36 PM

Its a corrupted filesystem - the modes are ? meaning that it couldn't be interpreted, and very likely cannot be written.

This usually happens when a block goes bad, but there are a few other situations that can cause it also (gvfs causes this for non-owners of the mountpoint).

PabloTwo · #7 2nd December 2011, 06:48 PM

Quote:

Originally Posted by jpollard

Its a corrupted filesystem - the modes are ? meaning that it couldn't be interpreted, and very likely cannot be written.

That may very well be true. But then again, in the past on more than one occasion, I have had directories go all

Code:

d????????? ? ?     ?        ?            ?

on me and just resetting the permissions on those directories brought them back to normal, fully accessible and readable. This seems to happen when all exec bits are removed from the permissions on a directory.

jpollard · #8 2nd December 2011, 09:17 PM

Quote:

Originally Posted by PabloTwo

That may very well be true. But then again, in the past on more than one occasion, I have had directories go all

Code:

d????????? ? ?     ?        ?            ?

on me and just resetting the permissions on those directories brought them back to normal, fully accessible and readable. This seems to happen when all exec bits are removed from the permissions on a directory.

No... all bits removed is a file/directory that looks like ---------, if the file is a directory then it is d---------.

Question marks occur only when the the stat system call returns with an I/O error.

The other cases are those of gvfs/fuse improper termination. The directory is hung, and an I/O operation fails due to the server process (gvfs/fuse) doesn't respond. After a timeout (and I don't know how long that is) then the file reverts. Sometimes this will happen when something refreshes the dcache entry (such as a chmod). But otherwise, an I/O error on an inode is not correctable. Especially one on disk.

stevea · #9 2nd December 2011, 10:09 PM

you have a bad situation on your hands. The lost+found fs is there specifically to hold any lost inodes from a file-system error and it;s used by fsck.

So my advice in this special case:
1/ Unless you have a good backup - then make an image backup of the damaged partition before trying anything more including mounting that FS.
2/ DO NOT try to repair or remove the damaged lost&found - replace it.
I suggest (from the root directory of that FS).

Quote:

mv lost+found lost+forever
mklost+found

Then umount the fs and give it a fsck (which should use the NEW lost+found.

The 'mv' should just re-write the directory entry and likely NOT allocate any blocks. The mklost_+found will allocate space and may ruin the FS further - but frankly mounting the rw and mucking with DB could destroy what's left too.

Ridd1ck · #10 5th December 2011, 04:23 PM

Thanks guys for your help!
At this time the SSD is in a productive mode and this is very bad.
I try to stop the node and format the SSD and then i will start the node with initial.
At this time i don't have the possibility to try something because the data is critical for me.
Thanks