The Case of the Disappearing Files (NTFS Style)

[AceRoom]

I've recently noticed that a lot of files are quite literally disappearing from my NTFS partitions. First, I thought it might have been my fault, maybe accidentally deleting the files. Later, I noticed that this especially happened after file transfers but other times, random files disappear. Sometimes, I could retrieve the files by running checkdisk from Windows but other times, I can't seem to retrieve the files at all!

I'm wondering if it has anything to do with the way I mount these NTFS partitions in my fstab which is as follows:

/dev/sda5	/media/Other		ntfs	rw,auto,users,async,exec,nls=utf8,umask=003,gid=1000,uid=1000	0	2
/dev/sda6	/media/Work		ntfs	rw,auto,users,async,exec,nls=utf8,umask=003,gid=1000,uid=1000	0	2
/dev/sda7	/media/Entertainment	ntfs	rw,auto,users,async,exec,nls=utf8,umask=003,gid=1000,uid=1000	0	2

I also thought it could be due to Fedora not dismounting the partitions properly or not writing back all the data.

This allows me to have a trash can in the NTFS partitions. I'm thinking of actually switching to ext4 partitions for my data as well if this doesn't stop (Cause some important work code just disappeared) and hoping that ext2fds may allow me to access those files in the few rare times I do use Windows.

Is it possible to fix this issue or should I try using ext? Or worse yet, is it a problem with the hard drive itself?

If I could get some help with this, it would be much appreciated.

Thanks in advance.

[Dutchy]

I could be wrong but the "2" you have there to fsck it after the root partition has been checked isn't really useful for ntfs although that shouldn't be related to the lost files.
The only occasion where I saw files "randomly" disappearing was when I enabled the hide_hid_files mount option, but the files were actually still there, just not visible.

When do the files get lost?
Solely running Linux or do you also still use Windows?
If so, it could be a character problem (Windows doesn't support certain characters where Linux does) and you could work around that by using the windows_names mount option.

[AceRoom]

I tried the 2 option later hoping it would help but I guess it doesn't do much.

I use Linux most of the time and Windows only rarely. Most of the files go missing when using Linux which makes me think that it is a mount problem. It's not a character problem. Some of the files were movies and entire music albums and they didn't have any "wrong" file name characters. But like I said, if I realize that the files are missing, run chkdsk from Windows and then go to the found.00x folder (From Linux), I can retrieve some of the files. But some don't come back! This makes me feel that maybe the file tables haven't been updated.

---------------------------
Turns out the random work files disappearing was some error due to differences in Dropbox Windows and Linux (I use sym links in Linux).

So the main issue then is files missing when I shutdown Linux.

[szilagyic]

You probably can't rule out this being a Linux problem, but I'd suggest using something other than NTFS if you can. I've seen countless times where even in Windows, NTFS filesystems become corrupted and files disappear, even on Windows file servers. FAT32 (Microsoft's other filesystem) is worse as it is not a journalling filesystem. In all the years I've never had disappearing files in native Linux filesystems like XFS, EXT2-4, etc.

[Jean Pierre]

Is it possible to fix this issue or should I try using ext? Or worse yet, is it a problem with the hard drive itself?

There is no report of disappearing files on the Fedora bugzilla or on the developers' forum (tuxera.com). What version of ntfs-3g are you using ("ntfs-3g --help") ?

Apart from not unmounting (unplugging a USB drive, power loss), do you have any hint about the circumstances in which files disappear ? Do they disappear from both Linux and Windows ?

I could be wrong but the "2" you have there to fsck it after the root partition has been checked isn't really useful for ntfs

Indeed. Nobody has developed an open source ntfs checker. There are proprietary checkers such as chkdsk on Windows.

But like I said, if I realize that the files are missing, run chkdsk from Windows and then go to the found.00x folder (From Linux), I can retrieve some of the files. But some don't come back!

If you were using chkdsk for Windows 7, an explanation was given in /System Volume Information/Chkdsk/Chkdsk*, please post a sample of its contents. Note that the chkdsk for Windows 7 moves to the found.00 folder the files whose name has a character considered illegal in Windows ('?' and ':' appear frequently in multimedia files), but these files should not disappear from Linux (unless moved by chkdsk).

Jean-Pierre

** Forgot to add : is Windows hibernated when you start Linux ? In such a situation, Windows reuses its saved state and may ignore modifications done by another OS (this situation cannot be detected if you do not mount the Windows system partition on Linux).

[AceRoom]

It's ntfs-3g 2011.4.12.
Would the fact that I type ntfs instead of ntfs-3g in the fstab make a difference?

I don't notice files disappearing other than after a reboot and only from Linux. My Windows isn't hibernated and files don't disappear after transfer.

Actually the main case is after file transfers, especially large ones. I usually make sure that the file transfer is complete (By checking in the destination folder) before declaring it to be done. So any shutdowns or reboots are done only after that. Even with this, it yields error.

You were right about the 2 (fsck) in the fstab. I was following the boot sequence today and noticed an error after the mounting of the three ntfs partitions stating that it could not perform fsck so I removed that.

Here's the file inside System Volume Information folder from one of the drivers.
http://db.tt/q12LzZIn
(Linked because the log made the post too long)

They show some papers which I found in the found.000 folder after running chkdsk. One partition didn't have any of the files and the third also contained something similar.

About what you said at the end, it may be of significance. I'm actually using the Windows 8 Consumer preview and the boot is hellishly fast (Less than 10 seconds) which could be due to some differences in hard disk mount itself or something like Windows 8 putting itself in some sort of hibernate state on shutdown so it can load fast.

It looks like this guess was right. I found this on a site:

"The speedy boot is thanks to a new hybrid system that mixes processes used in cold boots and hibernation mode."

Here's the article:
http://www.pcworld.com/article/23973...windows_8.html

And as you guessed, I don't mount the Windows partitions from Linux. Have no files of use on it.

Let me try this. I'll transfer a large file from one partition to another and reboot and see what happens. Then I'll try the same but boot into Windows and post back.

Thanks again for all the help.

---------- Post added at 07:16 PM ---------- Previous post was at 07:14 PM ----------

Also as you said, the log states that some of the files had incorrect file names (Probably too long). But the large file transfer is still an issue. I'll perform that test and get back.

---------- Post added at 07:22 PM ---------- Previous post was at 07:16 PM ----------

Come to think of it, this never happens with my external disc (Nor with my ext4 Linux partition) so this Windows boot thing is mostly likely the issue.

---------- Post added at 07:56 PM ---------- Previous post was at 07:22 PM ----------

I just tried that last with three different files, each of size 4 GB. Once, I shutdown and restarted back into Fedora. The next two times, I copied the files in Fedora, shutdown and booted into Windows. The files copied properly all three times.

[Jean Pierre]

Would the fact that I type ntfs instead of ntfs-3g in the fstab make a difference?

No difference, "ntfs" is a symlink to "ntfs-3g".

Here's the file inside System Volume Information folder from one of the drivers.

All these errors are about file names with characters considered invalid in Windows. In the samples, there were a '\\' (reverse-slash), a '\n' (new-line) and a '\r' (carriage return). By mounting with option "windows_names", you will get an error when trying to create such a file.

Also as you said, the log states that some of the files had incorrect file names (Probably too long).

You cannot create too long file names, up to 255 characters are allowed in ntfs and the number of characters is stored in a single byte.

About what you said at the end, it may be of significance. I'm actually using the Windows 8 Consumer preview and the boot is hellishly fast (Less than 10 seconds) which could be due to some differences in hard disk mount itself or something like Windows 8 putting itself in some sort of hibernate state on shutdown so it can load fast.

Hmm. This is technically possible, and if you guessed right I see no workaround. The state at Windows shutdown could be partially stored on the swap file of the Windows system disk... In this scenario, I see no relation to the size of files.

Are there different ways of closing down Windows 8 ? (I do not have Windows 8)

I just tried that last with three different files, each of size 4 GB. Once, I shutdown and restarted back into Fedora. The next two times, I copied the files in Fedora, shutdown and booted into Windows. The files copied properly all three times.

I suggest you first copy a big file in Fedora, then boot into Windows and copy a big file into the same directory in Windows, then boot again into Fedora and check what files are present. This is to determine whether the Windows copy is based on data from the partition or based on data from a saved state from the previous shutdown.

[Doug G]

Also as you said, the log states that some of the files had incorrect file names (Probably too long). But the large file transfer is still an issue. I'll perform that test and get back.

Maybe you are using characters in the filename that are invalid in windows. I don't think your can start a filename with a dot in windows, and you can't use colons & some other characters in windows but you can in linux.

[AceRoom]

I tried out the large file transfer into a folder in Linux, then in Windows. Both were done perfectly.

Then I copied a small folder with 3 sub-folders inside it, each with a single file inside it, not for any particular reason but a backup (This was in Linux). When i rebooted into Windows, those files were missing.

Ran a chkdsk and rebooting into Linux to see if I can find the files. As a bit of background info, these were Windows profiles for a particular application so the file name issue shouldn't be there.

But actually, the Consumer Preview of Windows 8 that I'm using has become unbearably unstable. When I mounted both a hard drive and a pen drive, it just shuts down and sometimes it shuts down randomly. So I'll mostly be reverting back to Win7.

Once I do, I'll try the same things and post back. If it is a problem with Win8, then hopefully Microsoft fixes it before the final release so there are going to be a lot of problems in the future for a lot of users...

You can have filenames starting with a dot in Windows. All the .* files from Mac and Linux computers do appear. You can notice this especially on pen drives.

There is only one way to shut down Win8 and it's quite unintuitive. Other people using my comp (Which is the main reason I have windows) had a difficult time shutting down. If anything, it's more complicated than Gnome3 default!

But the MetroUI should be good, especially for touchscreens.

---------- Post added at 12:50 PM ---------- Previous post was at 12:25 PM ----------

Nope, those files didn't show up even after the chkdsk.

[Jean Pierre]

Then I copied a small folder with 3 sub-folders inside it, each with a single file inside it, not for any particular reason but a backup (This was in Linux). When i rebooted into Windows, those files were missing.

Is this repeatable ? Which files were missing (the original or the backup ones) ?

As a bit of background info, these were Windows profiles for a particular application so the file name issue shouldn't be there.

This could be some roaming profile synchronization feature. Could you retry with similar files in another directory ?

So I'll mostly be reverting back to Win7.

Can you wait until the matter is clarified ?

I would also like to know the header of the hiberfile.sys file from the Windows 8 system partition. Can you post the output of (assuming you mount the partition on /windows, you can mount readonly, also assuming Windows 8 uses the same file name as older Windows) :

head -c 4 /windows/hiberfil.sys | od -t x1

You can have filenames starting with a dot in Windows.

I agree. However the *last* character of file names is documented not to be allowed to be a space or dot, but chkdsk does not complain or relocate such files.

** Oh, another useful point :

If the files are intentionally deleted by Windows 8, they must be undeletable, whereas they will probably not be undeletable if this is caused by using obsolete data saved at the previous shutdown.

After a file or directory disappears from /dev/sdax, what is the output of (run as root)

/sbin/ntfsundelete -f /dev/sdax 2> /dev/null | grep '2012-05-23'

where 2012-05-23 is the date when the file disappeared, and /dev/sdax is unmounted.

[AceRoom]

The backup files were missing. Just to be clear, I copied the files, not moved.

I'm holding on to the Win8.

The head of the file hiberfil.sys:

0000000 57 41 4b 45
0000004

Without the od command, it prints the

WAKE

.

Here is the output of ntfsundelete:

482      FR..   100%  2012-05-23         0  <none>
630      FR..   100%  2012-05-23         0  <none>
631      FR..   100%  2012-05-23       109  ArchLinux.iso.trashinfo

That was one of the files I had copied earlier. The copy worked perfectly. This is the same drive as the one the files recently disappeared from.

I'll try that test again and get back.

Exact same test with the same files succeeds. A similar test with different files also succeeds. I'm wondering if it's an OS problem at all. Could it be related to disk caching?

Both the failed and succeeded tests were followed immediately by a shutdown.

[Jean Pierre]

The head of the file hiberfil.sys:

Without the od command, it prints the
Code:
WAKE

Argh ! Never saw that before, must be a new feature in Windows 8.

Exact same test with the same files succeeds. A similar test with different files also succeeds. I'm wondering if it's an OS problem at all. Could it be related to disk caching?

Well, reuse of data saved while shutting down is likely to lead to a behavior uneasy to reproduce...

No more tries to suggest, I have to think it over.

---------- Post added at 04:08 PM ---------- Previous post was at 02:30 PM ----------

There is only one way to shut down Win8 and it's quite unintuitive.

On http://blogs.msdn.com/b/b8/archive/2...windows-8.aspx
there is an indication of a way to fully shutdown Windows 8 :

shutdown /s /full / t 0

This might be the only workaround for now, and please post what is the hiberfil.sys header after such a shutdown.
[thanks to Anton for the pointer]

[AceRoom]

Same output even after the full shutdown.

[Jean Pierre]

Same output even after the full shutdown.

Which output ?

Do you still have disappearing files ? in what circumstances ?

What is the hiberfil.sys header after a full shutdown ?

[AceRoom]

I didn't try copying any files with a full shutdown. When I said the same output, I mean the hiberfil.sys

The first four bytes are the same as it was before.