[SOLVED] Error copying large files

[Bazu135]

When I attempt to copy large files (over 4GB) between two external hard drives, the process fails, telling me the file is too large. I've experienced this using cp and rsync, and graphically through thunar.

I doubt this is an issue with the filesystems since they're both NTFS, and I can't imagine it'd be an issue with memory since I have 4GB RAM and 6GB swap (surely more than enough for a 5.2GB file?)

I'm using 32-bit F16; is this likely to be the source of the problem? If so, I'd guess that installing a PAE kernel would resolve it, but does this pose any risk of causing conflicts with the rest of my system? (i.e. is there a risk of PAE not playing nice with any already-installed 32-bit packages, or would it be just like any other kernel update?)

Thanks in advance for any responses!

[DBelton]

Well, I'm not sure that your issue will be resolved by using the PAE kernel, but to answer your question on that.. You can install the PAE kernel without a problem with your other packages. the PAE kernel is still a 32 bit kernel.

If you have the kernel-devel or th kernel-modules-extra packages installed, you will also need to install the PAE version of those. Also, if you have some of the kmods installed (like kmod-nvidia) you will need the PAE version of those as well.

To get your PAE kernel, all you need to do is a yum install

yum install kernel-PAE

and if you have the kernel-devel and kernel-modules-extra packages installed:

yum install kernel-PAE-devel kernel-PAE-modules extra

You should then have the PAE kernel as an option in your grub2 boot menu.

[Bazu135]

Good to hear that the PAE kernel won't break things, at the very least I still find it strange that >4GB files won't copy though!

---------- Post added 29th September 2012 at 09:01 AM ---------- Previous post was 28th September 2012 at 09:05 PM ----------

Had a trawl through every result for a "copy large files" search, and got nothing I can split+cat the files to copy them, but that doesn't resolve doing it through rsync/cp. I suppose doing it with dd might work, but they're 1TB drives connected by USB - that somewhat ruins the concept of doing it quickly! (My reason for using rsync)

One point that did come up in one thread was ulimit values; however:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 28168
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I can't see anything in there that suggests it could be causing a problem?

[jpollard]

actuall, dd in some cases will be faster - you can specify larger buffers for use.

Cutting off at 4G looks more like a filesystem limit. How do you have the volume mounted?

[Bazu135]

The 'source' drive automounts on boot through /etc/fstab:

UUID=24586C8B586C5D94    /mnt/Storage    ntfs     defaults,nofail     0 0

The second drive doesn't have any special settings - it just automounts when I plug the USB in (it usually sits on a shelf, rather than attached to the PC - it's a 'monthly back-up' sort of drive).



Is it actually possible that dd could be faster? That may be the case for the first time the back-up's created, but I'd have thought that rsync would be faster from then on owing to the amount of files it can skip - it wouldn't do the 30+gb of Linux iso's or 8+gb of music the second time, for example? That said, I've only used dd once before, and that was a straight "if=[src] of=[dest]" case, so I'm not exactly qualified to make judgements on it

[kkshethin]

is correct fstab entry should read ntfs-3g instead of ntfs?

[jpollard]

Try using "ntfs-3g". If you have the >4G files already existing on input, and copying to the USB disk, you might check how it is mounting the output.

As for dd It depends entirely on what you are copying. If you are copying iso files, that don't change, then rsync is better on the second copy. Creating an iso is only good for a quick image copy - it wastes a lot of space due to the unused data blocks and metadata in the input. I think an iso would be ok for /boot for instance, but not for anything else (it would copy the partition boot block, but won't copy the MBR unless you are copying a full disk device)

The problem rsync has is that it does not use large buffers - it was originally designed for remote sync (hence the "r"sync). dd, can use large buffers (largest I've ever used was 16MB or so) and that allows the disk controller to be busy copying/filling the buffer in one go- then busy emptying it in one go (scatter/gather works well for this). For synching a lot of small files, rsync would be very hard to beat.

If you are copying single, small files, it doesn't matter much about the buffering..

[Bazu135]

I'll have a go at the ntfs-3g setting - thanks

Would I be correct in thinking that I'd want that setting on both of them, since they're both ntfs filesystems? (They're shared with my Windows partition ) If so, would I be correct in thinking that setting an fstab entry for the second external drive:

UUID=24586C8B586C5D94    /mnt/Storage    ntfs-3g     defaults,nofail     0 0
UUID=################    /mnt/Backup     ntfs-3g     defaults,nofail     0 0

would result in that drive mounting as ntfs-3g when I plug it in?

---------- Post added at 03:37 PM ---------- Previous post was at 03:18 PM ----------

I've possibly found the source of the problem - plugged in my Backup drive, and ran fdisk -l:

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1953520064   976760001    7  HPFS/NTFS/exFAT

That result is what made me assume it was an ntfs volume. However, when I run blkid:

/dev/sdc1: LABEL="NEKO~ROAMIN" UUID="F830-8F13" TYPE="vfat"

And if I recall correctly, vfat can only handle up to 4gb files?

Time to format the drive and try again...!

---------- Post added at 03:55 PM ---------- Previous post was at 03:37 PM ----------

With the backup volume changed to ntfs, the files copy correctly. Problem solved!

Thank you for your replies, everyone