File transfers (cp, mv and rsync) slow system tremendously

[cruisecho]

I have a problems that is becoming very annoying. When ever I transfer large files using cp, mv, rsync or dolphin the system will slow down to the point that it's unusable. It will sometime hang completely and not accept any input until the file is transferred. I have no clue what could be causing this problem but I know it shouldn't be happening.

I am using Fedora 15 (2.6.40.3-0.fc15.x86_64) with KDE 4.6. have a Phenom II 955 processor, 6 GB of system ram and the OS and swap file is on an 80 GB SSD. Copying files in the SSD doesn't cause any problem, but moving files between my other two large HDDs causes the extreme slow down. Using htop I can see that my system load jumps to between 3 and 4, but my RAM and CPU usage stays low during the transfer.

Here are two commands that take about 10 mins to run and make the system unusable while it's running. It usually transferring around 2-20GB worth of data during the transfers:


cp -a /media/data.1.5/backup/Minecraft/backups/* /media/data.0.5/backup/Minecraft/backups/

rsync -a /media/data.1.5/backup/ /media/data.0.5/backup/


/media/data.1.5/ is the mount point for a 1.5 TB internal SATA drive, and /media/data.0.5/ is the mount point for a 500 GB internal SATA drive.

[jpollard]

How are you starting these commands? On a console terminal, or within an X terminal session?

The reason I'm asking is that one of the things F15 uses are cgroups. This gives higher priority to GUI type logins, and this could affect how the scheduling gets done for these copies.

[cruisecho]

I'm using Konsole.

[jpollard]

Try logging in at a console terminal (ctl-alt-F2 through F6). Login and start a copy. Then switch back to the GUI (alt-F7 I think it is, but it could be F1).

Let us know if that changes the situation a bit. I'm not sure it will, but what should happen is that the console terminal will be put in a separate cgroup from the GUI login. That MAY give the GUI suitable CPU time, and I/O priority, for normal operation.

[cruisecho]

I transferred a 3.8GB file in the other terminal (F2) like this.
cp -a /media/data.1.5/largefile /media/data.0.5/
The CPU usage was higher than before, but the system load never went over 1 and the transfer was very fast. There was also no noticeable slowdown during the transfer.

I then deleted the file on /media/data.0.5/ and did ran the same command in konsole.

Within less than a minute the system load was over 4 and the systems was VERY sluggish. The file was also transferring MUCH slower than before (Like 1/10th of the speed). I stopped the job with ctrl-Z but the system stayed sluggish for a good 3 or 4 minutes after I stopped the job.



I'd really like to be able to run commands while in a gui. It's very useful to have several consoles up at once on the same desktop. Do you have any ideas on what I can do to resolve this? Maybe I can just SSH into localhost. lol.

[jpollard]

Yup. It is an issue with cgroups.

You can report it as a bug in the use of cgroups, and call for being able to create non-hierarchical cgroups.

The way F15 uses cgroups is to put each login in a new cgroup, as well as put each service into it's own cgroup. This can give better performance by having all of the GUI collected into a cgroup for scheduling.

The problem with that is what you found. CPU or IO bound processes tend to lock other processes in the same cgroup out.

Something you might try:

at now
<command>
^D

This will run the command via cron (right now). I don't know if this will create a new cgroup, but it should.

[cruisecho]

After a few large files transfer through another terminal, I started to get the same slow down. I did some more reading and started to watch what was happening with vmstat. I noticed I didn't get any slowdowns until free memory started to get down into the 90,000 KB range. Looking at Htop I saw that only half of my system RAM was actually used, the other half was all cache and buffer.

I was able to run this command as root to clear the cache:
sync; echo 3 > /proc/sys/vm/drop_caches

Upon clearing the RAM cache the system sped up considerably, the problem is as soon as the cache fills back up (which happens quickly while moving files) the system slows down to a crawl again. It takes much longer for the system to start slowing down when I make a transfer in another terminal though.

The system seems to get unstable after doing large file transfers. I think it has to do with the RAM since the same thing happens when I open a large number of windows in firefox and eat up all the system RAM. I noticed it with moving large files first because of how quickly doing that eats up the system RAM with cache and buffers.


E:
The at now command worked the same as running through another terminal. The system did eventually slow down, it just took longer than when doing it through konsole. I might try moving my desktop over to CentOS with KDE just to see how well that works.