Linux – Why the rsync slow down over time

flash-memorylinuxperformancersyncusb

I'm trying to backup some files from my PC to an external hard-drive using rsync. The copy works fine but takes a lot of time even for the standard USB 2.0. For an USB 2.0 the real transfer speed should be round 30MB/s, in my test the speed reach sometimes 2MB/s. The pen-drive I'm using is a good 32GB USB3.0 with FAT32 fs.

Note: before each test I erased the whole content of the pen-drive and I also tried on different USB port.

Part of the script I'm using is as follow:

mkdir /media/eusbd0
mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0 

# copy only data between date1 to date2
for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do
    if [ "/home/records" != "$dir" ]; then # skip parent dir
        rsync -ravP --inplace --modify-window=2 --log-file=/var/log/download_records "$dir" /media/eusbd0 > /var/log/rsync_report.txt
    fi     
done
#sync    
umount -l /media/eusbd0

Here is part of the output of rsync:

    sending incremental file list 2016-08-11/ 2016-08-11/gps_00000

     44,179 100%   10.88MB/s    0:00:00 (xfr#1, to-chk=34/36) 2016-08-11/log

    106,792 100%  101.84MB/s    0:00:00 (xfr#2, to-chk=33/36) 2016-08-11/log_cam0

      5,532 100%    5.28MB/s    0:00:00 (xfr#3, to-chk=32/36) 2016-08-11/rec_2016-08-11.13_57_22.mov

  3,513,782 100%  108.10MB/s    0:00:00 (xfr#4, to-chk=31/36) 2016-08-11/rec_2016-08-11.13_59_00.mov

  4,281,177 100%   63.79MB/s    0:00:00 (xfr#5, to-chk=30/36) 2016-08-11/rec_2016-08-11.14_02_09.mov

  3,359,207 100%   36.82MB/s    0:00:00 (xfr#6, to-chk=29/36) 2016-08-11/rec_2016-08-11.14_04_16.mov

  1,877,863 100%   17.91MB/s    0:00:00 (xfr#7, to-chk=28/36) 2016-08-11/rec_2016-08-11.14_05_42.mov


........

 50,482,791 100%  114.90MB/s    0:00:00 (xfr#20, to-chk=15/36) 2016-08-11/rec_2016-08-11.15_14_53.mov

 19,291,527 100%   34.52MB/s    0:00:00 (xfr#21, to-chk=14/36) 2016-08-11/rec_2016-08-11.15_18_42.mov

 50,700,461 100%   58.68MB/s    0:00:00 (xfr#22, to-chk=13/36) 2016-08-11/rec_2016-08-11.15_20_20.mov

.......

sent 802,030,914 bytes  received 685 bytes  94,356,658.71 bytes/sec

total size is 801,832,718  speedup is 1.00 tail:

/var/log/rsync_report.txt: file truncated sending incremental file

list 2016-08-16/ 2016-08-16/log

     41,966 100%    8.77MB/s    0:00:00 (xfr#1, to-chk=16/18) 2016-08-16/obd_00000

     46,798 100%   44.63MB/s    0:00:00 (xfr#2, to-chk=15/18) 2016-08-16/rec_2016-08-16.16_24_12.mov

 50,649,317 100%   18.64MB/s    0:00:02 (xfr#3, to-chk=14/18) 2016-08-16/rec_2016-08-16.16_25_50.mov

 25,602,242 100%    7.78MB/s    0:00:03 (xfr#4, to-chk=13/18) 2016-08-16/rec_2016-08-16.16_57_42.mov

 50,496,580 100%   15.18MB/s    0:00:03 (xfr#5, to-chk=12/18) 2016-08-16/rec_2016-08-16.16_59_20.mov

 50,617,906 100%   13.29MB/s    0:00:03 (xfr#6, to-chk=11/18) 2016-08-16/rec_2016-08-16.17_00_58.mov

 50,759,115 100%   10.16MB/s    0:00:04 (xfr#7, to-chk=10/18) 2016-08-16/rec_2016-08-16.17_02_36.mov

 50,883,325 100%    6.81MB/s    0:00:07 (xfr#8, to-chk=9/18) 2016-08-16/rec_2016-08-16.17_04_14.mov

 47,995,074 100%    4.41MB/s    0:00:10 (xfr#9, to-chk=8/18) 2016-08-16/rec_2016-08-16.17_09_44.mov

 50,813,636 100%    2.35MB/s    0:00:20 (xfr#10, to-chk=7/18) 2016-08-16/rec_2016-08-16.17_11_22.mov

 50,953,015 100%    2.52MB/s    0:00:19 (xfr#11, to-chk=6/18) 2016-08-16/rec_2016-08-16.17_13_00.mov

 50,221,069 100%    2.93MB/s    0:00:16 (xfr#12, to-chk=5/18) 2016-08-16/rec_2016-08-16.17_14_38.mov

 50,445,757 100%    2.87MB/s    0:00:16 (xfr#13, to-chk=4/18) 2016-08-16/rec_2016-08-16.17_16_16.mov

 51,036,959 100%    3.16MB/s    0:00:15 (xfr#14, to-chk=3/18) 2016-08-16/rec_2016-08-16.17_17_54.mov

 51,056,196 100%    3.19MB/s    0:00:15 (xfr#15, to-chk=2/18) 2016-08-16/rec_2016-08-16.17_19_32.mov

 50,434,922 100%    3.12MB/s    0:00:15 (xfr#16, to-chk=1/18) 2016-08-16/rec_2016-08-16.17_21_10.mov

 19,456,000 100%    2.71MB/s    0:00:06 (xfr#17, to-chk=0/18)

sent 701,682,372 bytes  received 343 bytes  4,512,429.04 bytes/sec

total size is 701,509,877  speedup is 1.00

What can be noted in the output is that at the beginning the speed is quiet high whereas at the end it gets slower and slower. Why is this happening?

The file I'm copying are pretty much the same (except the size that may vary a bit) and the amount of them is quiet low, less than 100 files. I also tried with other pen-drives but I got the same result.

Thanks!

NEW EXPERIMENT:

I tried cpio instead of rsync to copy the same files. The modified script:

mkdir /media/eusbd0
mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0 

# copy only data between date1 to date2
for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do
    if [ "/home/records" != "$dir" ]; then # skip parent dir
        find "$dir" -print | cpio -pdm /media/eusbd0 > /var/log/rsync_report.txt
    fi     
done
#sync    
umount -l /media/eusbd0

The amount of data to copy is around 3GB and the time required with cpio is 900 seconds which translates to an average transfer speed of 3-4 MB/s. It seems that the problem concerns the hardware or the drives since also rsync speed is close to that value.

UPDATE
I tried with other 3 pen-drive formatted in FAT32 but I got the same behavior. Then I tried the NTFS fs and I got a good result of >30MB/s but unfortunately only on one of the 4 USB sticks in my possession. Other experiment with external HDD and SSD have shown also some reasonably good results >20MB/s.

To sum up, my motherboard (or drivers) probably do not like USB sticks nor FAT32 fs. External hard drives (powered or not) work as expected at least in NTFS. And, I have no clues why this happen.

Best Answer

rsync is not the best solution for copy. It is the best solution to synchronize files. Speed for most data write/read depends on a buffer. In the case of writing files, buffers get filled and you see a higher speed for that. As soon the buffer will be written to the device (buffer flush) the speed is going down. Write speed to the device depends not only on specification (USB 2.0 or 3.0 ...), also on protocol (implementation), driver, block sizes. Sometimes it is better to copy the whole file instead of letting sync programs first scan for changes. This brings overhead and is lowering the speed in case of local attached devices. If you doing this via network the story may be different.

Related Question