I'm trying to backup some files from my PC to an external hard-drive using rsync. The copy works fine but takes a lot of time even for the standard USB 2.0. For an USB 2.0 the real transfer speed should be round 30MB/s, in my test the speed reach sometimes 2MB/s. The pen-drive I'm using is a good 32GB USB3.0 with FAT32 fs.
Note: before each test I erased the whole content of the pen-drive and I also tried on different USB port.
Part of the script I'm using is as follow:
mkdir /media/eusbd0
mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0
# copy only data between date1 to date2
for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do
if [ "/home/records" != "$dir" ]; then # skip parent dir
rsync -ravP --inplace --modify-window=2 --log-file=/var/log/download_records "$dir" /media/eusbd0 > /var/log/rsync_report.txt
fi
done
#sync
umount -l /media/eusbd0
Here is part of the output of rsync:
sending incremental file list 2016-08-11/ 2016-08-11/gps_00000 44,179 100% 10.88MB/s 0:00:00 (xfr#1, to-chk=34/36) 2016-08-11/log 106,792 100% 101.84MB/s 0:00:00 (xfr#2, to-chk=33/36) 2016-08-11/log_cam0 5,532 100% 5.28MB/s 0:00:00 (xfr#3, to-chk=32/36) 2016-08-11/rec_2016-08-11.13_57_22.mov 3,513,782 100% 108.10MB/s 0:00:00 (xfr#4, to-chk=31/36) 2016-08-11/rec_2016-08-11.13_59_00.mov 4,281,177 100% 63.79MB/s 0:00:00 (xfr#5, to-chk=30/36) 2016-08-11/rec_2016-08-11.14_02_09.mov 3,359,207 100% 36.82MB/s 0:00:00 (xfr#6, to-chk=29/36) 2016-08-11/rec_2016-08-11.14_04_16.mov 1,877,863 100% 17.91MB/s 0:00:00 (xfr#7, to-chk=28/36) 2016-08-11/rec_2016-08-11.14_05_42.mov ........ 50,482,791 100% 114.90MB/s 0:00:00 (xfr#20, to-chk=15/36) 2016-08-11/rec_2016-08-11.15_14_53.mov 19,291,527 100% 34.52MB/s 0:00:00 (xfr#21, to-chk=14/36) 2016-08-11/rec_2016-08-11.15_18_42.mov 50,700,461 100% 58.68MB/s 0:00:00 (xfr#22, to-chk=13/36) 2016-08-11/rec_2016-08-11.15_20_20.mov ....... sent 802,030,914 bytes received 685 bytes 94,356,658.71 bytes/sec total size is 801,832,718 speedup is 1.00 tail: /var/log/rsync_report.txt: file truncated sending incremental file list 2016-08-16/ 2016-08-16/log 41,966 100% 8.77MB/s 0:00:00 (xfr#1, to-chk=16/18) 2016-08-16/obd_00000 46,798 100% 44.63MB/s 0:00:00 (xfr#2, to-chk=15/18) 2016-08-16/rec_2016-08-16.16_24_12.mov 50,649,317 100% 18.64MB/s 0:00:02 (xfr#3, to-chk=14/18) 2016-08-16/rec_2016-08-16.16_25_50.mov 25,602,242 100% 7.78MB/s 0:00:03 (xfr#4, to-chk=13/18) 2016-08-16/rec_2016-08-16.16_57_42.mov 50,496,580 100% 15.18MB/s 0:00:03 (xfr#5, to-chk=12/18) 2016-08-16/rec_2016-08-16.16_59_20.mov 50,617,906 100% 13.29MB/s 0:00:03 (xfr#6, to-chk=11/18) 2016-08-16/rec_2016-08-16.17_00_58.mov 50,759,115 100% 10.16MB/s 0:00:04 (xfr#7, to-chk=10/18) 2016-08-16/rec_2016-08-16.17_02_36.mov 50,883,325 100% 6.81MB/s 0:00:07 (xfr#8, to-chk=9/18) 2016-08-16/rec_2016-08-16.17_04_14.mov 47,995,074 100% 4.41MB/s 0:00:10 (xfr#9, to-chk=8/18) 2016-08-16/rec_2016-08-16.17_09_44.mov 50,813,636 100% 2.35MB/s 0:00:20 (xfr#10, to-chk=7/18) 2016-08-16/rec_2016-08-16.17_11_22.mov 50,953,015 100% 2.52MB/s 0:00:19 (xfr#11, to-chk=6/18) 2016-08-16/rec_2016-08-16.17_13_00.mov 50,221,069 100% 2.93MB/s 0:00:16 (xfr#12, to-chk=5/18) 2016-08-16/rec_2016-08-16.17_14_38.mov 50,445,757 100% 2.87MB/s 0:00:16 (xfr#13, to-chk=4/18) 2016-08-16/rec_2016-08-16.17_16_16.mov 51,036,959 100% 3.16MB/s 0:00:15 (xfr#14, to-chk=3/18) 2016-08-16/rec_2016-08-16.17_17_54.mov 51,056,196 100% 3.19MB/s 0:00:15 (xfr#15, to-chk=2/18) 2016-08-16/rec_2016-08-16.17_19_32.mov 50,434,922 100% 3.12MB/s 0:00:15 (xfr#16, to-chk=1/18) 2016-08-16/rec_2016-08-16.17_21_10.mov 19,456,000 100% 2.71MB/s 0:00:06 (xfr#17, to-chk=0/18) sent 701,682,372 bytes received 343 bytes 4,512,429.04 bytes/sec total size is 701,509,877 speedup is 1.00
What can be noted in the output is that at the beginning the speed is quiet high whereas at the end it gets slower and slower. Why is this happening?
The file I'm copying are pretty much the same (except the size that may vary a bit) and the amount of them is quiet low, less than 100 files. I also tried with other pen-drives but I got the same result.
Thanks!
NEW EXPERIMENT:
I tried cpio instead of rsync to copy the same files. The modified script:
mkdir /media/eusbd0
mount -t vfat -o shortname=mixed,iocharset=utf8 /dev/sdb1 /media/eusbd0
# copy only data between date1 to date2
for dir in $(find /home/records -type d -newermt "2016-08-04" ! -newermt "2016-08-18" 2>&1); do
if [ "/home/records" != "$dir" ]; then # skip parent dir
find "$dir" -print | cpio -pdm /media/eusbd0 > /var/log/rsync_report.txt
fi
done
#sync
umount -l /media/eusbd0
The amount of data to copy is around 3GB and the time required with cpio is 900 seconds which translates to an average transfer speed of 3-4 MB/s. It seems that the problem concerns the hardware or the drives since also rsync speed is close to that value.
UPDATE
I tried with other 3 pen-drive formatted in FAT32 but I got the same behavior. Then I tried the NTFS fs and I got a good result of >30MB/s but unfortunately only on one of the 4 USB sticks in my possession. Other experiment with external HDD and SSD have shown also some reasonably good results >20MB/s.
To sum up, my motherboard (or drivers) probably do not like USB sticks nor FAT32 fs. External hard drives (powered or not) work as expected at least in NTFS. And, I have no clues why this happen.
Best Answer
rsync
is not the best solution for copy. It is the best solution to synchronize files. Speed for most data write/read depends on a buffer. In the case of writing files, buffers get filled and you see a higher speed for that. As soon the buffer will be written to the device (buffer flush) the speed is going down. Write speed to the device depends not only on specification (USB 2.0 or 3.0 ...), also on protocol (implementation), driver, block sizes. Sometimes it is better to copy the whole file instead of letting sync programs first scan for changes. This brings overhead and is lowering the speed in case of local attached devices. If you doing this via network the story may be different.