Why is copying to my USB drive so slow in Linux (and faster in Windows)?
Reason 1. File caching can make writes appear slower or faster
The problem I seem to see in the GUI is that the progress bar goes to 90% almost instantly, completes to 100% a little slower and then hangs there for 10 minutes.
One thing you need to understand is file caching. Linux (and Windows) will use otherwise "empty" RAM to cache read/write operations and make them faster on subsequent accesses. Caching copy operations to slow devices results in the behavior you see -- the "fast completion" is actually writing to the cache, and then it slows and stops because the actual flushing of the data in the cache (sync) to the slow device is taking very long. If you abort at that point, the data is corrupted (as you noted) since the sync never finished.
Such copying in Windows may seem faster (including the reported MB/sec speeds) because sometimes Windows will not wait for the sync, and declare the job completed as soon as the data is written to cache.
Reason 2. Writing lots of files, especially small ones, is slow
To copy 1.8GB
Because of the way flash memory and filesystems work, the fastest throughput (speed) is achieved when writing very large files. Writing lots of small files, or even mixed data containing a number of small files can slow the process down a lot. This affects hard drives too, but to a somewhat lesser extent.
Reason 3. Write speeds of a USB stick and an SSD cannot be compared
I have a super talent 32GB USB SSD in the neighboring port and it works at expected speeds.
A garden variety USB stick usually consists of flash memory chips that are written to serially (sequentially), and does not have any cache of its own.
An SSD, on the other hand, contains a controller which writes to the flash memory chips parallel, increasing the throughput by a factor of 2 or more over the USB stick.
- If your 32GB SSD had 4x 8GB chips, it would still be 4x faster than the USB stick at any write operation.
- The SSD also contains RAM cache (like hard disks), so it can quickly store incoming data in the cache and tell the OS that it's done, while it still has to actually write that data to the flash memory.
So, with one large file, your 32GB GB with the 4x structure we assumed, would be 4x as fast; with many small files, it would be 10x or more faster because it could intelligently store them in its cache.
To sum up, these are the reasons why file copying to USB sticks may appear slower in Linux. Is it actually slower because of a hardware/driver issue or whatever....
Doing a proper comparison of write speeds between Linux and Windows
- First of all, forget about the SSD because of reason 3. It's like oranges and apples.
- To negate the effects of reason 1 (caching) and reason 2 (small files), you need to test with a single large file, larger than the amount of RAM on the test system.
- In Linux you can create it with
dd if=/dev/urandom of=largetest bs=1M count=7500, which gives you a 7500 MB test file. Assuming your system has less than 4GB or so of RAM, it's good enough. Copy that to a freshly formatted Sandisk 8GB stick, and time it.
- Reboot in Windows, and copy
largetest from the USB stick to your hard disk. Reboot again (to remove it from the cache). Then format the USB stick (same vfat/FAT32!), and copy largetest from the hard disk to the stick.
- How do the times compare?
You should not worry about this too much.
Maybe your USB pendrive is of "lower quality".
The dialog output regarding the time is misleading.
Other copy tools may have a different output behavior,
but they do not improve the writing speed of the USB drive.
Most important of all is that finally the copy action succeeds.
What you can do is optimizing the input/output schedulers of disks.
Install gksu (when you haven't already) to edit files with gedit as root :
sudo apt-get install gksu
Now optimize the priority of all running processes for various disks types :
gksudo gedit /etc/udev/rules.d/60-schedulers.rules
Paste the following lines into this empty file and save the file afterwards :
# set cfq scheduler for rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="cfq"
# set deadline scheduler for non-rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"
The changes you made take effect after a restart of the operating system.
Also ... an alternative copy solution is available in the Ubuntu repositories :
sudo apt-get install dirdiff
Now launch the tool from a terminal to compare or copy files and folders :
dirdiff
dirdiff is a GUI for diff and can handle up to 5 trees. It displays a main window with a list of files which are different between the trees, with colored squares to indicate the relative ages of the versions. A menu allows you to display the differences between any two of the versions in another window. Another menu allows you to copy files and folders from one tree to another.
Summary : Most relevant of all is the quality of the USB disk and its writing speed capabilities !
Best Answer
This is because many filesystems buffer and then when they
flushthey have to wait for the device to do the actual writing.... Which takes a while.I don't know that there's a good solution so that you see the actual write speed and progress.