Ubuntu – Moving large amount of files (~ 100 000)

batchnautilustransfer

I work with folders that contain a lot of files, like 100 000 or even 1 000 000 files per folder. When I try to move the content of a folder into another, my computer always gets stuck. Even when the process seems finished, I can't see the content of any folder because nautilus seems completely frozen and I have to force my computer to restart. I noticed that this happens also when I try to move like 10 000 files.

Is that a problem of my computer or is it normal when working with these numbers?

Any smart way of performing this file transfer?

Best Answer

Perhaps consider using a pure command line method to transfer very large amounts files, you will undoubtedly find the process is substantially faster than using a gui.

There are many different ways to accomplish this, but the following worked quickly, safely and efficiently on my system:

find . -maxdepth 1 -type f -print0 | xargs -0 mv -t <destination>

Some explanation for this command:

  1. Your input directory is the '.' character and for this particular command you need to be in that directory
  2. Your output directory is the <destination> in my example. Obviously modify this to suit your own needs and leave out the brackets.
  3. This syntax allows for filenames with spaces as a bonus :)

Endless permutations are possible but this should work well and much more efficiently than the gui. One permutation for example: if you wanted to move only pdf files you could run:

find . -iname "*.pdf" -maxdepth 1 -type f -print0 | xargs -0 mv -t <destination>

Use of xargs opens many possibilities particularly with the movement of such a large number of files. Many, many possibilities....

Potential Problems:

There are at least 2 potential pitfalls to ponder, thanks to the commenters below for these thoughts:

  1. Your destination directory could be corrupt, in a subsequently unreachable location, mistyped etc. mv will still move the files there! Be careful here...
  2. If the -t option (--target-directory) is missing and the destination folder is actually a file you will move one file and fail on the rest. mv has 2 uses: rename source to destination or move source to directory. Again be careful...
Related Question