Ubuntu – Why does moving some files in a folder take longer than moving the whole folder

mv

I have millions of images on my ubuntu cloud server. When I move a complete folder containing 12 million images using mv command, it happens almost instantaneously. However, when I mv only images(not the folder) then it takes some time. Is there a way to move all the images as quickly as folders ?

This is what is happening:

  1. src folder has 12 million images and I move this to dst folder using

    $ mv  src ../dst

    Happens immediately

  2. Inside src folder I do this to move: 

    find -maxdepth 1 -name '*.jpg' -exec mv -t ../../dst/ {} +

    This takes some time.

Is there a way to speed up the second process ?

Best Answer

TL;DR: No

For a smaller amount of files, you would not need find but, even in this simplified and smaller case, if you just 

mv *.jpg ../../dst/

it will take more time than moving the whole directory at once.

Why? The point is to understand what mv does.

Briefly speaking, mv moves a number (that identifies a directory, or a file) from an inode (the directory containing it) to another one, and these indices are updated in the journal of the file system or in the FAT (if the file system is implemented in such a way).

If source and destination are on the same file system, there is no actual movement of data, it just changes the position, the point where they are attached to.

So, when you mv one directory, you are doing this operation one time.

But when you move 1 million files, you are doing this operation 1 million times.

To give you a practical example, you have a tree with a many branches. In particular, there is one node to which 1 million branches are attached.
To cut down these branches and move them somewhere else, you can either cut each one of them, so you make 1 million cuts, or you cut just before the node, thus making just one cut (this is the difference between moving the files and the directory).

Related Question