Does rsync Verify Files Copied Between Local Drives – Verification Guide

rsyncverification

I want to make a fresh new copy of a large number of files from one local drive to another.

I've read that rsync does a checksum comparison of files when sending them to a remote machine over a network.

Will rsync make the comparison when copying the files between two local drives?
If it does do a verification – is it a safe bet? Or is it better to do a byte by byte comparison?

Best Answer

rsync always uses checksums to verify that a file was transferred correctly. If the destination file already exists, rsync may skip updating the file if the modification time and size match the source file, but if rsync decides that data need to be transferred, checksums are always used on the data transferred between the sending and receiving rsync processes. This verifies that the data received are the same as the data sent with high probability, without the heavy overhead of a byte-level comparison over the network.

Once the file data are received, rsync writes the data to the file and trusts that if the kernel indicates a successful write, the data were written without corruption to disk. rsync does not reread the data and compare against the known checksum as an additional check.

As for the verification itself, for protocol 30 and beyond (first supported in 3.0.0), rsync uses MD5. For older protocols, the checksum used is MD4.

While long considered obsolete for secure cryptographic hashes, MD5 and MD4 remain adequate for checking file corruption.

Source: the man page and eyeballing the rsync source code to verify.

Related Solutions

Using rsync to move (not copy) files between directories

You can pass --remove-source-files to rsync to move files instead of copying them.

But in your case, there's no point in using rsync, since the destination is empty. A plain mv will do the job as fast as possible.

In your case, what could make a difference to performance is the choice of network protocol, if you have a choice among NFS, Samba, sshfs, sftp, rsync over ssh, tar piped into ssh, etc. The relative speed of these methods depends on the file sizes, the network and disk bandwidth, and other factors, so there's no way to give general advice, you'll need to run your own benchmarks.

Make rsync move (not copy) files on local file system

Rsync copies files. That's what it does. Even if you tell it to remove source files, it still copies them first, it never moves them, even when the destination happens to be on the same filesystem.

The mv utility from GNU coreutils has an option -u to move files only if the destination is older than the source or doesn't exist yet. This is similar to rsync's -u.

If you want to move a directory tree into an existing one, you can use a recursive traversal that calls mv on every file. If you don't want to overwrite existing files:

cd d1
find . -depth -exec sh -c 'test -e "../d2/$0" || mv "$0" "../d2/$0"' {} \;

If you do want to overwrite existing files, you'll need to distinguish between existing directories and other files, and to decide what to do when the source has a regular file where the destination has a directory.

Best Answer

Related Solutions

Using rsync to move (not copy) files between directories

Make rsync move (not copy) files on local file system

Related Question