How to copy large files over the network

rsync

I need to copy large files over the network, but I am not sure about the command line to use for it. I have been using rsync, but I feel maybe I could zip the files first?

Best Answer

rsync already provides compression when you specify option -z. You should avoid applying this option for already compressed files or binaries as it will just introduce additional overhead.

When using rsync with the ssh protocol (the default) and you have only limited CPU resources and security is not an issue, you can also try specifying -e ssh -c arcfour to use a less expensive encryption algorithm (RC4). On modern CPUs this won't make any big difference, though.

Other interesting arguments you may want to use:

-vP for a nice progress bar and keep partially transfered files when interrupted, so you can resume from the partial file the next time you run rsync
-a to preserve most (but not all!) file meta data such as permissions and modification times

Related Solutions

Using rsync to move (not copy) files between directories

You can pass --remove-source-files to rsync to move files instead of copying them.

But in your case, there's no point in using rsync, since the destination is empty. A plain mv will do the job as fast as possible.

In your case, what could make a difference to performance is the choice of network protocol, if you have a choice among NFS, Samba, sshfs, sftp, rsync over ssh, tar piped into ssh, etc. The relative speed of these methods depends on the file sizes, the network and disk bandwidth, and other factors, so there's no way to give general advice, you'll need to run your own benchmarks.

Inotify and rsync on large number of files

If the server has a slow processor avoid checksums and compression with rsync. I would remove ht "-z" option in the rsync command.

rsync --update -alvr --exclude '*cache*' --exclude '*.git*' /var/www/* root@secondwebserver:/var/www/

Note that it will not avoid rsync to compare the 650k files. You could rsync subdirectories of /var/www one by one to reduce the number of files checked at one time.

Best Answer

Related Solutions

Using rsync to move (not copy) files between directories

Inotify and rsync on large number of files

Related Question