Ssh – How to copy big data over network

rsync already provides compression when you specify option -z. You should avoid applying this option for already compressed files or binaries as it will just introduce additional overhead.

When using rsync with the ssh protocol (the default) and you have only limited CPU resources and security is not an issue, you can also try specifying -e ssh -c arcfour to use a less expensive encryption algorithm (RC4). On modern CPUs this won't make any big difference, though.

Other interesting arguments you may want to use:

• -vP for a nice progress bar and keep partially transfered files when interrupted, so you can resume from the partial file the next time you run rsync
• -a to preserve most (but not all!) file meta data such as permissions and modification times

ssh provides a connected stream from the local system running your rsync to another instance running on the remote server. The local rsync starts it as part of the ssh connection process - it is not the same instance of rsync that might be running as a daemon on that remote server.

What is an ssh stream?

An ssh stream is what you use when you ssh to a remote system. You might prefer to call it an ssh session. That's fine. As an example, ssh snow@grumpy.example.net would connect you to that remote system and give you an interactive session. Anything you typed locally would be received and actioned remotely, and anything generated remotely would be presented to you locally.

Similarly, ssh -t snow@doc.example.net vi /etc/hosts would set up a connection to run vi /etc/hosts on the remote system as if you were using it locally. (The -t flag tells ssh to ensure that the remote command, vi, is run on a pseudo-terminal; otherwise it would skip that layer of complexity.)

Now, for rsync, the connection is set up by the local instance with something like ssh snow@bashful.example.net rsync --server -nlogDtpre.iLsf {local_path} {remote_path}. This allows the local rsync to talk with the remote rsync over a connection just like you would use to access that same remote server. The remote instance of rsync knows it's a server because of the flags it has been provided when started by the local instance.

You can see how rsync has been called remotely by running something like ssh snow@dopey.example.net 'ps -ef | grep [r]sync' while the rsync is in progress.

Why not just have rsync talk directly without ssh?

The advantage of using ssh is twofold

1. It is a well known protocol and the mechanisms for authentication and logging in to a remote system are well understood (by those who need to understand it) and trusted
2. It provides a secure and encrypted channel over which any sensitive data can be safely passed. Without this layer rsync would either have to implement its own encryption or ignore the issue. Both are hazardous in today's Internet connected world.