Bash – How to copy between two remote hosts using tar piped into SSH from remote server when behind a firewall

backupbashsshtar

I'd like to transfer a directory between two servers, but compress the directory on the remote host before transfer, and then uncompress to another host. I'm sure it's possible to pipe everything all the way through and do it in a one liner.

I realise it would be better of course if I could transfer between the hosts directly but that would involve transferring keys and what not, and I love Unix one line power tools. I'm sure people can come up with a few different ways to do this. I'm looking for the shortest syntax and most bandwidth conservative.

To start off I have 

ssh -n REMOTEHOST 'tar zcvf - DIRTOCOPY' | localZip.tar.gz

Best Answer

Similar to what jw013 suggested in the comments with separate compression/decompression steps, i.e. combine two ssh commands with a pipe:

compress=gzip
decompress=gunzip

ssh remote1 "cd srcdir; tar cf - dir | $compress" |
ssh remote2 "cd destdir; $decompress | tar xvf -"

Note that the compression/decompression is configurable without depending on the version of tar.

Update

You could also add checksum verification into the pipe:

compress=gzip
decompress=gunzip

ckprg=md5sum
cksum=/tmp/cksum

ssh remote1 "cd srcdir; tar cf - dir | $compress | tee <($ckprg > $cksum)" |
ssh remote2 "cd destdir; tee <($ckprg > $cksum) | $decompress | tar xvf -"

ssh remote1 cat $cksum
ssh remote2 cat $cksum
Related Question