SCP – Best Methods to Copy Large Numbers of Small Files

scptar

I have a directory that's got several gigabytes and several thousand small files. I want to copy it over the network with scp more than once. CPU time on the source and destination machines is cheap, but the network overhead added by copying each file individually is huge. I would tar/gzip it up and ship it over, but the source machine is short on disk.

Is there a way for me to pipe the output of tar -czf <output> <directory> to scp? If not, is there another easy solution? My source machine is ancient (SunOS) so I'd rather not go installing things on it.

Best Answer

You can pipe tar across an ssh session:

$ tar czf - <files> | ssh user@host "cd /wherever && tar xvzf -"

Related Solutions

Copying large tree from one machine to another, maintaining ownership

As root, set up a named pipe:

# mkfifo /tmp/fifo
# chmod o+w /tmp/fifo

Then, transfer your data as me:

$ tar cfzp - foldertocopy | ssh me@machine "cat > /tmp/fifo"

But read it as root:

# tar -xfzp /tmp/fifo

How to copy only new files using “scp” command

You can use rsync for it. rsync is really designed for this type of operation.

Syntax:

rsync -avh /source/path/ host:/destination/path

rsync -a --ignore-existing /local/directory/ host:/remote/directory/

When you run it first time it will copy all content then it will copy only new files.

If you need to tunnel the traffic through a SSH connection (for example, for confidentiality purposes), as indicated by you originally asking for a SCP-based solution, simply add -e ssh to the parameters to rsync. For example:

rsync -avh -e ssh /source/path/ host:/destination/path

Best Answer

Related Solutions

Copying large tree from one machine to another, maintaining ownership

How to copy only new files using “scp” command

Related Question