How to rsync only a specific list of files to the same location on another server

awkrsyncscp

I have a list of files:
/location/file.txt

Content of file:

/location1/file2.abc
/location2/location3/file1.abc
/location4/location5/fileX.cde

I'd like to copy files to another sever like this:

cat /location/file.txt | xargs -i scp {} user@server:/location

but in the above example all of files from the list are copying to specific location, while I'd like to copy them to the same locations (accurate path as in the txt file /location1/, /location2/location3/ etc.

How can I do it? I can also use rsync, but I'd like to enter password (to remote server) only once.

Best Answer

rsync -av --files-from=/location/file.txt / user@server:/location/

That will copy the local file /location1/file2.abc to /location//location1/file2.abc etc.

As rsync will invoke ssh only once, you only have to enter the password once. You may want to investigate using ssh keys though.

How to know where the file is?

a) Use SSH in another terminal

Just open a second terminal, SSH to bar, find your file and copy/paste the path to the first one.

b) Use SFTP

SFTP (not related to FTP nor FTPS in any way!) is implemented in OpenSSH and is available by default. Just SFTP to the server and use the FTP-like commands to find you files and get them.

c) Use a GUI

Filezilla or Nautilus for instance can browse remote SFTP/SSH shares.

d) Set up certificates

When you set up certificate connection, you can do tab completion on the local side as well as on the remote side! For instance, with your buzz example, you can do:

bob@foo$ scp bob@bar:/guest/[tab][tab]

and wait a little for the list of files contained in the remote /guest/ folder.

How to set up SSH with certificates?

a) If not already done, generate your personal RSA key pair

If you've installed OpenSSH client, you can do it by typing

bob@foo$ ssh-keygen -t rsa

(look at the manual or online for all the available options). It may ask you for a password. This is not your local account password but an optional password that can be used to encrypt the private key you are about to generate.

Actually, you will generate 2 files:

/Users/[yourusername]/.ssh/id_rsa
/Users/[yourusername]/.ssh/id_rsa.pub

The first one, *id_rsa* ought to be private. By default, ssh-keygen will do everything it can to avoid making it public (using filesystem access permission). That's why it will also ask you for an (optional) password. Don't be too paranoid with that, but just remember *id_rsa* == personal key == private. This key should never ever leave your computer.

The second one is public. It requires a huge amount of computer power to get back your private key from this public certificate (I really mean HUUUUUUGE). This is perfectly safe to share it with the whole world. Even in the very unlikely event that the NSA or the likes really want to spend millions of dollars cracking your public key, your macbook will still be safe … (or not. Thinking about it, if someone wants to spend that much, you are in trouble :)

This public certificate is actually what you will put on the remote server bar.

b) How do I put my public certificate on the server?

Two options.

Use ssh-copy-id if available: bob@foo$ ssh-copy-id bob@bar. Done.
If it is not, copy ~/.ssh/id_rsa.pub to bar:

bob@foo$ sftp ~/.ssh/id_rsa.pub bob@bar:pub_cert

(here, you copied your public cert id_rsa.pub from .ssh/, in your personal ~ folder to the remote computer bar in the home folder of the user bob. This is the default. Also note that id_rsa.pub has been renamed to pub_cert in the process. I used sftp just to show you that it can be used exactly as scp).

Now, we shall copy this certificate to the right location:

bob@foo$ ssh bob@bar

Now you are in bob's personal folder in bar.

bob@bar$ cat pub_cert >> .ssh/known_hosts

(here, you displayed the content of pub_cert with cat. But instead of printing it to the screen, you redirect this output to a file: .ssh/known_hosts. Note that a redirection with > would mean "replace the content of the file with this stream" while >> means "append the stream at the end of the existing file").

c) Result?

Now you can scp/sftp/ssh to bar as much as you want without having to provide a password. You can also autocomplete local and remote paths using the [tab] key.

d) What about my mac security?

With this way of doing this, you don't even need a running SSH server on your computer. Only an SSH client (the scp/sftp/ssh programs). This is safe for you even if bar is compromised.

e) What did I do exactly with these keys/certificates?

First you generated a couple of files: a private key and a public certificate. You can do a lot of things related to security and authentication with them. But in our case, with a fair bit of simplification, these are used this way:

When you try to connect to bar, you will advertise that you've got a certificate that you can use for the connection. bar will inspect various locations in the system, including ~/.ssh/known_hosts. It will find the certificate that you advertised and use it to send you encrypted data.

Actually, public certificates can encrypt stuff!

Now this is great, but how can foo understand that? Using your private key.

Private keys can decrypt stuff encrypted with the corresponding public certificate!

This is what is called asymmetric encryption.

Then, basically, the server will send a complicated password to you encrypted with your public certificate. You will receive it, decrypt it with your private key and start to use it to encrypt data with the server both ways.

Now, what if you really really want to do the things your way and SCP back to foo?

You are just asking for troubles. But to mitigate the effect of a possible compromission, you can set up a chrooted SFTP-only server. scp and ssh won't work any more, but sftp, Filezilla and stuff are gonna work.

ref: https://www.allthingsdigital.nl/2013/05/12/setting-up-an-sftp-only-account-with-openssh/

The most robust remote file copy

I would definitely suggest rsync. I use rsync to copy files anytime I think that the connection has any possibility of being interrupted. If the copy fails, I know I can simply start it again.

It's easy to put it in a while loop if you need it to automatically restart until it succeeds.