Scp all files ending ‘-123.jpg’ – recursion, spaces, wildcards …

scpsshterminal

There's too much going on for me to get my noobie head around here. I'm wanting to download all files that end -123.jpg from a multitude of nested directories on a remote server. Some of these directories have spaces in their names. I'm thinking that the command should be along the lines of:

scp -r user@server:/path/to/parent\ directory/*/*/*123\.jpg ./

… where "parent\ directory" is a directory name with a space, and the specified path goes as deep as it can before it splits off to various sub-directories, for example dir/sub dir/[uniquely_id]-123.jpg file. (Note that these sub-directories often contain spaces too, should that affect the * wildcard)

I'm getting 'no match' returned for this, or 'no such file or directory' if I meddle with the space escaping. I'm thinking therefore that it's the recursion or the wildcard that I've got wrong.

Thanks in advance.

Best Answer

Filename escaping with scp is tricky; your arguments get (re)expanded on the remote side. "Interesting" characters have to be double quoted, to avoid special handing by your local shell and by the remote. I would expect both the following to work for you:

scp user@server:'/path/to/parent\ directory/*/*/*123.jpg' ./
scp user@server:'"/path/to/parent directory"/*/*/*123.jpg' ./

If that's confusing, rsync is pretty easy to use (as lothar mentions), and piping tar or cpio through ssh is pretty easy too.

ssh user@server 'cd /path/to/parent\ directory;
                 find -name "*123.jpg" -print0 | cpio -0 -o' |
    cpio -i -d -v

Related Solutions

Why do you need to escape spaces when passing SCP filenames

In the first case, you are specifying two files as copy target, both residing on remote.net. This is clearly ambiguous, as the program cannot determine where to copy the file.

Consider the following, working example specifying two source files and one destination (directory) using the same format:

scp 'me@example.org:file1 file2' .

This will copy both file1 and file2 from example.org to the local directory.

Your local scp program connects to the remote host and starts another instance of of scp with a parameter determining whether you're transferring files from it (-f), or to it (-t). You can see this when you add the argument -v for verbose output. The source/destination parameter you enter (host:file) is split and the file names part simply appended as additional parameters specifying the files to use.

When using it as source:

debug1: Sending command: scp -v -f -- file1 file2

This works without problem, it transfers multiple source files over the network.

When using it as sink:

debug1: Sending command: scp -v -t -- file1 file2

This fails, as you're intending to write to multiple files. Now the way around this is to escape the file name again (once for the local shell, once for the remote call):

$ scp foo 'me@example.org:foo\ bar'
[...]
debug1: Sending command: scp -v -t -- foo\\ bar

And FWIW, scp foo 'me@example.org:"foo bar"' works just as well.

Bash – Using wildcards in scp command

Arrays are not a feature found in /bin/sh, so use #!/bin/bash

scp by default shows a progress of the files transferred so I question whether you really need to print out each filename by yourself.

Your wildcards are not expanding because you add them into the array with quotes, and then you quote the array expansion in the for loop, so the wildcards are not expanded then either.

Let the wildcards expand when you store them in the array, and send all the filenames separately as scp arguments:

#!/bin/bash
files=( 
    path1/subpath/file.*.ext 
    path2/subpath2/nowildcard.ext2 
    path3/subpath3/file3.*.ext3
)
scp -P12345 "${files[@]}" name@host:/permanent/path/

You can further simplify by omitting the array altogether, in which case you can fall back to /bin/sh:

#!/bin/sh
scp -P12345                         \
    path1/subpath/file.*.ext        \
    path2/subpath2/nowildcard.ext2  \
    path3/subpath3/file3.*.ext3     \
    name@host:/permanent/path/

Best Answer

Related Solutions

Why do you need to escape spaces when passing SCP filenames

Bash – Using wildcards in scp command

Related Question