Linux – FTP download list of absolute paths

command lineftplinux

I have a list of (a couple thousand) absolute paths to files on a remote server that I need to download to my PC.

I also need the files to keep the directory structure intact for those files.

Example:

/* UNIX Server File-System  */
/home/username/
    stuff/
    more-stuff/
    data/
    file1.txt

/* Local Windows File-System After Transfer  */
C:\Users\username\Documents\home\username\
    stuff\
    more-stuff\
    data\
    file1.txt

Ideally, I would use some type of FTP to get those files to my PC. However, I am unaware of a program or CLI command that supports getting a list of files. I need to get specific files from specific directories, I can't just download whole directories.

My Question: How can I use a list of absolute paths to automatically download the files to my localhost? (while keeping the directory structure intact)

Additionally, I have these files in a PHP array. So it is possible for me to export the list as JSON, CSV, XML, etc.

Best Answer

If you mind you can use rsync with something like 

rsync -av --files-from=/path/yourlist.txt / remote:/backup

where in

  • /path/yourlist.txt you can put your list of files with the full path
  • / The path to add to the filename in your list (If they are full pathname /)
  • remote:/backup the remote host name and its relative path

You can read more searching for --files-from from the man rsync [1]

--files-from=FILE 

Using this option allows you to specify the exact list of files to transfer (as
read from the specified FILE or - for standard input). It also tweaks the 
default  behavior of rsync to make transferring just the  specified files and 
directories  easier:
  • The --relative (-R) option is implied, which preserves the path information that is specified for each item in the file (use --no-relative or --no-R if you want to turn that off).
  • The --dirs (-d) option is implied, which will create directories specified in the list on the destination rather than noisily skipping them (use --no-dirs or --no-d if you want to turn that off).
  • The --archive (-a) option’s behavior does not imply --recursive (-r), so specify it explicitly, if you want it.
  • These side-effects change the default state of rsync, so the position of the --files-from option on the command-line has no bearing on how other options are parsed (e.g. -a works the same before or after --files-from, as does --no-R and all other options).

... in the man page there is more...

Related Question