Bash – LFTP mirror includes – including other directories other than what I’ve included

bashcentoslftpshell-script

My question:
How can I make sure LFTP only includes files AND directories using --include or --include-glob, and not download a bunch of extra directories from the root of the remote that aren't in the includes?

Some context:

I'm using LFTP and mirror to download a sites files, but I want to only download certain files and directories. So, I've given LFTP some includes and excludes to run from the root of the remote server.

It's my understanding according to LFTP man that it will include ONLY all files you tell it to if you specify includes, and then apply your excludes TO the include rules. This works just perfect for files in the root dir and not downloading any files that aren't included… however, it's downloading all directories in the root, not just the ones I've asked it to include.

My suspicion is that I'm having trouble with using --include and --include-glob… No matter what I've tried between both (glob or regex), I can't get LFTP mirror to not download directories outside of what I specify in my includes.

Here's my my bash script (somewhat new to bash scripting)

#!/bin/bash

# Credentials
protocol="ftp" # ftp or sftp
host="host"
user="user"
pass="pass"
localcd="/path/to/localdir"
remotecd="/remotedir"

# Set up FTP URL
ftpurl="$protocol://$user:$pass@$host"

# Default includes - helps only include core files in root in case there are other random files
includes="--include-glob wp-admin/*" # other dirs are still being downloaded with these three
includes+=" --include-glob wp-content/*"
includes+=" --include-glob wp-includes/*"

includes+=" --include ^wp-.*\.php$"  # works just fine
includes+=" --include ^index.php$"   # works just fine
includes+=" --include ^xmlrpc.php$"   # works just fine

# Exclude hidden files/directories
excludes="--exclude-glob .*"
excludes+=" --exclude-glob .*/"

# LFTP sets
lftp_sets="set cmd:fail-exit yes;"
if [ "$protocol" == "ftp" ]
then
    lftp_sets+=" set ftp:ssl-allow no;"
fi

# Run the LFTP
lftp -e "$lftp_sets
open '$ftpurl';
lcd $localcd;
cd $remotecd;
mirror --continue --only-newer --delete --verbose=3 --no-perms --parallel=8 $includes $excludes"

I've tried

includes="--include ^wp-.*/.**"

and

includes="--include ^wp-admin/.*"
includes+="--include ^wp-content/.*"
includes+="--include ^wp-includes/.*"

and so many more variations that I can't keep count. Everything I've googled about LFTP mirror 4 pages deep has a purple link. 🙁

Best Answer

Order of operations.

You need your excludes first, then your includes.

This example excludes everything, then includes certain folders, then excludes files within those folders that were included.

mirror --exclude '.*' --exclude '.*/' --include 'v*-stable/' -X '*.src.rpm'

in your case

mirror --continue --only-newer --delete --verbose=3 --no-perms --parallel=8 $excludes $includes"
Related Question