Bash – Running a command multiple times with arguments (filenames) from a file

bashcommand lineshell-script

I have a file with a long list of filenames (with full paths). I've also got a program I'd like to run multiple times, using one and one filename from this list as argument. The program I'd like to run, is sadly a self-made script; so it can only take one filename at a time, and it cannot accept input from stdin (like a list of filenames) – so no piping or input-redirection possible.

What I need, is a command that will run another command (my script), using the lines in the file as argument.

Something like find with the -exec-action… (find . -name "*.txt" -exec command \; ). I guess I actually could use find here, but I'd like for the input-files to be sorted…

So what I need is something like this:

for_each_command1 -f list_of files -c './my_command {}'
for_each_command2 -f list_of_files -exec ./my_command {} \;
for_each_command3 -c './my_command {}' 

My usual way to handle such tasks is with sed - unfortunately, there's a lot of overhead and it's not pretty (but it does work...):


$ wc -l list_of_files
232
$ for (( i=1; i do
> ./my_command "`sed -n "${i}p" list_of_files`"
> done

So is there a command or shell built-in to handle something like this?

Best Answer

Conveniently, xargs -I does exactly what you want:

$ xargs <my_file_list.txt -I filename ./my_command "filename"

-I replace-str
    Replace occurrences of replace-str in the initial-arguments with names read
    from standard input.
    Also, unquoted blanks do not terminate input items;
    instead the separator is the newline character.
    Implies -x and -L 1.

This means it takes exactly one newline-delimited input line and invokes your command with it as a single argument. Nothing apart from newline is treated as a delimiter, so spaces and other special characters are fine.

Note that if you wanted to allow newlines in your filenames, the nul terminator (as per Wildcard's answer) will also work in a file.

Related Solutions

Bash Pipe – Using Data Read from a Pipe Instead of a File in Command Options

This is dependent on the command. Some commands that read from a file expect that the file be a regular file, whose size is known in advance, and which can be read from any position and rewinded. This is unlikely if the contents of the file is a list of file names: then the command will probably be content with a pipe which it will just read sequentially from start to finish. There are several ways to feed data via a pipe to a command that expects a file name.

Many commands treat - as a special name, meaning to read from standard input rather than opening a file. This is a convention, not an obligation.
```
ls | command -r -
```
Many unix variants provide special files in /dev that designate the standard descriptors. If /dev/stdin exists, opening it and reading from it is equivalent to reading from standard input; likewise /dev/fd/0 if it exists.
```
ls | command -r /dev/stdin
ls | command -r /dev/fd/0
```
If your shell is ksh, bash or zsh, you can make the shell deal with the business of allocating some file descriptor. The main advantage of this method is that it's not tied to standard input, so you can use standard input for something else, and you can use it more than once.
```
command -r <(ls)
```
If the command expects the name to have a particular form (typically a particular extension), you can try to fool it with a symbolic link.
```
ln -s /dev/fd/0 list.foo
ls | command -r list.foo
```
Or you can use a named pipe.
```
mkfifo list.foo
ls >list.foo &
command -r list.foo
```

Note that generating a list of files with ls is problematic because ls tends to mangle file names when they contain unprintable characters. printf '%s\n' * is more reliable — it'll print every byte literally in file names. File names containing newlines will still cause trouble, but that's unavoidable if the command expects a list of file names separated by newlines.

Bash – Custom autocomplete: deal with spaces in filenames

Believe it might be better to use compgen instead of find in this case.

You probably already have a completion script with system. Try e.g.

locate bash_completion

On Debian variants this is probably:

/usr/share/bash-completion/bash_completion

where you find e.g. _filedir. So the simplest way then would be something in the direction of:

*)
    pushd "/some/path" >/dev/null
    _filedir
    popd >/dev/null

If that is not an option this could be a starter:

_comp_by_path()
{
    local opt cur dir
    local IFS=$'\n' x tmp
    local -a tokens

    opt="$1"
    cur="$2"
    dir="$3"

    # Enter target directory
    pushd "$dir" >/dev/null

    # Get directories, filtered against current
    [[ "$opt" != "-f" ]] && \
    x=$( compgen -d -- "$cur" ) &&
    while read -r tmp; do
        tokens+=( "$tmp" )
    done <<< "$x"

    # Get files, filtered against current
    [[ "$opt" != "-d" ]] && \
    x=$( compgen -f -- "$cur" ) &&
    while read -r tmp; do
        tokens+=( "$tmp" )
    done <<< "$x"

    # If anything found
    if [[ ${#tokens[@]} -ne 0 ]]; then
        # Make sure escaping is OK
        compopt -o filenames 2>/dev/null
        COMPREPLY+=( "${tokens[@]}" )
    fi

    # Go back
    popd >/dev/null
}

_GetOptMyCommand()
{
    local cur

    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"

    case "$cur" in
    -*)
        COMPREPLY=( $( compgen -W "-h -l --help --list --" -- "$cur" ) );;
    *)
        _comp_by_path "any" "$cur" "/some/path"
    esac
}

complete -F _GetOptMyCommand my_command

A variant using find could be something in direction of this:

_zaso()
{
    local dir="$1"

    pushd "$dir" >/dev/null
    find * -maxdepth 0 2>/dev/null
    popd >/dev/null
}

_comp_with_find()
{
    local cur dir      
    local IFS=$'\n'

    cur="$1"
    dir="$2"

    compopt -o filenames 2>/dev/null
    COMPREPLY=( $( compgen -W "$(_zaso "$dir")" -- "$cur" ) );
}

Also note that printf in Bash has a %q option. So to generate quoted strings this is an option to play with:

find * -maxdepth 0 2>/dev/null && \
while read -r tmp; do
    printf "%q\n" "$tmp"
done <<< "$x"

Also not that file names can have newline characters in which a lot of this will break. Have not found a way to use \0 with compgen.

Best Answer

Related Solutions

Bash Pipe – Using Data Read from a Pipe Instead of a File in Command Options

Bash – Custom autocomplete: deal with spaces in filenames

Related Question