Bash – Maintain filenames as separate arguments in successive commands

bashquotingshellwhitespace

A disclaimer: This isn't related to one specific command like ls or grep, although I use them in the examples, this is more a bash thing.

Let's say I want to see the file sizes, timestamps, and permissions/owner/group on a set of files (ls -l) that all contain "a certain word". Typically I'd do this:

ls -l $( grep -l 'a certain word' * )

And that works like magic until any of the file names contain a space, like a file named "file name". Then ls can't find what it thinks are two file names, "file" and "name".

Bash scripts have a familiar way of solving a variant of this problem, if you pass file names in on the command line to a script, they are referenced as ${1}, ${2}, etc. And if you wrap them in quotes, i.e. "${1}", "${2}", then if they have spaces you don't have a problem and you can even process all the file names at once by using "${@}" which will correctly list each of the parameters, and keep them intact even if they have spaces – I just haven't found such a mechanism for the grep/ls style problem I list above. If you wrap the expansion command, $( ) with quotes it treats all the files as one big parameter for the ls.

I've usually worked around it with something like grep -l .... | while read file; do ls -l "${file}"; done, but I really shouldn't have to do that. And doing that still doesn't allow things like sorting the files by date/size/etc (i.e. ls -ltr $( grep -l ....))

I keep thinking bash has a solution for this and I just haven't came across it yet and this problem only surfaces once in a while and I usually work around it, but it's now bugged me enough I think there must be a way.

And yes, I've seen the following, they are either questions solved by the specific tool being used (like tar), or the answers are "sorry, you can't get to there from here" because their question is command-specific. I'm asking for handling any space-embedded file name(s) given as results to be passed as parameters to another command, not more while or find magic to solve this issue. I would rather this question sit unanswered (a testament that there isn't a solution) rather than have any of the not-the-same-thing questions offered:

Best Answer

Hmm. @don_crissti already gave the answer for grep in a comment. But since you said it wasn't really about grep or ls, I'm going to rewrite the command in question to not use those commands.

What I think you want is:

do-something-with $(produce-list-of-files)

where the output of one command should be dropped in as command line parameters to another command. There just happens to be a utility just for that, it is called xargs (man page).

If the file names are "nice", we could do just

produce-list-of-files | xargs do-something-with

If the file names can contain spaces but are separated by newlines, we have to tell xargs to not split on any whitespace, but only newlines:

produce-list-of-files | xargs -d '\n' do-something-with

If the file names can contain newlines too, the list has to be separated by NULs ('\0', byte with value zero), and we need an xargs that supports it. At least some versions of various utilities support listing files separated by NULs instead of newlines, there's at least find -print0, sort -z and grep -Z in the GNU versions of those tools. In xargs the flag is --null or -0. So:

produce-list-of-files -0 | xargs -0 do-something-with

An example run with cat and, well, ls -l:

$ touch "abba acdc" "foo bar" $'new\nline'
$ echo -en "abba acdc\0foo bar\0new\nline\0" > list
$ cat list | xargs -0 ls -l
-rw-r--r-- 1 itvirta itvirta 0 Aug  2 01:23 abba acdc
-rw-r--r-- 1 itvirta itvirta 0 Aug  2 01:23 foo bar
-rw-r--r-- 1 itvirta itvirta 0 Aug  2 01:23 new?line

Related Solutions

Bash – Trouble w/ bash programmable completion of filenames matching patterns

Not sure if this is still an issue for you but I ran into something similar while trying to get mid-word filename completion and have some ideas for possible solutions.

Changing the IFS variable by adding local IFS=$'\n' fixed the problem with spaces in filenames for me, might be worth a try.

As for making things case-insensitive, you could force $curr to lowercase for compgen by using ${cur,,}. Full function would look like this after both changes:

_th()  {
  local IFS=$'\n'
  local curr=${COMP_WORDS[COMP_CWORD]}
  COMPREPLY=( $(compgen -f -X '!*.@(tar|tar.xz|tar.bz2|tar.gz|tbz|tgz)' -- "${curr,,}") )
}                
complete -F _th -o filenames th

That would make the whole filename matching case-insensitive. If you only want the extension case insensitive you could do some string manipulation to $cur to make only the extension lowercase.

Bash – Custom autocomplete: deal with spaces in filenames

Believe it might be better to use compgen instead of find in this case.

You probably already have a completion script with system. Try e.g.

locate bash_completion

On Debian variants this is probably:

/usr/share/bash-completion/bash_completion

where you find e.g. _filedir. So the simplest way then would be something in the direction of:

*)
    pushd "/some/path" >/dev/null
    _filedir
    popd >/dev/null

If that is not an option this could be a starter:

_comp_by_path()
{
    local opt cur dir
    local IFS=$'\n' x tmp
    local -a tokens

    opt="$1"
    cur="$2"
    dir="$3"

    # Enter target directory
    pushd "$dir" >/dev/null

    # Get directories, filtered against current
    [[ "$opt" != "-f" ]] && \
    x=$( compgen -d -- "$cur" ) &&
    while read -r tmp; do
        tokens+=( "$tmp" )
    done <<< "$x"

    # Get files, filtered against current
    [[ "$opt" != "-d" ]] && \
    x=$( compgen -f -- "$cur" ) &&
    while read -r tmp; do
        tokens+=( "$tmp" )
    done <<< "$x"

    # If anything found
    if [[ ${#tokens[@]} -ne 0 ]]; then
        # Make sure escaping is OK
        compopt -o filenames 2>/dev/null
        COMPREPLY+=( "${tokens[@]}" )
    fi

    # Go back
    popd >/dev/null
}

_GetOptMyCommand()
{
    local cur

    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"

    case "$cur" in
    -*)
        COMPREPLY=( $( compgen -W "-h -l --help --list --" -- "$cur" ) );;
    *)
        _comp_by_path "any" "$cur" "/some/path"
    esac
}

complete -F _GetOptMyCommand my_command

A variant using find could be something in direction of this:

_zaso()
{
    local dir="$1"

    pushd "$dir" >/dev/null
    find * -maxdepth 0 2>/dev/null
    popd >/dev/null
}

_comp_with_find()
{
    local cur dir      
    local IFS=$'\n'

    cur="$1"
    dir="$2"

    compopt -o filenames 2>/dev/null
    COMPREPLY=( $( compgen -W "$(_zaso "$dir")" -- "$cur" ) );
}

Also note that printf in Bash has a %q option. So to generate quoted strings this is an option to play with:

find * -maxdepth 0 2>/dev/null && \
while read -r tmp; do
    printf "%q\n" "$tmp"
done <<< "$x"

Also not that file names can have newline characters in which a lot of this will break. Have not found a way to use \0 with compgen.

Best Answer

Related Solutions

Bash – Trouble w/ bash programmable completion of filenames matching patterns

Bash – Custom autocomplete: deal with spaces in filenames

Related Question