Bash – Use asterisk in variables

bashescape-characterskshshell-scriptvariable

I want to cat all files to one new file in a bash script.

For example there are three files in my dir:
– file_a.txt
– file b.txt
– file(c).txt

When I write the following, it works without problems:

cat "file"*".txt" >> out_file.bak

No I want to make it more flexible/clean by using a variable:

input_files="file"*".txt"
cat $input_files >> out_file.bak

Unfortunately this doesn't work. The question is why? (When I echo input_files and run the command in terminal everything is fine. So why doesn't it work in the bash script?)

Best Answer

It depends when you want to do the expansion:

When you define the variable:

$ files=$(echo a*)
$ echo $files
a1 a2 a3
$ echo "$files"
a1 a2 a3

When you access the variable:

$ files=a*
$ echo "$files"
a*
$ echo $files
a1 a2 a3

Related Solutions

Bash Scripting – Use Filename Arguments or Default to stdin/stdout

I'd make heavier use of I/O redirection:

#!/bin/bash
[[ $1 ]] && [[ ! -f $1 ]] && echo "file $1 dne" && exit 1
[[ $1 ]] && exec 3<$1 || exec 3<&0
[[ $2 ]] && exec 4>$2 || exec 4>&1
fgrep -v "stuff" <&3 >&4

Explanation

[[ $1 ]] && [[ ! -f $1 ]] && echo "file $1 dne" && exit 1

Test if an input file has been specified as a command line argument and if the file exists.
[[ $1 ]] && exec 3<$1 || exec 3<&0

If $1 is set, i.e. an input file has been specified, the specified file is opened at file descriptor 3, otherwise stdin is duplicated at file descriptor 3.
[[ $2 ]] && exec 4>$2 || exec 4>&1

Similarly if the $2 is set, i.e. an output file has been specified, the specified file is opened at file descriptor 4, otherwise stdout is duplicated at file descriptor 4.
fgrep -v "stuff" <&3 >&4

Lastly fgrep is invoked, redirecting its stdin and stdout to the previously set file descriptors 3 and 4 respectively.

Reopening standard input and output

If you'd prefer not to open intermediate file descriptors, an alternative is to replace the file descriptors corresponding to stdin and stdout directly with the specified input and output files:

#!/bin/bash
[[ $1 ]] && [[ ! -f $1 ]] && echo "file $1 dne" && exit 1
[[ $1 ]] && exec 0<$1
[[ $2 ]] && exec 1>$2
fgrep -v "stuff"

A drawback with this approach is that you loose the ability to differentiate output from the script itself from the output of the command which is the target for the redirection. In the original approach, you can direct script output to the unmodified stdin and stdout, which in turn might have been redirected by the caller of the script. The specified input and output files could still be accessed via the corresponding file descriptors, which are distinct from the script stdin and stdout.

Bash – Using Shell Variables for Command Options

In general, it's a bad idea to demote a list of separate items into a single string, no matter whether it's a list of command line options or a list of pathnames.

Using an array instead:

rsync_options=( -rnv --exclude='.*' )

rsync_options=( -r -n -v --exclude='.*' )

and later...

rsync "${rsync_options[@]}" source/ target

This way, the quoting of the individual options is maintained (as long as you double quote the expansion of ${rsync_options[@]}). It also allows you to easily manipulate the individual entries of the array, would you need to do so, before calling rsync.

In any POSIX shell, one may use the list of positional parameters for this:

set -- -rnv --exclude='.*'

rsync "$@" source/ target

Again, double quoting the expansion of $@ is critical here.

Tangentially related:

How can we run a command stored in a variable?

The issue is that when you put the two sets of option into a string, the single quotes of the --exclude option's value becomes part of that value. Hence,

RSYNC_OPTIONS='-rnv --exclude=.*'

would have worked¹... but it's better (as in safer) to use an array or the positional parameters with individually quoted entries. Doing so would also allow you to use things with spaces in them, if you would need to, and avoids having the shell perform filename generation (globbing) on the options.

^{¹ provided that $IFS is not modified and that there's no file whose name starts with --exclude=. in the current directory, and that the nullglob or failglob shell options are not set.}

Best Answer

Related Solutions

Bash Scripting – Use Filename Arguments or Default to stdin/stdout

Bash – Using Shell Variables for Command Options

Related Question