Bash – How to make bash built-in “read” ignore commented or empty lines

bashreadshell-script

(For simplicity I'll assume the file to read is the first argument – $1.)

I can do what I want externally with:

tempfile=$(mktemp)
awk '/^#/ {next}; NF == 0 {next}; {print}' "$1" > $tempfile
while read var1 var2 var3 var4 < $tempfile; do
  # stuff with var1, etc.
done

However, it seems absurd to need to call awk every time I parse the config file. Is there a way to make read ignore commented or whitespace-only lines in a file, without external binaries/potential performance issues?

Answers so far are quite helpful! To clarify, I don't want to use a temp file, but I do want to read the config from a file, not from standard in. I'm well aware that I can use an input redirection when I call the script, but for various reasons that won't work in my circumstance.

I want to softcode the input to read from, e.g.:

configfile="/opt/myconfigfile.txt"
[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

while read var1 var2 var3 var4 < "$configfile" ; do
  ...

But when I try this, it just reads the first line of configfile over and over until I kill the process.

Maybe this should be its own question…but it's probably a single line change from what I'm doing. Where's my error?

Best Answer

You don't need a tempfile to do this, and sed (or awk) are far more flexible in comment processing than a shell case statement.

For example:

configfile='/opt/myconfigfile.txt'
[ $# -gt 0 ] && [ -r "$1" ] && configfile="$1"

sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' "$configfile" |
    while read var1 var2 var3 var4; do
      # stuff with var1, etc.
    done

This strips comments (with or without leading whitespace) and deletes empty lines from the input before piping it into the while loop. It handles comments on lines by themselves and comments appended to the end of the line:

# full-line comment
# var1 var2 var3 var4
abc 123 xyz def # comment here

Calling sed or awk for tasks like this isn't "absurd", it's perfectly normal. That's what these tools are for. As for performance, I'd bet that in anything but very tiny input files, the sed version would be much faster. Piping to sed has some startup overhead but runs very fast, while shell is slow.

Related Solutions

Bash – How to Read from Two Input Files Using While Loop

If you can guarantee that some character will never occur in the first file then you can use paste.

For example you know for sure that @ will never occur:

paste -d@ file1 file2 | while IFS="@" read -r f1 f2
do
  printf 'f1: %s\n' "$f1"
  printf 'f2: %s\n' "$f2"
done

Note that it is enough if the character is guaranteed to not occur in the first file. This is because read will ignore IFS when filling the last variable. So even if @ occurs in the second file it will not be split.

Example using some bash features for arguably cleaner code and paste using default delimiter tab:

while IFS=$'\t' read -r f1 f2
do
  printf 'f1: %s\n' "$f1"
  printf 'f2: %s\n' "$f2"
done < <(paste file1 file2)

Bash features used: ansi c string ($'\t') and process substitution (<(...)) to avoid the while loop in a subshell problem.

If you cannot be certain that any character will never occur in both files then you can use two file descriptors.

while true
do
  read -r f1 <&3 || break
  read -r f2 <&4 || break
  printf 'f1: %s\n' "$f1"
  printf 'f2: %s\n' "$f2"
done 3<file1 4<file2

Not tested much. Might break on empty lines.

File descriptors number 0, 1, and 2 are already used for stdin, stdout, and stderr, respectively. File descriptors from 3 and up are (usually) free. The bash manual warns from using file descriptors greater than 9, because they are "used internally".

Note that open file descriptors are inherited to shell functions and external programs. Functions and programs inheriting an open file descriptor can read from (and write to) the file descriptor. You should take care to close all file descriptors which are not required before calling a function or external program.

Here is the same program as above with the actual work (the printing) separated from the meta-work (reading line by line from two files in parallel).

work() {
  printf 'f1: %s\n' "$1"
  printf 'f2: %s\n' "$2"
}

while true
do
  read -r f1 <&3 || break
  read -r f2 <&4 || break
  work "$f1" "$f2"
done 3<file1 4<file2

Now we pretend that we have no control over the work code and that code, for whatever reason, tries to read from file descriptor 3.

unknowncode() {
  printf 'f1: %s\n' "$1"
  printf 'f2: %s\n' "$2"
  read -r yoink <&3 && printf 'yoink: %s\n' "$yoink"
}

while true
do
  read -r f1 <&3 || break
  read -r f2 <&4 || break
  unknowncode "$f1" "$f2"
done 3<file1 4<file2

Here is an example output. Note that the second line from the first file is "stolen" from the loop.

f1: file1 line1
f2: file2 line1
yoink: file1 line2
f1: file1 line3
f2: file2 line2

Here is how you should close the file descriptors before calling external code (or any code for that matter).

while true
do
  read -r f1 <&3 || break
  read -r f2 <&4 || break
  # this will close fd3 and fd4 before executing anycode
  anycode "$f1" "$f2" 3<&- 4<&-
  # note that fd3 and fd4 are still open in the loop
done 3<file1 4<file2

Bash – How to call bash function from within awk

You may be able to do what you want by piping awk's output into a while read loop. For example:

awk '/^#/ {next}; NF == 0 {next}; NF != 4 {exit 1} ; {print}' | 
    while read NAME METHOD URL TAG ; do
        :  # do stuff with $NAME, $METHOD, $URL, $TAG
        echo "$NAME:$METHOD:$URL:$TAG"
    done

if [ "$PIPESTATUS" -eq 1 ] ; then
    : # do something to handle awk's exit code
fi

Tested with:

$ cat input.txt 
# comment
NAME METHOD URL TAG
a b c d
1 2 3 4
x y z
a b c d

$ ./testawk.sh <input.txt 
NAME:METHOD:URL:TAG
a:b:c:d
1:2:3:4

Note that it correctly exits on the fifth x y z input line.

Best Answer

Related Solutions

Bash – How to Read from Two Input Files Using While Loop

Bash – How to call bash function from within awk

Related Question