Bash – How does the -d option to bash read work

bashoptionsquotingreadshell-script

I have a bash script where I'm trying to assign a heredoc string to a variable using read, and it only works if I use read with the -d ''
option, I.e.

read -d '' <variable>

script block

#!/usr/bin/env bash

function print_status() {
    echo
    echo "$1"
    echo 
}


read -d '' str <<- EOF
    Setup nginx site-config
    NOTE: if an /etc/nginx/sites-available config already exists for this
    website, this routine will replace existing config with template from
    this script. 
EOF

print_status "$str"

I found this answer on SO which is where I copied the command from, it works, but why?
I know the first invocation of read stops when it encounters the first newline character, so if I use some character that doesn't appear in the string the whole heredoc gets read in, e.g.

read -d '|' <variable> — this works
read -d'' <variable> — this doesn't

I'm sure it's simple but what's going on with this read -d '' command option?

Best Answer

I guess the question is why read -d '' works though read -d'' doesn't.

The problem doesn't have anything to do with read but is a quoting "problem". A "" / '' which is part of a string (word) simply is not recognized at all. Let the shell show you what is sees / executes:

start cmd:> set -x

start cmd:> echo read -d " " foo
+ echo read -d ' ' foo

start cmd:> echo read -d" " foo
+ echo read '-d ' foo

start cmd:> echo read -d "" foo
+ echo read -d '' foo

start cmd:> echo read -d"" foo
+ echo read -d foo

Related Solutions

Bash – How to use bash script to read binary file content

If you want to stick with shell utilities, you can use head to extract a number of bytes, and od to convert a byte into a number.

export LC_ALL=C    # make sure we aren't in a multibyte locale
n=$(head -c 1 | od -An -t u1)
string=$(head -c $n)

However, this does not work for binary data. There are two problems:

Command substitution $(…) strips final newlines in the command output. There's a fairly easy workaround: make sure the output ends in a character other than a newline, then strip that one character.
```
string=$(head -c $n; echo .); string=${string%.}
```
Bash, like most shells, is bad at dealing with null bytes. As of bash 4.1, null bytes are simply dropped from the result of the command substitution. Dash 0.5.5 and pdksh 5.2 have the same behavior, and ATT ksh stops reading at the first null byte. In general, shells and their utilities aren't geared towards dealing with binary files. (Zsh is the exception, it's designed to support null bytes.)

If you have binary data, you'll want to switch to a language like Perl or Python.

<input_file perl -e '
  read STDIN, $c, 1 or die $!;    # read length byte
  $n = read STDIN, $s, ord($c);   # read data
  die $! if !defined $n;
  die "Input file too short" if ($n != ord($c));
  # Process $s here
'

<input_file python -c '
  import sys
  n = ord(sys.stdin.read(1))      # read length byte
  s = sys.stdin.read(n)           # read data
  if len(s) < n: raise ValueError("input file too short")
  # Process s here
'

Bash – Read non-bash variables from a file into bash script

While you can transform this file to be a shell snippet, it's tricky. You need to make sure that all shell special characters are properly quoted.

The easiest way to do that is to put single quotes around the value and replace single quotes inside the value by '\''. You can then put the result into a temporary file and source that file.

script=$(mktemp)
sed <"config-file" >"$script" \
  -e '/^[A-Z_a-z][A-Z_a-z]*=/ !d' \
  -e s/\'/\'\\\\\'\'/g \
  -e s/=/=\'/ -e s/\$/\'/

I recommend doing the parsing directly in the shell instead. The complexity of the code is about the same, but there are two major benefits: you avoid the need for a temporary file, and the risk that you accidentally got the quoting wrong and end up executing a part of a line as a shell snippet (something like dangerous='$(run me)'). You also get a better chance at validating potential errors.

while IFS= read -r line; do
  line=${line%%#*}  # strip comment (if any)
  case $line in
    *=*)
      var=${line%%=*}
      case $var in
        *[!A-Z_a-z]*)
          echo "Warning: invalid variable name $var ignored" >&2
          continue;;
      esac
      if eval '[ -n "${'$var'+1}" ]'; then
        echo "Warning: variable $var already set, redefinition ignored" >&2
        continue
      fi
      line=${line#*=}
      eval $var='"$line"'
  esac
done <"config-file"

Best Answer

Related Solutions

Bash – How to use bash script to read binary file content

Bash – Read non-bash variables from a file into bash script

Related Question