Bash Heredoc – Understanding Strange Behavior of Nested Heredoc

The first line of the script is called the #! line (sometimes called the "shebang" line).

It is only used when a script is executed directly, as in

./test.sh

Explanation

When you run an executable, the operating system looks at the first two bytes to tell what type of program it is.

When you run

./test.sh

the operating system sees that the first two bytes of ./test.sh are #!, so it knows the program is a script.

It reads the rest of the line to see which shell¹ it should use to run the script. In this case, the rest of the line is /usr/bin/sh, so it runs /usr/bin/sh test.sh².

When you run

bash test.sh

it finds bash in /usr/bin/bash³. It sees that the first two bytes of /usr/bin/bash are not #!, so it runs /usr/bin/bash test.sh without even looking at the first line of test.sh.

In both cases, sh and bash actually ignore the shebang line because any line beginning with # is a comment. The # only has this magic effect because of the operating system, not the shell.

More info:

• #!
• The Whole Story on #! /usr/bin/ksh
• Solaris execve man page
• Linux execve man page

Footnotes:

1. In fact, it can be any program, it doesn't have to be a shell.
2. Actually more like execl("./test.sh", "/usr/bin/sh", "./test.sh", NULL).
3. /bin/sh on most Linux systems, but the question was about Solaris.

What needs to be explained is that the command appeared to work, not its exit code

'\n' is two characters: a backslash \ and a letter n. What you thought you needed was $'\n', which is a linefeed (but that wouldn't be right either, see below).

The -d option does this:

  -d delim  continue until the first character of DELIM is read, rather
            than newline

So without that option, read would read up to a newline, split the line into words using the characters in $IFS as separators, and put the words into the array. If you specified -d $'\n', setting the line delimiter to a newline, it would do exactly the same thing. Setting -d '\n' means that it will read up to the first backslash (but, once again, see below), which is the first character in delim. Since there is no backslash in your file, the read will terminate at the end of file, and:

Exit Status:
The return code is zero, unless end-of-file is encountered, read times out,
or an invalid file descriptor is supplied as the argument to -u.

So that's why the exit code is 1.

From the fact that you believe that the command worked, we can conclude that there are no spaces in the file, so that read, after reading the entire file in the futile hope of finding a backslash, will split it by whitespace (the default value of $IFS), including newlines. So each line (or each word, if a line contains more than one word) gets stashed into the array.

The mysterious case of the purloined backslash

Now, how did I know the file didn't contain any backslashes? Because you didn't supply the -r flag to read:

  -r                do not allow backslashes to escape any characters

So if you had any backslashes in the file, they would have been stripped, unless you had two of them in a row. And, of course, there is the evidence that read had an exit code of 1, which demonstrates that it didn't find a backslash, so there weren't two of them in a row either.

Takeaways

Bash wouldn't be bash if there weren't gotchas hiding behind just about every command, and read is no exception. Here are a couple:

1. Unless you specify -r, read will interpret backslash escape sequences. Unless that's actually what you want (which it occasionally is, but only occasionally), you should remember to specify -r to avoid having characters disappear in the rare case that there are backslashes in the input.

2. The fact that read returns an exit code of 1 does not mean that it failed. It may well have succeeded, except for finding the line terminator. So be careful with a loop like this: while read -r LINE; do something with LINE; done because it will fail to do something with the last line in the rare case that the last line doesn't have a newline at the end.

3. read -r LINE preserves backslashes, but it doesn't preserve leading or trailing whitespace.