Bash – Why do piping and redirection work differently with here documents

1. Yes. More precisely, in the shell, a pipe connects the standard output of the command on the left to the standard input of the command on the right. Even more precisely, what foo | bar means to the shell is:

   1. Create a pipe.
   2. Fork a child process, and connect the write end of the pipe to its standard output. Then execute foo in this child process.
   3. Fork a child process, and connect the read end of the pipe to its standard input. Then execute bar in this child process.
   4. Wait for both child processes to exit.

2. Yes. Ordinary redirections indicate a file by name: >foo. Another form of redirection indicates a file by descriptor. For example, >&2 means “redirect standard output to whatever file descriptor 2 is currently connected to”.

   On that topic, note that redirections are processed from left to right. For example, to redirect both standard output and standard error to the same file, use foo >filename 2>&1. The command foo 2>&1 >filename typed in a terminal would first connect standard error to the terminal (which both standard output and standard error are still connected to at that point, so it wouldn't make any difference), then connect standard output to the file.

3. Yes, but. The pipe construct in the shell creates an anonymous pipe; there are also named pipes.

   mkfifo f
   cat f
   # (in another terminal)
   echo hello >f
   

   Named pipes are a lot rarer than pipelines. They're used when it's necessary to connect two processes that are started independently; this doesn't happen very often, but it's nice to have it when necessary.

   Note that the replacement you gave: foo >tempfile; bar <tempfile is different: first foo writes all of its output to the temporary file, then bar starts running. With a pipeline, the commands run in parallel.

4. Yes, but it's not very useful.

   cat input_file | some_command   # a uselessly complicated way of writing some_command <input_file
   echo hello | tee output_file    # a uselessly complicated way of writing echo hello >output_file
   

There are multiple ways to do this. The simplest is probably this:

cat <<EOF | sh
touch somefile
echo foo > somefile
EOF

Another, which is nicer syntax in my opinion:

(
cat <<EOF
touch somefile
echo foo > somefile
EOF
) | sh

This works as well, but without the subshell:

{
cat <<EOF
touch somefile
echo foo > somefile
EOF
} | sh

More variations:

cat <<EOF |
touch somefile
echo foo > somefile
EOF
  sh

Or:

{ cat | sh; } << EOF
touch somefile
echo foo > somefile
EOF

By the way, I expect the use of cat in your question is a placeholder for something else. If not, take it out, like this:

sh <<EOF
touch somefile
echo foo > somefile
EOF

Which could be simplified to this:

sh -c 'touch somefile; echo foo > somefile'

or:

sh -c 'touch somefile
echo foo > somefile'

Redirecting output instead of piping

sh >out <<EOF
touch somefile
echo foo > somefile
EOF

Using cat to get the equivalent of echo test > out:

cat >out <<EOF
test
EOF

Multiple Here Documents

( cat; echo ---; cat <&3 ) <<EOF 3<<EOF2
hi
EOF
there
EOF2

This produces the output:

hi
---
there

Here's what's going on:

• The shell sees the ( ... ) and runs the enclosed commands in a subshell.
• The cat and echo are simple enough. The cat <&3 says to run cat with file descriptor (fd) 0 (stdin) redirected from fd 3; in other words, cat out the input from fd 3.
• Before the (...) is started, the shell sees the two here document redirects and substitutes fd 0 (<<EOF) and fd 3 (3<<EOF2) with the read-side of pipes
• Once the initial command is started, the shell reads its stdin until EOF is reached and sends that to the write-side of the first pipe
• Next, it does the same with EOF2 and the write-side of the second pipe