Shell – Command with pipeline and redirection

io-redirectionpipeshell

What is the order of commands executed which have both pipeline and output redirection in it?

Say we do the following:

Charles@myzone:/tmp$ mkdir /tmp/testdir      
Charles@myzone:/tmp$ cd /tmp/testdir   
Charles@myzone:/tmp/testdir$ touch file1 file2  
Charles@myzone:/tmp/testdir$ ls | wc -l
2
Charles@myzone:/tmp/testdir$ ls | wc -l > ls_result
Charles@myzone:/tmp/testdir$ cat ls_result
3

I know that if you do ls > result then result will contain the name of itself because the shell will do something like

1) create/open file named result
2) set the fd of result to be stdout
3) exec ls

I was expecting ls_result to have value 2, but it's 3.

Question

How is the command ls | wc -w > ls_result above executed ?

Is it equivalent to (ls | wc -w ) > ls_result?

Some links with concerning information ? (I've looked up the bash manual)

Best Answer

utility1 | utility2 >output

is not equivalent to

( utility1 | utility2 ) >output

but to

utility1 | { utility2 >output; }

The two utilities are started pretty much the same time, which means you would expect your command to sometimes return 3 and sometimes 2.

Example:

$ { [ -f test ] && echo exists >&2; } | { echo >test; }; rm test
$ { [ -f test ] && echo exists >&2; } | { echo >test; }; rm test
$ { [ -f test ] && echo exists >&2; } | { echo >test; }; rm test
exists
$ { [ -f test ] && echo exists >&2; } | { echo >test; }; rm test
exists
$ { [ -f test ] && echo exists >&2; } | { echo >test; }; rm test
$ { [ -f test ] && echo exists >&2; } | { echo >test; }; rm test

The above shows that the file created by the right hand side of the pipeline is sometimes detected by the left hand side of the pipeline.

Related Solutions

Shell IO Redirection – IO Redirection and the Head Command

When the shell gets a command line like: command > file.out the shell itself opens (and maybe creates) the file named file.out. The shell sets file descriptor 0 to the file file descriptor it got from the open. That's how I/O redirection works: every process knows about file descriptors 0, 1 and 2.

The hard part about this is how to open file.out. Most of the time, you want file.out opened for write at offset 0 (i.e. truncated) and this is what the shell did for you. It truncated .hgignore, opened it for write, dup'ed the filedescriptor to 0, then exec'ed head. Instant file clobbering.

In bash shell, you do a set noclobber to change this behavior.

Shell – redirection and pipeline

Yes. More precisely, in the shell, a pipe connects the standard output of the command on the left to the standard input of the command on the right. Even more precisely, what foo | bar means to the shell is:
1. Create a pipe.
2. Fork a child process, and connect the write end of the pipe to its standard output. Then execute foo in this child process.
3. Fork a child process, and connect the read end of the pipe to its standard input. Then execute bar in this child process.
4. Wait for both child processes to exit.
Yes. Ordinary redirections indicate a file by name: >foo. Another form of redirection indicates a file by descriptor. For example, >&2 means “redirect standard output to whatever file descriptor 2 is currently connected to”.

On that topic, note that redirections are processed from left to right. For example, to redirect both standard output and standard error to the same file, use foo >filename 2>&1. The command foo 2>&1 >filename typed in a terminal would first connect standard error to the terminal (which both standard output and standard error are still connected to at that point, so it wouldn't make any difference), then connect standard output to the file.
Yes, but. The pipe construct in the shell creates an anonymous pipe; there are also named pipes.
```
mkfifo f
cat f
# (in another terminal)
echo hello >f
```
Named pipes are a lot rarer than pipelines. They're used when it's necessary to connect two processes that are started independently; this doesn't happen very often, but it's nice to have it when necessary.

Note that the replacement you gave: foo >tempfile; bar <tempfile is different: first foo writes all of its output to the temporary file, then bar starts running. With a pipeline, the commands run in parallel.

Yes, but it's not very useful.

cat input_file | some_command   # a uselessly complicated way of writing some_command <input_file
echo hello | tee output_file    # a uselessly complicated way of writing echo hello >output_file

Best Answer

Related Solutions

Shell IO Redirection – IO Redirection and the Head Command

Shell – redirection and pipeline

Related Question