Bash – Process substitution with input redirection

bashfile-descriptorsio-redirectionprocess-substitution

I am trying to understand input redirection in combination with process substituation. I am using bash 3

An example with tr is the following

$ tr "o" "a" <(echo "Foo")
tr: extra operand `/dev/fd/63'
Try `tr --help' for more information.

I think I understand why this does not work. The process substitution <( ) creates a file descriptor, where tr only reads from standard input.

How can I make it work with proper redirection?

I know that I could simply use pipes:

$ echo "Foo" | tr "o" "a"
Faa

However, I am trying to get a better understanding. I tried some thing with the help of man bash, by using <&, but I don't really know what I am doing.

How can I use process substitution properly using tr?

Best Answer

You were really close:

tr "o" "a" < <(echo "Foo")

The substitution <() makes a file descriptor and just pastes the path to the shell. For comprehension just execute:

<(echo blubb)

You will see the error:

-bash: /dev/fd/63: Permission denied

That's why it just pastes /dev/fd/63 into the shell and /dev/fd/63 is not excutable, because it's a simple pipe. In the tr-example above, it's echo "Foo" that writes to the pipe and via input redirection < it's the tr command that reads from the file descriptor.

Related Solutions

Performance: Pipelines vs Process Substitution

Doing same strace, you can see the differences:

With pipe:

$ strace -c ./pipe.sh 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 57.89    0.103005           5     20000           clone
 40.81    0.072616           2     30000     10000 wait4
  0.58    0.001037           0    120008           rt_sigprocmask
  0.40    0.000711           0     10000           pipe

With proc-sub:

$ strace -c ./procsub.sh 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 85.08    0.045502           5     10000           clone
  3.25    0.001736           0     90329       322 read
  2.12    0.001133           0     20009           open
  2.03    0.001086           0     50001           dup2

With above statistics, you can see pipe create more child processes (clone syscall) and spending many times to wait child process (wait4 syscall) to finish for parent process to continue executing.

Process substitution is not. It can read directly from child processes. Process substitution is performed at the same time with parameter and variable expansion, the command in Process Substitution run in background. From bash manpage:

Process Substitution
       Process  substitution  is supported on systems that support named pipes
       (FIFOs) or the /dev/fd method of naming open files.  It takes the  form
       of  <(list) or >(list).  The process list is run with its input or out‐
       put connected to a FIFO or some file in /dev/fd.  The name of this file
       is  passed  as  an argument to the current command as the result of the
       expansion.  If the >(list) form is used, writing to the file will  pro‐
       vide  input  for list.  If the <(list) form is used, the file passed as
       an argument should be read to obtain the output of list.

       When available, process substitution is performed  simultaneously  with
       parameter  and variable expansion, command substitution, and arithmetic
       expansion.

Update

Doing strace with statistics from child processes:

With pipe:

$ strace -fqc ./pipe.sh 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 70.76    0.215739           7     30000     10000 wait4
 28.04    0.085490           4     20000           clone
  0.78    0.002374           0    220008           rt_sigprocmask
  0.17    0.000516           0    110009     20000 close
  0.15    0.000456           0     10000           pipe

With proc-sub:

$ strace -fqc ./procsub.sh 
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 52.38    0.033977           3     10000           clone
 32.24    0.020913           0     96070      6063 read
  5.24    0.003398           0     20009           open
  2.34    0.001521           0    110003     10001 fcntl
  1.87    0.001210           0    100009           close

Bash – What are the difference between those four commands (fifo, process substitution, redirection…)

The key here is that opening a FIFO is a blocking operation. The open only returns once both ends are connected, i.e. once the fifo is open for both reading and writing.

man fifo(7)

Normally, opening the FIFO blocks until the other end is opened also.

In the 1st case, the shell forks to execute the pipeline, so opening the fifo for reading (cat fifo) and opening the fifo for writing (> fifo) happen in separate processes, so happen independently.

In the 2nd case, the open for reading and open for writing (3<>fifo) happen in a single step.

In the 3rd case, <(cat fifo) expands to a filename, e.g. /dev/fd/42. So it's like you're running nc -l localhost 8888 /dev/fd/42 > fifo. You need an extra < for it to be equivalent, e.g. nc -l localhost 8888 < <(cat fifo) > fifo.

In the 4th case, the shell is trying to open the fifo for reading (< fifo) and open it for writing (> fifo) as part of the same process. The shell does them one at a time, left to right. So it tries to open fifo for reading, and blocks forever, waiting for something to open fifo for writing. I think you'll find that in this case, nc never even got started, and the port was never opened for listening.

Best Answer

Related Solutions

Performance: Pipelines vs Process Substitution

Bash – What are the difference between those four commands (fifo, process substitution, redirection…)

Related Question