Shell – Subshell and process substitution

process-substitutionsubshell

Apologies if this is a basic question – I'm stuck trying to solve a larger problem, and it's come down to how a shell script is invoked – directly (shellScript.sh) or using sh shellScript.sh.

Here's a model for the problem:

When I execute on bash:

cat <(echo 'Hello')

I see the output

Hello

But when I use:

sh -c "cat <(echo 'Hello')"

I see errors:

sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `cat <(echo 'Hello')'

I've tried escaping the <, ( and ) in various combinations, but I don't see the output anywhere. What am I missing here?

My actual problem is that I'm passing a <() as an input argument to a python script within a shell script, and while it works fine when I invoke the shell script using just the name, if I use sh to invoke it, I get errors similar to what I've shown above.

Thank you!

Best Answer

Process substitution is a feature that originated in the Korn shell in the 80s (in ksh86). At the time, it was only available on systems that had support for /dev/fd/<n> files.

Later, the feature was added to zsh (from the start: 1990) and bash (in 1993). zsh was using temporary named pipes to implement it, while bash was using /dev/fd/<n> where available and named pipes otherwise. zsh switched to using /dev/fd/<n> where available in 2.6-beta17 in 1996.

Support for process substitution via named pipes on systems without /dev/fd was only added to ksh in ksh93u+ in 2012. The public domain clone of ksh doesn't support it.

To my knowledge, no other Bourne-like shell supports it (rc, es, fish, non-Bourne-like shells support it but with a different syntax). yash has a <(...) construct, but that's for process redirection.

While quite useful, the feature was never standardized by POSIX. So, one can't expect to find it in sh, so shouldn't use it in a sh script.

Though the behaviour for <(...) is unspecified in POSIX, (so there would be no harm in retaining it), bash disables the feature when called as sh or when called with POSIXLY_CORRECT=1 in its environment.

So, if you have a script that uses <(...), you should use a shell that supports the feature to interpret it like zsh, bash or AT&T ksh (of course, you need to make sure the rest of the syntax of script is also compatible with that shell).

In any case:

cat <(cmd)

Can be written:

cmd | cat

Or just

cmd

For a command other than cat (that needs to be passed data via a file given as argument), on systems with /dev/fd/x, you can always do:

something | that-cmd /dev/stdin

Or if you need that-cmd's stdin to be preserved:

{ something 3<&- | that-cmd /dev/fd/4 4<&0 <&3 3<&-; } 3<&0

Related Solutions

Shell – Process Substitution and Pipe

A good way to grok the difference between them is to do a little experimenting on the command line. In spite of the visual similarity in use of the < character, it does something very different than a redirect or pipe.

Let's use the date command for testing.

$ date | cat
Thu Jul 21 12:39:18 EEST 2011

This is a pointless example but it shows that cat accepted the output of date on STDIN and spit it back out. The same results can be achieved by process substitution:

$ cat <(date)
Thu Jul 21 12:40:53 EEST 2011

However what just happened behind the scenes was different. Instead of being given a STDIN stream, cat was actually passed the name of a file that it needed to go open and read. You can see this step by using echo instead of cat.

$ echo <(date)
/proc/self/fd/11

When cat received the file name, it read the file's content for us. On the other hand, echo just showed us the file's name that it was passed. This difference becomes more obvious if you add more substitutions:

$ cat <(date) <(date) <(date)
Thu Jul 21 12:44:45 EEST 2011
Thu Jul 21 12:44:45 EEST 2011
Thu Jul 21 12:44:45 EEST 2011

$ echo <(date) <(date) <(date)
/proc/self/fd/11 /proc/self/fd/12 /proc/self/fd/13

It is possible to combine process substitution (which generates a file) and input redirection (which connects a file to STDIN):

$ cat < <(date)
Thu Jul 21 12:46:22 EEST 2011

It looks pretty much the same but this time cat was passed STDIN stream instead of a file name. You can see this by trying it with echo:

$ echo < <(date)
<blank>

Since echo doesn't read STDIN and no argument was passed, we get nothing.

Pipes and input redirects shove content onto the STDIN stream. Process substitution runs the commands, saves their output to a special temporary file and then passes that file name in place of the command. Whatever command you are using treats it as a file name. Note that the file created is not a regular file but a named pipe that gets removed automatically once it is no longer needed.

Bash – Is the command in a process substitution invoked in a subshell

Running the current shell under strace(1) and then executing e.g. <(command) gives:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fa6713d59d0) = 13305

From a purely definitional standpoint, since clone(2) is defined as

create a child process

and a subshell as

Running a shell script launches a new process, a subshell.

one could say that yes - running process substitution is invoked as a subshell.

Best Answer

Related Solutions

Shell – Process Substitution and Pipe

Bash – Is the command in a process substitution invoked in a subshell

Related Question