Shell – avoid multiple pipes broken due to missing command

pipeportabilityshell-script

In shell scripts I often use multiple pipes like this:

cmd1 | cmd2 | cmd3

Now if cmd2 is missing the pipe will result "broken" and cmd3 won't receive any input.

As workaround I could test explicitly if cmd2 exist, but this requires some code replication I'd like to avoid.

Instead I'd like a more concise test like this:

cmd1 | cmd2; [ $? -eq 127 ] && cat | cmd3

Best Answer

Based on your example it looks like you're looking for a way to group arguments to operations with lower precedence than |, (i.e. && and ||).

The { and } metacharacters can be used for grouping in this way.

Your example above could be written like this (using the exit status of which to determine whether cmd2 exists or not).

cmd1 | { which cmd2 >/dev/null && cmd2 || cat; } | cmd3

For example

printf "hi\n" | { : && rev || cat; } | cat

Will print ih

Your suggestion of using a subshell will also work, but uses another process unnecessarily

printf "hi\n" | ( : && rev || cat ) | cat

I would say it is probably less tricky and better for maintainability if you use an explicit if outside the pipe.

if [ which cmd2 >/dev/null ]
then
    cmd1 | cmd3
else
    cmd1 | cmd2 | cmd3
fi

Related Solutions

Non-blocking buffered named pipe

I think what you are looking for is GNU screen. It maintains a buffer to hold the last screen full or two of output from one or more programs and lets you disconnect and come back later.

Bash vs ksh pipes

Your question is a bit rambling. I'll answer what seems to be the central part, on the difference between ksh and bash that you observe.

You have encountered what is probably the #1 incompatibility between ksh and bash when it comes to scripts. ATT ksh (both ksh88 and ksh93) and zsh execute the last (rightmost) command in a pipeline in the parent shell, whereas other shells (Bourne, ash, bash, pdksh, mksh) execute all the commands including the last one in a subshell.

Here is a simple test program:

msg="a subshell"
true | msg="the parent shell"
echo "This shell runs the last command of a pipeline in $msg"

In ATT ksh and zsh, the second assignment to msg is executed in the parent shell so the effect is visible after the pipeline. In other shells, this assignment is executed in a subshell so the first assignment remains in place in the parent.

A workaround is to execute the rest of the script in the pipeline. This is a common idiom for reading data and doing some processing afterward:

output_some_stuff | {
  var=
  while IFS= read -r line; do
    var=$(process "$line")
  done
  use "$var"
}

You appear to have run into a ksh bug. I recommend upgrading to a non-buggy version. If that isn't possible, try Stephane Chazelas's workaround. While you can try running your scripts in bash, it is not (and does not pretend to be) a drop-in replacement for ksh; there are plenty of ksh features that bash doesn't have (and vice versa). Bash and ksh are only compatible in their POSIX core and some other central features (in particular arrays, [[ … ]], and local variables in functions declared by typeset).

You could also try zsh, which when invoked as ksh behaves in a way that's a bit closer to ksh than bash is. You may nonetheless run into incompatibilities.

Best Answer

Related Solutions

Non-blocking buffered named pipe

Bash vs ksh pipes

Related Question