Bash – How track progress of a command in a pipe if only the size of its input is known in advance

bashpipepvshell

I would like to track progress of a slow operation using pv. The size of the input of this operation is known in advance, but the size of its output is not. This forced me to put pv to the left of the operation in the pipe.

The problem is that the long-running command immediately consumes its whole input because of buffering. This is somewhat similar to the Turn off buffering in pipe question, but in my case it is the consuming operation that is slow, not the producing one and none of the answers to the other question seem to work in this case.

Here is a simple example demonstrating the problem:

seq 20 | pv -l -s 20 | while read line; do sleep 1; done
  20 0:00:00 [13.8k/s] [=====================================>] 100%

Instead of getting updated every second, the progress bar immediately jumps to 100% and stays there for the entire 20 seconds it takes to process the input. pv could only measure the progress if the lines were processed one by one, but the entire input of the last command seems to be read into a buffer.

A somewhat longer example that also demonstrates the unknown number of output lines:

#! /bin/bash
limit=10
seq 20 | \
  pv -l -s 20 | \
  while read num
do
  sleep 1
  if [ $num -gt $limit ]
  then
    echo $num
  fi
done

Any suggestions for a workaround? Thanks!

Best Answer

In your setup the data has passed pv while it is still processed on the right side. You could try to move pv to the rightmost side like this:

seq 20 | while read line; do sleep 1; echo ${line}; done | pv -l -s 20 > /dev/null

Update: Regarding your update, maybe the easiest solution is to use a named pipe and a subshell to monitor the progress:

#! /bin/bash
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
(rm /tmp/progress.pipe; mkfifo /tmp/progress.pipe; tail -f /tmp/progress.pipe | pv -l -s 20 > /dev/null)&
limit=10
seq 20 | \
  while read num
do
  sleep 1
  if [ $num -gt $limit ]
  then
    echo $num
  fi
  echo $num > /tmp/progress.pipe
done

Related Solutions

Bash – How to capture ordered STDOUT/STDERR and add timestamp/prefixes

You might use coprocesses. Simple wrapper that feeds both outputs of a given command to two sed instances (one for stderr the other for stdout), which do the tagging.

#!/bin/bash
exec 3>&1
coproc SEDo ( sed "s/^/STDOUT: /" >&3 )
exec 4>&2-
coproc SEDe ( sed "s/^/STDERR: /" >&4 )
eval $@ 2>&${SEDe[1]} 1>&${SEDo[1]}
eval exec "${SEDo[1]}>&-"
eval exec "${SEDe[1]}>&-"

Note several things:

It is a magic incantation for many people (including me) - for a reason (see the linked answer below).
There is no guarantee it won't occasionally swap couple of lines - it all depends on scheduling of the coprocesses. Actually, it is almost guaranteed that at some point in time it will. That said, if keeping the order strictly the same, you have to process the data from both stderr and stdin in the same process, otherwise the kernel scheduler can (and will) make a mess of it.

If I understand the problem correctly, it means that you would need to instruct the shell to redirect both streams to one process (which can be done AFAIK). The trouble starts when that process starts deciding what to act upon first - it would have to poll both data sources and at some point get into state where it would be processing one stream and data arrive to both streams before it finishes. And that is exactly where it breaks down. It also means, that wrapping the output syscalls like stderred is probably the only way to achieve your desired outcome (and even then you might have a problem once something becomes multithreaded on a multiprocessor system).

As far as coprocesses be sure to read Stéphane's excellent answer in How do you use the command coproc in Bash? for in depth insight.

Bash – How to Time a Pipe Command

It is working.

The different parts of a pipeline are executed concurrently. The only thing that synchronises/serialises the processes in the pipeline is IO, i.e. one process writing to the next process in the pipeline and the next process reading what the first one writes. Apart from that, they are executing independently of each other.

Since there is no reading or writing happening between the processes in your pipeline, the time take to execute the pipeline is that of the longest sleep call.

You might as well have written

time ( foo.sh & bar.sh &; wait )

Terdon posted a couple of slightly modified example scripts in the chat:

#!/bin/sh
# This is "foo.sh"
echo 1; sleep 1
echo 2; sleep 1
echo 3; sleep 1
echo 4

and

#!/bin/sh
# This is "bar.sh"
sleep 2
while read line; do
  echo "LL $line"
done
sleep 1

The query was "why does time ( sh foo.sh | sh bar.sh ) return 4 seconds rather than 3+3 = 6 seconds?"

To see what's happening, including the approximate time each command is executed, one may do this (the output contains my annotations):

$ time ( env PS4='$SECONDS foo: ' sh -x foo.sh | PS4='$SECONDS bar: ' sh -x bar.sh )
0 bar: sleep 2
0 foo: echo 1     ; The output is buffered
0 foo: sleep 1
1 foo: echo 2     ; The output is buffered
1 foo: sleep 1
2 bar: read line  ; "bar" wakes up and reads the two first echoes
2 bar: echo LL 1
LL 1
2 bar: read line
2 bar: echo LL 2
LL 2
2 bar: read line  ; "bar" waits for more
2 foo: echo 3     ; "foo" wakes up from its second sleep
2 bar: echo LL 3
LL 3
2 bar: read line
2 foo: sleep 1
3 foo: echo 4     ; "foo" does the last echo and exits
3 bar: echo LL 4
LL 4
3 bar: read line  ; "bar" fails to read more
3 bar: sleep 1    ; ... and goes to sleep for one second

real    0m4.14s
user    0m0.00s
sys     0m0.10s

So, to conclude, the pipeline takes 4 seconds, not 6, due to the buffering of the output of the first two calls to echo in foo.sh.

Best Answer

Related Solutions

Bash – How to capture ordered STDOUT/STDERR and add timestamp/prefixes

Bash – How to Time a Pipe Command

Related Question