How to grep in real time an output containing a progress bar

grepprogress-informationstdout

I am using a tool (openocd) which is printing a lot of garbage, then a basic progress bar slowly printing simple dots, then again some garbage.

I would like to filter this output with grep so only the line with progress bar is shown, and in real time (i.e. each dot outputted by openocd is immediately printed in the terminal):

openocd <args> |& grep '^\.'

The problem is that grep is line buffered (at best) so the progress bar will not be shown until it is finished.

How can I do with grep, or is there any standard alternative to achieve this?
If there is a way through openocd configuration, this would be useful though I would prefer a more general solution.

Best Answer

This is a kind of hacky/unusual answer, the fact being that this is most likely possible in a not very clean way.

grep itself seems to only print output when it encounters a newline character, your progress bar likely does not introduce a newline character when it updates, hence your issue.

strace is a tool used to view the system calls that a command is calling, this includes things like reading and writing things to memory/storage, as well as things like opening/closing file descriptors.

With strace you can view what a process is accessing, in the case of your pipe stout is being passed to grep, so with strace you can view the text that's being fed to grep. strace will regularly be sent the output coming from the piped command, and you can sniff that output and display it. I was testing with rsync --progress, which seems to run into a similar scenario. I used grep on the ##% because that's what rsync uses to show progress.

rsync --progress file1 file2 | strace -e trace=read grep "[0-9]*%"

If you run this command you'll find that strace doesn't have nice output, but that when I used it strace caught a couple of reads from the rsync that grep would not normally write, showing reads for 0%, 21%, 45%, 68%, 91% and 100%, seemed to update about every second (probably based on how often rsync updates the progress).

So with that you can grep the strace output, which is not very nice, by calling the same grep again.

rsync --progress file1 file2 | strace -e trace=read grep "[0-9]*%" 2>&1 > /dev/null | grep -o "[0-9]*%"

The 2>&1 is important because strace prints on stderr. The > /dev/null redirects stdout to /dev/null to prevent the output of the first grep being reported. The end result of this was the following output:

0%
21%
45%
68%
91%
100%

You'll have to swap out the grep, but it seems like it'll do the trick. It's not pretty, but it functions, and works around the restrictions of grep. Seems like a grep -f that works like tail -f would be handy (I know grep -f is already in use).

The first grep is mostly just to filter down the text that strace will be reading, since only the matching lines will be listed in straces read calls, but you also need something for the text to be moving through so strace can watch it.

Related Question