jq – Fix No Output in Pipe Chain

io-redirectionjqpipe

The issue of jq needing an explicit filter when the output is redirected is discussed all over the web. But I'm unable to redirect output if jq is part of a pipe chain, even when an explicit filter is in use.

Consider:

touch in.txt
tail -f in.txt | jq '.f1'
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt

As expected, the output in the original terminal from the jq command is:

1
3

But if I add any sort of redirection or piping to the end of the jq command, the output goes silent:

rm in.txt
touch in.txt
tail -f in.txt | jq '.f1' | tee out.txt
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt

No output appears in the first terminal and out.txt is empty.

I've tried hundreds of variations but it's an elusive issue. The only workaround I've found, as discovered through mosquitto_sub and The Things Network (which was where I also discovered the issue), is to wrap the tail and jq functions in a shell script:

#!/bin/bash
tail -f $1 | while IFS='' read line; do
echo $line | jq '.f1'
done

Then:

./tail_and_jq.sh | tee out.txt
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt

And sure enough, the output appears:

1
3

This is with the latest jq installed via Homebrew:

$ echo $SHELL
/bin/bash
$ jq --version
jq-1.5
$ brew install jq
Warning: jq 1.5_3 is already installed and up-to-date

Is this a (largely undocumented) bug in jq or with my understanding of pipe chains?

Best Answer

The output from jq is buffered when its standard output is piped.

To request that jq flushes its output buffer after every object, use its --unbuffered option, e.g.

tail -f in.txt | jq --unbuffered '.f1' | tee out.txt

From the jq manual:

--unbuffered

Flush the output after each JSON object is printed (useful if you're piping a slow data source into jq and piping jq's output elsewhere).