The issue of jq
needing an explicit filter when the output is redirected is discussed all over the web. But I'm unable to redirect output if jq
is part of a pipe chain, even when an explicit filter is in use.
Consider:
touch in.txt
tail -f in.txt | jq '.f1'
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt
As expected, the output in the original terminal from the jq
command is:
1
3
But if I add any sort of redirection or piping to the end of the jq
command, the output goes silent:
rm in.txt
touch in.txt
tail -f in.txt | jq '.f1' | tee out.txt
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt
No output appears in the first terminal and out.txt is empty.
I've tried hundreds of variations but it's an elusive issue. The only workaround I've found, as discovered through mosquitto_sub
and The Things Network (which was where I also discovered the issue), is to wrap the tail and jq functions in a shell script:
#!/bin/bash
tail -f $1 | while IFS='' read line; do
echo $line | jq '.f1'
done
Then:
./tail_and_jq.sh | tee out.txt
# in a different terminal:
echo '{"f1":1,"f2":2}' >> in.txt
echo '{"f1":3,"f2":2}' >> in.txt
And sure enough, the output appears:
1
3
This is with the latest jq
installed via Homebrew:
$ echo $SHELL
/bin/bash
$ jq --version
jq-1.5
$ brew install jq
Warning: jq 1.5_3 is already installed and up-to-date
Is this a (largely undocumented) bug in jq
or with my understanding of pipe chains?
Best Answer
The output from
jq
is buffered when its standard output is piped.To request that
jq
flushes its output buffer after every object, use its--unbuffered
option, e.g.From the
jq
manual: