Ubuntu – Piping with Moreutils ts

command linepipesed

I have an incoming stream at a serial port, with new lines appearing about once per second

wren@Raven:~$ cat /dev/ttyUSB0

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

A_Sensor1,B_22.00,C_50.00

I want to strip blank lines and timestamp the remainder.

sed will cull blank lines and add a timestamp, but I can't make the timestamp update, it just reports the time it was invoked:

wren@Raven:~$ cat /dev/ttyUSB0 | sed -e '/^$/d' -e "s/$/`date +\,%F\,%T`/"
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
A_Sensor1,B_22.00,C_50.00,2014-05-14,09:44:42
^C

I've found ts, part of Moreutils, and can pipe into it to get an updating timestamp.

wren@Raven:~$ cat /dev/ttyUSB0 |  ts
May 14 09:49:26 A_Sensor1,B_22.00,C_50.00
May 14 09:49:26
May 14 09:49:27 A_Sensor1,B_22.00,C_50.00
^C

However, I can't properly combine ts with sed.

This, which looks like it should do what I want, produces no output at all

wren@Raven:~$ cat /dev/ttyUSB0 | sed -e '/^$/d' | ts
^C
wren@Raven:~$

However reversing the order of the pipes does produce an output, but of course doesn't strip lines which are no longer blank. Other substitutions work fine, so I know the pipe to sed is working.

wren@Raven:~$ cat /dev/ttyUSB0 |  ts | sed -e '/^$/d'
May 14 10:07:25 A_Sensor1,B_22.00,C_50.00
May 14 10:07:25
May 14 10:07:26 A_Sensor1,B_22.00,C_50.00
May 14 10:07:26
^C

So I'm a bit baffled. I can presumably make sed remove the unwanted lines, but timestamping them prior to removal must be the wrong approach.

I would appreciate an explanation and some help.

Best Answer

To answer the question directly, sed is buffering and that's the sole problem.
You can fix this by telling it not to buffer with its -u/--unbuffered flag:

sed -u '/^$/d' /dev/ttyUSB0 | ts

With a test harness (but you'll need to run it for proof):

$ (echo -e 'banana\n\n'; sleep 2; echo 'cheese') | sed -u '/^$/d' | ts
May 14 11:26:05 banana
May 14 11:26:07 cheese

You can run into similar predicaments with other stream editors. They seemingly all want to buffer a little bit. They all have workarounds though. Here are a bunch of commands I've tested:

... | mawk -W interactive '/./' | ts
... | gawk '/./ { print $0; fflush(); }' | ts
... | grep --line-buffered '.' | ts
... | perl -n -e 'print if /./' | ts

Another idea is to just let gawk handle it. It can filter for non-empty lines and do the date-printing for you (thanks to SO's Kieron):

awk '/./ { print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }' /dev/ttyUSB0

That flushes straight after lines come in. gawk is especially helpful here if you want to do other things... If you want to check that the fourth column of output (pre-ts) matches a regex, you can (eg $4~/\d{4}/). Awk (and its variants) are very flexible for stream processing.

Another test harness:

$ gawk '/./ { print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }' <(
      echo -e 'banana\n\n';
      sleep 2;
      echo 'cheese'
  )
2014-05-14 11:13:59 banana
2014-05-14 11:14:01 cheese
Related Question