Determine if producer is outpacing consumer in pipeline

pipestdinstdiostdout

If I have:

node foo.js | node bar.js

is there a way to determine if there is a queue in between them that's building up?
in other words if the producer is outpacing the consumer w.r.t. stdio?

Best Answer

You can use ioctl(,FIONREAD,) on pipes in Linux to get the current number of bytes available to read. This might be found in man 7 pipe (FIONREAD is also in man tty_ioctl). You can either integrate this into your application or run something like this bit of perl in parallel with it, using the pipe as stdin:

#!/usr/bin/perl
# https://unix.stackexchange.com/a/440361/119298
use strict;
require 'sys/ioctl.ph';
my $v = pack("l",0);
$| = 1;
# while(1)
{
  die "ioctl: $!" if ioctl(STDIN, &FIONREAD, $v)!=0;
  my @r = unpack("l",$v);
  printf "%d ",$r[0];
#  sleep 1;
}

So the usage would be

node foo.js | (myperl & node bar.js)

Try it out with something like

while sleep 1;do date;done |
while sleep 1.1;do myperl 2>/dev/null; dd status=none count=1 bs=20;done

I get warnings from perl about _FORTIFY_SOURCE, hence the stderr redirection. The dd reads less than the length of date so you can see the result gradually increasing.

For completeness, you can get the capacity of a pipe with fcntl(,F_GETPIPE_SZ) and set it with fcntl(,F_SETPIPE_SZ,newvalue) up to /proc/sys/fs/pipe-max-size (unless root or have CAP_SYS_RESOURCE). See man fcntl.

Related Question