How Do Pipelines Limit Memory Usage? – Detailed Explanation

There's no way to peek at the content of a pipe using commonly available shell utilities, nor is there a way to read a character from the pipe then put it back. The only way to know that a pipe has data is to read a byte, and then you have to get that byte to its destination.

So do just that: read one byte; if you detect an end of file, then do what you want to do when the input is empty; if you do read a byte then fork what you want to do when the input is not empty, pipe that byte into it, and pipe the rest of the data.

first_byte=$(dd bs=1 count=1 2>/dev/null | od -t o1 -A n | tr -dc 0-9)
if [ -z "$first_byte" ]; then
  # stuff to do if the input is empty
else
  {
    printf "\\$first_byte"
    cat
  } | {
    # stuff to do if the input is not empty
  }      
fi

The ifne utility from Joey Hess's moreutils runs a command if its input is not empty. It usually isn't installed by default, but it should be available or easy to build on most unix variants. If the input is empty, ifne does nothing and returns the status 0, which cannot be distinguished from the command running successfully. If you want to do something if the input is empty, you need to arrange for the command not to return 0, which can be done by having the success case return a distinguishable error status:

ifne sh -c 'do_stuff_with_input && exit 255'
case $? in
  0) echo empty;;
  255) echo success;;
  *) echo failure;;
esac

test -t 0 has nothing to do with this; it tests whether standard input is a terminal. It doesn't say anything one way or the other as to whether any input is available.

Easiest way would be to pipe through some program which sets nonblocking output. Here is simple perl oneliner (which you can save as leakybuffer) which does so:

so your a | b becomes:

a | perl -MFcntl -e \
    'fcntl STDOUT,F_SETFL,O_NONBLOCK; while (<STDIN>) { print }' | b

what is does is read the input and write to output (same as cat(1)) but the output is nonblocking - meaning that if write fails, it will return error and lose data, but the process will continue with next line of input as we conveniently ignore the error. Process is kind-of line-buffered as you wanted, but see caveat below.

you can test with for example:

seq 1 500000 | perl -w -MFcntl -e \
    'fcntl STDOUT,F_SETFL,O_NONBLOCK; while (<STDIN>) { print }' | \
    while read a; do echo $a; done > output

you will get output file with lost lines (exact output depends on the speed of your shell etc.) like this:

12768
12769
12770
12771
12772
12773
127775610
75611
75612
75613

you see where the shell lost lines after 12773, but also an anomaly - the perl didn't have enough buffer for 12774\n but did for 1277 so it wrote just that -- and so next number 75610 does not start at the beginning of the line, making it little ugly.

That could be improved upon by having perl detect when the write did not succeed completely, and then later try to flush remaining of the line while ignoring new lines coming in, but that would complicate perl script much more, so is left as an exercise for the interested reader :)

Update (for binary files): If you are not processing newline terminated lines (like log files or similar), you need to change command slightly, or perl will consume large amounts of memory (depending how often newline characters appear in your input):

perl -w -MFcntl -e 'fcntl STDOUT,F_SETFL,O_NONBLOCK; while (read STDIN, $_, 4096) { print }' 

it will work correctly for binary files too (without consuming extra memory).

Update2 - nicer text file output: Avoiding output buffers (syswrite instead of print):

seq 1 500000 | perl -w -MFcntl -e \
    'fcntl STDOUT,F_SETFL,O_NONBLOCK; while (<STDIN>) { syswrite STDOUT,$_ }' | \
    while read a; do echo $a; done > output

seems to fix problems with "merged lines" for me:

12766
12767
12768
16384
16385
16386

(Note: one can verify on which lines output was cut with: perl -ne '$c++; next if $c==$_; print "$c $_"; $c=$_' output oneliner)