Bash Grep Performance – Why ‘tac file | grep foo’ is Faster than ‘grep foo < <(tac file)'

bashefficiencygrepperformance

This question was motivated by "Reverse grepping", about grepping a huge file from bottom up.

@chaos said:

tac file | grep whatever

Or a bit more effective:

grep whatever < <(tac file)

@vinc17 said:

The < <(tac filename) should be as fast as a pipe

There are also many interesting comments from other users.

My questions:

  • What is the difference between | and < <()?
  • Why is one faster than other?
  • And which is really faster?
  • Why did no one suggest xargs?

Best Answer

The construction <(tac file) causes to shell to:

  • Create a pipe with a name
    • On systems such as Linux and SysV which have /dev/fd, a regular pipe is used, and /dev/fd/<the-file-descriptor-of-the-pipe> is used as the name.
    • On other systems, a named pipe is used, which requires creating an actual file entry on disk.
  • Launch the command tac file and connect it to one end of the pipe.
  • Replace the whole construction on the command line with the name of the pipe.

After the replacement, the command line becomes:

grep whatever < /tmp/whatever-name-the-shell-used-for-the-named-pipe

And then grep is executed, and it reads its standard input (which is the pipe), reads it, and searches for its first argument in that.

So the end result is the same as with...

tac file | grep whatever

...in that the same two programs are launched and a pipe is still used to connect them. But the <( ... ) construction is more convoluted because it involves more steps and may involve a temporary file (the named pipe).

The <( ... ) construct is an extension, and is not available in the standard POSIX bourne shell nor on platforms that do not support /dev/fd or named pipes. For this reason alone, because the two alternatives being considered are exactly equivalent in functionality, the more portable command | other-command form is a better choice.

The <( ... ) construction should be slower because of the additional convolution, but it's only in the startup phase and I don't expect the difference to be easily measurable.

NOTE: On Linux SysV platforms, < ( ... ) does not use named pipes but instead uses regular pipes. Regular pipes (indeed all file descriptors) can be referred to by the special named /dev/fd/<file-descriptor-number so that's what the shell uses as a name for the pipe. In this way it avoids creating a real named pipe with a bona fide temporary filename in the real filesystem. Although the /dev/fd trick is what was used to implement this feature when it originally appears in ksh, it is an optimization: on platforms that don't support this, a regular named pipe in the real filesystem is used as described above.

ALSO NOTE: To describe the syntax as <<( ... ) is misleading. In fact it's <( ... ), which is replaced with the name of a pipe, and then the other < character which prefixes the whole thing is separate from this syntax and it's the regular well-known syntax for redirecting input from a file.