Bash – grep -v unexpected behaviour

bashgreprm

Let's say I have a simple stupid script that removes files by ending, looking like this:

rm *.uvw *.xyz

The script, or rm, to be precise, writes messages on stderr if it cannot find at least one file with the specified ending.

Now let's say the script is a bit bigger and does a bit more with a bit more file types and I'm not interested in which file types exist and which don't, but the complains about non-existent file types obstruct the rest of the output and error messages I am more interested in, so I want to filter the output:

rm *.uvw *.xyz 2>&1 | grep -v "No such file or directory"

This works fine for the most part, but it removes the message part of interactive dialogs, which for example ask, if a write-protected file should be deleted, so I get prompted without the according message.

I do not understand this behaviour and could not find any related information. Can someone explain this?

Best Answer

The problem

When rm prompts the use for input, it does not put a newline at the end of the prompt:

$ rm *.uvw *.xyz
rm: remove write-protected regular empty file 'a.xyz'?

grep is line-based. It can only process complete lines. It cannot tell whether the line should be printed until the line is complete. Thus, standard utilities for dealing with buffering, such as stdbuf, cannot help.

The solution

Use nullglob and remove the missing files messages.

Without nullglob, the messages you don't want appear:

$ rm *.uvw *.xyz
rm: cannot remove '*.uvw': No such file or directory
rm: remove write-protected regular empty file 'a.xyz'? n

With it, the "No such file or directory" message is suppressed:

$ shopt -s nullglob
$ rm *.uvw *.xyz
rm: remove write-protected regular empty file 'a.xyz'? n

Refinement

If there is no file at all that matches either glob, then a different error message appears:

$ shopt -s nullglob
$ rm *.uvw *.xyz
rm: missing operand
Try 'rm --help' for more information.

A simple way to avoid this is to make sure that at least one such file exists:

shopt -s nullglob
[ -e "deleteme.xyz" ] ||touch deleteme.xyz
rm *.uvw *.xyz

Since deleteme.xyz is going to be erased anyway, there is no harm in touching it before we run rm.

Related Solutions

Bash – How to implement deferred stdout in child process without using temporary files

inner.sh could be:

printf '%s\n' "$1" >&3

and in outer.sh, you could do:

{ inner=$(middle "$@" 3>&1 >&4 4>&-); } 4>&1
printf '%s\n' "$inner"

The inner text is passed via a pipe (in the command substitution) and stored in a shell variable. That assumes middle doesn't close that fd 3 before invoking inner.sh (no reason it would do that though).

Explanation:

{ ... } 4>&1. In that command group, initially, both fd 1 and 4 point to the original stdout. IOW, we've made a copy of outer.sh's stdout onto fd 4 to be able to recover it inside the command substitution
$(...). Inside that command substitution, stdout (fd 1) is a pipe. That's the point of command substitution. It wants to grab the output of the command. But here, we don't want the stdout of middle, we want what it (or more exactly its child inner.sh) writes on fd 3, so:
middle 3>&1 >&4 4>&-: for middle, we make fd 3 the cmdsubst pipe, so that what inner.sh writes there goes into $inner, and we restore middle's stdout onto the saved original stdout on fd 4. We close fd 4 after it has served it's purpose as middle doesn't need to do anything with it.

Optimizing GNU grep

No, there's no such thing. Generally the cost of starting grep (fork a new process, load the executable, shared library, dynamic linkage...) would be a lot greater than compiling the regexps, so this kind of optimisation would make little sense.

Though see Why is matching 1250 strings against 90k patterns so slow? about a bug in some versions of GNU grep that would make it particularly slow for a great number of regexps.

Possibly here, you could avoid running grep several times by feeding your chunks to the same grep instance, for instance by using it as a co-process and use a marker to detect the end. With zsh and GNU grep and awk implementations other than mawk:

coproc grep -E -f patterns -e '^@@MARKER@@$' --line-buffered
process_chunk() {
  { cat; echo @@MARKER@@; } >&p & awk '$0 == "@@MARKER@@"{exit};1' <&p
}
process_chunk < chunk1 > chunk1.grepped
process_chunk < chunk2 > chunk2.grepped

Though it may be simpler to do the whole thing with awk or perl instead.

But if you don't need the grep output to go into different files for different chunks, you can always do:

{
  cat chunk1
  while wget -qO- ...; done # or whatever you use to fetch those chunks
  ...
} | grep -Ef patterns > output