Grep –exclude Not Excluding File – How to Fix

grepsearch

I am searching a dir for a particular string (to see all instances of where the string is present and in which files). However, I want to exclude one particular file from being searched.

Here's what is happening-

$echo "searchstring" > ./old_folder/useless_file
$echo "searchstring" > ./new_folder/good_file
$grep -r --exclude="old_folder/useless_file" searchstring *
./old_folder/useless_file:searchstring
./new_folder/good_file:searchstring

Here is the output I want-

./new_folder/good_file:searchstring

Best Answer

The --exclude option takes globs that are matched against file names, not directories or full paths:

  --exclude=GLOB
          Skip   files  whose  base  name  matches  GLOB  (using  wildcard
          matching).  A file-name  glob  can  use  *,  ?,  and  [...]   as
          wildcards,  and  \  to  quote  a wildcard or backslash character
          literally.

So, you could do:

$ grep -r --exclude="*useless_file*" searchstring 
new_folder/good_file:searchstring

Or, to exclude all files in that directory:

$ grep -r --exclude-dir="old_folder" searchstring 
new_folder/good_file:searchstring

Related Solutions

Using grep with the –exclude-dir flag to exclude multiple directories

The man page description for that option is sort of misleading... It's pattern as in globs not pattern as in regex. Per the info page:

--exclude-dir=GLOB

Skip any command-line directory with a name suffix that matches the pattern GLOB. When searching recursively, skip any subdirectory whose base name matches GLOB. Ignore any redundant trailing slashes in GLOB.

So, in your case you could run:

grep -inRw -E 'direct' . --exclude-dir={git,log,assets}

to exclude directories named git, log and assets or e.g.

grep -inRw -E 'direct' . --exclude-dir={\*git,asset\*}

to exclude directory names ending in git or starting with asset.

Grep –exclude option doesn’t always skip named pipes

It seems grep still opens files even if the regex tells it to skip them:

$ ll
total 4.0K
p-w--w---- 1 user user 0 Feb  7 16:44 pip-fifo
--w--w---- 1 user user 4 Feb  7 16:44 pip-file
lrwxrwxrwx 1 user user 4 Feb  7 16:44 pip-link -> file

(Note: none of these have read permissions.)

$ strace -e openat grep foo --exclude='pip*' pip-file pip-link pip-fifo
openat(AT_FDCWD, "pip-file", O_RDONLY|O_NOCTTY) = -1 EACCES (Permission denied)
grep: pip-file: Permission denied
openat(AT_FDCWD, "pip-link", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
grep: pip-link: No such file or directory
openat(AT_FDCWD, "pip-fifo", O_RDONLY|O_NOCTTY) = -1 EACCES (Permission denied)
grep: pip-fifo: Permission denied
+++ exited with 2 +++

Granting read permissions, it appears that it doesn't try to read them after opening if they are excluded:

$ strace -e openat grep foo --exclude='pip*' pip-file pip-link pip-fifo
openat(AT_FDCWD, "pip-file", O_RDONLY|O_NOCTTY) = 3
openat(AT_FDCWD, "pip-link", O_RDONLY|O_NOCTTY) = -1 ENOENT (No such file or directory)
grep: pip-link: No such file or directory
openat(AT_FDCWD, "pip-fifo", O_RDONLY|O_NOCTTY^Cstrace: Process 31058 detached
 <detached ...>

$ strace -e openat,read grep foo --exclude='pip*' pip-file
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\25\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260`\0\0\0\0\0\0"..., 832) = 832
openat(AT_FDCWD, "pip-file", O_RDONLY|O_NOCTTY) = 3
+++ exited with 1 +++

$ strace -e openat,read grep foo --exclude='pipe*' pip-file
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\25\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260`\0\0\0\0\0\0"..., 832) = 832
openat(AT_FDCWD, "pip-file", O_RDONLY|O_NOCTTY) = 3
read(3, "foo\n", 32768)                 = 4
foo
read(3, "", 32768)                      = 0
+++ exited with 0 +++

And since openat wasn't called with O_NONBLOCK, the opening itself hangs, and grep doesn't reach the part where it excludes it from reading.

Looking at the source code, I believe the flow is like this:

If not recursive, call grep_command_line_arg on each file.
That calls grepfile if not on stdin.
grepfile calls grepdesc after opening the file.
grepdesc checks for excluding the file.

When recursive:

grepdirent checks for excluding the file before calling grepfile, so the failing openat never happens.

Best Answer

Related Solutions

Using grep with the –exclude-dir flag to exclude multiple directories

Grep –exclude option doesn’t always skip named pipes

Related Question