How to grep thousands of files in a directory for hundreds of strings in a file

grep

I am trying to compose a grep statement and it is killing me. I am also tired of getting the arguments list too long error. I have a file, let's call it subset.txt. It contains hundreds of lines with specific strings such as MO43312948. In my object directory I have thousands of files and I need to copy all the files that contain the strings listed in subset.txt into another directory.

I was trying to start with this to just return the matching files from the objects directory.

grep -F "$(subset.txt)" /objects/*

I keep getting `bash: /bin/grep: Argument list too long“

Best Answer

You can pass a directory as a target to grep with -R and a file of input patterns with -f:

  -f FILE, --file=FILE
          Obtain patterns from FILE, one per line.  If this option is used
          multiple  times  or  is  combined with the -e (--regexp) option,
          search for all patterns given.  The  empty  file  contains  zero
          patterns, and therefore matches nothing.

   -R, --dereference-recursive
          Read all files under each directory,  recursively.   Follow  all
          symbolic links, unlike -r.

So, you're looking for:

grep -Ff subset.txt -r objects/

You can get the list of matching files with:

grep -Flf subset.txt -r objects/

So, if your final list isn't too long, you can just do:

 mv $(grep -Flf subset.txt -r objects/) new_dir/

If that returns an argument list too long error, use: 

grep -Flf subset.txt -r objects/ | xargs -I{} mv {} bar/

And if your file names can contain spaces or other strange characters, use (assuming GNU grep):

grep -FZlf subset.txt -r objects/ | xargs -0I{} mv {} bar/

Finally, if you want to exclude binary files, use:

grep -IFZlf subset.txt -r objects/ | xargs -0I{} mv {} bar/
Related Question