How to remove lines included in one file from another file

text processing

I have two lists, a complete list and a partial list. I want a bash script that will search the complete list for any lines that are in the partial, if there are, they will be removed from the complete list. Can anyone help?

Best Answer

grep can read multiple patterns from a file, one per line. Combine with the options -v to output non-matching lines, and -F to match strings instead of regex and -x to require that the whole line matches.

grep -Fvx -f partial.list complete.list >remaining.list &&
mv remaining.list complete.list

Obviously the second command line is only if you want to overwrite the file containing the complete list.

If the partial list is huge and you don't mind reordering the list, then join may be faster.

Related Solutions

How to remove identical lines in one file from another, using sed

Here is my snippet:

remove_lines()
{
    # remove lines from a file 
    #  
    # $1 - source file with patterns of lines to be removed
    # $2 - destination file
    tmpfile=$(mktemp "$(dirname -- "$2")"/XXXXXXXX) &&
    grep -F -f "$1" -v -- "$2" >>"$tmpfile" &&
    mv -- "$tmpfile" "$2" &&
}

EDIT: I've just realized that there is no sed in it, but that wasn't critical, was it?

Best way to remove lines from a file where matching text (not whole line) exists in another file

You can use fgrep for this:

fgrep -v -f file1  file2  > unique_addresses

This task will be a lot easier if you have 1 email address per line in both files.

Traditionally fgrep exists as a separate program, but in GNU utilities, grep -F does the same thing.

Best Answer

Related Solutions

How to remove identical lines in one file from another, using sed

Best way to remove lines from a file where matching text (not whole line) exists in another file

Related Question