Text Processing – Remove Unique Strings from a Text File

sorttext processing

Sorry guys I had to edit my example, because I didn't express my query properly.
Let's say I have the .txt file:

Happy sad
Happy sad
Happy sad
Sad happy
Happy sad
Happy sad
Mad sad
Mad happy
Mad happy

And I want to delete any string that is unique. Leaving the file with:

Happy sad
Happy sad
Happy sad
Happy sad
Happy sad
Mad happy
Mad happy

I understand that sort is able to get rid of duplicates (sort file.txt | uniq), so is there anyway we can do the opposite in bash using a command? Or would I just need to figure out a while loop for it?
BTW uniq -D file.txt > output.txt doesn't work.

Best Answer

Using awk:

$ awk 'seen[$0]++; seen[$0] == 2' file
Happy sad
Happy sad
Happy sad
Happy sad
Happy sad
Mad happy
Mad happy

This uses the text of each line as the key into the associative array seen. The first seen[$0]++ will cause a line that has been seen before to be printed since the value associated with the line will be non-zero on the second and subsequent times the line is seen. The seen[$0] == 2 causes the line to be printed again if this is the second time the line has been seen (without this, you'll miss one occurrence of each duplicated line).

This is related to awk '!seen[$0]++' which is sometimes used to remove duplicates without sorting (see e.g. How does awk '!a[$0]++' work?).

To only get one copy of the duplicated lines:

awk 'seen[$0]++ == 1' file

or,

sort file | uniq -d
Related Question