Print only unique lines from file not the duplicates

awkperlsedtext processing

I have a sorted words list, line by line file like so:

apple
apple
grapes
lime
orange
orange
pear
pear
peach
strawberry
strawberry

I only want to print out unique lines and drop duplicates:

grapes
peach
lime

How can I do this with sed, awk or perl?

Best Answer

That's the job for uniq:

$ LC_ALL=C uniq -u file
grapes
lime
peach

If you want other tools, like perl:

perl -nle '$h{$_}++; END {print for grep { $h{$_} == 1 } %h}' <file

Related Solutions

Print only lines where the first column is unique

awk normally reads each line of the input and invokes the script on it. The cases where you would use getline are few and far between. When your script is run with six lines of input, this is an overview of what happens:

Read line 1 normally

Set variables
Call getline, which reads line 2
Compare variables

Read line 3 normally

Set variables
Call getline, which reads line 4
Compare variables

Read line 5 normally

Set variables
Call getline, which reads line 6
Compare variables

Obviously this isn’t going to work.
Secondly, you made a common mistake in your awk code. In awk, fields from the input are referenced as $number and variables are referenced as variable_name. This is different from shell scripts, where command line arguments are referenced as $number and variables are referenced as $variable_name. Your test
```
if ($1 != $path)
```
should be
```
if ($1 != path)
```
Your overall approach is flawed. You can’t identify strings that occur only once in the file by looking at two lines at a time. I believe that you can do it by looking at three lines at a time (i.e., by keeping the two previous lines in variables), but things like that get complicated and messy. It’s probably simpler to count occurrences. Here’s a minimal modification on your script to do that.
```
awk '{
  if ($1 != path) {
    if (count == 1) {
      print prev
    }
    count=1
  }
  else count++
  prev=$0; path=$1
}
END {
    if (count == 1) {
      print prev
    }
}'
```
I deleted type, since you never used it.

Disclosure: This is essentially the same as the last part of glenn’s answer.

text-processing,sed,awk,grep – Print Only Unique Lines That Appear Last in Logfile Based on Date/Time

If you're going to do a second pass (which you pretty well have to), you may as well only store line numbers rather than full records. It makes the logic easier.

awk 'NR == FNR {if (z[$6]) y[z[$6]]; z[$6] = FNR; next} !(FNR in y)' logfile logfile

Proof of correctness:

At the end of processing each line, every line number processed so far is either a value in z, or an index (not value) in y, but never both.

The lines represented by values in z are, at the end of each iteration, exactly and only the latest records so far seen for each IP address.

The indices of y are, therefore, the exact lines which we wish not to print.

Best Answer

Related Solutions

Print only lines where the first column is unique

text-processing,sed,awk,grep – Print Only Unique Lines That Appear Last in Logfile Based on Date/Time

Related Question