Short version: collation doesn't really work in command line utilities.
Longer version: the underlying function to compare two strings is strcoll
. The description isn't very helpful, but the conceptual method of operation is to convert both strings to a canonical form, and then compare the two canonical forms. The function strxfrm
constructs this canonical form.
Let's observe the canonical forms of a few strings (with GNU libc, under Debian squeeze):
$ export LC_ALL=en_US.UTF-8
$ perl -C255 -MPOSIX -le 'print "$_ ", unpack("h*", strxfrm($_)) foreach @ARGV' b a A à 〼 〇
b d010801020
a c010801020
A c010801090
à 101010102c6b
〼 101010102c6b102c6b102c6b
〇 101010102c6b102c6b102c6b
As you can see, 〼 and 〇 have the same canonical form. I think that's because these characters are not mentioned in the collation tables of the en_US.UTF-8
locale. They are, however, present in a Japanese locale.
$ export LC_ALL=ja_JP.UTF-8
$ perl -C255 -MPOSIX -le 'print "$_ ", unpack("h*", strxfrm($_)) foreach @ARGV' 〼 〇
〼 303030
〇 3c9b
The source code for the locale data (in Debian squeeze) is in /usr/share/i18n/locales/en_US
, which includes /usr/share/i18n/locales/iso14651_t1_common
. This file doesn't have an entry for U3007
or U303C
, nor are they included in any range that I can find.
I'm not familiar with the rules to build the collation order, but from what I understand, the relevant phrasing is
The symbol UNDEFINED shall be interpreted as including all coded character set values not specified explicitly or via the ellipsis symbol. (…) If no UNDEFINED symbol is specified, and the current coded character set contains characters not specified in this section, the utility shall issue a warning message and place such characters at the end of the character collation order.
It looks like Glibc is instead ignoring characters that aren't specified. I don't know if there's a flaw of my understanding of the POSIX spec, if I missed something in Glibc's locale definition, or if there's a bug in the Glibc locale compiler.
This was a fun one.
First, we need to eliminate trailing comments, as in:
86.242.200.81 banana.domain.net # comment
We can do that with the following (assuming just spaces, no tabs):
sed 's/ *#.*//'
If you have tabs in your hosts file, maybe run this first:
tr '\t' ' '
Then we need to eliminate "comment out this line" comments, which I'm going to define as a single hash character preceding an ip address. We can remove those like this:
sed '/^#[0-9]/ s/^#//'
Putting the above together gets us:
### Comments
# Comments
86.242.200.81 banana.domain.net
86.242.200.3 orange.domain.net
31.28.225.81 monkey.anotherdomain.net
51.18.33.4 puffin.domainz.com
31.28.220.80 monkey.anotherdomain.net
86.242.201.3 orange.domain.net
If we sort this on the second column (sort -k2
), we get a list sorted by name:
86.242.200.81 banana.domain.net
# Comments
### Comments
31.28.220.80 monkey.anotherdomain.net
31.28.225.81 monkey.anotherdomain.net
86.242.200.3 orange.domain.net
86.242.201.3 orange.domain.net
51.18.33.4 puffin.domainz.com
And now we can apply uniq
to find duplicates, if we tell uniq
to ignore the first field:
uniq -c -f 1
Which gives us:
2
1 86.242.200.81 banana.domain.net
1 # Comments
1 ### Comments
2 31.28.220.80 monkey.anotherdomain.net
2 86.242.200.3 orange.domain.net
1 51.18.33.4 puffin.domainz.com
So if we look for lines with a count of 2 or higher, we have found our duplicates. Putting this all together we get:
#!/bin/sh
tr '\t' ' ' |
sed '
/^#[0-9]/ s/^#//
s/ *#.*//
/^ *$/ d
' |
sort -k2 |
uniq -f 1 -c |
awk '$1 > 1 {print}'
The final awk
statement in the above script looks for lines where the count from uniq
(field1 ) is > 1
.
Running the above script looks like this.
Best Answer
There are 3 occurrences of the line above. The first two occurrences both have a carriage return at the end. The 3rd occurrence doesn't. you can use the following command to visualize it:
It's caused by the fact that this text file lacks an empty line at the end.
I think it's a convention in the Unix/Linux world to always have an empty line at the end of text files.
Some other things I found interesting while I was testing it:
grep home american.txt | sort| uniq -u
works just fine.