ɛ
("Latin epsilon") is a letter used in certain African languages, usually to represent the vowel sound in English "bed". In Unicode it's encoded as U+025B, very distinct from everyday e
.
However, if I sort
the following:
eb
ed
ɛa
ɛc
it seems that sort
considers ɛ
and e
equivalent:
ɛa
eb
ɛc
ed
What's going on here? And is there a way to make ɛ
and e
distinct for sort
ing purposes?
Best Answer
No, it doesn't consider them as equivalent, they just have the same primary weight. So that, in first approximation, they sort the same.
If you look at /usr/share/i18n/locales/iso14651_t1_common (as used as basis for most locales) on a GNU system (here with glibc 2.27), you'll see:
e
,ɛ
andE
have the same primary weight,e
andE
same secondary weight, only the third weight differentiates them.When comparing strings,
sort
(thestrcoll()
standard libc function is uses to compare strings) starts by comparing the primary weights of all characters, and only go for the second weight if the strings are equal with the primary weights (and so on with the other weights).That's how case seems to be ignored in the sorting order in first approximation.
Ab
sorts betweenaa
andac
, butAb
can sort before or afterab
depending on the language rule (some languages have<MIN>
before<CAP>
like in British English, some<CAP>
before<MIN>
like in Estonian).If
e
had the same sorting order asɛ
,printf '%s\n' e ɛ | sort -u
would return only one line. But as<BAS>
sorts before<PCL>
,e
alone sorts beforeɛ
.eɛe
sorts afterEEE
(at the secondary weight) even thoughEEE
sorts aftereee
(for which we need to go up to the third weight).Now if on my system with glibc 2.27, I run:
You'll notice that there are quite a few characters that have been defined with the exact same 4 weights. In particular, our ɛ has the same weights as:
And sure enough:
That can be seen as a bug of GNU libc locales. On most other systems, locales make sure all different characters have different sorting order in the end. On GNU locales, it gets even worse, as there are thousands of characters that don't have a sorting order and end up sorting the same, causing all sorts of problems (like breaking
comm
,join
,ls
or globs having non-deterministic orders...), hence the recommendation of usingLC_ALL=C
to work around those issues.As noted by @ninjalj in comments, glibc 2.28 released in August 2018 came with some improvements on that front though AFAICS, there are still some characters or collating elements defined with identical sorting order. On Ubuntu 18.10 with glibc 2.28 and in a en_GB.UTF-8 locale.
(why would U+00B7 be considered equivalent as U+0387 only when combined with
L
/l
?!).And:
(still over 1 million characters (95% of the Unicode range, down from 98% in 2.27) sorting the same as other characters as their sorting order is not defined).
See also: