PostgreSQL – Trigram Match Acting Strange for Specific Characters

full-text-searchpattern matchingpostgresql

I'm using pg_trgm to run a query, but I'm getting a lot of 1 matches when searching with a symbol difference. I have the following query:

SELECT my_column, similarity('$ Hello', my_column) AS sml
FROM my_table
WHERE my_column % '$ Hello'
ORDER BY sml DESC, my_column;

In my_table, I have the following:

- Hello
? Hello
| Hello
$ Hello
! Hello
!? Hello

They all return with a similarity match of 1. Do I need to escape the '$' or something similar?

Best Answer

Other answers clarified that trigram similarity is based on alphanumeric characters only. That's why all your examples match 100 %.

You may still be able to make use of a trigram GiST or GIN index and establish your desired sort order with additional ORDER BY expressions. For your demonstrated case:

SELECT my_column, similarity('$ Hello', my_column) AS sml
FROM   my_table
WHERE  my_column % '$ Hello'
ORDER  BY sml DESC
        , my_column <> '$ Hello'       -- !
        , my_column;

The boolean expression my_column <> '$ Hello' evaluates to FALSE, TRUE, or NULL - with this sort order. So the exact match (considering all characters) comes first. And this query can still use a trigram index. A trigram GiST index would (still) even support a "nearest neighbor" (KNN) search with LIMIT. Related:

You can do more, depends on your exact use case and requirements. Example:

...
ORDER  BY sml DESC
        , my_column <> '$ Hello'
        , my_column !~ '\$ Hello'            -- note $ escaped with \$
        , levenshtein(my_column , '$ Hello')
        , my_column;

Within the same trigram similarity, this sorts exact matches first, then strings containing the exact phrase. And within each subgroup matches the shorter Levenshtein distance first. Alphabetical as final tiebreaker. Related:

Last but not least, you tagged full text search. But your example is based on trigram similarity (provided by the additional module pg_trgm), which is a largely different concept with completely separate infrastructure and operators. You may want actual full text search instead (and phrase search with that?):

But punctuation characters are considered noise and stripped in FTS all the same. Same "problem". ts_debug() shows how your text search configuration classifies identified tokens in a given string (simple configuration in the example). 

SELECT * FROM ts_debug('simple', '? Hello %&/( 123');

It starts with the "default" parser (currently the only one), parsing all these as noise to begin with ...

Related Question