Postgresql – Mixed search in multiple columns

database-designfull-text-searchjsonpattern matchingpostgresql

I have a table which has a particular column of type JSON, where some user's contact information are stored. I can alter the table and add new columns with cached values to increase search performance if needed (this is a new system, which is in the state of designing/at the beginning of implementation).

The top-level keys in the JSON are 'name' (both first and last name as one value), 'address' (a hash containing street, state, zip etc) and 'other' (tel, email, www etc). These are the three categories of information we are storing of customers.

One property of this JSON column is, that I really can't tell what keys will be there (that's why it is a JSON, so it remains flexible in this sense), but I need to search all the values.

My main problem is, that I've been told to implement a search functionality with one single input (like google), which would search in every value, even combined (so if u search 'John Smith Slovakia', it would find customers with the name John Smith who lives in Slovakia').

Another required property is that it has to support partial searches, i.e. if you wan't to find every customer with the last name Smith who live in street named 'Somelongname', then it would be enough to type in 'smith somelong' and it would still find them.

I've looked into full text search and it looked fine, except it did not really support partial searches.

Is there a better, more efficient solution, than searching every input word separately accross all the values using LIKE '%search_token%' and then merging the results for every search token into the final result?

BTW I'm using PostgreSQL.

Best Answer

Pattern matching and operators

Full text search is not the right tool for pattern matching (and possibly even fuzzy, fault tolerant input). Typically, trigram-similarity search with the % operator is the superior approach here. You need to install the additional module pg_trgm once per database:

Possibly combine that with LIKE (also supported by trigram indexes), and text search like you had in mind. Related answer where text search and similarity are combined:

PostgreSQL FTS and Trigram-similarity Query Optimization

Overview of pattern-matching functionality in Postgres:

Pattern matching with LIKE, SIMILAR TO or regular expressions in PostgreSQL

Database design

json is a very bad choice for your intended functionality. It is very inefficient to have to plow through huge columns holding everything in a semi-structured big string (json).

Consider normalizing your schema, which would result in superior performance (when done properly).

If you are hellbent on json, consider the new jsonb in the upcoming (and soon to be released) Postgres 9.4. More (last paragraph):

How to remove known elements from a JSON[] array in PostgreSQL?

And be sure to use functional indexes like:

Further optimization is likely possible. But that would require more detailed information including table definitions, cardinalities, sample data and typical queries.

Related Solutions

Does the ERD account for the business logic

There are a few issues with your ERD.

A. Your ERD does not account for these business requirements:

Certain customers might wish some different information displayed on their emails, so my database design need to account for this. E.g. Customer John Smith might want his emails to also list the 'Item Color', where as Emma Watson is not too fussed about that detail, and does not want that listed in the emails.

If customers have not explicitly mentioned that they want customized emails sent, then a Standard Template email is sent.

You need to add a location to maintain your customer email preferences. The model for that depends on what the rules are for recording the preferences. Your question isn't explicit enough on this point.

B. You might also want to include an EMAIL_TEMPLATE table with a foreign key from EMAIL to this new table. That will allow you to record which template was used for each email.

C. You aren't clear about what your m:n relationship is recording. Are these columns: EMAIL.TO, EMAIL.CC, EMAIL.BCC foreign keys? If so, are they single or multivalued? (i.e. can you have multiple email addresses in any or all of these fields?) If they are single valued foreign keys, then you can do away with the many-to-many and make them each a many-to-one from EMAIL to CUSTOMER (so you will have three 1:m relationships) If they are multi-valued foreign keys, then you need to create an explicit intersection table instead of using a simple m:n relationship and you will want to have an ADDRESSEE_TYPE column ("TO", "CC", "BCC") as part of the composite key.

EDIT: Entity Relationship Diagram

This would be a logical data model that could support your business rules:

ERD

The TEMPLATE_FIELD_TYPE table is optional. You could use it if you have a standard list of fields that can be added to a template. If you don't want to lock down this list or reuse fields then you can skip this table and just use TEMPLATE_FIELD by itself.

Postgresql – Querying JSONB in PostgreSQL

This does not work because it's trying to cast a jsonb value to integer.

select data->'name' as name from persons where cast(data->'age' as int) > 25

This would actually work:

SELECT data->'name' AS name FROM persons WHERE cast(data->>'age' AS int) > 25;

Or shorter:

SELECT data->'name' AS name FROM persons WHERE (data->>'age')::int > 25;

And this:

SELECT data->'name' AS name FROM persons WHERE data->>'name' > 'Jenny';

Seems like confusion with the two operators -> and ->> and operator precedence. The cast :: binds stronger than the json(b) operators.

Figure out type dynamically

This is the more interesting part of your question:

the type of age in the JSON document is number anyway, so why can't PostgreSQL figure out that by itself?

SQL is a strictly typed language, it does not allow the same expression to evaluate to integer in one row and to text in the next. But since you are only interested in the boolean result of the test, you can get around this restriction with a CASE expression that forks depending on the result of jsonb_typeof():

SELECT data->'name'
FROM   persons
WHERE  CASE jsonb_typeof(data->'age')
        WHEN 'number'  THEN (data->>'age')::numeric > '25' -- treated as numeric
        WHEN 'string'  THEN data->>'age' > 'age_level_3'   -- treated as text
        WHEN 'boolean' THEN (data->>'age')::bool           -- use boolean directly (example)
        ELSE FALSE                                         -- remaining: array, object, null
       END;

An untyped string literal to the right of the > operator is coerced to the respective type of the value to the left automatically. If you put a typed value there, the type has to match or you have to cast it explicitly - unless there is adequate implicit cast registered in the system.

If you know that all numeric values are actually integer, you can also:

... (data->>'age')::int > 25 ...