Postgresql – Query performance of index scans slower than parallel seq scan on Postgres

execution-planoptimizationperformancepostgresqlquery-performance

I've been trying to debug a performance problem on PostgreSQL 10 on AWS Aurora.

Specifically I'm trying to understand performance problems on "cold" queries where the buffer cache does not contain cached pages.

I have a table with an index and when the planner uses the index the query is substantially slower (spending more time on IO from what I can tell). However, when I drop an index the query is faster, it uses a parallel sequential scan.

The Schema of the table:

CREATE TABLE max_test_2018_3
(
   timestamp_         timestamp,
   person_alias       varchar(128),
   visitor_id         varchar(128),
   session_id         varchar(24),
   page_title         varchar(1024),
   location_host      varchar(256),
   location_path      varchar(1024),
   location_query     varchar(1024),
   location_fragment  varchar(1024),
   referrer_host      varchar(256),
   referrer_path      varchar(1024),
   referrer_query     varchar(1024),
   referrer_fragment  varchar(1024),
   duration           integer,
   location_secure    boolean,
   referrer_secure    boolean,
   id                 uuid,
   person_id          uuid,
   custom_properties  jsonb
);

The index DDL is:

CREATE INDEX max_test_2018_3_ix   ON max_test_2018_3
  USING btree  ( location_path varchar_pattern_ops);

And the query is:

explain (analyze, buffers)
SELECT O.person_id, 1 as step1, min(timestamp_) AS timestamp_ 
FROM max_test_2018_3 O
WHERE O.location_path  like '/tax-calculator%' 
GROUP BY O.person_id

Check the following two explains – note that both explains show minimal or no buffer hits. One shows the execution with the index and the other shows execution after the index has been dropped:

No index -> https://explain.depesz.com/s/N4uo
With index -> https://explain.depesz.com/s/69WS

The plan that includes the index scan takes 193 seconds vs 43 seconds for a sequential scan.

Why is step 2 in the "With-index" plan so slow? It reads in only 118372 buffers (about 118 MB of data), yet this step took 193 seconds. Why? When the buffer cache is warmed up the same with-index query takes a couple seconds. So we must concede that the IO is responsible for this extra time.

I've run this experiment several times on multiple similar datasets. Here is another example that demonstrates index scan operations that require IO and do not benefit from buffer cache.

No index -> https://explain.depesz.com/s/g6JU
with Index ->https://explain.depesz.com/s/Sw6Q

Again, you can see that the index scan is unreasonably slow. The planner attempts to use the index because it believes it will speed up the query, yet such queries are consistently slower.

With regard to parameters that affecting the planner I've set the following two:

random_page_cost = 1    
seq_page_cost = 1

The rationale being that for an SSD drive the random_page_cost does not require seeking and so can be set to the same value as seq_page_cost.

The table is vacuumed with statistics up-to date. The table max_test_2018_3 contains approximately 18 million rows and is about 9GB on disk. The above index on that table is 440 MB.

Why do the plans that use an index scan perform so poorly?

I/O Timing enabled Plans

with index https://explain.depesz.com/s/84kE no
no index(parallel scan) https://explain.depesz.com/s/CRBp

Tweeking parallelism settings

The explains above have default parameter settings with respect to parallelism. The only two exceptions are:

max_parallel_workers = 8
max_parallel_workers_per_gather = 4

As per the suggestions, I've experimented with with the following settings:

parallel_setup_cost = 10
parallel_tuple_cost = 0.001

Defaults for these are 1000 and 0.1 respectively. These changes did not affect the plans and PostgreSQL still chose to use the index even though a parallel sequential scan is faster.

The index size is 440MB, far larger than the default min_parallel_index_scan_size of 512kB

Best Answer

All your time is spent doing table I/O, which is to be expected in a test with a cold cache.

The index scan takes longer, but only because it is not parallelized. The sequential scan uses more total time, but since it is run on 5 cores in parallel, it finishes faster.

PostgreSQL v10 can perform parallel index scans, so the remaining riddle is why it doesn't. A side riddle is why the sequential scan uses so many parallel workers.

I suspect that you messed up the configuration parameters. The relevant parameters are:

max_parallel_workers_per_gather: limits how many workers can be used. You must have set this to 4 or more.
min_parallel_table_scan_size: if a table is bigger than that, a parallel worker is planned. If the table size exceeds 3^n-1 times that value, n parallel workers are planned. So either your table is very big, or you reduced the parameter. Alternatively:
The storage parameter parallel_workers on the table overrides the calculation based on min_parallel_table_scan_size as described above, so maybe you set that.
Finally, min_parallel_index_scan_size governs when a parallel index scan is considered. Either the index is small, or you lowered the parameter.

I'd expect that the index scan will be faster once you set these parameters to good values.

Related Solutions

PostgreSQL Sequential Scan instead of Index Scan Why

Notice this line:

->  Index Scan using data_area_pkey on data_area  (cost=0.00..52.13 rows=1 width=8) 
    (actual time=0.006..0.008 rows=0 loops=335130)

If you compute the total cost, considering loops, it is 52.3 * 335130 = 17527299. This is larger than 14857017.62 for the seq_scan alternative. That is why it does not use the index.

So the optimizer is overestimating the cost of the index scan. I'd guess that your data is sorted on the index (either due to a clustered index or to how it was loaded) and/or you have plenty of cache memory and/or a nice fast disk. Hence there is little random I/O going on.

You should also check the correlation in pg_stats, that is used by the optimizer to assess clustering when computing the index cost, and finally try changing random_page_cost and cpu_index_tuple_cost, to match your system.

Postgresql – Query planner thinks doing a sequential scan on a very large table is faster than using an index

From looking at your query, I think the planner is making the right choice, you have no WHERE clause or LIMIT, so the database has to return every row anyway, so it has the choice look at the entire table and look at the indexes or look at the entire table.

Have you tried restricting the rows with a WHERE clause or used a limit? I cant think of many time critical uses cases where you would want to look at 1.5 million rows.