PostgreSQL – Inserting Result of Parallelized Select Query

parallelismperformancepostgresqlquery-performance

According to https://www.postgresql.org/docs/current/static/when-can-parallel-query-be-used.html,

"Even when it is in general possible for parallel query plans to be
generated, the planner will not generate them for a given query if any
of the following are true:
The query writes any data or locks any database rows. If a query contains a data-modifying operation either at the top level or within
a CTE, no parallel plans for that query will be generated. This is a
limitation of the current implementation which could be lifted in a
future release."

Indeed, when I try to insert result of a parallel SELECT query into a table (either by SELECT.. INTO either by WITH..SELECT..INTO, the query is not executed as parallel query.

My question is: Is there any way to trick the Postgresql so that a SELECT query is executed as parallel query and then its result inserted into a table?

Best Answer

I think the only even worth trying is to use PostgreSQL FDW which may or may not work. Your use case would also have to be super-niche for that to pay off, like a complex GIS aggregation query.

You can see this restriction is still very much noted in the code,

The most significant restriction imposed by parallel mode is that all operations must be strictly read-only; we allow no writes to the database and no DDL. We might try to relax these restrictions in the future.

And, again here

 * Don't allow writes in parallel mode.  Supporting UPDATE and DELETE
 * would require (a) storing the combocid hash in shared memory, rather
 * than synchronizing it just once at the start of parallelism, and (b) an
 * alternative to heap_update()'s reliance on xmax for mutual exclusion.
 * INSERT may have no such troubles, but we forbid it to simplify the
 * checks.

I think it's being stopped by a call to ExecCheckXactReadOnly here in ExecCheckXactReadOnly

if (plannedstmt->commandType != CMD_SELECT || plannedstmt->hasModifyingCTE)
    PreventCommandIfParallelMode(CreateCommandTag((Node *) plannedstmt));

Shy of that Maybe in PostgreSQL 11, or 12?

Related Solutions

PostgreSQL – Why ALTER VIEW Not Possible with CREATE OR REPLACE

It's an implementation limitation. It's theoretically possible, of course, but nobody's written the code to handle it yet.

To cope with column removals or type changes, PostgreSQL would have to scan every view that references the view being modified (using pg_catalog.pg_depend) to see if any of them relied on the column. It'd also need to look for whole-row references and disallow changes in those cases.

It's less clear why adding a column is not permitted. Again, I suspect that's down to whole-row references. If pg_depend was checked for whole-row references without finding any, and the new column appeared at the end, it'd be OK to add it.

However, views created with SELECT * FROM wouldn't "inherit" the new column, because * gets expanded into a column-list during view creation, though. So if you had view_A that does a SELECT * FROM view_B, and you added a column to view_B, it wouldn't appear in view_A. Yet if you dropped and re-created view_A, the column would appear. Needless to say, that's not good. To cope with that, PostgreSQL would have to keep track of whether a given view's column list ("targetlist" in PostgreSQL internal terms) came from a * wildcard. Which is more complicated than you'd think because you can write someview.* too.

All in all - it's complicated, and nobody's wanted it enough to do the work to implement it.

PostgreSQL – Why Query is Slow When Subquery Added

My colleague has found a way to change the query so that it needs a simple rewrite and does what it needs to do, i.e. doing the subselect in one step, and then doing the further operations on the result:

SELECT mtid FROM publication 
WHERE 
  mtid = ANY( (SELECT ARRAY(SELECT 9762715))::bigint[] )
  OR last_modifier=21321
LIMIT 5000;

The explain analyze now is:

 Limit  (cost=92.58..9442.38 rows=2478 width=8) (actual time=0.071..0.074 rows=1 loops=1)
   InitPlan 2 (returns $1)
     ->  Result  (cost=0.01..0.02 rows=1 width=0) (actual time=0.010..0.011 rows=1 loops=1)
           InitPlan 1 (returns $0)
             ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.001..0.002 rows=1 loops=1)
   ->  Bitmap Heap Scan on publication  (cost=92.56..9442.36 rows=2478 width=8) (actual time=0.069..0.070 rows=1 loops=1)
         Recheck Cond: ((mtid = ANY (($1)::bigint[])) OR (last_modifier = 21321))
         Heap Blocks: exact=1
         ->  BitmapOr  (cost=92.56..92.56 rows=2478 width=0) (actual time=0.060..0.060 rows=0 loops=1)
               ->  Bitmap Index Scan on publication_pkey  (cost=0.00..44.38 rows=10 width=0) (actual time=0.046..0.046 rows=1 loops=1)
                     Index Cond: (mtid = ANY (($1)::bigint[]))
               ->  Bitmap Index Scan on publication_last_modifier_btree  (cost=0.00..46.94 rows=2468 width=0) (actual time=0.011..0.011 rows=0 loops=1)
                     Index Cond: (last_modifier = 21321)
 Planning time: 0.704 ms
 Execution time: 0.153 ms

It seems we can create a simple parser that finds and rewrites all subselects this way, and add it to a hibernate hook to manipulate the native query.

Best Answer

Related Solutions

PostgreSQL – Why ALTER VIEW Not Possible with CREATE OR REPLACE

PostgreSQL – Why Query is Slow When Subquery Added

Related Question