This is a bit of a FAQ. You'd find more information if you searched for ON DUPLICATE KEY UPDATE
(the MySQL syntax), MERGE
(the SQL-standard syntax), or UPSERT
. It's surprisingly hard.
The best article I've seen on it yet is Depesz's "why is upsert so complicated". There's also the SO question Insert, on duplicate update (postgresql) which has suggestions but lacks explanation and discussion of the issues.
The short answer is that, yes:
It seems the only option I have is to abort the entire transaction and
try again.
When using SERIALIZABLE
transactions you just have to re-issue them when they fail. Which they will. By design - and much more frequently on Pg 9.1 and above because of greatly improved conflict detection. Upsert-like operations are very high conflict, so you may land up retrying quite a bit. If you can do your upserts in READ COMMITTED
transactions instead it'll help, but you should still be prepared to retry because there are some unavoidable race conditions.
Let the transaction fail with a unique violation when you insert the conflicting row. If you get a SQLSTATE 23505 unique_violation
failure from the transaction and you know you were attempting an upsert, re-try it. If you get a SQLSTATE 40001 serialization_failure
you should also retry.
You fundamentally cannot do that retry within a PL/PgSQL function (without dirty hacks like dblink), it must be application side. If PostgreSQL had stored procedures with autonomous transactions then it'd be possible, but it doesn't. In READ COMMITTED
mode you can check for conflicting inserts made since the transaction started, but not after the statement that calls the PL/PgSQL function started, so even in READ COMMITTED
your "detect conflict with select" approach simply will not work.
Read depesz's article for a much better and more detailed explanation.
You will be able to avoid a lot of your race conditions by performing many of your steps in a single statement. By using a TOP()
cluse will be able to set the flag on at most one row. By using the OUTPUT
cluse you can return this to the application automically.
I define a simple test table and populate it:
create table dbo.T(id int, IsProcessing bit default 0);
insert dbo.T(id) values (1),(2),(3),(4);
The output clause needs a table variable to receive the changed values:
declare @U table (id int);
A bit of debug code to make the "before" and "after" states obvious:
select 'Before' as B, * from dbo.T;
And the statement itself:
update top (1) dbo.T
set IsProcessing = 1
OUTPUT INSERTED.id
INTO @U(id)
where IsProcessing = 0;
And the results:
select 'After' as A, * from dbo.T;
select * from @U;
This is the output from the second execution of the above.
B id IsProcessing
Before 1 1
Before 2 0
Before 3 0
Before 4 0
A id IsProcessing
After 1 1
After 2 1
After 3 0
After 4 0
id
2
The row with id=2
had gone from IsProcessing=0
to IsProcessing=1
and that id
is returned in the table variable.
With this trivial data the rows are processed in the sequence they were created. In a more complex environment the optimizer may choose any row that matches the where clause. If you have a requirement to process rows in, say, time sequence further qualification will be required.
I haven't thought it all the way through but I believe his will work at any isolation level with or without explicit transactions.
It is of course possible that there are no rows with IsProcessing=0
at the time the statement runs. In this case the table variable will have zero rows.
To fully isolate each transaction from the other you could try sp_getapplock
. This will add overhead and reduce concurrency. You have to be careful to release the app lock as soon as you can in both success and failure scenarios.
Best Answer
A function takes part of an existing transaction. To avoid this context error, the transaction mode should be changed before the call to the function. Therefore you must remove the SET TRANSACTION statement function, and then use the code (for example):
Using your example...
function code -
calling code to function -
Or yet:
And
I hope have contributed!