Constraint based rules engine

CRUD is meant to define the characteristics necessary for a database as it relates to persistent storage. It is not meant to describe everything that could be done by a database engine.

To make a comparison, fundamentally a vehicle is a device used for transport. While true, this definition certainly doesn't include all the detail entailed in a modern automobile.

A database engine might handle multiple users, transactions, MVCC (Multiversion Concurrency Control), buffers and caches, ACID (atomicity, consistency, isolation, durability), as well as different isolation levels. A read may pull data from memory, remote databases, and multiple tables on disk processing it using SQL through multiple explicit and/or implicit code paths in order to present it to the requesting application. A create may allocate storage, provision structures, assign values, and do it's own processing before storing data. Etc.

A foreign key can not be "conditional". The only "exception" to that rule are null values which can't reference another table by definition.

If I understand your question correctly, you are trying to implement a constraint that says "if refunded is true then refund_id must reference an existing transaction".

I don't think you need the refunded column at all. Because the flag refunded can be derived from the value of the refund_id using the expression: refund_id is not null.

If you do want a column like that, just create a view with that expression.

If you do insist on having the redundant refunded flag, you could setup a check constraint like this:

CREATE TABLE transactions(
    id SERIAL PRIMARY KEY,
    type TXN_TYPE NOT NULL,
    amount BIGINT,
    . . . .,
    refunded boolean DEFAULT FALSE,
    refund_id integer null REFERENCES transactions,
    constraint check_refund 
       check ( (refunded and refund_id is not null or
               (not refunded and refund_id is null) ) 
);