Mysql – Synchronise data from one table to another & vice-versa using triggers

MySQLtrigger

We are migrating from one database structure to another – very slowly. The system is MySQL. There are two databases. For example's sake, we will call the databases old_db and new_db. Both databases have tables that will need to be synchronised for around a year whilst the business's application is migrated from the old db to the new db.

Again to slim down the example, we have one table in the old database, called: old_db . myoldertable, new_db . mynewtable.

Structure:

old_db.myoldertable
primary_id, Date, amount, unneedded_column1, unneedded_column2 ... and so-on

new_db.mynewtable
primary_id, date, amount, old_table_primary_id

They look similar in structure, but this is simplified.

Problem:

If I put an insert/update/delete trigger on both tables, I will end up in a recursion loop. This is because the tables need to be synchronised, so the insert / update / delete triggers need to be on both tables.

Requirements

Must be 100% MySQL. Cannot use a programming language for this.
The new and old database tables must be synchronised immediately if either have rows that are inserted/updated/deleted.
Daily / hourly updates are NOT an option. Again, must be synchronised immediately.

Solutions?

Ideas to break possibility of recursion for all three insert / update / delete scenarios.
Any other ideas.

EDIT :

So far I have looked at

One solution I have explored is replacing the old table with a view. However, our new database has two tables (or more) where the old database had only one table. Views, in MySQL cannot be writeable to more than one base table. Also views cannot have triggers in MySQL.
- I also looked at using a UUID that represents the last update. I would create a column in both tables to contain a Unique ID. Making sure that both tables contain the same UUID as part of the triggers. However, this does not stop recursion.

Best Answer

I concur with @NathanJolly, this is a potential data nightmare and seems ill-advised... but there is a way to implement this that is, at least conceptually, fairly straightforward.

You need not worry about the possibility of recursion in triggers, because this won't happen... MySQL doesn't support it.

I built this fiddle that creates tables t1 and t2, each of which has an AFTER INSERT trigger that tries to insert the record into the other table... then tries to do an insert. Theoretically, there's a potential for recursion, but MySQL prevents this:

ERROR 1442 (HY000): Can't update table 't1' in stored function/trigger because it is already
                    used by statement which invoked this stored function/trigger.

So it won't run in a loop, but it still throws an exception. Technically, it's not trigger recursion causing the exception, because we haven't actually gotten that far... it's the fact that the trigger in t2 tries to modify table t1, which was involved in the currently-executing statement.

Then again, technically, it's not just the recursion of the triggers we need to avoid -- we have to stop one step before that -- if the client modifies t1, the t1 trigger modifies t2, then t2's trigger needs to not even try to do anything to t1 (including, but not limited to, firing the trigger again on t1).

Implementing this is actually fairly straightforward... in each trigger, we toggle the value of a session to tell the other trigger not to run.

CREATE TRIGGER t1_ai AFTER INSERT ON t1 FOR EACH ROW BEGIN
  IF @__disable_trigger_t1t2 = 1 THEN
    SET @__disable_trigger_t1t2 = NULL;
  ELSE
    SET @__disable_trigger_t1t2 = 1;
    -- trigger logic goes in here
    INSERT INTO t2 (id,things) VALUES (NEW.id,NEW.stuff);
  END IF;
END //

CREATE TRIGGER t2_ai AFTER INSERT ON t2 FOR EACH ROW BEGIN
  IF @__disable_trigger_t1t2 = 1 THEN
    SET @__disable_trigger_t1t2 = NULL;
  ELSE
    SET @__disable_trigger_t1t2 = 1;
    -- trigger logic goes in here
    INSERT INTO t1 (id,stuff) VALUES (NEW.id,NEW.things);
  END IF;
END //

A test case for this logic working can be seen in this fiddle.

Essentially, when a trigger fires, it checks the value of the @__disable_trigger_t1t2 variable (this variable name is, of course, arbitrary). If '1', it sets it back to NULL (the default for session variables) and does no further processing, because the '1' means this row is being manipulated by the trigger on the other table. If already NULL, then it sets the value to 1 and executes the trigger logic -- which will update the other table, whose trigger will see the '1', will not execute its own logic, and will reset the variable back to NULL, so that the next execution of either trigger -- whether for the next row in the same statement, or for a subsequent statement -- will see the NULL and execute normally.

In MySQL, triggers are only supported FOR EACH ROW, meaning that we can define a MySQL trigger as "a stored program that is executed before or after each row is inserted, updated, or deleted"... which means that no matter how many rows your query affects, the trigger will run once for each row. Actions against other tables will cause the triggers on those tables to fire, and the execution of the trigger on t1 for the first row will not complete until all of the triggers it caused to fire have also completed... and this is why we can use this variable like this -- only one row is actually being handled at a time, so we set and clear this variable for each row, always leaving it NULL after each statement terminates.

A significant caveat is that this approach fails if you need to trigger on cascading deletes or updates related to foreign keys, because:

"Currently, cascaded foreign key actions do not activate triggers."

^{-- http://dev.mysql.com/doc/refman/5.6/en/create-trigger.html}

So, while I have misgivings about the long-term viability of maintaining database synchronization with this approach because of the number of things that could go wrong with two applications manipulating each other's data or logical errors in the trigger definitions... it seems technically possible.

You also mentioned:

views cannot have triggers in MySQL

No, but updating an updatable view will fire the triggers on the underlying base table as if the table had been directly manipulated.

Related Solutions

Mysql – Disable trigger for just one table

Actually, if you place an if then block in every trigger, you could effectively shutdown all triggers. Here is such a code block

IF @TRIGGER_DISABLED = 0 THEN
    ...trigger body
END IF;

In the mysql environment, you could

run SET @TRIGGER_DISABLED = 1;
do your data maintenance
run SET @TRIGGER_DISABLED = 0;

So your trigger for table A should look like this:

BEGIN 
    IF @TRIGGER_DISABLED = 0 THEN
    IF (OLD.status != 1 AND NEW.status = 2) THEN 
        IF (OLD.geo_lat IS NOT NULL AND OLD.geo_long IS NOT NULL) THEN 
            DELETE FROM geo WHERE datatype IN (3,4) AND foreignid = NEW.id; 
        END IF; 
    ELSEIF (OLD.Status = 1 AND NEW.Status != 2) THEN 
        IF (NEW.geo_lat IS NOT NULL AND NEW.geo_long IS NOT NULL) THEN 
            INSERT INTO geo (datatype, foreignid, long, lat, hostid, morton, status) VALUES  (IF(NEW.groupType=1,3,4), NEW.id, NEW.geo_long, NEW.geo_lat, NEW.hostid, 0, NEW.Status); 
        END IF; 
    ELSEIF (NEW.status != 3) THEN   
        IF (OLD.geo_lat IS NOT NULL AND OLD.geo_long IS NOT NULL AND (NEW.geo_lat IS NULL OR NEW.geo_long IS NULL)) THEN 
            DELETE FROM geo WHERE datatype IN (3,4) AND foreignid = NEW.id; 
        ElSEIF ((OLD.geo_lat IS NULL OR OLD.geo_long IS NULL) AND NEW.geo_lat IS NOT NULL AND NEW.geo_long IS NOT NULL) THEN 
            INSERT INTO geo (datatype, foreignid, longitude, latitude, hostid, morton, status) VALUES  (IF(NEW.groupType=1,3,4), NEW.id, NEW.geo_long, NEW.geo_lat, NEW.hostid, 0, NEW.Status); 
        ELSEIF (OLD.geo_lat!=NEW.geo_lat OR OLD.geo_long != NEW.geo_long OR OLD.status != NEW.status) THEN 
            UPDATE geo SET lat = NEW.geo_lat, long = NEW.geo_long, status = NEW.status WHERE datatype IN (3,4) AND foreignid = NEW.id; 
        END IF; 
    END IF; 
    END IF; 
END

So your trigger for table B should look like this:

CREATE TRIGGER `usergroups_comments_insert` AFTER INSERT ON `usergroups_comment`     
    FOR EACH ROW     
    BEGIN     
        IF @TRIGGER_DISABLED = 0 THEN
            CALL sp-set-comment_count(NEW.`gid`);     
        END IF;
    END;

If you want the triggers for table A to launch but not to table B, then add the code block only to table B's trigger.

MySQL Triggers Different Database Different Server

There's the FEDERATED storage engine, which makes for a way to create a "proxy" table in your database server, which points to a real table on another server.

FEDERATED is disabled by default. To enable it, add this to your my.cnf configuration file, under the [mysqld] section:

federated = 1

So your trigger on some table T would then INSERT into your FEDERATED table FT, which would connect to the remote server, pass credentials, and perform the operation.

Do note the overhead: triggers are by themselves performance killers. Add to that the connection to the remote machine, and you now also have latency to worry about.