Db2 – IBM DB2 v9.7 HADR TSA failure on the standby server

This is actually an interesting scenario because I addressed this in the DBA StackExchange back on March 29, 2011.

The basic idea would be to use MySQL Replication in conjunction with DRBD and ucarp

Here is what you would need:

• Two DRBD Clusters, one in each DataCenter
• DRBD Pair at each DataCenter connected via a Crossover Cable (perhaps using 192.168.x.x subnet)
• DRBD Primary has MySQL Running
• DRBD Primary at one DataCenter running MultiMaster Replication with the DRBD Primary at other DataCenter via ucarp (in both directions)
• Any needed read slaves at each Data Center would the DBVIP established for use within MySQL at that DataCenter via ucarp

Having this setup gives you block-level replication within both data centers. If you are using MySQL 5.5, you can have Semisynchronous Replication send SQL to the other DataCenter without waiting for the SQL to be executed, only acknowledged. This keeps any intermittency that standard MySQL Asynchronous Replication would normally cause a MySQL Master to a minimum. In the event of any sudden network latency, MySQL Replication between DataCenters would switch from Semisync to Async. Once conditions improve, then MySQL Replication would it switch back to Semisynch.

CAVEAT

This setup will not prevent table corruption, particularly if MyISAM is involved. Even with DRBD, block-replication of a corrupt MyISAM table would produce...you guessed it, a corrupt MyISAM table in the DRBD Secondary. Therefore, it is preferable if all user data were InnoDB.

I wrote up an interesting layout last year which features DRBD pairs in two data centers (DC1,DC2) with as follows

• DRBD Pair in DC1 (db1 and db2)
• DBVIP for Primary of DRBD Pair1 is 10.1.2.30
• DRBD Pair in DC2 (db3 and db4)
• DBVIP for Primary of DRBD Pair2 is 10.1.2.40
• Have MySQL Circular Replication Between DRBD Primaries
• Have the 10.1.2.40 as Master_Host for DBRD Pair 1
• Have the 10.1.2.30 as Master_Host for DBRD Pair 2

MySQL high availability, failover and replication with Latency

MySQL Replication : 1 Slave / Multiple Masters

Here is why I suggested this: Using two data centers, you setup automatic failover for DRBD Pair in one data center. Let the other DRBD Pair in the second data center be for DB disaser site with it own local redundancy and failover. Should you ever loses one data center, the other data center is fully read with it own local failover setup. Your app would just have to use the DBVIP of the other database center in such a catastrophic case.

Please keep in mind that using DRBD in conjunction with MySQL is only beneficial if all of your data uses the InnoDB Storage Engine. Hard failovers in DRBD could easily result in crashed MyISAM tables.

Here is another setup to consider:

As with DRBD setups, any DRBD Secondary would provide just a Disk-Level copy of your MySQL Folder. It is available as a warm standby. MySQL is not being run on the DRBD Secondary. If you want the third DB server to become hot standby, ditch DRBD altogether and use pure MySQL Replication. With three DB servers, using db3 at a remote site, simply setup the following:

+--> db1 --> db2 --> db3 -->+
^                           |
|                           V
+<--------------------------+

Using your rudimentary failover, now you have two hot standby servers. You just have to make sure each DB server has a unique server_id value. I also recommend using MySQL 5.5 because it uses SemiSync Replication which is more sensitive to communication failures and stop Replication better. You will have to setup the appropriate heartbeats and timeouts.