There are three(3) suggestions I have that you may want to look into...
SUGGESTION #1
You could use MySQL 5.5 both locally and in AWS. You would also setup Semisync Replication. That way, every SQL statement executed on the Master has its latest recorded SQL statement sent over to the Slave.
In my past post (Jul 26, 2012 : Mysql database replication on different vlan/subnet/another site) I had recommended SemiSynchronous Replication over a geographic distance. Interestingly, @AaronBrown points out by experimentation that High Latency would degrade Semisynch Replication (See his blog on this one).
SUGGESTION #2
If all your data is InnoDB, you could use Percona XtraDB Cluster (preferable to have two Slaves in AWS but one would do). Why Percona XtraDB Cluster? You have synchronized writes. You only major bottleneck would strictly be network communication (again High Latency would have an adverse affect)
Here are my past posts on using Percona XtraDB Cluster
SUGGESTION #3
If you want to strengthen each data center's local copy of the data, I would suggestion using DRBD, not over a geographic distance, but having a DRBD cluster in each data center. In your case, at least have DRBD in the local data center.
Here is my past post on this suggestion :
Given the fact that you mentioned you have Master-Master replication mode, I would not recommend any automatic failover unless you properly account for Replication Lag. After all, MySQL Replication is asynchronous.
It is theoretically possible to have the following:
DBServer1 as Master to DBServer2DBServer2 as Master to DBServer1- DBVIP pointing at DBServer1
DBServer2 is 180 seconds behindDBServer1 goes down- Automatic Failover moves DBVIP to
DBServer2
With this scenario, DBServer2 could have auto increment keys that do not exist yet. Upon failover, the DBVIP will allow WebServers to connect to DBServer2 and ask for data that does not exist yet.
This would therefore require background processes running on each DBServer.
For the above scenario:
- DBVIP is on
DBServer1
DBServer1 runs HeartBeatDBServer2 runs HeartBeat- Background Process on
DBServer1 to monitor
- a) Data Mount Availability
- b) Data Mount Writeability
- c) MySQL Connectivity
- Once a,b, or c fail, kill HeartBeat
Background Process on DBServer2 to make sure DBVIP is pingable
What should killing HeartBeat do? Trigger the startup script defined for it.
What should the startup script on DBServer2 look for?
- Loop until DBVIP is unreachable via ping
- Connect to MySQL and
- Run
SHOW SLAVE STATUS\G in a Loop until Seconds_Behind_Master is NULL
- RUn
SHOW SLAVE STATUS\G in a Loop until `Exec_Master_Log_Pos stops changing
- Assign DBVIP to
DBServer2 via ip addr add
This is essentailly the algorithm for failing over safely to a Passive Master in a Master/Master Replication Cluster.
ALTERNATIVE
If ALL your data is InnoDB, I recommend something with less rigor. Perhaps you should look into using DRBD and HeartBeat. Here is why:
DRBD provides network RAID-1 for a Block Device on two servers.
You would essentially do this:
- Have
DBServer1's DRBD Block Device as Primary
- Have
DBServer2's DRBD Block Device as Secondary
- Mount
DBServer1's DRBD Device on /var/lib/mysql
- Startup MySQL on
DBServer1
- Have HeartBeat Monitor Ping Activity Between Servers
What would startup script look like in a DRBD scenario?
- Loop until DBVIP is unreachable via ping
- Kill HeartBeart
- Disconnect DRBD
- Promote DRBD to Primary
- Mount DRBD on /var/lib/mysql
- Start MySQL (InnoDB Crash Recovery Fills in Missing Data)
- Assign DBVIP via
ip addr add
This is a lot more straightforward because only one side is Active. The Passive side (DRBD Secondary) is a Synchronous Disk Copy of the Active Side (DRBD Primary).
CAVEAT
If all or most of the working set data is MyISAM, do not touch DRBD. Crash scnearios quickly result in MyISAM tables being marked crashed and need auto-repair (which can be paintfully slow to wait for).
UPDATE 2012-12-29 08:00 EDT
Here are my past posts on using DRBD with MySQL
Best Answer
You could created a Replica Set with a Hidden Member. The hidden member will take a read/only copy of your data. However, if the hidden member goes down then the primary steps down to secondary, i.e. your MongoDB becomes read/only.
You can define an Arbiter. It is not necessary to create the Arbiter service on a dedicated host, define it on your primary node. It is just a small mongod service which consumes almost no CPU/memory resources or disc space.
If you have a Primary + Arbiter on one host and the Hidden Secondary on another host then the second may stop and your application continues running normal. Of course, is your primary goes down then the application is not available anymore.