Mysql – Breaking Semisynchronous Replication in MySQL 5.5

MySQLmysql-5.5replicationsemi-sync-replication

I've set up Semisynchronous Replication between two MySQL 5.5 servers running on Windows 7.

My application is running and updating the database of the master server and same is being updated in the slave database server.

But due to some unknown reasons sometimes, Replication breaks.

On running the command:

SHOW STATUS LIKE 'Rpl_semi_sync%';

It gives this status:

'Rpl_semi_sync_master_no_times', '0'
'Rpl_semi_sync_master_no_tx', '0'
'Rpl_semi_sync_master_status', 'ON'     <<-------------
'Rpl_semi_sync_master_timefunc_failures', '0'
'Rpl_semi_sync_master_tx_avg_wait_time', '338846'
'Rpl_semi_sync_master_tx_wait_time', '29479685'
'Rpl_semi_sync_master_tx_waits', '87'
'Rpl_semi_sync_master_wait_pos_backtraverse', '0'
'Rpl_semi_sync_master_wait_sessions', '0'
'Rpl_semi_sync_master_yes_tx', '3106'

Ideally, in semi synchronization, when the sync breaks the status should come as OFF since master is not able to receive any acknowledgement from the slave.
Please help us in this regard.

Best Answer

The one thing you can probably do is to increase the sensitivity of the acknowledgement

Please look for this variables in /etc/my.cnf

[mysqld]
rpl_semi_sync_master_timeout=5000

When rpl_semi_sync_master_timeout (Default is 10000 or 10 seconds) is set to 5000, Replication should switch a Master from SemiSync to Async if no acknowledgement is received from the Slave in 5 seconds (5000 milliseconds). You may want to lower this even less than 5000.

You may need to check your network performance. As long as you not doing geographic distance replication, you should the following: On a Separate NIC, use a crossover cable over 192.168.xx.xx so as not have faster replication response. Also, check the switch for dropped packets.

You should not have to restart the server every time. As a quick-and-dirty band-aid, make up this SQL script:

STOP SLAVE IO_THREAD;
SELECT SLEEP(5);
START SLAVE IO_THREAD;

Simply write a cronjob to execute these commands every 5 minutes. Again, this is a band-aid. Otherwise, due diligence on the networking side is in order.

Related Solutions

Mysql – Replication Broken, List Out of Sync Tables

With regard to the first point

check (and alert) replication is broken?

just check the following

MYSQL_CONN="-uroot -ppassword"
SLAVESTATUS_OUTPUT=/tmp/ShowSlaveStatus.txt
mysql ${MYSQL_CONN} -ANe"SHOW SLAVE STATUS\G" > ${SLAVESTATUS_OUTPUT}
IO_RUNNING=`grep "Slave_IO_Running" < ${SLAVESTATUS_OUTPUT} | awk '{print $2}'`
SQL_RUNNING=`grep "Slave_SQL_Running" < ${SLAVESTATUS_OUTPUT} | awk '{print $2}'`

Just check for both IO_RUNNING and SQL_RUNNING being Yes.

If either value is No, check the error message.

You could check for other things such as Seconds_Behind_Master ( Does `Seconds_Behind_Master` show exact Slave Lag from Master? )

With regard to the second point

get a list of tables that are out of synch across various databases?

The best tool in the world for this is pt-table-checksum. Please read the documentation on it because the output will be all the tables in the specified databases. You will have to parse the text output to compare the checksum values between Master and Slave.

I wrote a part post about using this tool:

In a MySQL Master/Slave replication, what would happen if I write to the Slave?
Fixing tables out of sync as reported by pt-table-checksum, pt-table-sync is not working (This one has shell scripting I made to do the parsing for you)

Mysql – Replicate MySQL 5.0 Master to 5.5 Slave

I have done that dozens of times for my employer's clients. In fact, I just did this a week ago. The client was using MySQL 5.0.81 and had degraded performance in InnoDB.

I create a new DB Cluster
- running MySQL 5.5.27
- Two DB Servers (ServerA and ServerB)
- Configure InnoDB for Multicore Engagement
Master/Slave (ServerA Master and ServerB Slave)
I enable binary logging and log_slave_updates on Production's Slave
mysqldump all databases (except mysql schema) from Production's Slave to /root/MySQLData.sql
run pt-show-grants on Production's Master to /root/MySQLGrants.sql
load ServerA with /root/MySQLGrants.sql
load ServerA with /root/MySQLData.sql
I replicate from Production's Slave (MySQL 5.0.81) to ServerA (MySQL 5.5.27)

All that was left to do was

Have the client shutdown the app
Remove DBVIP from Master running MySQL 5.0.81
Bring up DBVIP on Master running MySQL 5.5.27
Have the client start up the app

CONCLUSION

MySQL 5.5 can handle replication from previous major releases, and not the other way around.

I have discussed this before

Feb 04, 2011 : MySQL master binlog corruption
Jun 09, 2011 : Yum update mysql from 5.1 to 5.5
Feb 08, 2012 : will replication from 5.5.20 to 5.0.XX server work?

GOOD NEWS !!!

I already answered a post back on Feb 06, 2012 ( How to setup replication(Master/slave) in MySQL 5.5.20? ) with all the steps you need.

BTW with regards to the step about dumping the MYSQL grants as SQL commands, here is my personal emulation of pt-show-grants:

mysql -hhostaddr -umyuserid -pmypassword --skip-column-names -A -e"SELECT CONCAT('SHOW GRANTS FOR ''',user,'''@''',host,''';') FROM mysql.user WHERE user<>''" | mysql -hhostaddr -umyuserid -pmypassword --skip-column-names -A | sed 's/$/;/g' > MySQLUserGrants.sql

Give it a Try !!!

UPDATE 2012-11-27 11:07 EDT

I wanted to add additional posts I made for setting up Circular Replication should you decide to setup the two DB servers as Master/Master

Sep 24, 2012 : Setting up MySQL circular replication in existing replication topology?
May 07, 2012 : Setting Circular Replication in mysql