MySQL – Master to Master Replication Without Downtime

master-master-replicationmulti-masterMySQLreplication

I want to setup 2 node master-master replication. Im aware of the point of failiures in this replication. Its a huge database around 1.5TB.
Heavy OLTP is going on. But its mandatory to implement right now.

I have 2 servers.

 1. Master1 
 2. Master2

Initially what I did is,

Master1:

Make changes my.cnf for replication.
Take mysqldump with –master-data=2 (For getting exact binlog
position)

Master2:

Restore dump to Master2.
Make changes on my.cnf.
set replication to Master1 (Change master='Master1').
Start Slave.

Now Master1 is replicating to Master2.

Then I need to replicate the Master2 to Master1.
Just take the binlog position and set replication on Master1 will work, but the problem is binog file and position is changing frequently.
So before execute the Change master command on Master1 the binog get changed.

How can I achive this without downtime.

Best Answer

I am glad you were able to mysqldump 1.5TB database. You are sure patient.

Now for the reality check. You will some downtime for reboots and pausing.

No need to mysqldump anymore.

STEP 01

Did you put this line in /etc/my.cnf on both Master1 and Master2 ?

log-slave-updates

If you not sure, run this on both Master1 and Master

select @@global.log_slave_updates;

If you get this on both Master1 and Master2

mysql> select @@global.log_slave_updates;
+----------------------------+
| @@global.log_slave_updates |
+----------------------------+
|                          1 |
+----------------------------+
1 row in set (0.00 sec)

then goto STEP 02.

This needs to be in /etc/my.cnf on both Master1 and Master2.

If you did not, please add it and restart mysqld on both Master1 and Master2.

STEP 02

Did you enable binary logging on Master2 ?

You should have a line in Master1's /etc/my.cnf like this

log-bin = ....

Please make sure Master2 has it.

If you are not sure if it has it, login to MySQL on Master2 and run

mysql> show binary logs;

If it echoes

mysql> show binary logs;
ERROR 1381 (HY000): You are not using binary logging

Then, add that log-bin line to /etc/my.cnf in Master2 and restart mysql in Master2.

Once log-slave-updates and log-bin have been enabled on both Master1 and Master2, you are ready to setup replication in the other direction

STEP 03

On Master2

STOP SLAVE;
FLUSH TABLES;
FLUSH LOGS;
SHOW BINARY LOGS;

STEP 04

On Master1

CHANGE MASTER TO
    MASTER_HOST='IP or Hostname of Master2',
    MASTER_PORT=3306,
    MASTER_USER='...',
    MASTER_PASSWORD='...',
    MASTER_LOG_FILE='First Binary Log On Master2',
    MASTER_LOG_POS=4
;
SHOW SLAVE STATUS\G

STEP 05

Test Connectivity from Master1 to Master2

On Master1

START SLAVE IO_THREAD;
DO SLEEP(10);
SHOW SLAVE STATUS\G

Look for Slave_IO_Running. It should be 'Yes'.

Run SHOW SLAVE STATUS\G like 3-5 times. Relay_Log_Space should be increasing.

STEP 06

Start processing SQL from Relay Logs

Run this on Master1

START SLAVE SQL_THREAD;
DO SLEEP(10);
SHOW SLAVE STATUS\G

Look for Slave_SQL_Running. It should be 'Yes'.

Look for Seconds_Behind_Master. It should be 0. If it greater than 0, keep running SHOW SLAVE STATUS\G until Seconds_Behind_Master is 0.

Related Solutions

MySQL – Master Binlog Corruption

Surprisingly, that's not gibberish.

That indeed appears at the top of binlogs whenever you do mysqlbinlog to a binary log generated using MySQL 5.1 and MySQL 5.5. You will not see that gibberish in binary logs for MySQL 5.0 and back.

This is why the start point for replication from an empty binary log is

107 for MySQL 5.5
106 for MySQL 5.1
98 for MySQL 5.0 and back

This is good to remember if you do MySQL Replication where the Master if MySQL 5.1 and the slave is MySQL 5.0. This could present a really big headache.

Replication from Master using 5.0 and Slave using 5.1 works fine, not the other way around.(According to MySQL Documentation, it is generally not supported for 3 reasons: 1) Binary Log Format, 2) Row-based Replication, 3) SQL Incompatibility).

Anyway, do a mysqlbinlog on the offending binary log on the master. If the resulting dump produces gibberish in the middle of the dump (which I have seen a couple of times in my DBA career) you may have to skip to position 98 (MySQL 5.0) or 106 (MySQL 5.1) or 107 (MySQL 5.5) of the master's next binary log and start replicating from there (SOB :( you may need to use MAATKIT tools mk-table-checksum and mk-table-sync to reload master changes not on the slave [if you want to be a hero]; even worse, mysqldump the master and reload the slave and start replication totally over [if you don't want to be a hero])

If the mysqlbinlog of the master is completely readable after the top gibberish you saw, it is possible the master's binary log is fine but the relay log on the slave is corrupt (due to transmission/CRC errors). If that's the case, just reload the relay logs by issuing the CHANGE MASTER TO command as follows:

STOP SLAVE;
CHANGE MASTER TO
MASTER_HOST='< master-host ip or DNS >',
MASTER_PORT=3306,
MASTER_USER='< usernmae >',
MASTER_PASSWORD='< password >',
MASTER_LOG_FILE='< MMMM >',
MASTER_LOG_POS=< PPPP >;
START SLAVE;

Where

MMMM is the last file used from the Master that was last processed on the Slave
PPPP is the last position used from the Master that was last processed on the Slave

You can get MMMM and PPPP by doing SHOW SLAVE STATUS\G and using

Relay_Master_Log_File for MMMM
Exec_Master_Log_Pos for PPPP

Try it out and let me know !!!

BTW running CHANGE MASTER TO command erases the slave's current relay logs and starts fresh.

Mysql – adding another table to existing MySQL replication scenario

Are you using InnoDB? Xtrabackup will allow you to do "hot" or "online" copies of your running master.

Propagate the backup made via xtrabackup to the slave. There is no need to disrupt the relay (blackhole) server, just add the extra tables to be replicated to the relay server config before you "START SLAVE" on your slave.