MySQL slave is continuously lagging behind master

MySQLreplication

I was given two fairly new servers and setup master and slave running MySQL 5.7.17, Linux RedHat 6. Both servers are using binlog_format=STATEMENT.

I can't understand why my slave is processing so slowly though I know the bottleneck with the single thread process from my slave. I believe our master server does not contain a lot of parallel queries though.

Slave > show processlist;

Id: 22593 User: system user Host: db: Database1 Command: Connect Time: 2364340 State: updating Info: update TableA set colA.....
This transaction is processing like 0.5s for every update. The total rows count for the table is just under 100,000 rows. I am not sure if my expectation is right that it should be fast.
I turn on general_log to capture for several seconds and switch off back.
Excerpt:
2016-12-28T10:26:53.937824Z 22093 Query COMMIT /* implicit, from Xid_log_event */ 2016-12-28T10:26:53.938293Z 22093 Query BEGIN 2016-12-28T10:26:54.404481Z 22093 Query update Table10 set COLA='XXX', COLB='XXXX', COLC='', COLD='', COLE='XXXX', COLF='XXXX' where COLB='YY' and COLF='ZZZZZZ' 2016-12-28T10:26:54.404576Z 22093 Query COMMIT /* implicit, from Xid_log_event */

slave > show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: masterA Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: masterA-bin.000117 Read_Master_Log_Pos: 818913948 Relay_Log_File: slaveB-relay-bin.001168 Relay_Log_Pos: 42738489 Relay_Master_Log_File: masterA-bin.000116 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 42738264 Relay_Log_Space: 1939474024 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 60598 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 10 Master_UUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXxx Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: updating Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec)

masterA> show master status\G;
*************************** 1. row ***************************
File: masterA-bin.000117
Position: 818913948
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set: 1 row in set (0.00 sec)

Based on the readings, my I/O thread is doing not too bad. Catching up with master log at masterA-bin.000117. But the SQL thread is lagging behind. masterA-bin.000116. Relay_Log_Pos: 42738489

SQL thread is replaying at slaveB-relay-bin.001168. But when I check my OS directory /log, I can see slaveB-relay-bin.001341 as the latest generated.

The slave is continuously lagging behind master. I have set

sync_binlog=1
Comment log-slave-updates
innodb_flush_log_at_trx_commit=2

But it helps a little only, reducing probably 1-2 seconds of lag.

I am using STATEMENT binlog, will it matters if some tables does not have primary/unique key?

I'd to know if there are other means to speed up the replication without multithreaded slave?

Best Answer

Let's try to speed up the queries.

Table10 needs a composite INDEX(COLB, COLF) (in either order). If it already has that, then let's see some more of the 'slow' queries.

On the slave, turn on log_slow_slave_statements, lower long_quer_time to 0.5, turn on the slowlog, preferably to a FILE. Wait a day, then use mysqldumpslow -s t or pt-query-digest to summarize the slowlog. Then let's discuss the first few.

Related Solutions

Mysql – Create a thesql slave database from master database

Going by your question, I will like to review what I believe you did thus far:

You stopped mysql on the Master
You copied Master's /var/lib/mysql to the Slave's /var/lib/mysql
I surmise the binlogs on the Master were copied as well

Look at the Slave's last binlog. From the question, it should be

mydbm1-bin.008524
Filesize 1330529

Believe it or not, you have to do a few things:

1) On the Master, create a replication user like this:

GRANT REPLICATION SLAVE,REPLICATION CLIENT
ON *.* TO replicator@'%'
IDENTIFIED BY 'r3plic4t0R';

2) Make /var/lib/mysql on the Slave owned by mysql user

chown -R mysql:mysql /var/lib/mysql

3) Make sure Master's server_id is explicitly set in my.cnf

[mysqld]
server_id = 1

4) Make sure Slave's server_id is explicitly set in my.cnf

[mysqld]
server_id = 2

5) Startup mysql on the Slave

service mysql start

6) Setup replication by running this on the Slave

CHANGE MASTER TO
MASTER_HOST='IPAddressOfMaster',
MASTER_PORT=3306,
MASTER_USER='replicator',
MASTER_PASSWORD='r3plic4t0R',
MASTER_LOG_FILE='mydbm1-bin.008524',
MASTER_LOG_POS=1330529;
START SLAVE;
SELECT SLEEP(5);
SHOW SLAVE STATUS\G

You will see something like this:

mysql> show slave status\G
*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 10.48.20.253
                Master_User: replicant
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysql-bin.000254
        Read_Master_Log_Pos: 858190247
             Relay_Log_File: relay-bin.066069
              Relay_Log_Pos: 873918
      Relay_Master_Log_File: mysql-bin.000254
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB:
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 858190247
            Relay_Log_Space: 873772
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
          Master_SSL_Cipher:
             Master_SSL_Key:
      Seconds_Behind_Master: 0
1 row in set (0.00 sec)

If Slave_IO_Running and Slave_SQL_Running are both Yes, CONGRATULATIONS !!!

I already answered a post back on Feb 06, 2012 ( How to setup replication(Master/slave) in MySQL 5.5.20? ) with essentially the same steps.

I wanted to add additional posts I made for setting up Circular Replication should you decide to setup the two DB servers as Master/Master

Sep 24, 2012 : Setting up MySQL circular replication in existing replication topology?
May 07, 2012 : Setting Circular Replication in mysql

MySQL Slave not updated

The replication filter you have in place

Replicate_Wild_Do_Table: zo_dev_matrix.*

is actually a little misleading. In the MySQL Documentation, the legal characters for wildcards are %, _ and \_. (If you want to interpret a literal underscore). The asterisk character is not listed. The above filter is actually looking for a table called zo_dev_matrix.*. Therefore, you have two options

OPTION #1

Correct the filter in my.cnf to have this

[mysqld]
replicate_wild_do_table = zo_dev_matrix.%

OPTION #2

Change the filter to the database zo_dev_matrix

replicate_do_db = zo_dev_matrix

only if all your queries do no preface the table with the db and you explicit set the current database to zo_dev_matrix.

CAVEAT

With MySQL 5.6 and back, you must restart mysql after changing the filter in my.cnf. Starting with MySQL 5.7, you can create a dynamic replication filter so a mysql restart is not necessary.