Mysql – Deleting bin logs in replication environment

MySQLmysql-5mysql-5.5replication

I have a question about deleting binary logs in the replication environment:

We have an environment with 1 master and 2 slaves (running mysql 5.5). Sometimes, we run into a space issues during heavy processing times, whereby bin log directory gets full. Logs are expired every 3 days. I was wondering, is there a reason why logs should be kept for 3 days on all the boxes – master and both slaves? Would it make sense to, for example, keep logs for 3 days on a master, but for 1 day on slaves? What is the best way to go about it?

Thank you!

Best Answer

SLAVE

If your Slaves are not Masters, then Slaves do not need binary logging at all. You can put a cap on the amount of relay log space accumulated by a Slave. In order to throttle relay logs at 4G, add relay_log_space_limit to /etc/my/.cnf on every Slave

[mysqld]
relay_log_space_limit=4G

and restart mysql

If you cannot set this, at least you should have some kind of alerting that does SHOW SLAVE STATUS\G and check the value of Relay_Log_Space (total bytes consumed by relay logs).

MASTER

As for the Master, you could set expire_logs_days to 1, but there is a severe warning I have for you...

If replication breaks, you have 1 day to fix it. Otherwise, a binary log on the Master may rotate away and you cannot run any CHANGE MASTER TO command to realign replication. I would leave expire_logs_days at 3 on the Master.

SUGGESTION #1

If you have any overnight bulk processing to do, maybe should run the bulk processes on on the Master with SET SQL_LOG_BIN=0; at the Start of the Session. This, of course, will not replicate to the Slave. You can perform the Same Bulk Load in Parallel to both Slaves.

SUGGESTION #2

Another thing you could do to manage the Master binary logs accumulation is this.

Run SHOW SLAVE STATUS\G on both Slaves. Look at Relay_Master_Log_File. That represents the binary log on the Master whose last command was executed on the Slave.

*************************** 1. row ***************************
             Slave_IO_State: Waiting for master to send event
                Master_Host: 10.4.92.250
                Master_User: replicant
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysql-bin.009677
        Read_Master_Log_Pos: 855227755
             Relay_Log_File: relay-bin.000674
              Relay_Log_Pos: 757296783
      Relay_Master_Log_File: mysql-bin.009590
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB:
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 757296646
            Relay_Log_Space: 94274010765
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
          Master_SSL_Cipher:
             Master_SSL_Key:
      Seconds_Behind_Master: 80561
1 row in set (0.00 sec)

In this example, Relay_Master_Log_File is mysql-bin.009590. All binary logs before this one can be removed from Master. You could run this on the Master:

PURGE BINARY LOGS TO 'mysql-bin.009590';

This will erase older logs and still leave replication in tact.

CAVEAT

Binary Logs are files that serially compiles (like a FIFO queue) all completed SQL transactions as either a SQL statement or a row change. A relay log is a file that collects binary log entries from a remote server (aka Master).

In MySQL Replication

Master must have its binary logs enabled
Slave compiles relay logs
When all SQL in a relay log is processed, it is deleted
On a Slave, when there is more that one relay log on a DB Server, it may indicate replication is falling behind because the IO thread is collecting SQL from a Master faster that the SQL thread can process the relay logs.
Using relay_log_space_limit prevents replication from piling up and potentially filling up a disk. Relay logs rotate out based on rule #3
It is possible for a DB Server to be both a Master and Slave. That's the only circumstance under which a Slave must have binary logs enabled. In that scenario, a DB Server will have both Binary Logs and Relay Logs.

If you failover to a Slave, and you want to Make it a Master

service mysql stop
Add log-bin=mysql-bin to /etc/my.cnf on the Slave
service mysql start

You will have to setup replication of other Slaves to the newly promoted Master and make sure the data on the Slave match up with the newly promoted Master

UPDATE 2012-08-13 17:47 EDT

According to the MySQL Documentation on relay-log option, you should define it. Here is why:

Due to the manner in which MySQL parses server options, if you specify this option, you must supply a value; the default basename is used only if the option is not actually specified. If you use the --relay-log option without specifying a value, unexpected behavior is likely to result; this behavior depends on the other options used, the order in which they are specified, and whether they are specified on the command line or in an option file. For more information about how MySQL handles server options, see Section 4.2.3, “Specifying Program Options”.

Question 1

Does the DML operations committed by db2 during the replication process gets included in its own bin-log?

Answer to Question 1

Yes it will, provided you have this in /etc/my.cnf on both db1 and db2

[mysqld]
log-slave-updates

If you do not have this, add it and restart mysql

Question 2

Would the resulting bin-log in db2 be exactly the same with the bin-log of db1, to the letter?

Answer to Question 2

Yes. Make sure the clocks on both DB servers are synchronized

Question 3

What happens to the entries in db2 relay-log once they are committed to the database during the replication process, are they discarded? What role does the relay-log info log has in this?

Answer to Question 3

In MySQL Replication, the IO Thread of a Slave will read its Master's bin-log entries and store them in a FIFO queue. For each relay log in a slave, when every entry in the currently processed relay is executed it is rotated out and discarded. If relay logs are piling up, this quickly indicates that the SQL thread died because of any SQL error. Just do SHOW SLAVE STATUS\G to find out what stopped the SQL thread. The IO Thread would conitnue collecting completed SQL statements from its Master.

Question 4

How does db1 know where in the bin-log of db2 (somehow dependent on the answer of Question 2), it will start the replication process?

Answer to Question 4

When you do SHOW SLAVE STATUS\G, look for the following lines:

Master_Log_File : The latest binary log whose most recently command was copied to the Slave
Read_Master_Log_Pos : The latest position of the latest binary log whose most recently command was copied to the Slave
Relay_Master_Log_File : The latest binary log whose most recently command was executed on the Slave
Exec_Master_Log_Pos : The latest binary log whose most recently command was executed on the Slave
Relay_Log_Space : The sum total (in bytes) of all relay logs. By default, each relay log is the default size of a binary log (1G). If Relay_Log_Space starts to significant exceed 1G, this indicates one of two things:
- SQL thread died due to SQL Error
- SQL thread is busy with a long-running query

Question 4.1

If you enable log-slave-updates on both databases i.e. dB1 & dB2, then that would mean all items from the binary log of dB1, which was successfully replicated by dB2 will be written into dB2's binary log and vice-versa. Would not this result to some sort of infinite circular replication or duplications of entries on both databases, if it's possible at all, considering the possible key-collision issues that would arise? What I'm trying to say is, How would dB1 know once it checks on the binary log of dB2 that, "I should not replicate those entries in there because they all just came from me"?

Answer to Question 4.1

You must have log-slave-updates available on both DB servers in order to have an audit trail that the SQL executed on on DB server made it to the other. If you don't, you would have to do your due diligence to compare the data explicitly. Such ways would include:

Running CHECKSUM TABLE on every table you have in both DB servers to compare their contents.
Using pt-table-checksum, which is an automated version of running CHECKSUM TABLE between Master and one or more Slaves

You need not worry about infinite circular replication unless you are dealing with more that two masters. There have been rare times when someone with, let's say four Masters, removes one of the four servers from circular rep cluster. Let's suppose the the server_id is 13. It is remotely, but still, possible for binary log entries whose server_id belongs to the server that removed to be inside the relay logs on other servers. Only in such a scenario would you worry about infinite circular replication.

To circumvent such situations, MySQL 5.5 has a new option for the CHANGE MASTER TO command called IGNORE_SERVER_IDS. You would do the following to repair things on all the remaining servers:

STOP SLAVE;
CHANGE MASTER TO IGNORE_SERVER_IDS = (13);
START SLAVE;

In fact, here is what the MySQL Documentation says on this:

IGNORE_SERVER_IDS was added in MySQL 5.5. This option takes a comma-separated list of 0 or more server IDs. Events originating from the corresponding servers are ignored, with the exception of log rotation and deletion events, which are still recorded in the relay log.

In circular replication, the originating server normally acts as the terminator of its own events, so that they are not applied more than once. Thus, this option is useful in circular replication when one of the servers in the circle is removed. Suppose that you have a circular replication setup with 4 servers, having server IDs 1, 2, 3, and 4, and server 3 fails. When bridging the gap by starting replication from server 2 to server 4, you can include IGNORE_SERVER_IDS = (3) in the CHANGE MASTER TO statement that you issue on server 4 to tell it to use server 2 as its master instead of server 3. Doing so causes it to ignore and not to propagate any statements that originated with the server that is no longer in use.

Question 5

On INSERT queries on the master, what form of the query is written into the binary log? Is it the 'raw' form of the query, or the one which already has the auto-generated value of the auto-increment key?

Answer to Question 5

Whichever form is presented. Here is what I mean: The raw form would usually not include the auto_increment column expressed explicitly. On the other hand, it you import a mysqldump into a DB server with binary logging, the rows being inserted would explicitly be given. Either version of INSERT would be allowed execution in mysqld. In like fashion, either version of INSERT would be recorded AS IS...