Surprisingly, that's not gibberish.
That indeed appears at the top of binlogs whenever you do mysqlbinlog to a binary log generated using MySQL 5.1 and MySQL 5.5. You will not see that gibberish in binary logs for MySQL 5.0 and back.
This is why the start point for replication from an empty binary log is
- 107 for MySQL 5.5
- 106 for MySQL 5.1
- 98 for MySQL 5.0 and back
This is good to remember if you do MySQL Replication where the Master if MySQL 5.1 and the slave is MySQL 5.0. This could present a really big headache.
Replication from Master using 5.0 and Slave using 5.1 works fine, not the other way around.(According to MySQL Documentation, it is generally not supported for 3 reasons: 1) Binary Log Format, 2) Row-based Replication, 3) SQL Incompatibility).
Anyway, do a mysqlbinlog on the offending binary log on the master. If the resulting dump produces gibberish in the middle of the dump (which I have seen a couple of times in my DBA career) you may have to skip to position 98 (MySQL 5.0) or 106 (MySQL 5.1) or 107 (MySQL 5.5) of the master's next binary log and start replicating from there (SOB :( you may need to use MAATKIT tools mk-table-checksum and mk-table-sync to reload master changes not on the slave [if you want to be a hero]; even worse, mysqldump the master and reload the slave and start replication totally over [if you don't want to be a hero])
If the mysqlbinlog of the master is completely readable after the top gibberish you saw, it is possible the master's binary log is fine but the relay log on the slave is corrupt (due to transmission/CRC errors). If that's the case, just reload the relay logs by issuing the CHANGE MASTER TO command as follows:
STOP SLAVE;
CHANGE MASTER TO
MASTER_HOST='< master-host ip or DNS >',
MASTER_PORT=3306,
MASTER_USER='< usernmae >',
MASTER_PASSWORD='< password >',
MASTER_LOG_FILE='< MMMM >',
MASTER_LOG_POS=< PPPP >;
START SLAVE;
Where
- MMMM is the last file used from the Master that was last processed on the Slave
- PPPP is the last position used from the Master that was last processed on the Slave
You can get MMMM and PPPP by doing SHOW SLAVE STATUS\G and using
- Relay_Master_Log_File for MMMM
- Exec_Master_Log_Pos for PPPP
Try it out and let me know !!!
BTW running CHANGE MASTER TO command erases the slave's current relay logs and starts fresh.
DISCLAIMER : Not a user of mha4mysql.
SUGGESTION #1
Hang the Lag Slave off from the Standby Server. How?
- Step 01: Stop all Writes to the Master
- Step 02: Run
SHOW MASTER STATUS; on the StandByMaster
- Step 03: Run mysqlbinlog against the binlog from
Step 02
- Step 04: Acquire timestamp from given position
- Step 05: Subtract 43200 sec (12 hrs) from the Timestamp acquired
- Step 06: Locate position just before timestamp acquired
- Step 07: Using StandByMaster Host IP, Log File and Position, run this on Lag Slave
STOP SLAVE;CHANGE MASTER TO MASTER_HOST='...',MASTER_LOG_FILE='...',MASTER_LOG_POS=...;START SLAVE;
All manual steps. I'm sorry.
SUGGESTION #2
Setup the Lag Slave under one of the Slaves PERMANENTLY. You'll never have worry about moving the Lag Slave between Masters again.
SUGGESTION #3
Setup the Lag Slave under a Blackhole Slave. You'll never have worry about moving the Lag Slave between Masters again.
I hope these suggestions helped or at least got you thinking.
Best Answer
Based on your comments- using Linux with a non-trivially small sized database, I will recommend you to use Percona XtraBackup ( https://www.percona.com/software/mysql-database/percona-xtrabackup ) to generate a copy of your current active database in a fast way and in an almost 100% hot way (no read or write blocking except to gather the binlog position, which normally should only take less than 2 seconds).
You can follow the steps at: https://www.percona.com/doc/percona-xtrabackup/LATEST/howtos/setting_up_replication.html (How to setup a slave for replication in 6 simple steps with Percona XtraBackup).
I asked those questions because the alternative is mysqldump/mydumper, but that will take more to recover for your data size (also XtraBackup didn't use to support non-linux). Here is how to do it ina hot way just in case: https://dev.mysql.com/doc/refman/5.5/en/replication-howto.html
A common mistake in both cases is thinking you need to stop writes or shutdown your master- you don't, just make sure you start replicating from the coordinates corresponding to the moment you took your backup (the end of XtraBackup run or the start of mysqldump/mydumper).
To prevent data drifts in the future consider performing regular consistency checks with tools such as pt-table-checksum or migrating to row based replication.