Mysql – Where is pt-table-checksum finding this hostname

master-slave-replicationMySQLpercona-toolkit

The Setup: An EC2 instance running MySQL 5.6 locally replicating to RDS MySQL. The VPC IP address range is 172.0.0.0/8.

Somehow the slave got out of sync and I'm getting this error when I do SHOW SLAVE STATUS: 

Could not execute Update_rows event on table db37708.general; Can't find record in 'general', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log newprod.000106, end_log_pos 285712203

I compared the master and slave tables and yes, there is some data out of sync. Not schema, just text data. Okay, fine, I'll just run my script that wraps pt-table-checksum and I get this:

[ubuntu@fabersubuntu utils ]$ ./checksumSpecificTable.sh db37708 general
Replica ip-10-3-2-119 is stopped.  Waiting.
Replica ip-10-3-2-119 is stopped.  Waiting.
^C# Caught SIGINT.
            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
04-19T12:09:50      0      0        1       1       0  32.282 db37708.general

So I run the script again after adding PTDEBUG=1 to it and I see things like this:

# DSNParser:1638 30974 DBI::db=HASH(0x21aae58) SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION'*/
# DSNParser:1652 30974 DBH info:  DBI::db=HASH(0x21aae58) $VAR1 = {
#   '@@hostname' => 'ip-10-3-2-119',
#   'CONNECTION_ID()' => '351452',
#   'DATABASE()' => undef,
#   'VERSION()' => '5.6.40-log'

My questions are:

  1. Where is this "hostname" coming from? Assuming it's an IP address format, it makes sense that it is waiting because I don't have any hosts in a 10.0.0.0 range.

  2. If I CALL mysql.skip_repl_error, I get another error; apparently I have lots of errors all of a similar type (so far). I assume pt-table-checksum can't do anything for me if replication isn't working, so what would be next step be other than rebuild my RDS slave?

Best Answer

Your Question 1

Hostname is from the output of SHOW PROCESSLIST. From that output it tries to detect the slaves. If this is not the right outcome for you, then you can define the dsns table with the list of hosts and use --recursion-method=dsn

Part two of your question will need more information, but you may be able to help yourself using PTDEBUG=1 pt-table-checksum <params>

You are welcome to update your original question with the output from the debug if you would like help with understanding the output.

Disclosure: I work for Percona

Related Question