I feel terrible for forgetting about this question!!!
I have located and fixed the problem as follows.
When the DNS names were added to DNS, the corresponding reverse lookup was not created.
This resulted in MySQL not being able to perform the reverse lookup from the IP address to the correct DNS name, and therefore rejecting the connection.
So, we added a set of reverse lookups from IP address to DNS names, ran FLUSH HOSTS;
on all of the MySQL boxes, and everything started working.
We require the use of DNS names for all connections, due to our disaster recovery solution being in a separate data centre, and a virtually identical VM farm, but with the IP addresses of all the machines modified only slightly. If / When we have a complete fail-over to the other data centre, all the software and communications will just "work", as the DNS resolution will always give the correct address based on the data centre.
You just need to be careful not to have TWO (or more) names reverse looking up from the same IP address, as there is no guarantee which of the two names will be returned - sometimes your connection will work, and other times it won't.
Hope this helps someone with the same problem!
Regards,
Dave
The system user
is the user defined to execute MySQL Replication.
There are two DB Connections dedicated to performing MySQL Replication
- IO Thread : This thread is responsible for downloading the latest Binary Log Entries from the Master and Storing those Entries in the Slave's Relay Logs.
- SQL Thread : This thread is responsible for processing the relay logs like a FIFO queue. It basically reads the next available Relay Log Entry and processes the SQL of that Entry.
The system user
is referred to as a non-client thread in the MySQL Documentation about SHOW PROCESSLIST;
As far seeing localhost:#####, that number after the colon is a really a port number within mysqld assigned to localhost.
UPDATE 2012-04-27 18:00 EDT
Questions from your comment
Can I rename the system user? Or any other properties of the system user? Also, is the system user is dedicated only for replication? Or any other MySQL processes are spawned by System User? I understand that system user cannot be accessed by a client, its an internal process spawned by MySQL.
Answers to Questions from your comment
No, you cannot rename the system user. It is dedicated to handle MySQL Replication only. The only way to manipulate properties of of the system user
would be throught the GRANT command issued to create the replication user.
For example, when you setup a replication user, you issue a command like this:
GRANT REPLICATION SLAVE,REPLICATION CLIENT ON *.* to 'repl'@'%';
When START SLAVE
is issued on a Slave, the Master authenticates the DB Thread coming from the Slave and assign one thread on the Master. The thread on the Master will ship binary log entries to the I/O Thread on the Slave. The I/O Thread on the Slave is assigned to system user
for handling communcation between Master and Slave. The SQL Thread on the Slave is also assigned to system user
for handling intracommunication of local relay log entries to be processed FIFO (Frist In, First Out) by mysqld running on the Slave. No direct access is permitted via the MySQL Client on the Slave except for
STOP SLAVE;
(Kills both I/O Thread and SQL Thread)
STOP SLAVE IO_THREAD;
STOP SLAVE SQL_THREAD;
START SLAVE;
(Creates both I/O Thread and SQL Thread)
START SLAVE IO_THREAD;
START SLAVE SQL_THREAD;
Of course, you could issue KILL ####;
where #### is the process ID of either the I/O Thread or SQL Thread. You would be totally respsonsible for reestablishing replication at the risk of losing the correct log file and position if the KILL
command misses any communication because of an unnatural stoppage of a replicaton thread.
Best Answer
Check the secure log in the event it was run via a sudo service call
Check to see if there's any mysql related stuff going on in cronjobs
Check shell histories for mysqladmin calls
cd /home; for u in *: do; sudo grep mysql /home/$u/.bash_history; done
Check with people you know that either have sudo or mysql root access on this machine