I feel terrible for forgetting about this question!!!
I have located and fixed the problem as follows.
When the DNS names were added to DNS, the corresponding reverse lookup was not created.
This resulted in MySQL not being able to perform the reverse lookup from the IP address to the correct DNS name, and therefore rejecting the connection.
So, we added a set of reverse lookups from IP address to DNS names, ran FLUSH HOSTS;
on all of the MySQL boxes, and everything started working.
We require the use of DNS names for all connections, due to our disaster recovery solution being in a separate data centre, and a virtually identical VM farm, but with the IP addresses of all the machines modified only slightly. If / When we have a complete fail-over to the other data centre, all the software and communications will just "work", as the DNS resolution will always give the correct address based on the data centre.
You just need to be careful not to have TWO (or more) names reverse looking up from the same IP address, as there is no guarantee which of the two names will be returned - sometimes your connection will work, and other times it won't.
Hope this helps someone with the same problem!
Regards,
Dave
Best Answer
You should consider what distance does to any DB connection.
I have some posts from myself and others on this subject
Jul 25, 2012
: Mysql database replication on different vlan/subnet/another site (From Me)Aug 26, 2012
: Mysql database replication on different vlan/subnet/another site (Shlomi Noach)Aug 20, 2012
: Best solution for cross-datacenter MySQL master-slave replication (From Me)Another member of this StackExchange, Aaron Brown, wrote a nice blog about measuring latency even with semisynchronous replication. Although his blog is 8 years old and technology has improved since, you should write the same test Aaron did to test the latency yourself. After all, London to Tokyo is intercontinental where US West to US East is within the same country. Latency should be expected, but only your testing will reveal how tolerable it will be.
Please review them and see if you can setup semisynchronous replication.
CAVEAT : if you are using Amazon RDS or Google GCP, you are at their mercy when it comes to latency.