OBSERVATION #1

You have max_connect_errors to 1000000 (One Million). If you ever reach that many consecutive connection failures, you would simply run FLUSH HOSTS; to clear up blocked connections. Given the high value of that setting, you will never need to run FLUSH HOSTS;

OBSERVATION #2

I am glad you mentioned your TIME_WAITs. I wrote this post in ServerFault back on Feb 01, 2012 : MySQL lowering wait_timeout value to lower number of open connections. This is a bit of a hack but it is a necessary evil in this instance.

In short, just run this on every PXC node

SEC_TO_TIMEWAIT=1
echo ${SEC_TO_TIMEWAIT} > /proc/sys/net/ipv4/tcp_tw_recycle
echo ${SEC_TO_TIMEWAIT} > /proc/sys/net/ipv4/tcp_tw_reuse

This will make TIME_WAITs timeout in 1 second. You have to run these lines every time a node reboots.

• I got this idea from this post: How to forcibly close a socket in TIME_WAIT?
• The accepted answer has a pictorial representation of when a TIME_WAIT comes into existence.
• The answer with the idea that I liked is the one I am now suggesting.

Galera/Percona XtraDB Cluster is an solution that provides high availability, read scaling and full consistency. Although in some cases it can provide some write scaling (mainly due to poor SQL writing), it is not focused on it, as it does not provide sharding, so it has to write to all nodes in a (virtually) synchronous way. This is important to decide if Galera is for you.

Having said that, some comments in your setup:

• Virtualisation, while not a problem per-se, can add some component of indeterminism, which is critical for Galera as it will perform as bad as the worst performing node. Make sure all nodes have a predictable performance.

• you can reduce some durability parameters in the cluster (for example, setting innodb_flush_log_at_trx_commit = 2), as HA will be provided by node redundancy in the case of a crash. This will increase performance.

• Certain workloads may need client changes, as they are hurtful for Galera performance. In particular, frequent updates on the same rows can case collision and frequent rollbacks of transactions, reducing the throughout. Identify hotspots with the Percona/XtraDB cluster statistics and profiling and try to avoid them from client side. In some cases, some commercial applications may need to change some code slightly to adapt to a cluster infrastructure.

• Network performance is critical. You say that the latency between nodes is 0.1 ms. A transaction will take an overhead at least equal to double the latency of the slowest node.