Firstly, I must point out that variable tuning is very application specific and highly depends on what problems you are running into. That being said, you need to know your dataset and which engines you are using (and so does anyone answering). Here is a query to get you that, but be aware it could potentially run a while and might lock up your application (anecdotal: I can run it on 80GB in less than 5 seconds):
SELECT IFNULL(B.engine,'Total') "Storage Engine", CONCAT(LPAD(FORMAT(
B.DSize/POWER(1024,pw),3),17,' '),' ',SUBSTR(' KMGTP',pw+1,1),'B') "Data Size",
CONCAT(LPAD(FORMAT(B.ISize/POWER(1024,pw),3),17,' '),' ',
SUBSTR(' KMGTP',pw+1,1),'B') "Index Size",CONCAT(LPAD(FORMAT(B.TSize/
POWER(1024,pw),3),17,' '),' ',SUBSTR(' KMGTP',pw+1,1),'B') "Table Size"
FROM (SELECT engine,SUM(data_length) DSize,
SUM(index_length) ISize,SUM(data_length+index_length) TSize FROM information_schema.tables
WHERE table_schema NOT IN ('mysql','information_schema','performance_schema') AND
engine IS NOT NULL GROUP BY engine WITH ROLLUP) B,(SELECT 2 pw) A ORDER BY TSize;
I stole this query from Rolando. It is very nice!
This will output the storage engines you are using and the amount of data, index and total space is being used by each.
This is important because you are allocation 5GB for MyISAM indexes:
key_buffer_size 5242880000
Do you have that much MyISAM index usage?
For InnoDB, here are some quick suggestions.
innodb_buffer_pool_size 16106127360 is 15GB. Normally if you run all InnoDB (no MyISAM) and MySQL is the only thing running on the server, this can be most of your memory (saving room for per-thread buffers). But 15GB is decent so I wouldn't start your tuning there.
InnoDB Log files are low:
innodb_log_buffer_size 1048576 #1MB
innodb_log_file_size 5242880 #5MB
Since you mention database sizes up to 84GB, I would start by tuning this up to innodb_log_file_size = 100MB and innodb_log_buffer_size = 8MB. Here is how to change this directive safely.
innodb_flush_log_at_trx_commit is at 1, which helps make your InnoDB as ACID compliant as MySQL can get. It means that after every update commit (insert, deleted, update), InnoDB with flush the transaction to the logfile. Depending on your disk speeds, this can be a bottleneck. By setting it to 2, InnoDB will flush it to the OS cache, and every second will flush it to the logfile. So if your OS crashes, you lose 1-2 seconds, but if MySQL crashes, you won't lose anything.
This is a good article to read to get started, and in general anything posted on that blog.
Best Answer
A good starting point is the MySQL Slow Query Log instead of the general query log. You can set the
You'll want to log queries that aren't using indexes
Update In your question, you state that the system is 'nice and responsive' over local network, but that you haven't done any performance tuning. The slow query log I pointed out will help you identify queries that are taking a long time to run (over 1 second, if configured that way). IMO, this is a great starting point. The longer a query takes, it is much worse when the response has to be transmitted over a WAN.
One tool I've recently discovered is mk-tcp-model that analyzes output from tcpdump to help measure how long a request takes to respond. You can see how many request/responses are coming in and how long each takes. The best tuning over a WAN is to reduce the amount of requests you need to make.