Mysql – Optimize MySQL Server Performance

MySQLmysql-5.5mysqltuneroptimizationperformanceperformance-tuning

I have a VPS Server at Vultr with 8GB Ram and 4vCPUs and am looking to get optimized mysql settings. I have a DB of around 170 InnoDB tables, but they mostly don't contain a lot of data. There are 2 MyISAM tables however with 150k rows each which I would like to increase to 350k.

The challenge here is that I do geo calculations in the table with long/lat data for every single calculation it has to go through the full 150k rows, determine which are the closest (within 5 miles) of distance and than give out only that data. This is obviously very resource hungry. Therefor increasing from 150k rows to 200k rows means the query takes 1sec longer while increasing to 350k rows takes an additional 3-4sec per query. I can't update the queries itself at this stage, but can only work on improving mysql as much as possible.

I already started updating my MySQL settings and greatly improved the performance. Here is the output from tuning-primer:

MySQL Version 5.5.55-0ubuntu0.14.04.1 x86_64

Uptime = 2 days 12 hrs 1 min 56 sec
Avg. qps = 61
Total Questions = 2665033
Threads Connected = 3

SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 10.000000 sec.
You have 0 out of 2665054 that take longer than 10.000000 sec. to complete
Your long_query_time seems to be fine

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 8
Current threads_cached = 5
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 3
Historic max_used_connections = 9
The number of used connections is 5% of the configured maximum.
You are using less than 10% of your configured max_connections.
Lowering max_connections could help to avoid an over-allocation of memory
See "MEMORY USAGE" section to make sure you are not over-allocating

INNODB STATUS
Current InnoDB index space = 35 M
Current InnoDB data space = 56 M
Current InnoDB buffer pool free = 25 %
Current innodb_buffer_pool_size = 128 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE
Max Memory Ever Allocated : 4.48 G
Configured Max Per-thread Buffers : 1.53 G
Configured Max Global Buffers : 4.39 G
Configured Max Memory Limit : 5.92 G
Physical Memory : 7.79 G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = 79 M
Current key_buffer_size = 256 M
Key cache miss rate is 1 : 9866
Key buffer free ratio = 79 %
Your key_buffer_size seems to be fine

QUERY CACHE
Query cache is enabled
Current query_cache_size = 4.00 G
Current query_cache_used = 889 M
Current query_cache_limit = 4 M
Current Query cache Memory fill ratio = 21.71 %
Current query_cache_min_res_unit = 4 K
Your query_cache_size seems to be too high.
Perhaps you can use these resources elsewhere
MySQL won't cache query results that are larger than query_cache_limit in size

SORT OPERATIONS
Current sort_buffer_size = 2 M
Current read_rnd_buffer_size = 256 K
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 4.00 M
You have had 72592 queries where a join could not use an index properly
join_buffer_size >= 4 M
This is not advised
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.

OPEN FILES LIMIT
Current open_files_limit = 2209 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 1024 tables
Current table_definition_cache = 1024 tables
You have a total of 702 tables
You have 1024 open tables.
Current table_cache hit rate is 55%
, while 100% of your table cache is in use
You should probably increase your table_cache

TEMP TABLES
Current max_heap_table_size = 16 M
Current tmp_table_size = 16 M
Of 162009 temp tables, 1% were created on disk
Created disk tmp tables ratio seems fine

TABLE SCANS
Current read_buffer_size = 3 M
Current table scan ratio = 66078 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 696683
Your table locking seems to be fine

Thanks for your help. If you need more information please ask!
Would it help to upgrade to mysql 5.6 or 5.7?

Best Answer

Step 1 in solving lat/lng performance problems:

INDEX(latitude),
INDEX(longitude)

If that does not work adequately, then see http://mysql.rjweb.org/doc.php/latlng

For more discussion, please provide SHOW CREATE TABLE and the SELECT that is doing the table scan.

innodb_buffer_pool_size = 128 M -- the most important performance tuning is to set that to 70% of available RAM (assuming all tables are InnoDB).

Why do you still have 2 MyISAM tables? Let's talk about converting them.

query_cache_size = 4.00 G -- This is really bad. Don't set it to more than 50M !

ASPECT #1

The MEMORY table behaves like a MyISAM table in that it performs a full table lock when doing INSERTs, UPDATEs, and DELETEs (MySQL Documentation says Locking granularity : Table). I would be very concerned with a high lock ratio if it bottlenecks your I/O performance. If 300 INSERTs per second is coming at this table, you can be fully confident that there will be 300 full table locks per second.

ASPECT #2

No matter how much data you load into a MEMORY table, mysqld will always contact the .frm file of the MEMORY table to verify the table's existence before query parsing. Although the .frm is not a large file, it will incur a very tiny amount of disk I/O. Thousands, or even just hundreds, of queries against a MEMORY table can reveal a noticeable, even a significant, amount of disk I/O.

Since this is the case with the MEMORY storage engine, you must also avoid MyISAM because it has the same locking granularity (Full Table Locking).

Here are my recommedations

Recommedation #1

Cleanup InnoDB so that it has innodb_file_per_table enabled.

Recommedation #2

Use a large InnoDB Buffer Pool

Since the table is 200MB, you can make InnoDB Buffer Pool 5 times as large (restart required)

[mysqld]
innodb_buffer_pool_size=1G

Recommedation #3

Upgrade to MySQL 5.5 in order to have multithreaded reads and write that engages multiple CPUs for InnoDB. If you have InnoDB 5.1.38+ and are using the InnoDB Plugin, the same options that engages multiple CPUs for InnoDB already exists.

Recommedation #4

Common sense would tell me to recommend turning the table back to InnoDB

Mysql – Optimising MySQL

Coming at this from the DB point of view, rather than WordPress.

Executive Summary

For each recommendation your optimiser script has generated, check out the appropriate MySQL documentation to see if it will impact your setup. Then tweak your my.conf little by little.

Your overall objective is for MySQL to load as much as it can into memory and not hit the disk. So increasing various caches will probably help. As long as you don't exceed the physical memory in your server (you need to take into account other processes running on the server). Then, make sure your tables are indexed appropriately.

Your Specific Issues

But to address each of the items you've listed:

Query cache is supported but not enabled; Perhaps you should set the query_cache_size

The query cache is like a big hash table of Select-Statement -> Result-Table. MySQL checks to see if the same query exists in the cache and returns the cached result without re-running the query. If you enable it, run SHOW VARIABLES to see how well utilised it is (having a huge cache which isn't used is just a waste of memory). My experience of the query cache is that it seems really good in theory, but didn't provide much help in practice.

You have had 11025 queries where a join could not use an index properly; You should enable "log-queries-not-using-indexes" then look for non indexed joins in the slow query log.

Indexes are what makes or breaks a database. Enable the slow query log and use EXPLAIN to see what queries are not using indexes. Fix the slow ones first.

You have a total of 834 tables; You have 528 open tables; Current table_cache hit rate is 8%, while 132% of your table cache is in use; You should probably increase your table_cache

MySQL ISAM databases exist as pairs (I think) of files on disk, but MySQL doesn't keep them open all the time, only ones used recently. The table_cache setting controls how many files it will keep open. This appears to be related to the number of connections, so be careful setting this too high, but it seems you have stacks of memory available so increase this until all tables are cached.

Current Lock Wait ratio = 1 : 529; You may benefit from selective use of InnoDB.

MyISAM tables are great for reading, but whenever you UPDATE, INSERT or DELETE from them MySQL locks the whole table. So if, for example, you have a Page table with a HitCount field which is incremented whenever the page is loaded, the entire Page table is locked and no other connections can read from it. I've seen some particular nasty combinations of read / write queries which would lock tables for minutes or even hours! Effectively killing the site.

InnoDB isn't as fast at reading, but supports more granular write operations (only locking the one record being updated). So is a better fit for tables which are written to more frequently. Converting tables with large numbers of UPDATE, INSERT and DELETE operations to InnoDB may decrease locking and increase performance. Many apps which use MySQL default to InnoDB across the board for this very reason.

~~I think you need to drop the old MyISAM tables and re-create them as InnoDB, so there will be downtime involved.~~

Apparently an ALTER TABLE statement is all you require to change the engine type. Although full table locks will be required, so you'll have some time when you can't run queries.

I don't know if WordPress assumes InnoDB or MyISAM tables. Please check WordPress before altering the tables' engine.

Current max_heap_table_size = 16 M; Current tmp_table_size = 16 M; Of 107071 temp tables, 25% were created on disk; Perhaps you should increase your tmp_table_size and/or max_heap_table_size to reduce the number of disk-based temporary tables

MySQL requires memory to do sorting (ORDER BY), JOINs and other operations involving large chunks of data, but only up to a certain limit, after that limit the operation spills out onto disk (so that one giant ORDER BY doesn't use up gigabytes of memory which could better be spent doing other things). Increasing max_heap_table_size and tmp_table_size means less operations run on the slow disk and more in fast memory (a discussion about these variables). 64M should be large enough for most cases, and making these too big means you're just wasting memory.

You have 1290 out of 1145245 that take longer than 2.000000 sec. to complete

Use the slow query log to figure out what these slow queries are. Although that ratio (0.11%) is pretty low, there may be a few really slow queries which are causing bigger problems.

Current max_connections = 500; Current threads_connected = 501; You should raise max_connections

Once you have more threads_connected than max_connections new connections are rejected. Increase max_connections (as you already have done).

As you can see, its not always obvious what to do without knowing what sorts of queries WordPress is generating.