Mysql optimization help

my.cnfMySQLmysql-5.5optimization

I've tried optimizing mysql as good as I could, but apparently I'm not to good at it 🙂

So I come to you guys for help.

Server specs are:
AMD Opteron CPU 8 cores
16 GB RAM
2×2.000 GB HDD 7.200 RPM Software raid 1
Cloudlinux + Cpanel
Around 60 websites hosted, half of them Joomla driven
Total traffic for last month : 218 GB (both incoming and outgoing)

My current my.cnf looks like this:

[mysqld]
local-infile=0
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
#skip-innodb
query_cache_limit=64M
query_cache_size=64M
query_cache_type=1
max_user_connections=100
max_connections=150
interactive_timeout=10
wait_timeout=20
connect_timeout=20
thread_cache_size=128
key_buffer=256M
join_buffer=4M
max_allowed_packet=128M
table_cache=163840
table_definition_cache=163840
#record_buffer=1M
sort_buffer_size=2M
read_buffer_size=8M
max_connect_errors=10
thread_concurrency=8
myisam_sort_buffer_size=64M
server-id=1
innodb_buffer_pool_size=6G 
innodb_file_per_table=1
tmp_table_size=3G
max_heap_table_size=3G
low_priority_updates=1
concurrent_insert=ALWAYS
log-slow-queries=/var/lib/mysql/slow.log
log-queries-not-using-indexes = /var/log/mysql-indexes.log

#[mysql.server]
#user=mysql
#basedir=/var/lib 

open_files_limit=50000
[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/lib/mysql/mysql.pid
open_files_limit=8192 

#[mysqldump]
#quick
#max_allowed_packet=16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M 

[myisamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=16M
write_buffer=16M

[mysqlhotcopy]
interactive-timeout

The output from mysqltuner is this:

>> MySQLTuner 1.2.0 - Major Hayden <major@mhtx.net>
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.30-cll
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 1G (Tables: 3186)
[--] Data in InnoDB tables: 250M (Tables: 8508)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 0B (Tables: 266)
[!!] Total fragmented tables: 40

-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 2d 5h 15m 11s (6M q [34.844 qps], 157K conn, TX: 43B, RX: 1B)
[--] Reads / Writes: 62% / 38%
[--] Total buffers: 5.1G global + 14.5M per thread (150 max threads)
[OK] Maximum possible memory usage: 7.2G (46% of installed RAM)
[OK] Slow queries: 0% (169/6M)
[OK] Highest usage of available connections: 26% (39/150)
[OK] Key buffer size / total MyISAM indexes: 256.0M/187.6M
[OK] Key buffer hit rate: 99.9% (357M cached / 236K reads)
[OK] Query cache efficiency: 82.0% (4M cached / 4M selects)
[!!] Query cache prunes per day: 95307
[OK] Sorts requiring temporary tables: 1% (2K temp sorts / 233K sorts)
[!!] Joins performed without indexes: 3349
[!!] Temporary tables created on disk: 38% (186K on disk / 479K total)
[OK] Thread cache hit rate: 99% (39 created / 157K connections)
[OK] Table cache hit rate: 32% (24K open / 74K opened)
[OK] Open file limit used: 2% (9K/327K)
[OK] Table locks acquired immediately: 99% (1M immediate / 1M locks)
[OK] InnoDB data size / buffer pool: 250.3M/1.8G

-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Adjust your join queries to always utilize indexes
Temporary table size is already large - reduce result set size
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
query_cache_size (> 64M)
join_buffer_size (> 4.0M, or always use indexes with joins)

Something that I've notced is that 90 % of Swap is being used, while the server has over 8GB of free RAM.

 total       used       free     shared    buffers     cached
Mem:         15851      15733        118          0        569       8094
-/+ buffers/cache:       7069       8782
Swap:         4047       3664        383

Normally the server load stays around 0.5-0.8 but when the load spike occur it goes u to 5.00-6.00 and stays like that until I restart mysql, after which it goes back to normal (until the next spike).

As for PHP I'm using fcgi as the handler and Nginx ( via nginxcp ). I've tried removing nginxcp, thinking that it may be the cause, but the spikes still occurred.

All the software on the server is up to date, including the Joomla websites.

Fcgi is configures with the following settings, made according to Cloudlinux recommendations:

FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 16
FcgidMaxProcesses 300
FcgidIdleTimeout 60
FcgidProcessLifeTime 120
FcgidIdleScanInterval 30

I've also checked for network issues or DDOS attacks, but that is not the case. Hardware is brand new and in perfect state. So what could it be ?

Hope anyone can help out, I'll be forever grateful.

Thanks.

Best Answer

With multi-cores on MySQL the single mutex will create contention on the query cache. Set query_cache_type=0 and query_cache_size=0 in my.cnf.

Next, make the innodb_buffer_pool_size close to the size of the database; looks like you still have memory left. This helps reduce overall IO.

Outside of my.cnf, the queries hitting your server should use indexes. If not, they can create more IO. And finally, to handle high IO, you want to spread IO across disk drives. The first candidate for that is your binary logs. If /var is mounted on /dev/sda1 then add log_bin = {directory mounted on /dev/sda2}.

Related Solutions

Mysql – How to debug a db memory-leak causing thesql to go before it’s own limits

...even surpassing it's theorically maximum possible allocation.

[OK] Maximum possible memory usage: 7.3G (46% of installed RAM)

There is not actually a way to calculate maximum possible memory usage for MySQL, because there is no cap on the memory it can request from the system.

The calculation done by mysqltuner.pl is only an estimate, based on a formula that doesn't take into account all possible variables, because if all possible variables were taken into account, the answer would always be "infinite." It's unfortunate that it's labeled this way.

Here is my theory on what's contributing to your excessive memory usage:

thread_cache_size       = 128

Given that your max_connections is set to 200, the value of 128 for thread_cache_size seems far too high. Here's what makes me think this might be contributing to your problem:

When a thread is no longer needed, the memory allocated to it is released and returned to the system unless the thread goes back into the thread cache. In that case, the memory remains allocated.

^{http://dev.mysql.com/doc/refman/5.6/en/memory-use.html}

If your workload causes even an occasional client thread to require a large amount of memory, those threads may be holding onto that memory, then going back to the pool and sitting around, continuing to hold on to memory they don't technically "need" any more, on the premise that holding on to the memory is less costly than releasing it if you're likely to need it again.

I think it's worth a try to do the following, after first making a note of how much memory MySQL is using at the moment.

Note how many threads are currently cached:

mysql> show status like 'Threads_cached';
+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| Threads_cached | 9     |
+----------------+-------+
1 row in set (0.00 sec)

Next, disable the thread cache.

mysql> SET GLOBAL thread_cache_size = 0;

This disables the thread cache, but the cached threads will stay in the pool until they're used one more time. Disconnect from the server, then reconnect and repeat.

mysql> show status like 'Threads_cached';

Continue disconnecting, reconnecting, and checking until the counter reaches 0.

Then, see how much memory MySQL is holding.

You may see a decrease, possibly significant, and then again you may not. I tested this on one of my systems, which had 9 threads in the cache. Once those threads had all been cleared out of the cache, the total memory held by MySQL did decrease... not by much, but it does illustrate that threads in the cache do release at least some memory when they are destroyed.

If you see a significant decrease, you may have found your problem. If you don't, then there's one more thing that needs to happen, and how quickly it can happen depends on your environment.

If the theory holds that the other threads -- the ones currently servicing active client connections -- have significant memory allocated to them, either because of recent work in their current client session or because of work requiring a lot of memory that was done by another connection prior to them languishing in the pool, then you won't see all of the potential reduction in memory consumption until those threads are allowed to die and be destroyed. Presumably your application doesn't hold them forever, but how long it will take to know for sure whether there's a difference will depend on whether you have the option of cycling your application (dropping and reconnecting the client threads) or if you'll have to just wait for them to be dropped and reconnected over time on their own.

But... it seems like a worthwhile test. You should not see a substantial performance penalty by setting thread_cache_size to 0. Fortunately, thread_cache_size is a dynamic variable, so you can freely change it with the server running.

Mysql – mariadb / thesql wont change table_cache / table_open_cache

The operating system has a low limit. In Unix, see ulimit -n, and change it. Some versions have an unnecessarily low limit of 1024.

Most, but not all, things set by my.cnf (my.ini) stay set. table_open_cache is one that is changed based on the environment. Use SHOW VARIABLES to see the real values. Or SHOW VARIABLES LIKE 'table_open_cache' to see the one setting.

Best Answer

Related Solutions

Mysql – How to debug a db memory-leak causing thesql to go before it’s own limits

Mysql – mariadb / thesql wont change table_cache / table_open_cache

Related Question