MySQL creates temporary tables on disk. How to stop it

database-tuningMySQLperformancetemporary-tableswindows

We are running a site (Moodle) that the users currently find slow. I think I have tracked down the problem to MySQL creating temporary tables on disk. I watch the variable created_tmp_disk_tables in Mysql Workbench server administration and the number increases with roughly 50 tables/s. After a days usage, created_tmp_disk_tablesis >100k. Also, the memory does not seem to be released. The usage keeps increasing until the system becomes pretty much unusable and we have to re-start MySQL. I need to re-start it almost every day and it begins with using about 30-35% of available memory and finishing the day with 80%.

I have no blobs in the database and no control over the queries either so I can't attempt to optimise them. I have also used the Percona Confirguration Wizard to generate a configuration file but that my.ini didn't solve my problem either.

Questions

What should I change to stop MySQL from creating temporary tables on disk? Are there settings I need to change? Should I throw more memory at it?
How can I stop MySQL from eating up my memory?

Edit

I enabled slow_queries log and discovered that the query SELECT GET_LOCK() was logged as slow. A quick search revealed that I had allowed persistent connections in the PHP configuration (mysqli.allow_persistent = ON). I turned this off. This reduced the rate at which MySQL consumes memory.It is still creating temporary tables though.

I also checked that the key_buffer size is large enough.
I looked at the variable key_writes. This should be zero. If not, increase the key_buffer_size.I have zero key_reads and zero key_writes so I assume that the key_buffer_size is large enough.

I increased the tmp_table_size and max-heap-table-size to 1024M as an increase in created_tmp_disk_tables may indicate that the tables can't fit in memory. This didn't solve it.

Ref:
http://www.mysqlperformanceblog.com/2007/08/16/how-much-overhead-is-caused-by-on-disk-temporary-tables/

Edit 2

If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the sort_buffer_size value. I had 2 sort_merge_passes in an hour so I consider the sort_buffer_size to be large enough.

Ref: Mysql Manual on sort_buffer_size

Edit 3

I have modified the sort and join buffers as suggested by @RolandoMySQLDBA. The result is displayed in the table below but I think the created_tmp_tables_on_disk is still high. I restarted the mysql server after I changed the value and checked the created_tmp_tables_on_disk after a day (8h) and calculated the average. Any other suggestions? It seems to me that there is something that doesn't fit inside some kind of container but I can't work out what it is.

+---------------------+-------------+-------------+--------------------+
| Tmp_table_size,     | Sort_buffer | Join_buffer | No of created      |
| max_heap_table_size |             |             | tmp_tables on disk |
+---------------------+-------------+-------------+--------------------+
| 125M                | 256K        | 256K        |  100k/h            |
+---------------------+-------------+-------------+--------------------+
| 125M                | 512K        | 512K        |  100k/h            |
+---------------------+-------------+-------------+--------------------+
| 125M                | 1M          | 1M          |  100k/h            |
+---------------------+-------------+-------------+--------------------+
| 125M                | 4M          | 4M          |  100k/h            |
+---------------------+-------------+-------------+--------------------+

This is my configuration:

+-----------------------+-----------------------+
|DATABASE SERVER        |WEB SERVER             |
+-----------------------+-----------------------+
|Windows Server 2008 R2 |Windows Server 2008 R2 |
+-----------------------+-----------------------+
|MySQL 5.1.48           |IIS 7.5                |
+-----------------------+-----------------------+
|4 Core CPU             |4 Core CPU             |
+-----------------------+-----------------------+
|4GB RAM                |8GB RAM                |
+-----------------------+-----------------------+

Additional information

+--------------------+---------+
|PARAM               |VALUE    |
+--------------------+---------+
|Num of tables in Db |361      |
+--------------------+---------+
|Size of database    |2.5G     |
+--------------------+---------+
|Database engine     |InnoDB   |
+--------------------+---------+
|Read/write ratio    |3.5      |
|(Innodb_data_read/  |         |
|innodb_data_written)|         |
+--------------------+---------+
|Avg table size      |15k rows |
+--------------------+---------+
|Max table size      |744k rows|
+--------------------+---------+

This setup was given to me so I have limited control over it. The web server is using very little CPU and RAM so I have excluded that machine as a bottleneck. A majority of the MySQL settings originates from a config auto-generation tool.

I have monitored the system using PerfMon over a few representative days. From that, I conclude that it is not the OS that is swapping to disk.

My.ini

[client]
port=3306
[mysql]
default-character-set=utf8

[mysqld]
port=3306
basedir="C:/Program Files/MySQL/MySQL Server 5.1/"
datadir="D:/DBs/Data/"
default-character-set=utf8
default-storage-engine=INNODB
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
max_connections=125
query_cache_size=350M
table_cache=1520
tmp_table_size=125M
table-definition-cache= 1024
max-heap-table-size= 32M
thread_cache_size=38

MyISAM Specific options
myisam_max_sort_file_size=100G
myisam_sort_buffer_size=125M
key_buffer_size=55M
read_buffer_size=1024K
read_rnd_buffer_size=256K
sort_buffer_size=1024K
join_buffer_size=1024K


INNODB Specific options
innodb_data_home_dir="D:/DBs/"
innodb_additional_mem_pool_size=32M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_buffer_pool_size=2G
innodb_log_file_size=407M
innodb_thread_concurrency=8

Best Answer

Looking at the my.ini, I have two suggestions

SUGGESTION #1

I would bump up the following settings in your my.ini

sort_buffer_size=4M
join_buffer_size=4M

This will make some joins and sort stay in memory. Of course, once a JOIN or an ORDER BY needs more than 4M, it will page to disk as a MyISAM table.

If you cannot login as root@localhost, then restart mysql with

C:\> net stop mysql
C:\> net start mysql

If you can login as root@localhost, you do not have to restart mysql to use these settings.

Just run this in the MySQL client:

SET @FourMegs = 1024 * 1024 * 4;
SET GLOBAL sort_buffer_size = @FourMegs;
SET GLOBAL join_buffer_size = @FourMegs;

SUGGESTION #2

Since your Data is on Drive D:, you may have Disk I/O on Drive C:.

Please run this query:

mysql> show variables like 'tmpdir';
+---------------+-----------------+
| Variable_name | Value           |
+---------------+-----------------+
| tmpdir        | C:\Windows\TEMP |
+---------------+-----------------+
1 row in set (0.00 sec)

Since I run mysql on my Desktop with defaults, my temp tables are being written to Drive C:. If Drive D is a better disk than Drive C:, perhaps you can map temp tables to Drive D: by setting tmpdir in my.ini as follows:

tmpdir="D:/DBs/"

You will have to restart mysql since tmpdir is not a dynamic variable.

Give it a Try !!!

UPDATE 2013-11-29 10:09 EST

SUGGESTION #3

Given the fact that MySQL is running in Windows and you cannot touch the queries in the core package, I have two ideas tat must be done together.

IDEA #1 : Move the Database to a Linux Machine

You should be able to

Setup a Linux machine
Install MySQL on the Linux machine
Enable Binary Logging for MySQL in Windows
mysqldump the database to a text SQL file
Load SQL file to MySQL running in Linux
Setup replication from MySQL/Windows to MySQL/Linux

IDEA #2 : Reconfigure Moodle to point to the Linux Machine

Moodle was designed for LAMP in the first place. Just change the config files to point to the Linux machine instead of localhost.

Here is a link to an old Moodle 2.3 doc on setting up MySQL : http://docs.moodle.org/23/en/Installing_Moodle#Create_an_empty_database

I am sure the latest docs are available as well.

What is the Point of Moving the Database to Linux ???

How does this help the temp table situation ???

I would then suggestion setting up a RAM disk as the target folder for your temp tables

Jan 04, 2013 : Is there a MySQL engine or trick to avoid writing so many temp tables to disk?
Dec 17, 2012 : Why does MySQL produce so many temporary MYD files? (Actual instructions)
Nov 30, 2012 : Is it bad to create many mysql temporary tables simultaneously?

Temp table creation will still happen, but it will be written to RAM rather than disk. reducing Disk I/O.

UPDATE 2013-11-29 11:24 EST

SUGGESTION #4

I would suggest revisiting SUGGESTION #2 with a fast RAID-0 disk (32+ GB), configuring it as Drive T: (T for Temp). After installing such a disk, add this to my.ini:

[mysqld]
tmpdir="T:\"

MySQL restart would be required, using

net stop mysql
net start mysql

BTW I said RAID-0 on purpose so that you can get good write performance over a RAID-1, RAID-10. A tmp table disk is not something I would make redundant.

Without optimizing the queries as @RaymondNijland has been commenting on, you cannot reduce the temp table creation count in any way. SUGGESTION #3 and SUGGESTION #4 offer speeding up temp table creation and temp table I/O as the only alternative.

Related Solutions

MySQL Replication: Insert generating VERY huge temporary tables

Algorithmically, it is doing what your colleague says. But, do you see what it is doing ???

It is generating 10,000 temp tables each containing 1 row after traversing 317 million rows through in the InnoDB internal index. Each temp table is a complete regeneration of the rowids in sernumbers_results_2009 table along with executing handler_read_prev commands internally to sort the data by an index scan from the back of the internal rowid index. Also, please remember you are dealing with InnoDB. Who knows what Multiversioning (via MVCC) is going on so that the INSERT is completed without interference and with rollback capabilities.

Is there any reason why this query wouldn't work for you ???

INSERT INTO sernumbers_results_2009
SELECT * FROM sernumbers_results
ORDER BY rowid DESC LIMIT 10000;

This will definitely generate one temp table.

Give it a Try !!!

Mysql – How to debug a db memory-leak causing thesql to go before it’s own limits

...even surpassing it's theorically maximum possible allocation.

[OK] Maximum possible memory usage: 7.3G (46% of installed RAM)

There is not actually a way to calculate maximum possible memory usage for MySQL, because there is no cap on the memory it can request from the system.

The calculation done by mysqltuner.pl is only an estimate, based on a formula that doesn't take into account all possible variables, because if all possible variables were taken into account, the answer would always be "infinite." It's unfortunate that it's labeled this way.

Here is my theory on what's contributing to your excessive memory usage:

thread_cache_size       = 128

Given that your max_connections is set to 200, the value of 128 for thread_cache_size seems far too high. Here's what makes me think this might be contributing to your problem:

When a thread is no longer needed, the memory allocated to it is released and returned to the system unless the thread goes back into the thread cache. In that case, the memory remains allocated.

^{http://dev.mysql.com/doc/refman/5.6/en/memory-use.html}

If your workload causes even an occasional client thread to require a large amount of memory, those threads may be holding onto that memory, then going back to the pool and sitting around, continuing to hold on to memory they don't technically "need" any more, on the premise that holding on to the memory is less costly than releasing it if you're likely to need it again.

I think it's worth a try to do the following, after first making a note of how much memory MySQL is using at the moment.

Note how many threads are currently cached:

mysql> show status like 'Threads_cached';
+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| Threads_cached | 9     |
+----------------+-------+
1 row in set (0.00 sec)

Next, disable the thread cache.

mysql> SET GLOBAL thread_cache_size = 0;

This disables the thread cache, but the cached threads will stay in the pool until they're used one more time. Disconnect from the server, then reconnect and repeat.

mysql> show status like 'Threads_cached';

Continue disconnecting, reconnecting, and checking until the counter reaches 0.

Then, see how much memory MySQL is holding.

You may see a decrease, possibly significant, and then again you may not. I tested this on one of my systems, which had 9 threads in the cache. Once those threads had all been cleared out of the cache, the total memory held by MySQL did decrease... not by much, but it does illustrate that threads in the cache do release at least some memory when they are destroyed.

If you see a significant decrease, you may have found your problem. If you don't, then there's one more thing that needs to happen, and how quickly it can happen depends on your environment.

If the theory holds that the other threads -- the ones currently servicing active client connections -- have significant memory allocated to them, either because of recent work in their current client session or because of work requiring a lot of memory that was done by another connection prior to them languishing in the pool, then you won't see all of the potential reduction in memory consumption until those threads are allowed to die and be destroyed. Presumably your application doesn't hold them forever, but how long it will take to know for sure whether there's a difference will depend on whether you have the option of cycling your application (dropping and reconnecting the client threads) or if you'll have to just wait for them to be dropped and reconnected over time on their own.

But... it seems like a worthwhile test. You should not see a substantial performance penalty by setting thread_cache_size to 0. Fortunately, thread_cache_size is a dynamic variable, so you can freely change it with the server running.

Questions

Edit

Edit 2

Edit 3

Best Answer

SUGGESTION #1

SUGGESTION #2

Give it a Try !!!

UPDATE 2013-11-29 10:09 EST

SUGGESTION #3

IDEA #1 : Move the Database to a Linux Machine

IDEA #2 : Reconfigure Moodle to point to the Linux Machine

What is the Point of Moving the Database to Linux ???

How does this help the temp table situation ???

UPDATE 2013-11-29 11:24 EST

SUGGESTION #4

Related Solutions

MySQL Replication: Insert generating VERY huge temporary tables

Mysql – How to debug a db memory-leak causing thesql to go before it’s own limits

Related Question