Mysql – How to get the most out of MySQL on a QuadCore machine with 16 GB of RAM

innodbMySQLmysql-5mysql-5.5performance

I'm running a MySQL 5.5 server on my workstation for scientific data analysis and wonder how to configure MySQL in order to get the most out of it performance-wise. The types of query that I typically run involve joins of 10-20 tables and can run for quite long, one to several minutes being no exception at all. Only very few users access the database at the same time (5 being the maximum).
I moved the server from a Lenovo Thinkpad T61 with a 2.2 GHz Dual Core and 4 GB of RAM to the following brand-new machine with hand-selected components:

Intel i7 3770, 4x 3.4 GHz (running @ 4×3.7 GHz)
Z77 chipset
16 GB of DDR3 1600 RAM
Windows 7 Prof 64-bit
Windows and MySQL server run on a Intel 520 series SSD drive.

First tests (running the same query on both machines) showed a definitive improvement in speed for the new one, but the queries still take a lot of time and I had expected more of a boost. The queries in question are fairly well optimized, i.e. all tables have proper key that are also being used as of "explain extended".

Now to my current MySQL settings:
First I should mention that I moved from MyISAM to Innodb long time ago.

Some of my my.ini tweaks (i.e. departures from default settings):

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
#tmp_table_size=35M
tmp_table_size=4000M
max_heap_table_size=4000M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
#innodb_buffer_pool_size=96M
innodb_buffer_pool_size=800M

general-log
expire_logs_days = 60
general_log_file = "F:/my_query_mysql.log"
log-output = TABLE
optimizer_search_depth = 0 #meant to cure the "statistics state" bug in some queries

I'd like to know whether somebody would suggest changes to the above numbers or even further settings that I do not know of.

I'd appreciate any helpful remark.

Steve

EDIT: I have two queries involving joins across 10-20 tables and ran them on my Lenovo notebook and the new PC. Query #1 took 3m36s on the new machine vs 9m11s on the laptop; Query #2 took 22.5s on the workstation vs 48.5s on the laptop. So the execution speed was improved by roughly the factor 2-2.5.
On the workstation, not even 50% of the RAM was used. The average CPU load across the four cores (as reported by Windows Task Manager) was only about 13%. The load on a per-core basis (as reported by Core Temp) was about 25-40% for ONE core, while it was <=10% for the others, indicating that MySQL does not make use of multiple cores for a single query.

Best Answer

Since you are running MySQL 5.5, you may want to consider configuring InnoDB to access multiple cores

Here are the settings you should be using

innodb_thread_concurrency sets the upper bound on number of concurrent threads that InnoDB can hold open. Best round number to set for this is (2 X Number of CPUs) + Number of Disks. UPDATE : As I learned firsthand from the Percona NYC Conference, you should set this to 0 in order to alert InnoDB Storage Engine to find the best number of threads for the environment it is running in.

innodb_concurrency_tickets sets the number of threads that can bypass concurrency checking with impunity. After that limit is reached, thread concurrency checking becomes the norm again.

innodb_commit_concurrency sets the number of concurrent transactions that can be committed. Since the default is 0, not setting this allows any number of transactions to commit simultaneously.

innodb_thread_sleep_delay sets the number of milliseconds an InnoDB thread can be dormant before reentering the InnoDB queue. Default is 10000 (10 sec).

innodb_read_io_threads and innodb_write_io_threads (both since MySQL 5.1.38) allocate the specified number of threads for reads and writes. Default is 4 and maximum is 64.

innodb_replication_delay imposes thread delay on a slave is innodb_thread_concurrency is reached.

Here are my past posts on MySQL 5.5 and activating multiple cores for InnoDB

Jun 01, 2012 : I've got 16GB of ram, how should I configure MySQL Server?
May 07, 2012 : MySQL Server Performance
Apr 26, 2012 : Is the CPU performance relevant for a database server?
Mar 16, 2012 : Using multiple cores for single MySQL queries on Debian
Oct 07, 2011 : Should I use a storage engine other than MyISAM to optimise these tables or should I get better disks?
Sep 20, 2011 : Multi cores and MySQL Performance
Sep 12, 2011 : Possible to make MySQL use more than one core?

Related Solutions

MySQL Config on Mac

I have never used MySQL on a Mac, but I have for Windows.

The Windows version of MySQL came with my-medium.ini, my-small.ini, my-large.ini, my-huge.ini.

If the Mac version of MySQL came with similar .cnf files, I would choose my-medium.cnf

Once you copy my-medium.cnf to my.cnf, you could tune it as follows:

If the majority of your data is MyISAM do this:

SELECT CONCAT(CEILING(SumNDXs/POWER(1024,2)),'M') KeyBufferSize
FROM
(
    SELECT SUM(index_length) SumNDXs
    FROM information_schema.tables
    WHERE table_schema NOT IN ('information_schema','mysql')
    AND engine = 'MyISAM'
) A;

This will give you the proper size for key_buffer_size given the amount of MyISAM data you have. Obviously, if this number exceeds 1024M (1GB,25% of system RAM), then set key_buffer_size to your liking with good judgment, as long as it's less than 1024M.

If the majority of your data in InnoDB do this

SELECT CONCAT(CEILING(InnoDBDataAndIndexes/POWER(1024,2)),'M') KeyBufferSize
FROM
(
    SELECT SUM(data_length+index_length) InnoDBDataAndIndexes FROM
    information_schema.tables WHERE engine = 'InnoDB'
) A;

This will give you the proper size for innodb_buffer_pool_size given the amount of InnoDB data and indexes you have. Obviously, if this number exceeds 3072M (3GB,75% of system RAM), then set innodb_buffer_pool_size to your liking with good judgment, as long as it's less than 3072M.

If you have a mixture of InnoDB and MyISAM data, then strike a balance between these numbers and leave at least 25% of system RAM free for MacOS.

Once you have innodb_buffer_pool_size, set innodb_log_file_size to 25% of innodb_buffer_pool_size.

Then do three 3 things:

Shutdown mysql
delete ib_logfile0 and ib_logfile1
Startup mysql (mysqld process will recreate ib_logfile0 and ib_logfile1 to new size)

Give it a try, and let us know !!!

MySQL – Can MySQL Use More Than One Core?

I actually discussed innodb_thread_concurrency with a MySQL Expert at the Percona Live NYC conference back in May 2011.

I learned something surprising: In spite of the documentation, it is best to leave innodb_thread_concurrency at 0 (infinite concurrency). That way, InnoDB decides the best number of innodb_concurrency_tickets to open for a given MySQL instance setup.

Once you set innodb_thread_concurrency to 0, you can set innodb_read_io_threads and innodb_write_io_threads (both since MySQL 5.1.38) to the maximum value of 64. This should engage more cores.

Best Answer

Related Solutions

MySQL Config on Mac

MySQL – Can MySQL Use More Than One Core?

Related Question