Mysql – Stress Test to Saturate Memory for MySQL InnoDB

innodbmemoryMySQLmysql-5.5windows

I'm investigating windows 32bit memory problems (the 2G limit) and am after a test which will max out MySQL's memory including InnoDB Buffer Pool, per connection memory and/or any other uses of memory.

Perhaps a query I could use for mysqlslap?

Update

I thought other people would benefit from a simple way to load test mysql and this was an easy question where lots of people would have "tools". They would help my obscure situation and I could use to investigate our real problem.

We have our configuration tuned, and it was fine for months but recently we eventually get mysqld: Out of memory (Needed 220428 bytes) and MySQL will crash minutes later.

I'm not researching your usual "whoops I allocated too much to my buffer pool" problem. That memory is allocated at start up and improper setting causes immediate shutdown. I'm really after the other 20%. Just creating the maximum number of connections with a high buffer pool hasn't proven to reproduce the problem so I want to stress everything. I am not ruling out stressing the buffer pool in case we are actually suffering a MySQL bug relating to it.

Here is an information dump for people interested:

I'm no authority on any of the above I just want to be able to produce high memory activity so we can A) reproduce the problem and then B) solve it in the medium term before moving off 32bit, which is the real solution but it won't happen tomorrow.

Update 2

Turns out Mysql 5.5's mysqld.exe is already built to be /LARGEADDRESSAWARE but cannot set the innodb engine size equal to or greater than 1600M. Connections do use the extra memory from /3GB (proven with mysqlslap).

vcvars32.bat
dumpbin.exe /headers mysqld.exe
Microsoft (R) COFF/PE Dumper Version 10.00.40219.01
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file mysqld.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
             14C machine (x86)
               5 number of sections
        4E6A3CEF time date stamp Sat Sep 10 04:21:03 2011
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
             122 characteristics
                   Executable
                   Application can handle large (>2GB) addresses
                   32 bit word machine

My conclusion. If you don't use /3GB or otherwise set connection memory + innodb buffer memory greater than available memory you are bound to eventually get mysqld: Out of memory… although mysql seems to do it's best to reject connections to avoid this problem.

Take home message, test that you can actually achieve max_connections with mysqlslap and the rest comes down to maintaining OS memory (and other resources like free system PTE).

Best Answer

From Windows Platform Limitations in the MySQL 5.5 Reference Manual:

On Windows 32-bit platforms, it is not possible by default to use more than 2GB of RAM within a single process, including MySQL.

What's to investigate? We already know what happens when you max the memory: Really Bad Things™.

If there's a remote chance that your setup will periodically require this much memory, 32-bit Windows is the wrong platform.

Almost universally, the biggest memory consumer is of course the InnoDB Buffer Pool, but there's no need for any queries to test this... you don't need any activity at all, or any tables, because the entire amount of memory declared for innodb_buffer_pool_size is allocated immediately when the server starts. The buffer pool never grows, never shrinks, never changes, ever. The number of free pages changes, but those are not "free" from the operating system's perspective -- they're still just as allocated, merely marked as containing nothing of interest within InnoDB.

If the operating system refuses to allocate the amount of memory provisioned for the buffer pool, MySQL will simply refuse to start.

This is illustrated here, where the OP mistakenly believed that the server was crashing and restarting "because" memory couldn't be allocated for the buffer pool, but was in fact crashing for a Linux-specific reason but then refusing to restart because the system would not allocate the total amount of memory required for the pool, due to overuse of available memory by something else... but this allocate-all-at-startup behavior for the InnoDB buffer pool is not platform-specific.

So, you should be able to set this value near the max and then find that taking the server process over the edge should not require very much additional effort at all. But I'm still not sure what the point is.

As you realize, MySQL uses memory for a variety of different purposes, several of which are dynamically-sized, definable on a per-connection basis, and allocated on demand, which makes it virtually impossible to provision a server based on limiting memory usage to some worst-case scenario absolute value, yet expecting that server to be able to handle its typical load efficiently.

The simplest illustration of this is the fact that you can obviously reduce the theoretical maximum memory utilization of a given instance by restricting the maximum number of simultaneous client connections... but any given application needs a certain number of available connections to perform efficiently, and if that number is below your target value, then you're not really solving anything -- it just feels like you have.

I say, either your server has enough memory for the workload, or it doesn't. If it doesn't, then attempting to "tune" your way out of potential trouble is unlikely to offer much in the way of solutions.

Some ideas on how to easily generate demand for more memory... 

SELECT * FROM large_table ORDER BY non_indexed_column;

SELECT * FROM large_table WHERE non_indexed_column = some_value;

SELECT * FROM large_table WHERE some_column LIKE '%a_freqent_match%';

Queries like these could trigger the allocation of a sort buffer, a read buffer and/or a random read buffer, which should make a new request to the OS for the memory that buffer requires.

Simple solution:

DELIMITER $$

DROP PROCEDURE IF EXISTS `test`.`eat_memory_until_server_crashes` $$
CREATE PROCEDURE `test`.`eat_memory_until_server_crashes`()
BEGIN

-- this procedure is intended to eat as much memory as it can
-- it creates a series of consecutively-numbered session variables as large
-- as your configuration will allow them to be.

-- do not run this unless you intend to crash your server

-- also, do not run from a gui tool -- use the mysql command line client:

-- mysql> CALL test.eat_memory_until_server_crashes;

-- if you kill the query or thread before the server crashes, 
-- the memory consumed will be returned to the OS

  DECLARE counter INT DEFAULT 0;
  LOOP
    SET counter = counter + 1;
    SET @qry = CONCAT('SET @crash_me_',counter,' := REPEAT(\'a\', @@max_allowed_packet)');
    SELECT counter, @qry;
    PREPARE hack FROM @qry;
    EXECUTE hack;
    DEALLOCATE PREPARE hack;
    -- adjust timing or remove this entirely depending on how quickly you want this to happen
    DO SLEEP(0.1);
  END LOOP;

END $$

DELIMITER ;

Inspiration for this: Schwartz, Baron; Zaitsev, Peter; Tkachenko, Vadim (2012-03-05). High Performance MySQL: Optimization, Backups, and Replication (Kindle Location 12194). OReilly Media - A. Kindle Edition.

Related Question