MySQL – Understanding Queries Per Second Average

Disabling the binary log on the slave fixed it. This makes me worry about when we upgrade the master to 5.5, but for now things are working much much better.

Edit: The issue was sync_binlog; now that it's set to zero, everything's happy.

That is 100% normal: what you are seeing is the background threads performing the following operations:

• Flushing dirty pages from the buffer pool into the tablespace files. You can see that is happening on your case due to the difference between your LSN and the latest checkpoint on the SHOW ENGINE INNODB STATUS.
• Merging non-unique secondary indexes modifications from the change buffer (inside the buffer pool) into disk. It is almost empty now, but it did 169928 merge inserts since the server started.
• Purging deleted rows so that free space inside the tablespace can be retrieved. Your history lenght is now 604, so it may not be executing right now.
• As a less intensive operation, preloading pages from disk into the buffer pool. There are 22 of those pending, and some of them are being done right now.
• Additionally, if you cancelled an on-going transaction, a thread may be rollbacking those results, and depending on the size of the transaction, it can take very long. Not happening right now, this is a foreground operation.

The first four are background threads to allow better performance on multi-cores systems and its usage kicks in with more intensity during the server idle time, so it is normal that you observe extra IO after stopping a rather intensive benchmark. Normally you should not worry about them. You can monitor them on SHOW ENGINE INNODB STATUS, they do not compromise the durability of your statements (if you were using innodb_flush_log_at_trx_commit = 1, once everything is committed, nothing is lost) and you can reduce its activity at the cost of worse performance:

• innodb_max_dirty_pages_pct can be reduced for more "syncronous I/O". Many other changes can be changed about the flushing policy.
• the purge threads can be reduced and configured to kick in more frequently
• the change buffer can be disabled
• the buffer readers can be configured so it does not load so many pages

None of this changes are recommended unless they are a problem or you have very fast I/O like flash cards/SSDs.

The last point mentioned (rollbacks) is as easy to fix as: don't do them if you have large transactions (or just do not do large transactions if you can suffer rollbacks). The rollback operations is way more costly than a commit.

So your problem is probably in other place. If you are just testing, you can just kill your server and generate a new installation.

There are some comments that I would do on the shown configuration, but I can't do reliably without seeing your hardware and knowing what your test load was. I am just achieving 200.000 inserts/s on 5.6.20, but on tables <100,000,000 rows and doing some dirty tricks. InnoDB tends to be worse the larger the table is, please consider partitioning or a larger buffer pool (it is 100% full here).