Mysql – Few Insert queries rollbacked due to system crash

myisamMySQLmysql-5.5

Well, this problem occured to me first time. Yesterday, one of our mysql system server crashed and when it restarted last few inserts were completely rollbacked.

I am sure that they were proper inserts as we fetch results after inserts to print a PDF and PDF was generated properly. and system was crashed after 2-3 minutes of that INSERT operation.

System is PHP – MySQL based and we are using MyISAM as storage engine for our tables with autocommit ON

This is unique as earlier, after a system crash we normally had crashed tables but nothing like rollback operation. all the table are intact and operational after system crash.

Can anyone explain to me how is that possible if autocommit is ON. Thanks in advance

Best Answer

MyISAM does not support transactions, commits, or rollbacks. You have to switch to the InnoDB.

InnoDB Architecture

InnoDB has mechanisms, log files, and associative memory structures for rollback, commit, transaction isolation, and MVCC.

MyISAM has no infrastructure like this. Each MyISAM table is its own entity.

Here is a great reason to switch to InnoDB: MyISAM does not cache data. It lets the OS do that. That's a scary thought for MySQL/Linux, and horrific nightmare for MySQL/Windows. I wrote about this difference in the past : What are the main differences between InnoDB and MyISAM?

Thus, in the event of a crash, not only do you lose any data the OS did not flush to disk, but you also may have to run CHECK TABLE/REPAIR TABLE to fix open file counts in the headers of the MyISAM tables affected.

In your case, you are playing Russian Roulette relying on data from a MyISAM tables after a server crash or a mysqld crash.

Please see my past posts on using and tuning InnoDB

When to switch from MyISAM to InnoDB?
Which first: upgrade mysql version or convert storage engine? (Code Provided to Convert to InnoDB)

Give it a Try !!!

UPDATE 2014-01-03 10:29 EST

In your comment, you asked

is it possible that data that was inserted 2-3 minutes before crash also lost in MyISAM?

Absolutely !!! I actually saw this happen three ago. I create a MyISAM table in /dev/shm (Shared Memory). I loaded it with a 3GB .MYD and 2GB .MYI. While loading it, I ran ls -l on the table and it kept saying 0 bytes for .MYD and 1024 bytes for .MYI. After 5 min, I killed the mass INSERT. Suddenly, data and index pages started to pour into the table files. Had mysqld actually crashed, those data and index pages would never have made it to the table. In your case, this would especially be true for all open MyISAM tables in a write-heavy database environment.

ASPECT #1

The first thing that caught my eye was this line

InnoDB: Error: trying to load index PRIMARY for table /

This indicates you have a table using the InnoDB Storage Engine

What is interesting about InnoDB is the way a PRIMARY KEY is stored. It is stored in a structure called the gen_clust_index, or more commonly known as the Clustered Index.

My immediate guess is that a certain PRIMARY KEY entry is too big

Please consider some articles on the good, the bad, and the ugly of using long PRIMARY KEYs:

then see if the <DB Hidden>.<Table Hidden> needs to be redesigned.

ASPECT #2

In terms of your conjecture concerning a parallel truncate table, that sounds kind of dangerous. Why? InnoDB performs TRUNCATE TABLE as DDL not DML. I have written about this before:

Jul 09, 2012 : What can cause TRUNCATE TABLE to take a really long time?
Jan 17, 2012 : Problem with InnoDB "per table" file sizes
Sep 28, 2011 : How to Recover an InnoDB table whose files were moved around

ASPECT #3

Some tuning suggestions

Please add the following to my.ini

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0

Start mysql

In another session, run tail -f <errorlogfile> and watch InnoDB Crash Recovery.

If mysql is fully started back up and InnoDB crash recovery has completed, try to shut mysql down immediately. You may need to resize your InnoDB Transaction Logs.

Sorry for these wild suggestions, but I am flying blind here.

Please post the following in the question:

your entire my.cnf
how much RAM is on board

UPDATE 2012-12-05 12:09 EDT

Please do the following:

STEP 01) Add these changes to my.cnf

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0
innodb_thread_concurrency=0

STEP 02) service mysql restart

to make sure mysql comes up

STEP 03) You need to resize ib_logfile0 and ib_logfile1 (24M might be too small)

service mysql stop
cd /var/lib/mysql
mv ib_logfile0 ib_logfile0.bak
mv ib_logfile1 ib_logfile1.bak

STEP 04) Add these changes to my.cnf

[mysqld]
innodb_log_file_size=512M
innodb_log_buffer_size=8M

STEP 05) service mysql start

mysqld will recreate ib_logfile0 and ib_logfile1 512M each

Now, try and see what happens....

UPDATE 2012-12-05 12:18 EDT

In the meantime, please read my ServerFault post on the mysql packet and its sizing implication with regard to the innodb_log_file_size and innodb_log_buffer_size as I learned from someone else's ServerFault post.

UPDATE 2012-12-05 14:28 EDT

I edited all references to customer tables out of this question.

The root cause was a damaged page in ibdata1 with data and index pages mixed inside. I helped Andrew migrate data out, recreate ibdata1 with innodb_file_per_table, and Andrew reloaded the data.