InnoDB: We intentionally generate a memory trap.
It looks like InnoDB is deliberately crashing itself because it's encountering a problem with your on-disk data structures that it cannot safely recover from.
What right happened before this? Unexpected power loss, hard reboot, lock-up? Upgrade? Updates? Restoring files from a backup? Is there anything interesting at the OS level in /var/log/messages
?
Knowing that information might change the way you proceed somewhat, but the net effect is that this portion of the error message is leading you in the right direction:
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
You will need to force recovery to have any chance of getting to your data. Be patient and use the lowest value possible to get the server started so you can make a backup of as much data as possible.
Of course, if there are any underlying hardware issues, those need to be addressed first.
Update: suggestion: put innodb_force_recovery = 6
into the [mysqld]
section of your my.cnf file instead of passing it on the command line. I don't know how the initscripts will handle it. Or, you can leave it where it is but put a nonsense value like --innodb_force_recovery=kitten
... just to make sure the option is really being seen by mysqld.
At some point, you should get a message in the log about the option being set, but I'm not sure where it is, in time, related to what you're seeing... so I want to be sure it's really getting set.
Here's a normal 5.5.28 startup with innodb_force_recovery
= 1. (Sorry, I don't have a 5.1 that I can take offline at the moment).
130131 12:47:16 InnoDB: Completed initialization of buffer pool
130131 12:47:16 InnoDB: highest supported file format is Barracuda.
130131 12:47:21 InnoDB: Waiting for the background threads to start
130131 12:47:22 InnoDB: 1.1.8 started; log sequence number 2872409775300
130131 12:47:22 InnoDB: !!! innodb_force_recovery is set to 1 !!!
It seems to me that this message is appearing a little too late in the show, which may be the simple explanation for why you aren't seeing it.
OMG !!! Look at the output of df -h
Size: 20G Used: 19G Available 0 Use Percent: 100%
YOU HAVE NO DISK SPACE !!!
Get more diskspace freed up or have linode
increase diskspace.
According to MySQL 5.0 Certification Study Guide
Page 408,409 Section 29.2 Bulletpoint 11 says:
If you run out of disk space while adding rows to a MyISAM table, no
error occurs. The server suspends the operation until space becomes
available, and then completes the operation.
In reality, mysqld is not dead. It is waiting for free diskspace to continue whatever it normally does. Even during startup, if mysqld makes a temp table, it is a MyISAM table. No space left on the disk? No problem. MyISAM just patiently waits for you to make free space.
If you have an error file like /var/log/mysql/log
, clear its contents like this:
echo -n > /var/log/mysqld.log
If the error file in /var/lib/mysql/my-machine.err
then do this:
echo -n > /var/lib/mysql/my-machine.err
Give it a Try !!!
Best Answer
When you stop or start MySQL, your concern should be on the state of data and connectivity during the process.
PROCESSES (Shutdown)
sync-binlog
, mysqld totally relies on the OS for flushing final binary log entries.PROCESSES (Startup)
skip-innodb
enabled, all InnoDB startup is bypassedREPLICATION
If MySQL Replication is up and running at the time of the shutdown, note the following Protocols:
MASTER
Protocol : mysqld is only concerned withSLAVE
Protocol : mysqld will runSTOP SLAVE;
master.info
file as to which SQL command from the Master was last downloaded, recorded in its relay logs and executed.MASTER/SLAVE
Protocol : If a MySQL Instance is both Master and Slave, individual Master and Slave protocols both applyOn startup, if MySQL Replication is configured on a Slave,
master.info
master.info
and last relay logmaster.info
DATA (MyISAM)
Since MyISAM tables are not transactional and the only caching done is for index lookups, the MyISAM key cache is simply discarded. All open file handles to the
.frm
,.MYD
, and.MYI
are closed and open file handle count for any MyISAM is is decremented. If mysqld crashes, the open file handle count for any open MyISAM table are the time of a crash that is greater than 0 causes mysqld to view the MyISAM table as crashed. RunningREPAIR TABLE
on that MyISAM may be necessary on startup (Must be done manually or you can have mysqld configured to do that automatically)On startup, if MyISAM recovery option is enabled, then
REPAIR TABLE
is executed for any closed MyISAM table that has a nonzero value for open file handles.DATA (InnoDB)
Since InnoDB is a Transactional Storage Engine, there are steps to doing shutdown.
If innodb_fast_shutdown is enabled, any lingering transactions within the InnoDB infrastructure (double write buffer, InnoDB Log Files) are retained in the files on shutdown. The changes are acted upon during the next startup of mysqld.
If innodb_fast_shutdown is disabled, any lingering transactions within the InnoDB infrastructure (double write buffer, InnoDB Log Files) are completed on shutdown. This makes for a faster mysqld startup.