Please review my answer to this recent question. I believe the circumstances are identical.
Do not change your MySQL configuration at this point, as MySQL is not the problem -- it's only a symptom of the problem... which is that you appear to have a system with a small amount of memory and zero swap space.
Your server is not crashing "because" memory can't be allocated for the buffer pool. Your server is crashing... and then is unable to subsequently restart due to the unavailability of system memory. All of the memory configured for the InnoDB buffer pool is requested from the system at mysql startup.
When you see this log message...
120926 08:00:51 mysqld_safe Number of processes running now: 0
...your server has already died. If it hasn't logged anything prior to this, it's not going to log anything about the first crash. The subsequent logs are from after the automatic attempt to restart.
Check your syslog and you should find messages where the kernel went looking for processes to kill due to an extreme out-of-memory condition.
Step 1 would probably be to add some swap space and/or allocating RAM if at all possible.
If that isn't possible, you might actually consider decreasing the innodb-buffer-pool size in your configuration. (I never thought I'd actually hear myself say that). As long as your database is small and your traffic is light, you may not need a buffer pool that large... and since the InnoDB Buffer Pool memory is all allocated at startup whether it's needed or not, this would free up some of your system's memory for whatever else is demanding it. (The 75% to 80%-of-total-RAM recommendation for sizing the buffer pool is only true if the whole server is dedicated to MySQL.)
Step 2 will be to review Apache's forking model and what you might need to do differently in the configuration to prevent it from overwhelming your server. It is pretty likely that uncontrolled growth in quantity or memory requirements of the Apache child processes is starting a cascade of events, resulting in the kernel killing MySQL to try to avoid a complete crash of the entire server.
Depending on how much flexibility you have, you might even consider two separate virtual machines for Apache and MySQL.
Check the MySQL error log. It looks like trying to read table 44 is actually crashing the server.
"Lost connection to MySQL server during query" is (or should be) heart-stopping time for a MySQL DBA because it often means that whatever your query just did has actually crashed the server.
The subsequent messages seem to bear this out:
mysqldump: Got error: 2003: Can't connect to MySQL server on 'localhost' (10061) when
trying to connect
Operation failed with exitcode 2
A quick test of mysqldump on Windows 7 against a MySQL server that is not running returns exactly that same error and exits with %ERRORLEVEL%
set to 2. The nature of error 10061 can be found with the perror
utility.
C:\>perror 10061
Win32 error code 10061: No connection could be made because the target machine actively refused it.
...so, check the MySQL error log. What's likely happening is that you're hitting corruption severe enough that in spite of innodb_force_recovery
, the MySQL Server is crashing. I will then usually restart on its own, but this takes time so may not complete before mysqldump gets an IP connection refused on the subsequent connection attempts. This should all show up in the error log.
I assume that when the dump does not yield any error, the table's data is non-corrupted.
Incorrect assumption. If the dump does not yield any error, it could also mean that that innodb_force_recovery
allowed the server to retrieve some of the data. A "successful" dump of a table only means it's not corrupted so severely that innodb_force_recovery
can't survive it.
1 (SRV_FORCE_IGNORE_CORRUPT)
Let the server run even if it detects a corrupt page. Try to make SELECT * FROM tbl_name
jump over corrupt index records and pages, which helps in dumping tables.
http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
Again, you should see chatter from InnoDB in the MySQL error log.
Final suggestion, don't use workbench. Use the mysql
command line client to explore the server schemata and use mysqldump
directly from the command line to try to restore individual schemas or individual databases.
update/additional:
how do you recover stored procedures? I get an error because they access corrupt tables
It shouldn't be because the procedures access corrupt tables, because stored procedures and functions aren't validated against their referenced tables in the way views are (drop a table that a view references, and then SHOW CREATE VIEW
doesn't work -- the same isn't the case with procs or functions)... MySQL doesn't care whether tables referenced by stored procedures and functions even exist (except at runtime)... however, unlike view definitions, which are stored in individual files, the definitions of procedures and functions are stored in the proc
table in the mysql
schema, which is a MyISAM table... so I would expect that you'd see error messages referencing that table and CHECK TABLE mysql.proc;
and REPAIR TABLE mysql.proc;
might get you going again. Note that the version of MySQL Server that you're using for your recovery needs to be the same release series (e.g. 5.5) or the table definition of mysql.proc
will be wrong... that shouldn't be the issue here, since if I read the question correctly, the crashed server and the recovery server are both 5.5.31.
If you can SELECT * FROM mysql.proc
then you can retrieve the definitions that way if for some reason the SHOW CREATE PROCEDURE
statement doesn't work.
And what options for a (partial) rescue are there for the tables that yielded an error in the dumps?
It depends on the nature and scope of the errors encountered. One option, discussed here, involves repeated INSERT ... SELECT ... LIMIT ... OFFSET
, extracting blocks of rows from readable pages into a MyISAM table (because you can't write to an InnoDB table while innodb_force_recovery
is on). There's also the Percona Data Recovery Tool for InnoDB, though I have never had occasion to use it.
Best Answer
You can run the command
sc delete [MySQL]
where [MySQL] is the name of the service.