Mysql – InnoDB table SELECT returns ERROR 2006 (HY000): MySQL server has gone away (after power outage)

innodbmyisamMySQLmysql-5.5

This server is a virtual machine running CentOS 6.0, MySQL 5.5.21

There is a database named devSystem. There are InnoDB tables within. Running the following commands causes error ERROR 2006 (HY000): MySQL server has gone away. I have not previously used InnoDB extensively and so have not yet experienced this type of issue. I can only assume it's specific to InnoDB as it has not surfaced using MyISAM. Anyway, the commands I attempt to run are as follows.

mysql -u root -p
mysql> USE `devSystem`;
-- database changed
mysql> SHOW TABLES;

Full error as returned by MySQL client

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    1
Current database: devSystem

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
ERROR:
Can't connect to the server

Now trying to SELECT from table baskets which exists in devSystem

mysql> select * from baskets\G
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    1
Current database: devSystem

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
ERROR:
Can't connect to the server

For comparison, I tried a second database which also has InnoDB tables on same server

mysql> USE `testSystem`;
-- database changed
mysql> SHOW TABLES;

This output the tables correctly. For this reason I surmise that it's an issue with devSystem specifically.

I have tried searching on here however other similar queries don't seem to help, does anyone have any suggestions/advice that will help me solve this? This has so far wasted my whole morning!

Current options seem to be to remove the database and start over (however will lose a significant amount of work. I have an almost-current database design however no backup of the data that had been generated to-date)

update 1
Adding innodb_force_recovery = 6 into my.cnf allows SHOW TABLE STATUS to execute successfully, values of <= 5 still result in the error as shown above. With this flag SELECT * FROM baskets operates, however one table specifically has the error returning still indicating it is one table that perhaps is causing the problem?

mysql> SELECT * FROM supplierOptionalExtras_relationships;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    1
Current database: devSystem

ERROR 2013 (HY000): Lost connection to MySQL server during query

CONCLUSION

Ultimately following more hours of scanning through recovery files I can only conclude that my data was lost (as RolandoMySQLDBA had said). To get as far as I did I followed DTest's suggestion on the Percona Recovery Tools, however the data corruption meant that the tools were unable to extract the data from my ibdata1 file for the specific table.

In the end I used RolandoMySQLDBA's answer and did the following

  1. Followed instructions here https://stackoverflow.com/questions/3927690/howto-clean-a-mysql-innodb-storage-engine/4056261#4056261 excluding the crashed database
  2. Used innodb_force_recovery = 6 to get all the non-crashed table data from the database with the crashed table
  3. Shut down MySQl, removed innodb_force_recovery = 6, removed the ibdata/iblog files (as detailed in link in step 1)
  4. Started MySQL and loaded dumped data
  5. Recreated bad table from design files
  6. Manually repopulated data

Of course this meant total data loss from the effected table which is a concern, however I can only hope that adding innodb_file_per_table will help with data recovery should this recur – I intend to kill the power at some point to attempt to try and reproduce this on test database.

Below is some more verbose information from the logs.

my.cnf

This is just a dev system so the my.cnf is VERY basic, in fact unchanged from default

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
old_passwords=1
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

MySQL logs from server start to before failing query executed

120302 10:57:42 [Note] /usr/libexec/mysqld: Normal shutdown

120302 10:57:42 [Note] Event Scheduler: Purging the queue. 0 events
120302 10:57:42  InnoDB: Starting shutdown...
120302 10:57:42  InnoDB: Shutdown completed; log sequence number 285938465
120302 10:57:42 [Note] /usr/libexec/mysqld: Shutdown complete

120302 10:57:42 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
120302 10:57:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
120302 10:57:43 [Note] Plugin 'FEDERATED' is disabled.
120302 10:57:43 InnoDB: The InnoDB memory heap is disabled
120302 10:57:43 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120302 10:57:43 InnoDB: Compressed tables use zlib 1.2.3
120302 10:57:43 InnoDB: Using Linux native AIO
120302 10:57:43 InnoDB: Initializing buffer pool, size = 128.0M
120302 10:57:43 InnoDB: Completed initialization of buffer pool
120302 10:57:43 InnoDB: highest supported file format is Barracuda.
120302 10:57:43  InnoDB: Waiting for the background threads to start
120302 10:57:44 InnoDB: 1.1.8 started; log sequence number 285938465
120302 10:57:44 [Note] Event Scheduler: Loaded 0 events
120302 10:57:44 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.15'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi

MySQL log rows following failing query

120302 10:58:39  InnoDB: Assertion failure in thread 140030446421760 in file btr0btr.c line 695
InnoDB: Failing assertion: (ibool)!!page_is_comp(buf_block_get_frame(block)) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
120302 10:58:39 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338483 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x30396e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f5b61043d98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x33)[0x77b6e3]
/usr/libexec/mysqld(handle_segfault+0x465)[0x50fc35]
/lib64/libpthread.so.0[0x37a9e0f4c0]
/lib64/libc.so.6(gsignal+0x35)[0x37a9a329a5]
/lib64/libc.so.6(abort+0x175)[0x37a9a34185]
/usr/libexec/mysqld[0x82a4bb]
/usr/libexec/mysqld[0x82a60c]
/usr/libexec/mysqld[0x85d133]
/usr/libexec/mysqld[0x862207]
/usr/libexec/mysqld[0x7e37a3]
/usr/libexec/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x3d)[0x66903d]
/usr/libexec/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x537)[0x5edf67]
/usr/libexec/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0xc33)[0x54d2d3]
/usr/libexec/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x683)[0x54e043]
/usr/libexec/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x4b)[0x54e61b]
/usr/libexec/mysqld(_Z18mysqld_list_fieldsP3THDP10TABLE_LISTPKc+0x23)[0x5c5623]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x174f)[0x581f2f]
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0xd2)[0x60e9e2]
/usr/libexec/mysqld(handle_one_connection+0x50)[0x60eaf0]
/lib64/libpthread.so.0[0x37a9e077e1]
/lib64/libc.so.6(clone+0x6d)[0x37a9ae18ed]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f5b50004c20): is an invalid pointer
Connection ID (thread ID): 2
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120302 10:58:39 mysqld_safe Number of processes running now: 0
120302 10:58:39 mysqld_safe mysqld restarted
120302 10:58:39 [Note] Plugin 'FEDERATED' is disabled.
120302 10:58:39 InnoDB: The InnoDB memory heap is disabled
120302 10:58:39 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120302 10:58:39 InnoDB: Compressed tables use zlib 1.2.3
120302 10:58:39 InnoDB: Using Linux native AIO
120302 10:58:39 InnoDB: Initializing buffer pool, size = 128.0M
120302 10:58:39 InnoDB: Completed initialization of buffer pool
120302 10:58:39 InnoDB: highest supported file format is Barracuda.
120302 10:58:39  InnoDB: Waiting for the background threads to start
120302 10:58:40 InnoDB: 1.1.8 started; log sequence number 285938465
120302 10:58:40 [Note] Event Scheduler: Loaded 0 events
120302 10:58:40 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.15'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi
120302 10:58:47  InnoDB: Assertion failure in thread 140051237820160 in file btr0btr.c line 695
InnoDB: Failing assertion: (ibool)!!page_is_comp(buf_block_get_frame(block)) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
120302 10:58:47 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1
max_threads=151
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338483 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x231b6e0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f603847cd98 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x33)[0x77b6e3]
/usr/libexec/mysqld(handle_segfault+0x465)[0x50fc35]
/lib64/libpthread.so.0[0x37a9e0f4c0]
/lib64/libc.so.6(gsignal+0x35)[0x37a9a329a5]
/lib64/libc.so.6(abort+0x175)[0x37a9a34185]
/usr/libexec/mysqld[0x82a4bb]
/usr/libexec/mysqld[0x82a60c]
/usr/libexec/mysqld[0x85d133]
/usr/libexec/mysqld[0x862207]
/usr/libexec/mysqld[0x7e37a3]
/usr/libexec/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x3d)[0x66903d]
/usr/libexec/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x537)[0x5edf67]
/usr/libexec/mysqld(_Z10open_tableP3THDP10TABLE_LISTP11st_mem_rootP18Open_table_context+0xc33)[0x54d2d3]
/usr/libexec/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x683)[0x54e043]
/usr/libexec/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x4b)[0x54e61b]
/usr/libexec/mysqld(_Z18mysqld_list_fieldsP3THDP10TABLE_LISTPKc+0x23)[0x5c5623]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x174f)[0x581f2f]
/usr/libexec/mysqld(_Z24do_handle_one_connectionP3THD+0xd2)[0x60e9e2]
/usr/libexec/mysqld(handle_one_connection+0x50)[0x60eaf0]
/lib64/libpthread.so.0[0x37a9e077e1]
/lib64/libc.so.6(clone+0x6d)[0x37a9ae18ed]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f6014004c20): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
120302 10:58:47 mysqld_safe Number of processes running now: 0
120302 10:58:47 mysqld_safe mysqld restarted
120302 10:58:47 [Note] Plugin 'FEDERATED' is disabled.
120302 10:58:47 InnoDB: The InnoDB memory heap is disabled
120302 10:58:47 InnoDB: Mutexes and rw_locks use GCC atomic builtins
120302 10:58:47 InnoDB: Compressed tables use zlib 1.2.3
120302 10:58:47 InnoDB: Using Linux native AIO
120302 10:58:47 InnoDB: Initializing buffer pool, size = 128.0M
120302 10:58:47 InnoDB: Completed initialization of buffer pool
120302 10:58:47 InnoDB: highest supported file format is Barracuda.
120302 10:58:47  InnoDB: Waiting for the background threads to start
120302 10:58:48 InnoDB: 1.1.8 started; log sequence number 285938465
120302 10:58:48 [Note] Event Scheduler: Loaded 0 events
120302 10:58:48 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.15'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL) by Remi

Best Answer

This looks oddly familiar.

I have seen this occur with one of my web hosting client's DB servers. There was a particular table that crashed mysqld every single time you accessed it, even with SHOW CREATE TABLE.

The problem stems from a corrupt data dictionary. There is really no way to correct it. You could attempt to alter the tablespace_id within the .ibd file but the headache stems from locating the tablespace_id list internal to ibdata1.

Even if you create a MyISAM table with the same name in the same database as the original InnoDB table, you cannot convert it to InnoDB because the tablespace_id is already associated with the table name. This, of course, is a corrupted state. It's like having a pidgeon hole in ibdata1 that you cannot patch up without some exploratory surgery.

You may have to mysqldump everything except the database that houses the corrupt table. You would then have to mysqldump every table in that database except the corrupt table. Remember, it is the data dictionary's view of the table that is screwed up, not necessarily the table's data.

The only sure way to clean everything up is to perform the mysqldumps as I just specified, shutdown mysql, rm -rf all DB folders except /var/lib/mytsql/mysql, delete ibdata1, delete ib_logfile0, delete ib_logfile1, startup mysql, reload all mysqldumps. See my StackOverflow post about cleaning up your InnoDB infrastructure.

Since you are not using innodb_file_per_table, any tables with this corrupt state of things within ibdata1 are lost as casualities of war. My condolences.

For future reference, click here to see an artistic conception of InnoDB and its Internals.

Related Question