MySQL InnoDB 5.6.23 – Cannot Start Segmentation Fault 11

innodbMySQL

I have a Linux Gentoo server which has OwnCloud application installed. It works for about 2 months with out any problems, and sudden this morning MySQL died. I have tried to restart mysql server and in mysqld.err log file a have this:

2015-04-16 21:39:59 28987 [Warning] No argument was provided to --log-bin, 
and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=mysqld-bin' to avoid this problem.
2015-04-16 21:39:59 28987 [Note] InnoDB: Using atomics to ref count buffer pool pages
2015-04-16 21:39:59 28987 [Note] InnoDB: The InnoDB memory heap is disabled
2015-04-16 21:39:59 28987 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2015-04-16 21:39:59 28987 [Note] InnoDB: Memory barrier is not used
2015-04-16 21:39:59 28987 [Note] InnoDB: Compressed tables use zlib 1.2.8
2015-04-16 21:39:59 28987 [Note] InnoDB: Using Linux native AIO
2015-04-16 21:39:59 28987 [Note] InnoDB: Using CPU crc32 instructions
2015-04-16 21:39:59 28987 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-04-16 21:39:59 28987 [Note] InnoDB: Completed initialization of buffer pool
2015-04-16 21:39:59 28987 [Note] InnoDB: Highest supported file format is Barracuda.
2015-04-16 21:39:59 28987 [Note] InnoDB: The log sequence numbers 50092945 and 50092945 in ibdata files do not match the log sequence number 447541274 in the ib_logfiles!
2015-04-16 21:39:59 28987 [Note] InnoDB: Database was not shutdown normally!
2015-04-16 21:39:59 28987 [Note] InnoDB: Starting crash recovery.
2015-04-16 21:39:59 28987 [Note] InnoDB: Reading tablespace information from the .ibd files...
2015-04-16 21:39:59 28987 [Note] InnoDB: Restoring possible half-written data pages
2015-04-16 21:39:59 28987 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 106756973, file name ./mysqld-bin.000032
18:39:59 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=151
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 134417 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x29)[0x857f39]
/usr/sbin/mysqld(handle_fatal_signal+0x350)[0x628700]
/lib64/libpthread.so.0(+0x11260)[0x7f27d6403260]
/usr/sbin/mysqld[0x9a643a]
/usr/sbin/mysqld[0x9a744b]
/usr/sbin/mysqld[0x920e29]
/usr/sbin/mysqld[0x8692f7]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x41)[0x58c251]
/usr/sbin/mysqld[0x6a131a]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x7b8)[0x6a4238]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x749)[0x5848a9]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f27d5a71db5]
/usr/sbin/mysqld[0x57a935]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

After searching and try everything I found on internet, including copying the database to another Linux server, I still cannot start mysql server.

I have even copied the entire database to an windows 7 OS with XAMPP MySQL server and when I try to start the MySQL server it give me the message "Application encountered an error and need to close", refering to mysqld.exe

How can I repair this database and make MySql server running again?

I want to mention that I'm not using any replication, regarding the first warning in mysqld.err file, so not problem there.

Best Answer

This entry in the log seems to indicate a possible corruption problem with mysqld-bin.000032. Those mysql bin files are part of the MySQL binary log process. They were probably being used for replication or for point-in-time backups.

InnoDB: Last MySQL binlog file position 0 106756973, file name ./mysqld-bin.000032

18:39:59 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

So whereever you move the /var/lib/mysql directory, you are also brining along the mysqld-bin files. Which would explain why changing servers doesn't help.

Try moving those mysqld-bin files to another location so MySQL can complete the recovery process.

Update

In the my.cnf file under the [mysqld] section.

Remove:

log-bin variable from my.cnf it is present

Add:

innodb_force_recovery=4

Restart MySQL and let me know. There are additional steps to take to completely restore MySQL, if your innodb tablespace is truly corrupt.

Related Question