Yesterday my database crashed and then restarted; after the automatic crash recovery started normally and it's working fine; is there a way to know what caused it? I tried searching in the binlogs for some "strange" query or something with the log, but can't find what happened. Here's what the error log shows:
2017-08-30 16:33:19 7f87ddacb700 InnoDB: Assertion failure in thread 140221516396288 in file row0sel.cc line 2840
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
21:33:19 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured.
This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.
key_buffer_size=268435456
read_buffer_size=2097152
max_used_connections=110
max_threads=250
thread_count=56
connection_count=54 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1289343 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x37913940
Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong...
stack_bottom = 7f87ddacaec0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x8e770b]
/usr/sbin/mysqld(handle_fatal_signal+0x491)[0x674bb1]
/lib64/libpthread.so.0(+0xf370)[0x7f906980d370]
/lib64/libc.so.6(gsignal+0x37)[0x7f90684081d7]
/lib64/libc.so.6(abort+0x148)[0x7f90684098c8]
/usr/sbin/mysqld[0x585801]
/usr/sbin/mysqld[0x585e6f]
/usr/sbin/mysqld[0x9c69ff]
/usr/sbin/mysqld[0x91e361]
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0xe4)[0x5b7804]
/usr/sbin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0x66)[0x5bbba6]
/usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0x86)[0x5b3736]
/usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x5a)[0x7f4aea]
/usr/sbin/mysqld[0x81887d]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0xdd)[0x6d013d]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x408)[0x6cf4a8]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_P10SQL_I_ListI8st_orderESB_S7_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x275)[0x7180b5]
/usr/sbin/mysqld(_Z13handle_selectP3THDP13select_resultm+0x195)[0x718945]
/usr/sbin/mysqld[0x57a508]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x269e)[0x6f35ee]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x368)[0x6f7838]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xf82)[0x6f8f62]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x152)[0x6c5fe2]
/usr/sbin/mysqld(handle_one_connection+0x40)[0x6c60a0]
/lib64/libpthread.so.0(+0x7dc5)[0x7f9069805dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f90684ca76d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f858e4f9420): is an invalid pointer
Connection ID (thread ID): 442657
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
The SO is CentOS Linux release 7.3.1611 (Core). Also searched for SO errors but the only thing found was something after it crashed (something from "sd.collector… Can't connect to MySQL server").
Is there something I'm missing? Tried searching some errors like that, but there are many and very different. I would like to know if I can know what caused it and if it can happen again.
Best Answer
We've seen this in my shop, and traced it to this bug, fixed here in mysqld 5.7-22.
https://dev.mysql.com/doc/relnotes/mysql/5.7/en/news-5-7-22.html
InnoDB: An assertion was raised when a thread attempted to read a record containing BLOB data while another thread was writing the same data to external pages. (Bug #26300119)