MySql InnoDB crash recovery – File ./ibdata1: ‘open’ returned OS error 71. Cannot continue operation

innodbMySQLmysql-8.0

Mysql 8.0.23 on CentOS 7.9 with WHM

I ended up with a corrupted InnoDB table. I gave up on a long query and killed it. Then the query was still blocking so I did a forceful server reboot. I know now that the query was still blocking because it was undoing the process. Killing it with a forceful server reboot corrupted my database. The rest is history.

So I've been trying to work through this process: https://dev.iserversupport.com/recover-crashed-innodb-tables-on-mysql-database-server/

But it seems MySQL doesn't want to recreate the ibdata1 file.

my.cnf:

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/8.0/en/server-configuration-defaults.html

[mysqld] disable-log-bin=1 default-authentication-plugin=mysql_native_password performance-schema=0
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove the leading "# " to disable binary logging
# Binary logging captures changes between backups and is enabled by
# default. It's default setting is log_bin=binlog
# disable_log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
#
# Remove leading # to revert to previous value for default_authentication_plugin,
# this will increase compatibility with older clients. For background, see:
# https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_default_authentication_plugin
# default-authentication-plugin=mysql_native_password

datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock

log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid innodb_file_per_table=1


default-storage-engine=MyISAM interactive_timeout=300 key_cache_block_size=4096 max_heap_table_size=5G max_join_size=1000000000 max_allowed_packet=268435456 open_files_limit=40000 thread_cache_size=100 tmp_table_size=5G wait_timeout=7800 max_user_connections=500 myisam_recover_options=FORCE innodb_file_per_table=1 innodb_flush_log_at_trx_commit=0 innodb_purge_threads=1 innodb_thread_concurrency=8 pid-file=/var/lib/mysql/mysqld.pid


performance-schema=0 sql_mode='' interactive_timeout=30 wait_timeout=90

collation-server = utf8_unicode_ci character-set-server=utf8

key_buffer_size = 4G innodb_buffer_pool_size = 8G innodb_buffer_pool_chunk_size = 256M innodb_buffer_pool_instances = 8

max_connections=300

And error log:

2021-01-19T18:18:18.543601Z 1 [ERROR] [MY-012592] [InnoDB] Operating system error number 2 in a file operation.
2021-01-19T18:18:18.543714Z 1 [ERROR] [MY-012593] [InnoDB] The error means the system cannot find the path specified.
2021-01-19T18:18:18.543792Z 1 [ERROR] [MY-012594] [InnoDB] If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2021-01-19T18:18:18.543895Z 1 [ERROR] [MY-012646] [InnoDB] File ./ibdata1: 'open' returned OS error 71. Cannot continue operation
2021-01-19T18:18:18.543986Z 1 [ERROR] [MY-012981] [InnoDB] Cannot continue operation.
2021-01-19T18:18:19.908326Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.23) starting as process 1777
2021-01-19T18:18:19.909547Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2021-01-19T18:18:19.909570Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8_unicode_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
2021-01-19T18:18:19.959205Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.

This makes me think I have an ownership or read/write permissions error. But I checked the ownership on the /var/lib/mysql directory and it is mysql:mysql, permission is 751.

I was able to recover all of my data by forcing innodb_force_recovery = 3. But now that I've deleted those ibd files I can't get the server to start again either way. I did back them up, and I've tried copying them back but no dice.

I also attempted to re-install mysql via WHM. That failed.

Thank you Matt

Best Answer

I ended up resolving it by deleting the /var/lib/mysql directory and running

mysqld --initialize --datadir=/var/lib/mysql

ASPECT #1

The first thing that caught my eye was this line

InnoDB: Error: trying to load index PRIMARY for table /

This indicates you have a table using the InnoDB Storage Engine

What is interesting about InnoDB is the way a PRIMARY KEY is stored. It is stored in a structure called the gen_clust_index, or more commonly known as the Clustered Index.

My immediate guess is that a certain PRIMARY KEY entry is too big

Please consider some articles on the good, the bad, and the ugly of using long PRIMARY KEYs:

then see if the <DB Hidden>.<Table Hidden> needs to be redesigned.

ASPECT #2

In terms of your conjecture concerning a parallel truncate table, that sounds kind of dangerous. Why? InnoDB performs TRUNCATE TABLE as DDL not DML. I have written about this before:

Jul 09, 2012 : What can cause TRUNCATE TABLE to take a really long time?
Jan 17, 2012 : Problem with InnoDB "per table" file sizes
Sep 28, 2011 : How to Recover an InnoDB table whose files were moved around

ASPECT #3

Some tuning suggestions

Please add the following to my.ini

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0

Start mysql

In another session, run tail -f <errorlogfile> and watch InnoDB Crash Recovery.

If mysql is fully started back up and InnoDB crash recovery has completed, try to shut mysql down immediately. You may need to resize your InnoDB Transaction Logs.

Sorry for these wild suggestions, but I am flying blind here.

Please post the following in the question:

your entire my.cnf
how much RAM is on board

UPDATE 2012-12-05 12:09 EDT

Please do the following:

STEP 01) Add these changes to my.cnf

[mysqld]
max_allowed_packet=1G
innodb_fast_shutdown=0
innodb_thread_concurrency=0

STEP 02) service mysql restart

to make sure mysql comes up

STEP 03) You need to resize ib_logfile0 and ib_logfile1 (24M might be too small)

service mysql stop
cd /var/lib/mysql
mv ib_logfile0 ib_logfile0.bak
mv ib_logfile1 ib_logfile1.bak

STEP 04) Add these changes to my.cnf

[mysqld]
innodb_log_file_size=512M
innodb_log_buffer_size=8M

STEP 05) service mysql start

mysqld will recreate ib_logfile0 and ib_logfile1 512M each

Now, try and see what happens....

UPDATE 2012-12-05 12:18 EDT

In the meantime, please read my ServerFault post on the mysql packet and its sizing implication with regard to the innodb_log_file_size and innodb_log_buffer_size as I learned from someone else's ServerFault post.

UPDATE 2012-12-05 14:28 EDT

I edited all references to customer tables out of this question.

The root cause was a damaged page in ibdata1 with data and index pages mixed inside. I helped Andrew migrate data out, recreate ibdata1 with innodb_file_per_table, and Andrew reloaded the data.

Best Answer

Related Solutions

Thesql wont start after increasing innodb_buffer_pool_size and innodb_log_file_size

MySQL INNODB corruption after server crash during concurrent truncate command