Mysql – Server with mariadb sometimes dies under high load

mariadbmariadb-10.5MySQL

I have a dedicated server i7-4770 with 32GB RAM. On server installed CentOs 7.8, MariaDB 10.5.8, PHP 7.4.14, Nginx 16.1. All tables in the database are InnoDB.

When the load on the database increases significantly and I see more than 20,000,000 requests per hour in PHPMyAdmin, the server dies unexpectedly. After I reboot the server from the hosting company's control panel – the server is working fine. But as soon as the load on the database becomes large, the server dies again.
In the logs of MariaDB 10.5.8, before the server dies, only 1 error appears:

[Warning] Detected table cache mutex contention at instance 1: 22% waits. Additional table cache instance cannot be activated: consider raising table_open_cache_instances. Number of active instances: 1.

I think the reason for the death of the server is in her. But the problem is that when I specify table_open_cache_instances = 8 and restart the MariaDB, the request

SHOW VARIABLES LIKE 'table_open_cache_instances'

gives me a value of 1. After reboot server value also remains 1. Other values such as max_connections are set correctly.

My /etc/my.cnf.d/server.cnf (=my.cnf)

slow_query_log      = 1
slow_query_log_file = /var/log/mariadb/slow_queries.log
long_query_time     = 1

max_connections = 836
key_buffer_size = 256M
read_buffer_size = 16M
join_buffer_size = 16M

query_cache_size = 0
query_cache_type = 0
query_cache_limit = 1M

tmp_table_size = 512M
max_heap_table_size = 512M

innodb_buffer_pool_size = 22G
innodb_log_file_size = 512M

innodb_flush_log_at_trx_commit = 0

thread_cache_size = 512

table_open_cache = 207000
open_files_limit = 207000

table_definition_cache     = 1400

table_open_cache_instances = 8

How can I solve this problem? How do I edit my MariaDB 10.5.8 config?

Best Answer

I assume you hit the open_files_limit. This should be indicated somewhere further up of your warning about the instances in your MariaDB Error Log.

The problem here is, that MariaDB behaves differently than MySQL:

In MySQL the number of FD (open_files_limit) used is: table_open_cache x 2 + ... In MariaDB the number of DD used is table_open_cache x table_open_cache_instances x 2 + ...

https://mariadb.com/kb/en/server-system-variables/#table_open_cache_instances

Why they made it different I do not know.

So in your case you have:

table_open_cache = 207000 x table_open_cache_instances = 8 = requiring 1.6M TOC entries but you only have defined 207k as open_files_limit = 207000

So either lower the table_open_cache by at least 8 times or increase the open_files_limit by at least 8 times.

3 additional thoughts:

  • Do you really need 200k or 1.6M FD??? That happens only very rarely when you have multi-tenant set-ups or 100ks of tables.

See also here: https://fromdual.com/mysql-shared-hosting-configuration

  • MariaDB: table_open_cache x table_open_cache_instances x 2 + max_connections + 40
  • MySQL: table_open_cache x 2 + max_connections + 10

Regards, Oli