I am working on a web app that uses a remote MySQL(5.1) server (dedicated server for database). At times, web server crashes due to high CPU usage. In our application logs, there are some traces that shows there is some problem in communicating with MySQL server.
But, no errors found in MySQL logs and server was accessible from a local machine.
So, I have used mysql tuner to check if there is any problem in database configuration. The tuner result is as follows,
[--] Skipped version check for MySQLTuner script
Please enter your MySQL administrative login: root
Please enter your MySQL administrative password: [OK] Currently running supported MySQL version 5.1.73
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/log/mysqld.log(304K)
[OK] Log file /var/log/mysqld.log exists
[OK] Log file /var/log/mysqld.log is readable.
[OK] Log file /var/log/mysqld.log is not empty
[OK] Log file /var/log/mysqld.log is smaller than 32 Mb
[!!] /var/log/mysqld.log contains 27 warning(s).
[!!] /var/log/mysqld.log contains 188 error(s).
[--] 87 start(s) detected in /var/log/mysqld.log
[--] 1) 180115 16:07:08 [Note] /usr/libexec/mysqld: ready for connections.
[--] 2) 180103 11:08:40 [Note] /usr/libexec/mysqld: ready for connections.
[--] 3) 180103 7:37:06 [Note] /usr/libexec/mysqld: ready for connections.
[--] 4) 180102 10:36:27 [Note] /usr/libexec/mysqld: ready for connections.
[--] 5) 171109 17:10:16 [Note] /usr/libexec/mysqld: ready for connections.
[--] 6) 171031 5:43:04 [Note] /usr/libexec/mysqld: ready for connections.
[--] 7) 171023 10:00:56 [Note] /usr/libexec/mysqld: ready for connections.
[--] 8) 171010 7:48:13 [Note] /usr/libexec/mysqld: ready for connections.
[--] 9) 170920 6:57:47 [Note] /usr/libexec/mysqld: ready for connections.
[--] 10) 170914 9:50:08 [Note] /usr/libexec/mysqld: ready for connections.
[--] 82 shutdown(s) detected in /var/log/mysqld.log
[--] 1) 180115 16:06:57 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 2) 180103 11:08:25 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 3) 180102 10:39:22 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 4) 171109 17:10:13 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 5) 171031 5:42:58 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 6) 171023 10:00:53 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 7) 171010 7:48:11 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 8) 170920 6:57:40 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 9) 170914 9:49:38 [Note] /usr/libexec/mysqld: Shutdown complete
[--] 10) 170804 13:23:47 [Note] /usr/libexec/mysqld: Shutdown complete
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +CSV +InnoDB +MRG_MYISAM
[--] Data in MyISAM tables: 2M (Tables: 851)
[--] Data in InnoDB tables: 9G (Tables: 4306)
[OK] Total fragmented tables: 0
-------- Security Recommendations ------------------------------------------------------------------
[!!] User '@localhost' is an anonymous account.
[!!] User '@localhost' has no password set.
[!!] User 'readonly@%' has no password set.
[!!] User 'root@127.0.0.1' has no password set.
[!!] User 'root@ip-xx-xxx-xxx-x' has no password set.
[!!] User '@localhost' has user name as password.
[!!] User 'readonly@%' hasn't specific host restriction.
[!!] There is no basic password file list!
-------- CVE Security Recommendations --------------------------------------------------------------
[--] Skipped due to --cvefile option undefined
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 28d 18h 49m 49s (265M q [106.609 qps], 28M conn, TX: 485G, RX: 40G)
[--] Reads / Writes: 88% / 12%
[--] Binary logging is disabled
[--] Physical Memory : 7.8G
[--] Max MySQL memory : 7.2G
[--] Other process memory: 147.1M
[--] Total buffers: 4.0G global + 2.7M per thread (1200 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 5.0G (64.32% of installed RAM)
[!!] Maximum possible memory usage: 7.2G (92.79% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (1K/265M)
[OK] Highest usage of available connections: 31% (372/1200)
[OK] Aborted connections: 0.00% (75/28625593)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[!!] Query cache efficiency: 0.0% (0 cached / 57M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (11K temp sorts / 3M sorts)
[!!] Joins performed without indexes: 58834
[OK] Temporary tables created on disk: 0% (147K on disk / 58M total)
[!!] Thread cache is disabled
[!!] Table cache hit rate: 0% (64 open / 8M opened)
[OK] Open file limit used: 0% (2/6K)
[OK] Table locks acquired immediately: 99% (42M immediate / 42M locks)
-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 18.3% (1M used / 8M cache)
[OK] Key buffer size / total MyISAM indexes: 8.0M/1.1M
[OK] Read Key buffer hit rate: 99.9% (251M cached / 180K reads)
[!!] Write Key buffer hit rate: 0.0% (96M cached / 542 writes)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 8
[!!] InnoDB File per table is not activated
[!!] InnoDB buffer pool / data size: 4.0G/9.1G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (0.244140625 %): 5.0M * 2/4.0G should be equal 25%
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 100.00% (818222561475 hits/ 818225469097 total)
[!!] InnoDB Write Log efficiency: 42.33% (2251283 hits/ 5318531 total)
[!!] InnoDB log waits: 0.00% (59 waits / 7569814 writes)
-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.
-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.
-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
Control warning line(s) into /var/log/mysqld.log file
Control error line(s) into /var/log/mysqld.log file
Remove Anonymous User accounts - there are 1 anonymous accounts.
Set up a Password for user with the following SQL statement ( SET PASSWORD FOR 'user'@'SpecificDNSorIp' = PASSWORD('secure_password'); )
Set up a Secure Password for user@host ( SET PASSWORD FOR 'user'@'SpecificDNSorIp' = PASSWORD('secure_password'); )
Restrict Host for user@% to user@SpecificDNSorIp
Reduce your overall MySQL memory footprint for system stability
Enable the slow query log to troubleshoot bad queries
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
Adjust your join queries to always utilize indexes
Set thread_cache_size to 4 as a starting value
Increase table_open_cache gradually to avoid file descriptor limits
Read this before increasing table_open_cache over 64:
Beware that open_files_limit (6000) variable
should be greater than table_open_cache (64)
Read this before changing innodb_log_file_size and/or innodb_log_files_in_group:
Variables to adjust:
*** MySQL's maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
query_cache_size (=0)
query_cache_type (=0)
query_cache_limit (> 1M, or use smaller result sets)
join_buffer_size (> 128.0K, or always use indexes with joins)
thread_cache_size (start at 4)
table_open_cache (> 64)
innodb_file_per_table=ON
innodb_buffer_pool_size (>= 9G) if possible.
innodb_log_file_size should be (=512M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
innodb_log_buffer_size (>= 1M)
With this result, I believe increasing innodb_buffer_pool will fix my problem. But, to increase this I need to increase RAM of the instance. (current RAM: 8GB, DB size: 9.1GB).
UPDATE:
my.cnf file
[client]
socket=/home/mysqldbloc/mysql.sock
[mysqld]
innodb_buffer_pool_size=4096M
datadir=/home/mysqldbloc
socket=/home/mysqldbloc/mysql.sock
user=mysql
lower_case_table_names=1
symbolic-links=0
max_connections=1200
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
log-slow-queries=/var/run/mysqld/mysqld-slow.log
So, how much RAM should I allocate to this instance so that there wont be any communication problem? Also, currently our web application is very slow. Will increase in RAM make it fast?
Best Answer
Mostly this Answer is asking for more details and making minor suggestion. However, some of this could lead to some concrete answers after another iteration.
No, you do not necessarily need more RAM. The buffer_pool is a cache. Recently used things tend to live there; infrequently used things tend not to.
High I/O would be an indicator of the cache not being big enough. But, since you are complaining about high CPU, the likely problem is inefficient queries. This is often solved by adding a composite index or reformulating the queries. Please find what queries are hogging the CPU, then let's discuss them.
851 MyISAM tables -- Consider migrating to InnoDB.
4306+851 tables -- Why so many? This may be a sign of poor schema design.
106qps is about the median -- that is not taxing the server.
Look at the actual log file, not mysqltuner's summary. It may indicate that you are running out of memory or something else. (The numbers you have presented should not lead to out of memory.) Is the server dedicated to MySQL, or are other memory-hogs running?
How much swap space do you have? Was it swapping when it "crashed". Swapping is terrible for MySQL performance, but it should not lead to a crash.
The 5M for
innodb_log_file_sizeis an old default; 100M would be better. Caution: It is a tricky setting to change in 5.1.Much of mysqltuner's suggestions are fake news. But I do agree that
table_open_cacheshould be increased. Perhaps to 500.Wilson -- I would say that a "few"
Handler_rollbackper hour is median. So 1394 in 29 days is not remarkable. (This is from surveying 127 disparate MySQL instances.)An
Aborted_clientevery 18 seconds -- that is high; about the 90th percentile.I think
log_warnings=2was new with version 5.7. 5.1 is antique; I recommend you work on upgrading before it becomes really difficult to.Further analysis of GLOBAL STATUS and VARIABLES
Observations:
The More Important Issues:
Increase
table_open_cacheto, say, 500.innodb_buffer_pool_sizecould be raised to 5500M if you are mostly using InnoDB and you do not have other apps running on the same server.Raise
innodb_log_buffer_sizeto 2M.Use the slowlog to find the slow queries; then let's work on them. Use
long_query_time=1. See http://mysql.rjweb.org/doc.php/mysql_analysis#slow_queries_and_slowlog for more on getting started.Are you using a framework that does
SHOW VARIABLESandSHOW COLLATIONS12 times a second?? What a mess.Change to
thread_cache_size = 20. (0 is inefficient for Linux.)Doing
CREATE PROCEDUREabout once a minute? Why? AndDROP PROCEDUREhalf as often.Details and other observations:
( innodb_buffer_pool_size / _ram ) = 4096M / 8192M = 50.0%-- % of RAM used for InnoDB buffer_pool( Opened_tables ) = 8,516,944 / 2569789 = 3.3 /sec-- Frequency of opening Tables -- increase table_open_cache( table_open_cache ) = 64-- Number of table descriptors to cache -- Several hundred is usually good.( innodb_buffer_pool_size ) = 4096M-- InnoDB Data + Index cache -- 128M (an old default) is woefully small.( Innodb_pages_read/Innodb_data_reads ) = 24,505,278/4361973 = 561.8%-- Seems like these values should be equal?( innodb_log_buffer_size ) = 1M-- Suggest 2MB-64MB, and at least as big as biggest blob set in transactions. -- Adjust innodb_log_buffer_size.( innodb_file_per_table ) = OFF-- Put each file in its own tablespace -- (Mildly recommended, especially for large tables)( innodb_additional_mem_pool_size ) = 1M-- (deprecated in 5.6.3, removed in 5.7.4.) -- A high value is unnecessary.( innodb_stats_on_metadata ) = ON-- Re-analyze table when touching stats. -- ON is likely to slow down certain SHOWs and information_schema accesses.( local_infile ) = ON-- local_infile = ON is a potential security issue( Created_tmp_tables ) = 60,089,656 / 2569789 = 23 /sec-- Frequency of creating "temp" tables as part of complex SELECTs.( Com_show_variables ) = 29,563,363 / 2569789 = 12 /sec-- SHOW VARIABLES ... -- Why are you requesting the VARIABLES so often?( Select_scan ) = 61,648,783 / 2569789 = 24 /sec-- full table scans -- Add indexes / optimize queries (unless they are tiny tables)( Select_scan / Com_select ) = 61,648,783 / 59135200 = 104.3%-- % of selects doing full table scan. (May be fooled by Stored Routines.) -- Add indexes / optimize queries( binlog_format ) = STATEMENT-- STATEMENT/ROW/MIXED. ROW is preferred; it may become the default.( expire_logs_days ) = 0-- How soon to automatically purge binlog (after this many days) -- Too large (or zero) = consumes disk space; too small = need to respond quickly to network/machine crash. (Not relevant if log_bin = OFF)( innodb_autoinc_lock_mode ) = 1-- Galera: desires 2 -- 2 = "interleaved"; 1 = "consecutive" is typical; 0 = "traditional".( log_slow_queries ) = OFF-- Whether to log slow queries. (Before 5.1.29, 5.6.1)( slow_query_log ) = OFF-- Whether to log slow queries. (5.1.12)( long_query_time ) = 10.000000 = 10-- Cutoff (Seconds) for defining a "slow" query. -- Suggest 2( Connections ) = 29,566,069 / 2569789 = 12 /sec-- Connections -- Increase wait_timeout; use pooling?( thread_cache_size ) = 0-- How many extra processes to keep around (Not relevant when using thread pooling) (Autosized as of 5.6.8; based on max_connections) 0 is inefficient for non-Windows; 10 is probably fine. >100 may lead to OOM.( Threads_created / Connections ) = 29,566,068 / 29566069 = 100.0%-- Rapidity of process creation -- Increase thread_cache_size (non-Windows)( Threads_created ) = 29,566,068 / 2569789 = 12 /sec-- Frequency of needing to create a new process. -- Increase thread_cache_size (non-Windows)( thread_cache_size / max_connections ) = 0 / 1200 = 0-- (0 for Windows)You have the Query Cache half-off. You should set both query_cache_type = OFF and query_cache_size = 0 . There is (according to a rumor) a 'bug' in the QC code that leaves some code on unless you turn off both of those settings.
Abnormally large:
Abnormal strings: