Mysql Server Crashing Regularly

MySQL

Error Log: MYSQLD log

140925 20:47:19 mysqld_safe Number of processes running now: 0
140925 20:47:19 mysqld_safe mysqld restarted    
140925 20:47:26  InnoDB: Initializing buffer pool, size = 56.0M    
140925 20:47:26  InnoDB: Completed initialization of buffer pool    
InnoDB: **The log sequence number in ibdata files does not match**    
InnoDB: **the log sequence number in the ib_logfiles!**    
140925 20:47:26  InnoDB: Database was not shut down normally!    
InnoDB: Starting crash recovery.    
InnoDB: Reading tablespace information from the .ibd files...    
InnoDB: Restoring possible half-written data pages from the doublewrite    
InnoDB: buffer...    
140925 20:47:32  InnoDB: Started; log sequence number 4 3219462995    
140925 20:47:32 [Note] Event Scheduler: Loaded 0 events    
140925 20:47:32 [Note] /usr/libexec/mysqld: ready for connections.    
Version: '5.1.71'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution    
Messages Log :

Sep 25 08:15:02 s1 kernel: Out of memory: Kill process 28796 (php-cgi) score 18 or sacrifice child        
Sep 25 08:15:02 s1 kernel: Killed process 28796, UID 528, (php-cgi) total-vm:340760kB, anon-         rss:40048kB, file-rss:21088kB    
Sep 25 08:15:03 s1 kernel: php-cgi invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0,     oom_score_adj=0    
Sep 25 08:15:03 s1 kernel: php-cgi cpuset=/ mems_allowed=0

Server Details:
Operating system: Centos 6.2 linux server,
Memory: 4GB
Swap Size: 2GB

Steps taken to resolve the Issue:

  • Enabled innodb_force_recovery=4 to commit the .ibd files and to mysql server.
  • Tried installing MYSQLTUNER to monitor query activities.However cudnt install it.

Behavior

Have to manually restart it after it shuts down. Not able to restart it automatically using mysqlsafe
Have been seeing this issue since past few days and also a suspect spike in number of hits which reflects in my AW stats.
Will appreciate if someone help me know where to look for solutions and help me configure the mysql config file for a proper performance and memory usage.

My.conf settings: 

[mysqld] datadir=/var/lib/        
mysql socket=/var/lib/mysql/mysql.sock    
 user=mysql    
 symbolic-links=0    
 innodb_file_per_table = 1    
 innodb_buffer_pool_size = 56M    
 thread_concurrency = 8    
 query_cache_size = 32M    
 thread_cache_size = 8    
 myisam_sort_buffer_size = 64M    
 read_rnd_buffer_size = 8M    
 read_buffer_size = 2M    
 sort_buffer_size = 2M    
 table_cache = 512    
 max_allowed_packet = 1M    
 key_buffer = 384M    
 [mysqld_safe] log-error=/var/log/mysqld.log    
 pid-file=/var/run/mysqld/mysqld.pid

Best Answer

Something is eating your Ram, possibly the mysql recover. Add more swap (dd if=/dev/zero count=1000 bs=1M of=swapfile; mkswap swapfile; swapon swapfile) go to singleuser mode (init 1) so that nothing else is running and start your mysql recover. Watch with top or htop for memory usage.

Related Question