Linux – What does “task thesqld:xxx blocked for more than 120 seconds” mean

linux-kernelmemoryMySQLprocess

We are troubleshooting a MySQL issue where some queries are taking a very long time complete and I see many of these entry in /var/log/messages:

Jan 28 05:52:15 64455-alpha01 kernel: [2529273.616327] INFO: task mysqld:4123 blocked for more than 120 seconds.
Jan 28 05:52:15 64455-alpha01 kernel: [2529273.616525] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 05:52:15 64455-alpha01 kernel: [2529273.616813] mysqld        D  000000000000000d     0  4123   3142 0x00000080

What does it mean? How does it affect that MySQL thread (4123 is the thread id?)

The value in /proc/sys/kernel/hung_task_timeout_secs when I checked now is:

$ cat /proc/sys/kernel/hung_task_timeout_secs
120

I specifically would like to know how does it affect the process?

I read in a forum that it means it happens when that process is holding up too much memory.

Best Answer

echo 0 > /proc/sys/kernel/hung_task_timeout_secs only silences the warning. Besides that it has no effect whatsoever. Any value above zero will cause this message to be issued whenever a task is blocked for that amount of time.

The warning is given to indicate a problem with the system. In my experience it means that the process is blocked in kernel space for at least 120 seconds usually because the process is starved of disk I/O. This can be because of heavy swapping due to too much memory being used, e.g. if you have a heavy webserver load and you've configured too many apache child processes for your system. In your case it may just be that there are too many mysql processes competing for memory and data IO.

It can also happen if the underlying storage system is not performing well, e.g. if you have a SAN which is overloaded, or if there are soft errors on a disk which cause a lot of retries. Whenever a task has to wait long for its IO commands to complete, these warning may be issued.

Related Question