Way to identify which process turns into Zombie process

procprocesszombie-process

One of the ubuntu server has 82 zombie processes. All processes shows '[sh] defunct' as process command. Is there a way to find out which process is becoming a zombie process?

I tried checking the /proc/PID/ directory to get some clue about zombie process but all files are empty. How to find who let this process as zombie.. . Is there any other way to find it out?

Updated/Solved: Made the question clearer, and Answered my own question as suggested by andcoz.

Best Answer

The audit subsystem of the Linux kernel can be very useful to figure out what processes are becoming zombie processes. I just had the following situation:

server ~ # ps -ef --forest
[...]
root     16385     1  0 17:04 ?        00:00:00 /usr/sbin/apache2 -k start
root     16388 16385  0 17:04 ?        00:00:00  \_ /usr/bin/perl -T -CSDAL /usr/lib/iserv/apache_user
root     16389 16385  0 17:04 ?        00:00:00  \_ /usr/bin/perl -T -CSDAL /usr/lib/iserv/apache_user
www-data 16415 16385  0 17:04 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 18254 16415  0 17:23 ?        00:00:00  |   \_ [sh] <defunct>
www-data 18347 16415  0 17:23 ?        00:00:00  |   \_ [sh] <defunct>
www-data 22966 16415  0 18:18 ?        00:00:00  |   \_ [sh] <defunct>
www-data 16583 16385  0 17:05 ?        00:00:01  \_ /usr/sbin/apache2 -k start
www-data 18306 16583  0 17:23 ?        00:00:00  |   \_ [sh] <defunct>
www-data 18344 16583  0 17:23 ?        00:00:00  |   \_ [sh] <defunct>
www-data 17561 16385  0 17:12 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 22983 17561  0 18:18 ?        00:00:00  |   \_ [sh] <defunct>
www-data 18318 16385  0 17:23 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 19725 16385  0 17:43 ?        00:00:01  \_ /usr/sbin/apache2 -k start
www-data 22638 16385  0 18:13 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 22659 16385  0 18:14 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 25102 16385  0 18:41 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 25175 16385  0 18:42 ?        00:00:00  \_ /usr/sbin/apache2 -k start
www-data 25272 16385  0 18:44 ?        00:00:00  \_ /usr/sbin/apache2 -k start

The cause for these zombie processes is most probably a PHP script, but as these Apache child processes are processing lots of HTTP requests and lots of different PHP scripts, it's very hard to figure out which one could be responsible. Linux has also already deallocated important information of these zombie processes, so we don't even have /proc/<pid>/cmdline to figure out which script or -c command /bin/sh may have been running:

server ~ # cat /proc/18254/cmdline 
server ~ #

To figure it out, I've installed auditd: https://linux-audit.com/configuring-and-auditing-linux-systems-with-audit-daemon/

I set up the following audit rules:

auditctl -a always,exit -F arch=b32 -S execve -F path=/bin/dash
auditctl -a always,exit -F arch=b64 -S execve -F path=/bin/dash

These rules audit all process creations of the /bin/dash binary. /bin/sh doesn't work here, because it's a symlink and audit apparently only sees the target file name:

server ~ # ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Nov  8  2014 /bin/sh -> dash*

A simple test should now produce audit logs in /var/log/audit/audit.log (I've taken the liberty and added a lot of line breaks to improve the readability):

server ~ # sh -c 'echo test'
test

server ~ # tail -f /var/log/audit/audit.log
[...]
type=SYSCALL msg=audit(1488219335.976:43871): arch=40000003 syscall=11 \
  success=yes exit=0 a0=ffdca3ec a1=f7760e58 a2=ffdd399c a3=ffdca068 items=2 \
  ppid=27771 pid=27800 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 \
  fsgid=0 tty=pts7 ses=7532 comm="sh" exe="/bin/dash" key=(null)
type=EXECVE msg=audit(1488219335.976:43871): argc=3 a0="sh" a1="-c" \
  a2=6563686F2074657374
type=CWD msg=audit(1488219335.976:43871):  \
  cwd="/var/lib/iserv/remote-support/iserv-martin.von.wittich"
type=PATH msg=audit(1488219335.976:43871): item=0 name="/bin/sh" inode=10403900 \
  dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PATH msg=audit(1488219335.976:43871): item=1 name=(null) inode=5345368 \
  dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PROCTITLE msg=audit(1488219335.976:43871): \
  proctitle=7368002D63006563686F2074657374

Lots of the information is encoded, but ausearch can translate it with -i:

server ~ # ausearch -i -x /bin/dash | tail                                      
[...]
----
type=PROCTITLE msg=audit(27.02.2017 19:15:35.976:43871) : proctitle=sh 
type=PATH msg=audit(27.02.2017 19:15:35.976:43871) : item=1 name=(null) \
  inode=5345368 dev=08:01 mode=file,755 ouid=root ogid=root rdev=00:00 \
  nametype=NORMAL 
type=PATH msg=audit(27.02.2017 19:15:35.976:43871) : item=0 name=/bin/sh \
  inode=10403900 dev=08:01 mode=file,755 ouid=root ogid=root rdev=00:00 \
  nametype=NORMAL 
type=CWD msg=audit(27.02.2017 19:15:35.976:43871) :  \
  cwd=/var/lib/iserv/remote-support/iserv-martin.von.wittich 
type=EXECVE msg=audit(27.02.2017 19:15:35.976:43871) : argc=3 a0=sh a1=-c \
  a2=echo test 
type=SYSCALL msg=audit(27.02.2017 19:15:35.976:43871) : arch=i386 \
  syscall=execve success=yes exit=0 a0=0xffdca3ec a1=0xf7760e58 a2=0xffdd399c \
  a3=0xffdca068 items=2 ppid=27771 pid=27800 auid=root uid=root gid=root \
  euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts7 \
  ses=7532 comm=sh exe=/bin/dash key=(null) 
----

If you don't want to restrict the ausearch filtering to /bin/dash, you can also use ausearch -i -m ALL to translate the complete log. Another good filter would be ausearch -i -p <PID of a zombie process>, in this case ausearch -i -p 27800.

Just leave these rules in place until new zombie processes show up, and then search for the process creation of a zombie PID:

ausearch -i -p <PID>

This should be very helpful to identify the root cause of the zombie processes. In my case it was a PHP script that used proc_open to spawn a Perl script without closing the handle with proc_close.

Related Solutions

Linux Process – How to Find Out Namespace of a Particular Process?

I'll try and answer both this and your earlier question as they are related.

The doors to namespaces are files in /proc/*/ns/* and /proc/*/task/*/ns/*.

A namespace is created by a process unsharing its namespace. A namespace can then be made permanent by bind-mounting the ns file to some other place.

That's what ip netns does for instance for net namespaces. It unshares its net namespace and bind-mounts /proc/self/ns/net to /run/netns/netns-name.

In a /proc mounted in the root pid namespace, you can list all the namespaces that have a process in them by doing:

# readlink /proc/*/task/*/ns/* | sort -u
ipc:[4026531839]
mnt:[4026531840]
mnt:[4026531856]
mnt:[4026532469]
net:[4026531956]
net:[4026532375]
pid:[4026531836]
pid:[4026532373]
uts:[4026531838]

The number in square brackets is the inode number.

To get that for a given process:

# ls -Li /proc/1/ns/pid
4026531836 /proc/1/ns/pid

Now, there may be permanent namespaces that don't have any process in them. Finding them out can be a lot trickier AFAICT.

First, you have to bear in mind that there can be several mount namespaces.

# awk '$9 == "proc" {print FILENAME,$0}' /proc/*/task/*/mountinfo | sort -k2 -u
/proc/1070/task/1070/mountinfo 15 19 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
/proc/19877/task/19877/mountinfo 50 49 0:3 / /run/netns/a rw,nosuid,nodev,noexec,relatime shared:2 - proc proc rw
/proc/19877/task/19877/mountinfo 57 40 0:3 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
/proc/1070/task/1070/mountinfo 66 39 0:3 / /run/netns/a rw,nosuid,nodev,noexec,relatime shared:2 - proc proc rw
/proc/19877/task/19877/mountinfo 68 67 0:3 / /mnt/1/a rw,nosuid,nodev,noexec,relatime unbindable - proc proc rw

Those /mnt/1/a, /run/netns/a may be namespace files.

We can get an inode number:

# nsenter --mount=/proc/19877/task/19877/ns/mnt -- ls -Li /mnt/1/a
4026532471 /mnt/1/a

But that doesn't tell us much other than it's not in the list computed above.

We can try and enter it as any of the different types:

# nsenter --mount=/proc/19877/task/19877/ns/mnt -- nsenter --pid=/mnt/1/a true
nsenter: reassociate to namespace 'ns/pid' failed: Invalid argument
# nsenter --mount=/proc/19877/task/19877/ns/mnt -- nsenter --mount=/mnt/1/a true
nsenter: reassociate to namespace 'ns/mnt' failed: Invalid argument
# nsenter --mount=/proc/19877/task/19877/ns/mnt -- nsenter --net=/mnt/1/a true
#

OK, that was a net namespace file.

So it would seem we have a method to list the name spaces: list the ns directories of all the tasks, then find all the proc mountpoints in all the /proc/*/task/*/mountinfo and figure out their type by trying to enter them.

Linux – sudo pgrep -f matches arbitrary strings and returns increasing pids

It's showing the sudo process i.e. the PID is the PID of the sudo process that is the parent of the pgrep command you are running by putting after sudo. As you are searching in the whole command line (by -f), the sudo process pops up in the output because the string (pattern) is also a part of the original sudo command.

By using the -l and -a (if your pgrep supports), you would get a better idea.

Test:

% sudo pgrep -af "asdf"
4560 sudo pgrep -af asdf

% sudo pgrep -lf "asdf"
4562 sudo

% pgrep -af "asdf" 
%

Best Answer

Related Solutions

Linux Process – How to Find Out Namespace of a Particular Process?

Linux – sudo pgrep -f matches arbitrary strings and returns increasing pids

Related Question