Centos – why nginx holds file descriptions of logs

centosfile-descriptorslogsnginx

on Red Hat Enterprise Linux Server release 6.6 (Santiago)
nginx version: nginx/1.0.15

I use common nginx logrotate config, logrotate works fine and nginx creates new log file like access.log or error.log

# cat /etc/logrotate.d/nginx 
/var/log/nginx/*log {
    daily
    rotate 4
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
       /bin/kill -USR1 $(cat /var/run/nginx.pid 2>/dev/null) 2>/dev/null || :
    endscript
}

However, one time my free space became low and after a while I found that nginx keeps file descriptors of deleted files. Only way to free space on the server is to restart nginx thereby free file descriptors.

Any ideas?

>     [srv2 nginx]# logrotate -f  /etc/logrotate.d/nginx 
>     [srv2 nginx]#  lsof +L1
>     COMMAND    PID        USER   FD   TYPE DEVICE SIZE/OFF NLINK NODE NAME
>     vmtoolsd  1125        root    3u   REG  253,3     4240     0   45 /tmp/vmware-root-2883746505/vmware-apploader-1125.log (deleted)
>     nginx    38748 nginx    2w   REG  253,4  1370362     0  674 /var/log/nginx/error.log.1 (deleted)
>     nginx    38748 nginx    4w   REG  253,4  1370362     0  674 /var/log/nginx/error.log.1 (deleted)
>     nginx    38748 nginx    5w   REG  253,4        0     0  220 /var/log/nginx/access.log (deleted)
>     nginx    38748 nginx    6w   REG  253,4    41819     0  693 /var/log/nginx/localhost.access.log.1 (deleted)
>     nginx    38749 nginx    2w   REG  253,4  1370362     0  674 /var/log/nginx/error.log.1 (deleted)
>     nginx    38749 nginx    4w   REG  253,4  1370362     0  674 /var/log/nginx/error.log.1 (deleted)
>     nginx    38749 nginx    5w   REG  253,4        0     0  220 /var/log/nginx/access.log (deleted)
>     nginx    38749 nginx    6w   REG  253,4    41819     0  693 /var/log/nginx/localhost.access.log.1 (deleted)
>     nginx    38750 nginx    2w   REG  253,4  1370362     0  674 /var/log/nginx/error.log.1 (deleted)
>     nginx    38750 nginx    4w   REG  253,4  1370362     0  674 /var/log/nginx/error.log.1 (deleted)
>     nginx    38750 nginx    5w   REG  253,4        0     0  220 /var/log/nginx/access.log (deleted)
>     nginx    38750 nginx    6w   REG  253,4    41819     0  693 /var/log/nginx/localhost.access.log.1 (deleted)

Best Answer

This is normal, you probably "rotated" the log without restarting, so the process (any process really) would keep the descriptor open.

Either use postrotate or copytruncate. This is pretty popular:

postrotate   
    kill -USR1 `cat /var/run/nginx.pid` &>/dev/null   
endscript

the USR1 signal tells to reload the log files (thus releasing the descriptors)