Ubuntu – Upstart not reopening log files on logrotation

initlogrotateupstart

We use upstart to manage our services on our Ubuntu servers. They produce logs which are logged out to /var/log/upstart/SERVICE_NAME.log

Then daily, the log files are rotated using the logrotation script which comes with 12.04 LTS:

/var/log/upstart/*.log {
        daily
        missingok
        rotate 7
        compress
        notifempty
        nocreate
}

The problem is that while logrotate moves the files, it doesn't appear to signal to upstart to close and reopen the files, leaving the upstart process writing to a delete PID.

init          1       root    8w      REG              202,1        64       2431 /var/log/upstart/dbus.log.1 (deleted)
init          1       root   13w      REG              202,1        95       2507 /var/log/upstart/acpid.log.1 (deleted)
init          1       root   14w      REG              202,1       127      17377 /var/log/upstart/whoopsie.log.1 (deleted)
init          1       root   36w      REG              202,1       122       6747 /var/log/upstart/SERVICE_NAME.log.1 (deleted)
init          1       root   37w      REG              202,1        30       6762

Obviously I could redirect the output from my own services to other log files, but the problem would still be there for the system processes. Also I would rather not have to build more infrastructure than what I need.

Best Answer

I believe you have 3 options.

  1. You modify the existing configuration by adding "copytruncate"

    /var/log/upstart/*.log { copytruncate daily missingok rotate 7 compress notifempty nocreate }

  2. If you can't or (are not allowed) to change the existing logrotate config because of other log files that don't suffer and the existing config works for them, then move your "SERVICE_NAME.log" files to a new folder under /var/log if your wish, create a new config with the "copytruncate" and add it to the cron.daily.

  3. a) If you're not allowed to change the host os logrotate config or add to the host OS's cron.daily, then your third option is to change the scripts or programs to either check that the file exists before writing out to the file. b) Another way is a bit of point 2 above which is to move your logfiles somewhere else and within you script or program, execute the logrotate command specific for that program's logfile.

Point 3b above is more tricky but more elegant and it is what I use most of the time as it means the program is self-sufficient and self-managed and doesn't need the OS's jobs to babysit it.

To find out how to manually run logrotate and add it to your program or script just type:

man logrotate

or

logrotate --help

If you are using Python for your programs you can check out how this program uses it to self-manage its log files. http://bazaar.launchpad.net/~ferncasado/keep.awake/trunk/files/head:/v4/

Related Question