Rails App Logs – How to Rotate Log Files Automatically or Manually

logrotatelogsrubyUbuntu

I am taking over administration for a server which has a custom made 3rd party Rails app. The app developer informed me that the ruby log files are getting large and pointed me to the following link:
– https://stackoverflow.com/questions/4883891/ruby-on-rails-production-log-rotation

The logs are located in /root/production/app/log. This folder contains several files which end in .log. The current log files are about 2GB in size and the folder also has archived log files with the format of "logname.log.2013_01_18.bz2".

I tried searching the ssh command history to see if I can see which command was used to create the archive files, but the commands don't go back that far. I also ran "cat /var/lib/logrotate/status" but it doesn't look like logrotate did not rotate any logs from the above folder.

Basically:

I would like to be able to run a command to rotate the logs manually and/or have the logs rotate automatically every Sunday night.
If I need to shutdown my rails app to rotate the logs, I have no problem with this as I have a maintenance window every Sunday night to perform this.
Finally, I am not sure if I should compress the log files weekly into bz2 or only compress the logs on a monthly basis for any rotated logs related to the previous month. This is mainly because I am not sure exactly how this app utilizes the logs and I have never utilized rails apps nor have I had to configure log rotation manually.

At this moment I do need to keep all log files and not discard any. Any information on concerns related to rotating logs, such as backing up logs before attempting to rotate are welcome.

Best Answer

logrotate is used by the system to rotate logs so you have 2 choices. You can either incorporate the rotation of these app logs into the systems rotations or setup your own and either run them manually or from the root user's crontab (Assuming the Rails app is run as root given it's directory is /root/...).

System rotation

To setup a logrotation within the system's pre-existing ones simply add a new file to the directory /etc/logrotate.d. Call it railsapp.conf. I'd use the other examples there to construct it. Also confrere with the logrotate man page.

User rotation

If you want to run your own instance of logrotate you only have to provide it with command line switches to do so.

First make a copy of /etc/logrotate.conf /root/rails_logrotate.conf
Edit the file so that it has the log rotation configured the way you want (i.e. keep all logs, rotate weekly, etc.)
Run it
```
# 1st time
$ logrotate -d -f -s $HOME/my_logrotate.state logrotate.conf

# afterwards
$ logrotate -d -s $HOME/my_logrotate.state logrotate.conf
```
If things look OK you can re-run these commands without the -d switch. This is for debugging purposes only and won't actually do any of the tasks, merely show you what it WOULD do.
```
$ logrotate -s $HOME/my_logrotate.state logrotate.conf     
```
You could also use the -v switch to make it verbose, similar to the output seen when using the -d switch.

Example

Start with this log file.

$ dd if=/dev/zero of=afile bs=1k count=10k
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 0.0702393 s, 149 MB/s

$ ll afile
-rw-rw-r-- 1 saml saml 10485760 Aug  6 14:37 afile

$ touch -t 201307010101 afile
$ ll afile
-rw-rw-r-- 1 saml saml 10485760 Jul  1 01:01 afile

Now run logrotate

$ logrotate -v -f -s $HOME/my_logrotate.state logrotate.conf
reading config file logrotate.conf
reading config info for /home/saml/afile 

Handling 1 logs

rotating pattern: /home/saml/afile  forced from command line (1 rotations)
empty log files are rotated, old logs are removed
considering log /home/saml/afile
  log needs rotating
rotating log /home/saml/afile, log->rotateCount is 1
dateext suffix '-20130806'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding old rotated logs failed
renaming /home/saml/afile to /home/saml/afile-20130806
creating new /home/saml/afile mode = 0664 uid = 500 gid = 501

Check the results

$ ll afile*
-rw-rw-r-- 1 saml saml        0 Aug  6 14:40 afile
-rw-rw-r-- 1 saml saml 10485760 Jul  1 01:01 afile-20130806

Weekly Cron

To make this run every Sunday you could create the following crontab entry for the root user.

$ crontab -e

Add the following lines:

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed
0 0 * * sun logrotate -v -f -s $HOME/my_logrotate.state $HOME/logrotate.conf

Then save the above.

You can also use these types of shortcuts instead of specifying the actual days, mins, secs, etc.

string         meaning
------         -------
@reboot        Run once, at startup.
@yearly        Run once a year, "0 0 1 1 *".
@annually      (same as @yearly)
@monthly       Run once a month, "0 0 1 * *".
@weekly        Run once a week, "0 0 * * 0".
@daily         Run once a day, "0 0 * * *".
@midnight      (same as @daily)
@hourly        Run once an hour, "0 * * * *".

References

Debugging the issue

Put this line in a shell script, logrotate.sh:

#!/bin/bash
/usr/sbin/logrotate -f -v /etc/logrotate.d/mail3-logs &>> /var/log/logrotate/rotate.log

Make it executable and run it like this from the cron:

03 00 * * * root strace -s 2000 -o /tmp/strace.log /path/to/logrotate.bash

In going through the output you should see what is getting tripped up by the permissions problems.

EDIT #1

After conversing with the OP he mentioned that the above debugging technique uncovered that SELinux was enabled. He was perplexed as to why this was the case since he had previously disabled it with the command setenforce 0.

Disabling SELinux in this fashion will only remain in this state until the next reboot. The default mode for SELinux is dictated by this file on Fedora/CentOS:

$ cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#   enforcing - SELinux security policy is enforced.
#   permissive - SELinux prints warnings instead of enforcing.
#   disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
#   targeted - Only targeted network daemons are protected.
#   strict - Full SELinux protection.
SELINUXTYPE=targeted

To permanently disable SELinux you'll want to change the line SELINUX=.. to one of the 3 states, enforcing, permissive, disabled.

I would encourage you however to take the time to understand why SELinux is disallowing the access to the directory these log files are within, and add the appropriate context's so that SELinux allows this access. SELinux is an important part of the layered security model that is facilitated on Linux distros that make use of it, and blindly disabling it is taking one of the critical layers away.

Best Answer

System rotation

User rotation

Example

Weekly Cron

References

Related Solutions

Apache – How to avoid apache reload when rotating logs

Error stat of log failed: Permission denied during LogRotation

Debugging the issue

EDIT #1

References

Related Question