Ubuntu – Very large log files, what should I do

filesystemlogsystem

(This question deals with a similar issue, but it talks about a rotated log file.)

Today I got a system message regarding very low /var space.

As usual I executed the commands in the line of sudo apt-get clean which improved the scenario only slightly. Then I
deleted the rotated log files which again provided very little improvement.

Upon examination I find that some log files in the /var/log has grown up to be very huge ones. To be specific, ls -lSh /var/log gives,

total 28G
-rw-r----- 1 syslog            adm      14G Aug 23 21:56 kern.log
-rw-r----- 1 syslog            adm      14G Aug 23 21:56 syslog
-rw-rw-r-- 1 root              utmp    390K Aug 23 21:47 wtmp
-rw-r--r-- 1 root              root    287K Aug 23 21:42 dpkg.log
-rw-rw-r-- 1 root              utmp    287K Aug 23 20:43 lastlog

As we can see, the first two are the offending ones. I am mildly surprised why such large files have not been rotated.

So, what should I do? Simply delete these files and then reboot? Or go for some more prudent steps?

I am using Ubuntu 14.04.

UPDATE 1

To begin with, the system is only several months old. I had to install the system from scratch couple of months back after a hard disk crash.

Now, as advised in this answer,
I first checked the offending log files using tail, no surprise there. Then, for deeper inspection, I executed this script from the same answer.

for log in /var/log/{syslog,kern.log}; do 
  echo "${log} :"
  sed -e 's/\[[^]]\+\]//' -e 's/.*[0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\}//' ${log} \
  | sort | uniq -c | sort -hr | head -10
done

The process took several hours.
The output was in the line of,

/var/log/syslog :
71209229  Rafid-Hamiz-Dell kernel:  sda3: rw=1, want=7638104968240336200, limit=1681522688
53929977  Rafid-Hamiz-Dell kernel:  attempt to access beyond end of device
17280298  Rafid-Hamiz-Dell kernel:  attempt to access beyond end of device
   1639  Rafid-Hamiz-Dell kernel:  EXT4-fs warning (device sda3): ext4_end_bio:317: I/O error -5 writing to inode 6819258 (offset 0 size 4096 starting block 54763121030042024)
       <snipped>

/var/log/kern.log.1 :
71210257  Rafid-Hamiz-Dell kernel:  attempt to access beyond end of device
71209212  Rafid-Hamiz-Dell kernel:  sda3: rw=1, want=7638104968240336200, limit=1681522688
   1639  Rafid-Hamiz-Dell kernel:  EXT4-fs warning (device sda3): ext4_end_bio:317: I/O error -5 writing to inode 6819258 (offset 0 size 4096 starting block 954763121030042024)

(/dev/sda3 is my home directory. As we can find,

lsblk /dev/sda
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 931.5G  0 disk 
├─sda1   8:1    0 122.1G  0 part /
├─sda2   8:2    0   7.6G  0 part [SWAP]
└─sda3   8:3    0 801.8G  0 part /home

Why a process will want to write beyond the limit is actually outside the scope of my comprehension. Perhaps I will want to ask a different question in this forum if this
continues even after a system update.)

Then, from this answer (you may want to check this for a deeper understanding), I
executed, 

sudo su -
> kern.log
> syslog

Now, these files have zero sizes. The system is running fine before and after a reboot.

I will watch these files (along with others) in the next few days and report back should
they behave out-of-line.

As a final note, both the offending files (kern.log and syslog), are set to be rotated, as inspection of the files (grep helped) inside
/etc/logrotate.d/ shows.

UPDATE 2

The log files are actually rotated. Looks like the large sizes were attained on a single day.

Best Answer

Simply delete these files and then reboot?

No. Empty them but do not use rm because it could end up crashing something while you are typing the touch command to recreate it.

Shortest method:

cd /var/log
sudo su
> lastlog
> wtmp
> dpkg.log 
> kern.log
> syslog
exit

If not root it will require sudo. Taken from another answer on AU.

BEFORE YOU DO THAT. Do a tail {logfile} and check if there is a reason for them to be so big. Unless this system is several years old there should be no reason for this and fixing the problem is better than letting this go on.

Both kern.log and syslog should normally not be that big. But like I said: if this system is up and running for years and years it might be normal and the files just need to be cleared.

And to prevent it to become that big in the future: setup logrotate. It is pretty straightforward and will compress the logfile when it becomes bigger then a size you set it to.

1 other thing: if you do not want to delete the contents you can compress the files by tarring or gzipping them. That will have you end up with files probably 10% of what they are now. That is if there is still room on the disk to do that.

Related Question