Delete old files but keep one each week/month/year

fileslogsrm

I have a folder with log-files each ~10 minutes:

alfred_140810-190001.json
alfred_140810-191001.json
alfred_140810-192002.json
alfred_140810-193001.json
alfred_140810-194001.json
alfred_140810-195001.json
alfred_140810-200002.json
alfred_140810-201119.json
alfred_140810-202002.json
...

How do I achieve this?

delete all files older than a week but keep one per week
delete all files older than a month but keep one per month
delete all files older than a year but keep one per year

So I want one file for the last four weeks (4 files) one file per month (12 Files) and one per year (the same system like rsnapshot is sorting his backups).

Best Answer

You want logrotate http://linuxcommand.org/man_pages/logrotate8.html.

It is probably already on your system. You just need to configure it. However it is mainly for the purpose of purging old log files and I don't know if you can configure it to keep one file.

What you can do

create several directories log, log.weekly, log.monthly, and log.yearly

log being where all the log files go. Create

a weekly cron job that copies the latest log file from log to log.weekly,
a monthly cron job that copies the latest log file from log to log.monthly, and
a yearly cron job that copies the latest log file from log to log.yearly.

Then configure logrotate appropriately for the different directories.

    #!/bin/bash

    NOW=$(date +%


    ls -rt1 ${LOG} | while read FILE
    do
        TVAL=$(stat --printf %W ${LOG}/${FILE})
        if [ $(ls -1 ${LOG.WEEKLY} | wc -l) ] -eq 0 ]
        then
             cp ${LOG}/${FILE} ${LOG.WEEKLY}/${FILE}
        else
             LAST_WEEKLY=$(ls -t1 ${LOG.WEEKLY} | head -n 1 | stat --printf %W)
             if [ $((${TVAL}-${LAST_WEEKLY})) -gt $((60*60*24*7)) ]
             then
                 cp ${LOG}/${FILE} ${LOG.WEEKLY}/${FILE}
             fi
        fi
   # repeat the above logic for month and year
   rm ${LOG}/${FILE} 
   done

Related Solutions

Bash – Remove all files except for last file of each month

With the file zz containing the list of file names, this works, so just replace cat zz.

cat zz | grep -vF -f <(cat zz|sort -r|uniq -w11)

e.g. echo *.sql | grep -vF -f <( echo *.sql | sort -r | uniq -w11 ) | xargs rm

As is, it won't work if spaces in file names, and very fragile to filename length.

Shell – How to delete the files in one folder which are more than 60 days old in UNIX

OK, I have remade this script, and by sorting it backwards it looks like it should work. It compares the year and month to the previous one, and if it is lower it should be the last entry for that month.

#!/bin/bash

#the tac reverses the listing, so we go from newest to oldest, vital for our logic below
FILES=`ls | tac`

#create a cutoff date by taking current date minus our 60 day limit
CUTOFFDATE=`date --date="60 days ago" +%Y%m%d`

#we are setting this value to month 13 of the current year 
LASTYEARMONTH=`date +%Y`13

for file in $FILES; do

    #get datestamp
    FILEDATE=`expr match "$file" '.*\(20[0-9][0-9][0-9][0-9][0-9][0-9]\)'`

    #get year and month, ie 201410
    THISYEARMONTH=${FILEDATE:0:6}

    if [ ! -z $FILEDATE ] && [ $THISYEARMONTH -lt $LASTYEARMONTH ]; then

            echo "$file IS THE LAST FILE OF THE MONTH. NOT DELETING"

    else

            #now we see if the file is older than 60 days (ie, has a LOWER unix timestamp than our cutoff)
            if [ ! -z $FILEDATE ] && [ $FILEDATE -lt $CUTOFFDATE ]; then

                    echo "$file has a file date of $FILEDATE which is older than 60 days."

                    #uncomment this next line to remove
                    #rm $file
            fi
    fi
    LASTYEARMONTH=$THISYEARMONTH
done

Best Answer

Related Solutions

Bash – Remove all files except for last file of each month

Shell – How to delete the files in one folder which are more than 60 days old in UNIX

Related Question