Linux – Best way to delete large amount of files by date

fileslinuxstorage

So, I have a folder with alot of files, they are generated by second and need to be kept in I only can delete them after 90 days, so as you may guess, I would generate tons of files and then after the 90 days I'm able to delete those files that is older than 90 days.
But, I'm stuck in the part where I search for those files, since I have alot, the system complains that the list is too large and thus I can't remove them.

What is the best solution so I can pass this? The file names are in timestamp mode, so I could start by it, but I want to make sure all files are deleted after some time….

I have tried these methods

rm -rf *
find /path/to/files/ -type f -name '*.ts' -mtime +90 -exec rm {} \;

I have also managed to create a script to delete by filename, but with this method I have no guarantee that all the files are deleted.

Best Answer

If the files are not modified after initial creation, you could delete if they have not been modified in over 90 days:

find /path/to/folder -mtime +90 -delete

find /path/to/folder -mtime +90 -exec rm {} +

(for versions of find which do not support the -delete action).

As a matter of safety, you should use a non-destructive version of this command first and ensure it will delete exactly what you want deleted, especially if you intend to automate this action via cron or similar, e.g.:

find /path/to/folder -mtime +90 -print

Related Solutions

Linux – Cleaner Way to Delete Files with Datestamp in Name

Another way to delete all except the last 30 files:

rm $(ls -r | tail -n +31)

Or here is a shorter version of the script in the original post:

cd /opt/backup/mysql/dumps
d=$(date -r $(($(date +%s)-30*86400)) +%Y%m%d)
for f in all-mysql-*; do
    [[ ${f#all-mysql-} < $d ]] && rm $f
done

Shell – How to delete the files in one folder which are more than 60 days old in UNIX

OK, I have remade this script, and by sorting it backwards it looks like it should work. It compares the year and month to the previous one, and if it is lower it should be the last entry for that month.

#!/bin/bash

#the tac reverses the listing, so we go from newest to oldest, vital for our logic below
FILES=`ls | tac`

#create a cutoff date by taking current date minus our 60 day limit
CUTOFFDATE=`date --date="60 days ago" +%Y%m%d`

#we are setting this value to month 13 of the current year 
LASTYEARMONTH=`date +%Y`13

for file in $FILES; do

    #get datestamp
    FILEDATE=`expr match "$file" '.*\(20[0-9][0-9][0-9][0-9][0-9][0-9]\)'`

    #get year and month, ie 201410
    THISYEARMONTH=${FILEDATE:0:6}

    if [ ! -z $FILEDATE ] && [ $THISYEARMONTH -lt $LASTYEARMONTH ]; then

            echo "$file IS THE LAST FILE OF THE MONTH. NOT DELETING"

    else

            #now we see if the file is older than 60 days (ie, has a LOWER unix timestamp than our cutoff)
            if [ ! -z $FILEDATE ] && [ $FILEDATE -lt $CUTOFFDATE ]; then

                    echo "$file has a file date of $FILEDATE which is older than 60 days."

                    #uncomment this next line to remove
                    #rm $file
            fi
    fi
    LASTYEARMONTH=$THISYEARMONTH
done

Best Answer

Related Solutions

Linux – Cleaner Way to Delete Files with Datestamp in Name

Shell – How to delete the files in one folder which are more than 60 days old in UNIX

Related Question