Bash – Rename Multiple files gzips them and delete older than 10 days

bashfilesgziprenamescripting

I'm new to Linux/Unix and slowly learning it step by step. Unfortunately on today’s job interview was surprised by a task to do (which of course I didn’t know). I was asked to prepare a bash script that:

changes the names of .log. files to (name).(date).log.(remaining part of the original name) and gzips them to a .gz archive.
deletes (name).(date).log.(remaining part of the original name).gz files older than 10 days.

Never too late for asking and would really appreciate if you could let me know how it should look like.

Best Answer

Assuming GNU tools, I'd do something like

find . -type f \( -name '*.[0-9][0-9][0-9][0-9]-[0-1][0-9]-[0-3][0-9].log.*.gz' \
       \( -mtime -10 -o -delete -o -true \) -o -name '*.log.*' ! -name '*.gz' \
       \( -mtime +9 \( -delete -o -true \) -o -printf '%TF/%p\0' \) \) |
  while IFS=/ read -rd '' date file; do
    basename=${file##*/}
    dirname=${file%/*}
    newfile=$dirname/${basename%.log.*}.$date.log.${basename##*.log.}
    mv "$file" "$newfile" && gzip "$newfile"
  done

Related Solutions

Shell – delete old log files except the last one (alphanumerically sorted)

find /home/foo -maxdepth 1 -iname log\* -type f -mmin +1800 |
  sort | head -n -1 | xargs rm

Or if you want to use mtime instead of the filename:

find /home/foo -maxdepth 1 -iname log\* -type f -mmin +1800 -exec ls -t {} + |
  tail -n +2 | xargs rm

From @Stephane's comments, a more robust approach would be to do:

IFS=$'\n'
set -f
rm $(
  find /home/foo -maxdepth 1 -iname log\* ! -name $'*\n*' -type f -mmin +1800 |
  sort | head -n -1 )

Or for POSIX shell (still requires GNU tools):

IFS='
'
ex_newline='*
*'
set -f
rm $(
  find /home/foo -maxdepth 1 -iname log\* ! -name "$ex_newline" -type f -mmin +1800 |
  sort | head -n -1 )

A single (robust) pipeline can be used with a recent version of GNU sed/sort (and GNU find as with all of the above):

find /home/foo -maxdepth 1 -iname log\* -type f -mmin +1800 -print0 |
  sort -z | sed -z '$d' | xargs -0 rm

Shell Script – Cron Job to Delete Files Older Than 3 Days

This is easy enough (although note that this goes by a modification time more than 3 days ago since a creation time is only available on certain filesystems with special tools):

find /a/b/c/1 /a/b/c/2 -type f -mtime +3 #-delete

Remove the # before the -delete once you are sure that it is finding the files you want to remove.

To have it run by cron, I would probably just create an executable script (add a shebang - #!bin/sh to the top line of the file and make executable with chmod a+x), then put it in an appropriate cron directory like /etc/cron.daily or /etc/cron.weekly. Provided of course that you do not need a more specific schedule and that these directories exist on your distro.

Update

As noted below, the -delete option for find isn't very portable. A POSIX compatible approach would be:

find /a/b/c/1 /a/b/c/2 -type f -mtime +3 #-exec rm {} +

Again remove the # when you are sure you have the right files.

Update2

To quote from Stéphane Chazelas comment below:

Note that -exec rm {} + has race condition vulnerabilities which -delete (where available) doesn't have. So don't use it on directories that are writeable by others. Some finds also have a -execdir that mitigates against those vulnerabilities.