Shell – Move huge number of files into date structured directory order

filesrenameshell-script

I have about 1 million files in this directory: /home/username/images/

Each of the files are called something like: 012345678910(Place)_0_20120414185957_28841.jpg with the timestamp part of the filename changing on each picture.

The code below contains code to sort/move the files into this date structure: /home/username/sorted/2012/04/14/18/name_of_file.jpg

For a small sample of files it works fine, but for the huge directory my putty terminal gets disconnected after outputting

Directory $newdir does not exist.  Creating same.

I had other code which always died with the error code argument list too long.

Here is the code:

#!/bin/bash
ALLFILES=(images/*)
for ((i=0; i<${#ALLFILES[*]}; i+=30000));
do
    set $(echo "${ALLFILES[@]:i:30000}" | awk -F_ '{print $1, $2, $3, $4, $5}')
    fullyear=$3
    year=$(echo $fullyear |cut -c1-4)
    month=$(echo $fullyear |cut -c5-6)
    day=$(echo $fullyear |cut -c7-8)
    hour=$(echo $fullyear |cut -c9-10)
    newdir=$(echo /home/username/sorted/$year/$month/$day/$hour/)
    if ! [ -d $newdir ]; then
        echo Directory $newdir does not exist.  Creating same.
        mkdir -p $newdir;
    fi
    mv "${ALLFILES[@]:i:30000}" $newdir;
done

Any ideas why the connection will not hold while performing the large loop?

Best Answer

Try to run it in screen session. Or even try another construction. I believe find + sed will work better then pure bash:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%'

This is just to show, how sed make commands to execute. Adding e after last % will force command executing:

find images/ -name "*.jpg" | sed 's%^[^_]*_[^_]*_\([0-9][0-9][0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).*%mkdir -p "/home/username/sorted/\1/\2/\3/\4" \&\& mv "&" "/home/username/sorted/\1/\2/\3/\4/"%e'

ps. You don't need to use in bash 

day=$(echo $fullyear |cut -c7-8)

Bash can do it itself without echo | cut :

day=${fullyear:6:2}
Related Question