How to move a file recursively based on size and keep structure

filesfindrenamescripting

I want to move files bigger than "300Mb" from one directory tree where each file is located in subfolders

Example: I have a directory structure:

dirA/
dirA/file1
dirA/x/
dirA/x/file2
dirA/y/
dirA/y/file3

Here is the expected result, a "move" of the directory tree where each file is a moved to the sub folders:

dirB/            #  normal directory
dirB/file1       #  moved from dirA/file1
dirB/x/          #  normal directory
dirB/x/file2     #  moved from dirA/x/file2
dirB/y/          #  normal directory
dirB/y/file3     #  moved from dirA/y/file3

The find /path/ -type f -size +300m but then what? and unfortunately some of the files have all sorts of characters you can find on your keyboard.

I have been looking at this thread where someone is talking about cpio but I don't know that program…

PS: have GNU Parallel installed if this could speed up things?

Best Answer

The easy way is with zsh. You can use glob qualifiers to match files according to criteria such as their type and size. The wildcard pattern **/ matches any level of subdirectories. The history modifiers h and t are easy ways of extracting the directory and the base part of a filename. Call mkdir -p to create the directories when needed.

cd dirA
for x in **/*(.Lm+300); do
  mkdir -p ../dirB/$x:h &&
    mv -- $x ../dirB/$x
done

The portable way is with find. Use -exec to invoke a shell snippet for every file.

cd dirA
find . -type f -size +300000k -exec sh -c 'for x do
  mkdir -p "../dirB/${x%/*}"
  mv "$x" "../dirB/$x"
done' sh {} +

Parallelization is rarely useful for input/output: it lets you take advantage of multiple CPUs but the CPU is rarely a bottleneck in I/O.

Related Solutions

Find and Symlink Files Containing Specific String Recursively

I created a short script to do this, with a nice output which should be easy check the results. It doesn't need to have the destination directory structure created. Use as follows:

$ ./recursive-symlink.sh --help
Usage:
  ./recursive-symlink.sh <source_path> <dest_path> <find_args...>

To show its usage, let's say I have the following files/dirs at the begining:

├── recursive-symlink.sh*
└── src/
    ├── dir1/
    │   ├── file_A_misc.txt
    │   └── file_B_sub.txt
    ├── dir3/
    │   ├── file_A3.txt
    │   ├── file_C.txt
    │   └── subsub_dir/
    │       ├── file_Asubsub.txt
    │       └── file_D.txt
    ├── dir_A/
    │   └── should_be_empty.dat
    ├── file_A.txt
    └── file_B.txt

If I run:

$ find -name '*_A*'
./src/file_A.txt
./src/dir3/file_A3.txt
./src/dir3/subsub_dir/file_Asubsub.txt
./src/dir_A
./src/dir1/file_A_misc.txt

I can see which files would be linked. I then run the script like this:

$ ./recursive-symlink.sh src/ dest/ -name '*_A*'
src/file_A.txt
mkdir: created directory 'dest'
'dest/file_A.txt' -> '../src/file_A.txt'

src/dir3/file_A3.txt
mkdir: created directory 'dest/dir3'
'dest/dir3/file_A3.txt' -> '../../src/dir3/file_A3.txt'

src/dir3/subsub_dir/file_Asubsub.txt
mkdir: created directory 'dest/dir3/subsub_dir'
'dest/dir3/subsub_dir/file_Asubsub.txt' -> '../../../src/dir3/subsub_dir/file_Asubsub.txt'

src/dir_A
'dest/dir_A' -> '../src/dir_A'

src/dir1/file_A_misc.txt
mkdir: created directory 'dest/dir1'
'dest/dir1/file_A_misc.txt' -> '../../src/dir1/file_A_misc.txt'

My final state will be then:

├── recursive-symlink.sh*
├── src/
│   ├── dir1/
│   │   ├── file_A_misc.txt
│   │   └── file_B_sub.txt
│   ├── dir3/
│   │   ├── file_A3.txt
│   │   ├── file_C.txt
│   │   └── subsub_dir/
│   │       ├── file_Asubsub.txt
│   │       └── file_D.txt
│   ├── dir_A/
│   │   └── should_be_empty.dat
│   ├── file_A.txt
│   └── file_B.txt
└── dest/
    ├── dir1/
    │   └── file_A_misc.txt -> ../../src/dir1/file_A_misc.txt
    ├── dir3/
    │   ├── file_A3.txt -> ../../src/dir3/file_A3.txt
    │   └── subsub_dir/
    │       └── file_Asubsub.txt -> ../../../src/dir3/subsub_dir/file_Asubsub.txt
    ├── dir_A -> ../src/dir_A/
    └── file_A.txt -> ../src/file_A.txt

You can see that dest directory is created automatically, as well all recursive subdirectories, and on the dest dir, only the files that matched the *_A* pattern were linked.

Here the script source code:

#!/bin/bash

verbose='-v'  # you may comment this line

if [ "$1" == '-h' ] || [ "$1" == '--help' ] || [ $# -lt 3 ]
then
    echo "Usage:"
    echo "  $0 <source_path> <dest_path> <find_args...>"
    exit
fi

src="${1%/}" ; shift
dest="${1%/}" ; shift
relflag='' ; [ "${src:0:1}" != '/' ] && relflag='-r'

find "$src" \( "$@" \) -print0 |
    while IFS= read -r -d '' f
    do
        base_fname="${f#$src}"
        [ "$verbose" ] && echo "${f}"
        dest_ln="$dest/${base_fname#/}"
        dest_dir="$(dirname "$dest_ln")"
        mkdir -p $verbose "$dest_dir"
        ln $relflag -s $verbose -t "$dest_dir" "$f"
        [ "$verbose" ] && echo
    done

Best Answer

Related Solutions

Find and Symlink Files Containing Specific String Recursively

Related Question