Shell Script – Distributing Thousands of Files Over Subfolders

filesshell-scriptzsh

I have a folder A with hundreds of thousands of files. I would like to move these files to new subfolders S_i, with, say, 100 files in each (the last folder may have less than 100 files)

In other words, if my folder A has:

file1
file2
...
file1000

I would later have:

S_1:    
    file_1
    ...
    file_100    
S_2:
    file_101
    ...
    file_200
...

Before I write a Zsh script manually to do this task (e.g. using variables to count files), I was wondering if there are any readily available tools (like split) that would facilitate this task.

Best Answer

With zsh:

autoload zmv
zmv 'file_(<->)' 'S_$((1 + ($1 - 1) / 100))'

If the files are not numbered, but you just want to split that list:

n=0; zmv -Q 'file_*(n)' 'S_$((n++/100+1))'

(n) is to toggle numerical ordering for the list (and you need -Q for that globbing qualifier).

Those call one mv per file. You can make it a bit more efficient by making mv builtin (zmodload zsh/files), or you could do:

files=(file_*(nN))
for ((n=1; $#files; n++)) {
  mv -- $files[1,100] S_$n
  files[1,100]=()
}

The (nN) above are zsh globbing qualifiers which further qualify the glob. n is for numerical sorting, N is to turn on the nullglob option for that glob, that is for the glob to expand to an empty list when there is no match.

(with that one, you could even throw in a mkdir S_$n in case those directories didn't exist beforehand).

Related Question