bash – Calculating Number of Files for Batch Processing in Bash

For example, I have directory with multiple files created by this way:

touch files/{1..10231}_file.txt

I want to move them into new directory new_files_dir.

The simplest way to do this is:

for filename in files/*; do
    mv "${filename}" -t "new_files_dir"
done

This script works for 10 seconds on my computer. It is slow. The slowness happens due execution of mv command for every file.

###Edit start###

I have understood, that in my example the simplest way will be just

mv files/* -t new_files_dir

or, if the "Argument list too long":

printf '%s\0' files/* | xargs -0 mv -t new_files_dir

but aforementioned case is a part of task. The whole task is in this question: Moving large number of files into directories based on file names in linux.
So, the files must be moved into corresponding subdirectories, the correspondence of which is based on a number in the filename. This is the cause of for loop usage and other oddities in my code snippets.

###Edit end###

There is possibility to speedup this process by passing bunch of files to mv command instead of a single file, like this:

batch_num=1000

# Counting of files in the directory
shopt -s nullglob
file_list=(files/*)
file_num=${#file_list[@]}

# Every file's common part
suffix='_file.txt'

for((from = 1, to = batch_num; from <= file_num; from += batch_num, to += batch_num)); do
    if ((to > file_num)); then
        to="$file_num"
    fi  

    # Generating filenames by `seq` command and passing them to `xargs`
    seq -f "files/%.f${suffix}" "$from" "$to" |
    xargs -n "${batch_num}" mv -t "new_files_dir"
done

In this case the script works for 0.2 seconds. So, the performance has increased by 50 times.

But there is a problem: at any moment the program can refuse to work due "Argument list too long", because I can't guarantee that the bunch of filenames length is less than max allowable length.

My idea is to calculate the batch_num:

batch_num = "max allowable length" / "longest filename length"

and then use this batch_num in xargs.

Thus, the question: How can max allowable length be calculated?

I have done something:

1. Overall length can be found by this way:

    $ getconf ARG_MAX
    2097152
   

2. The environment variables contributes into the argument size too, so probably they should be subtracted from ARG_MAX:

    $ env | wc -c
    3403
   

3. Made a method to determine the max number of files of equal sizes by trying different amount of files before the right value is found (binary search is used).

    function find_max_file_number {
        right=2000000
        left=1
        name=$1
        while ((left < right)); do
            mid=$(((left + right) / 2))
   
            if /bin/true $(yes "$name" | head -n "$mid") 2>/dev/null; then
                left=$((mid + 1))
            else
                right=$((mid - 1))
            fi
        done
        echo "Number of ${#name} byte(s) filenames:" $((mid - 1))
    }
   
    find_max_file_number A
    find_max_file_number AA
    find_max_file_number AAA
   

   Output:

    Number of 1 byte(s) filenames: 209232
    Number of 2 byte(s) filenames: 190006
    Number of 3 byte(s) filenames: 174248
   

   But I can't understand the logic/relation behind these results yet.

4. Have tried values from this answer for calculation, but they didn't fit.

5. Wrote a C program to calculate the total size of passed arguments. The result of this program is close, but some non-counted bytes are left:

    $ ./program {1..91442}_file.txt
   
    arg strings size: 1360534
    number of pointers to strings 91443
   
    argv size:  1360534 + 91443 * 8 = 2092078
    envp size:  3935
   
    Overall (argv_size + env_size + sizeof(argc)):  2092078 + 3935 + 4 = 2096017
    ARG_MAX: 2097152
   
    ARG_MAX - overall = 1135 # <--- Enough bytes are
                             # left, but no additional
                             # filenames are permitted.
   
    $ ./program {1..91443}_file.txt
    bash: ./program: Argument list too long
   

   program.c

    #include <stdio.h>
    #include <string.h>
    #include <unistd.h>
   
    int main(int argc, char *argv[], char *envp[]) {
        size_t chr_ptr_size = sizeof(argv[0]);
        // The arguments array total size calculation
        size_t arg_strings_size = 0;
        size_t str_len = 0;
        for(int i = 0; i < argc; i++) {
            str_len = strlen(argv[i]) + 1;
            arg_strings_size += str_len;
    //      printf("%zu:\t%s\n\n", str_len, argv[i]);
        }
   
        size_t argv_size = arg_strings_size + argc * chr_ptr_size;
        printf( "arg strings size: %zu\n"
                "number of pointers to strings %i\n\n"
                "argv size:\t%zu + %i * %zu = %zu\n",
                 arg_strings_size,
                 argc,
                 arg_strings_size,
                 argc,
                 chr_ptr_size,
                 argv_size
            );
   
        // The enviroment variables array total size calculation
        size_t env_size = 0;
        for (char **env = envp; *env != 0; env++) {
          char *thisEnv = *env;
          env_size += strlen(thisEnv) + 1 + sizeof(thisEnv);
        }
   
        printf("envp size:\t%zu\n", env_size);
   
        size_t overall = argv_size + env_size + sizeof(argc);
   
        printf( "\nOverall (argv_size + env_size + sizeof(argc)):\t"
                "%zu + %zu + %zu = %zu\n",
                 argv_size,
                 env_size,
                 sizeof(argc),
                 overall);
        // Find ARG_MAX by system call
        long arg_max = sysconf(_SC_ARG_MAX);
   
        printf("ARG_MAX: %li\n\n", arg_max);
        printf("ARG_MAX - overall = %li\n", arg_max - (long) overall);
   
        return 0;
    }
   

   I have asked a question about the correctness of this program on StackOverflow: The maximum summarized size of argv, envp, argc (command line arguments) is always far from the ARG_MAX limit.