Not only is it possible; it is also recommended in some situations.
GNU Parallel takes around 10 ms to run a job. So if you have 8 cores and the jobs you run take less than 70 ms, then you will see GNU Parallel use 100% of a single core, and yet there will be idle time on other cores. Thus you will not use 100% of all cores.
The other situation where it is recommended is if you want to run more jobs than -j0 will do. Currently -j0 will run around 250 jobs in parallel unless you adjust some system limits. It makes perfect sense to run more than 250 jobs if the jobs are not limited by CPU and disk I/O. This is for example true if network latency is the limiting factor.
However, using 2 lists is not the recommended way to split up jobs. The recommended way is to use GNU Parallel to call GNU Parallel:
cat list0 | parallel -j20 --pipe parallel -j100
That will run 2000 jobs in parallel. To run more adjust -j. It is recommended that the outer (the 20) is at least the number of cores, so that there will be at least one GNU Parallel process on each core.
Using this technique you should have no problem starting 20000 jobs in parallel; when you get over 32000 processes things start acting up.
By first running:
echo 4194304 | sudo tee /proc/sys/kernel/pid_max
I was able to run:
seq 1000000 2000000000 |
parallel -j16 --roundrobin --pipe parallel -j0 --pipe parallel -j0 sleep
which will start 1 million processes in parallel (it takes 300 G RAM on my system).
Order
Your sample program did not seem to care about the order of the *.png for the allframes array that you were constructing, but your comments led me to believe that order would matter.
I guess what I would want to do is to have parallel splitting the load like handing the first 100 frames to core 1, frames 100-199 to core 2 etc.?
Bash
Therefore I'd start with a modification to your script like so, changing the construction of the allframes array so that the files are stored in numeric order.
allframes=($(printf "%s\n" *.png | sort -V | tr '\n' ' '))
This can be simplified further to this using sort -zV:
allframes=($(printf "%s\0" *.png | sort -zV | tr '\0' ' '))
This has the effect on constructing your convert ... commands so that they look like this now:
$ convert "0.png 1.png 2.png 3.png 4.png 5.png 6.png 7.png 8.png 9.png \
10.png 11.png 12.png 13.png 14.png 15.png 16.png 17.png 18.png \
19.png 20.png 21.png 22.png 23.png 24.png 25.png 26.png 27.png \
28.png 29.png 30.png 31.png 32.png 33.png 34.png 35.png 36.png \
37.png 38.png 39.png 40.png 41.png 42.png 43.png 44.png 45.png \
46.png 47.png 48.png 49.png 50.png 51.png 52.png 53.png 54.png \
55.png 56.png 57.png 58.png 59.png 60.png 61.png 62.png 63.png \
64.png 65.png 66.png 67.png 68.png 69.png 70.png 71.png 72.png \
73.png 74.png 75.png 76.png 77.png 78.png 79.png 80.png 81.png \
82.png 83.png 84.png 85.png 86.png 87.png 88.png 89.png 90.png \
91.png 92.png 93.png 94.png 95.png 96.png 97.png 98.png 99.png" \
-evaluate-sequence "mean" -channel RGB -normalize ../out2/0.png
Parallels
Building off of eschwartz's example I put together a parallel example as follows:
$ printf '%s\n' *.png | sort -V | parallel -n100 --dryrun convert {} \
-evaluate-sequence 'mean' -channel RGB -normalize ../out2/{1}
again, more simply using sort -zV:
$ printf '%s\0' *.png | sort -zV | parallel -0 -n100 --dryrun "convert {} \
-evaluate-sequence 'mean' -channel RGB -normalize ../out2/{1}
NOTE: The above has an echo "..." as the parallel action to start. Doing it this way helps to visualize what's happening:
$ convert 0.png 1.png 2.png 3.png 4.png 5.png 6.png 7.png 8.png 9.png 10.png \
11.png 12.png 13.png 14.png 15.png 16.png 17.png 18.png 19.png \
20.png 21.png 22.png 23.png 24.png 25.png 26.png 27.png 28.png \
29.png 30.png 31.png 32.png 33.png 34.png 35.png 36.png 37.png \
38.png 39.png 40.png 41.png 42.png 43.png 44.png 45.png 46.png \
47.png 48.png 49.png 50.png 51.png 52.png 53.png 54.png 55.png \
56.png 57.png 58.png 59.png 60.png 61.png 62.png 63.png 64.png \
65.png 66.png 67.png 68.png 69.png 70.png 71.png 72.png 73.png \
74.png 75.png 76.png 77.png 78.png 79.png 80.png 81.png 82.png \
83.png 84.png 85.png 86.png 87.png 88.png 89.png 90.png 91.png \
92.png 93.png 94.png 95.png 96.png 97.png 98.png 99.png \
-evaluate-sequence mean -channel RGB -normalize ../out2/0.png
If you're satisfied with this output, simply remove the --dryrun switch to parallel, and rerun it.
$ printf '%s\0' *.png | sort -zV | parallel -0 -n100 convert {} \
-evaluate-sequence 'mean' -channel RGB -normalize
References
Best Answer
If the only issue is that the directories are hidden, you can just remove the
.from the beginning of their name to make them unhidden. For example, using perl-rename (calledrenameon Ubuntu):Or, with just shell tools:
Bot of these leave you with horrible directory names though, with spaces and slashes and other unsavory things. Since you're renaming, you may as well rename to something sane:
That will result in:
IMPORTANT: note that I am not checking for file name collision. If you rename one of these to something that already exists, then you will overwrite the existing file.