Run Command on Multiple Threads in Bash

bashcentosmultithreading

I am running a command (pngquant to be precise: https://github.com/pornel/pngquant) in a terminal window. I noticed, that if I open 4 terminal windows, and run pngquant command in each of them, I get 4x speed increase, effectively compressing 4 times as many images in the same time as before.

So I used this approach and assigned each pngqunat process a portion of images I want to compress, effectively creating multiple processes on multiple threads

Can you run command on multiple threads without doing this tricks that I did? I would like to just say "run pngquant compression on all this images and use all threads available."

Best Answer

Both moreutils parallel and GNU parallel will do this for you. With moreutils' parallel, it looks like:

parallel -j "$(nproc)" pngquant [pngquant-options] -- *.png

nproc outputs the number of available processors (threads), so that will run available-processors (-j "$(nproc)") pngquants at once, passing each a single PNG file. If the startup overhead is too high, you can pass more PNG files at once to each run with the -n option; -n 2 would pass two PNGs to each pngquant.

Related Solutions

Linux max threads count

Your issue is the max user processes limit.

From the getrlimit(2) man page:

RLIMIT_NPROC The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

Same for pthread_create(3):

EAGAIN Insufficient resources to create another thread, or a system-imposed limit on the number of threads was encountered. The latter case may occur in two ways: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of process for a real user ID, was reached; or the kernel's system-wide limit on the number of threads, /proc/sys/kernel/threads-max, was reached.

Increase that limit for your user, and it should be able to create more threads, until it reaches other resource limits.
Or plain resource exhaustion - for 1Mb stack and 20k threads, you'll need a lot of RAM.
See also NPTL caps maximum threads at 65528?: /proc/sys/vm/max_map_count could become an issue at some point.

Side point: you should use -pthread instead of -lpthread. See gcc - significance of -pthread flag when compiling.

Bash – Saving bash history from multiple Konsole not working correctly

I have concluded that it is not possible to append the new commands directly and at the same time to remove any duplicates. So my solution is to run the following command ones in a while. or to put it in my crontab.

tac $HOME/.bash_history | awk '!seen[$0]++' | tac > $HOME/.hist_Temp 
mv $HOME/.hist_Temp $HOME/.bash_history

The above command will keep the last occurrence of a command and remove every other repetitions above it from the history file without messing up the order.

Best Answer

Related Solutions

Linux max threads count

Bash – Saving bash history from multiple Konsole not working correctly

Related Question