Why using more threads makes it slower than using less threads

multithreading

Tried to run program X using 8 threads and it was over in n minutes.
Tried to run same program using 50 threads and it was over in n*10 minutes.

Why does this happen and how can I get optimal number of threads I can use?

Best Answer

This is a complicated question you're asking. Without knowing more about the nature of your threads it's difficult to say. Some things to consider when diagnosing system performance:

Is the process/thread

CPU bound (needs lots of CPU resources)
Memory bound (needs lots of RAM resources)
I/O bound (Network and/or hard drive resources)

All of these three resources are finite and any one can limit the performance of a system. You need to look at which (might be 2 or 3 together) your particular situation is consuming.

You can use ntop and iostat, and vmstat to diagnose what's going on.

Related Solutions

Linux max threads count

Your issue is the max user processes limit.

From the getrlimit(2) man page:

RLIMIT_NPROC The maximum number of processes (or, more precisely on Linux, threads) that can be created for the real user ID of the calling process. Upon encountering this limit, fork(2) fails with the error EAGAIN.

Same for pthread_create(3):

EAGAIN Insufficient resources to create another thread, or a system-imposed limit on the number of threads was encountered. The latter case may occur in two ways: the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of process for a real user ID, was reached; or the kernel's system-wide limit on the number of threads, /proc/sys/kernel/threads-max, was reached.

Increase that limit for your user, and it should be able to create more threads, until it reaches other resource limits.
Or plain resource exhaustion - for 1Mb stack and 20k threads, you'll need a lot of RAM.
See also NPTL caps maximum threads at 65528?: /proc/sys/vm/max_map_count could become an issue at some point.

Side point: you should use -pthread instead of -lpthread. See gcc - significance of -pthread flag when compiling.

Run Command on Multiple Threads in Bash

Both moreutils parallel and GNU parallel will do this for you. With moreutils' parallel, it looks like:

parallel -j "$(nproc)" pngquant [pngquant-options] -- *.png

nproc outputs the number of available processors (threads), so that will run available-processors (-j "$(nproc)") pngquants at once, passing each a single PNG file. If the startup overhead is too high, you can pass more PNG files at once to each run with the -n option; -n 2 would pass two PNGs to each pngquant.

Best Answer

Related Solutions

Linux max threads count

Run Command on Multiple Threads in Bash

Related Question