Linux – Cannot edit /proc/sys/kernel/threads-max

kernellinuxthreadsUbuntu

I am currently building a stress testing tool, and as such I need a pretty massive number of threads. I already went through all settings to raise the limit, but there is one last setting, the upper system-wide limit in /proc/sys/kernel/threads-max, that I can't seem to be able to change.

I tried

sysctl -w kernel.threads-max=200000

Editing manually with nano or echo

echo 200000 > /proc/sys/kernel/threads-max

Editing /etc/sysctl.conf and running

sysctl -f

If I run those as sudo, I have no error displayed (the new value is even displayed), but when checking again, the value hasn't changed either. When trying to edit the value with gedit, it spat an

invalid argument"

whatever the value I try, even the original one. I had no problem changing the pid_max value.

I really have no clue why it refuses my edits, and I haven't been able to find anybody with a similar problem, so I would be very grateful if someone could explain what is happening.

Best Answer

The repsonse lies in man proc(5), here is the interesting part:

 /proc/sys/kernel/threads-max (since Linux 2.3.11)
                     This file specifies the system-wide limit on the number
                     of threads (tasks) that can be created on the system.

                     Since Linux 4.1, the value that can be written to
                     threads-max is bounded.  The minimum value that can be
                     written is 20.  The maximum value that can be written
                     is given by the constant FUTEX_TID_MASK (0x3fffffff).
                     If a value outside of this range is written to threads-
                     max, the error EINVAL occurs.

                     The value written is checked against the available RAM
                     pages.  If the thread structures would occupy too much
                     (more than 1/8th) of the available RAM pages, threads-
                     max is reduced accordingly.

I assume your kernel version is > 4.1, so since 200000 (the number you try) is less than 0x3fffffff, the problem looks like the available RAM in not sufficient.

Related Solutions

Number of kernel threads = cores

No, you can set the maximum kernel threads to very high numbers.

Note that the word "threads" is used for many different things:

Most programmers use it to refer to independent threads of execution in the sense of POSIX threads. This is a way of organising programs and does not depend on hardware support. See Maximum number of threads per process in Linux?
Intel use it to refer to their "Hyperthreading" technology. See Why does my Intel i7-920 display 8 cores instead of 4 cores? and What does "thread" mean as related to CPUs?

It may be that Intels use causes confusion.

Update re kernel threads

Here are some Linux kernel threads running in CoLinux under Vista on AMD Athlon 64 X2 dual-core.

$ ps -eLf
UID        PID  PPID   LWP  C NLWP STIME TTY          TIME CMD
root         1     0     1  0    1 17:24 ?        00:00:00 init [2]
root         2     0     2  0    1 17:24 ?        00:00:00 [kthreadd]
root         3     2     3  0    1 17:24 ?        00:00:00 [ksoftirqd/0]
root         4     2     4  0    1 17:24 ?        00:00:00 [events/0]
root         5     2     5  0    1 17:24 ?        00:00:00 [khelper]
root        21     2    21  0    1 17:24 ?        00:00:00 [kblockd/0]
root        22     2    22  0    1 17:24 ?        00:00:00 [kseriod]
root        41     2    41  0    1 17:24 ?        00:00:00 [pdflush]
root        42     2    42  0    1 17:24 ?        00:00:00 [pdflush]
root        43     2    43  0    1 17:24 ?        00:00:00 [kswapd0]
root        44     2    44  0    1 17:24 ?        00:00:00 [aio/0]
root       727     2   727  0    1 17:24 ?        00:00:00 [kjournald]

LWP is the thread ID.

(See man ps: "-L Show threads, possibly with LWP and NLWP columns" … "LWP lwp (light weight process, or thread) ID of the lwp being reported. (alias spid, tid)")

kthreadd is the kernel thread daemon, I believe is is responsible for all the other kernel threads. Note I am not showing daemons like klogd which do not execute in ring 0 (as far as I know).

Number of kernel threads != number of CPU cores. (ref title of question)

Kernel threads consist of a set of registers, a stack, and a few corresponding kernel data structures.

…

The purported advantage of kernel threads over processes is faster creation and context switching compared with processes.

…

Kernel threads are considered “lightweight,” and one would expect the number of threads to only be limited by address space and processor time

…

In particular, operating system kernels tend to see kernel threads as a special kind of process rather than a unique entity. For example, in the Solaris kernel threads are called “light weight processes” (LWP’s). Linux actually creates kernel threads using a special variation of fork called “clone,” and until recently gave each thread a separate process ID. Because of this heritage, in practice kernel threads tend to be closer in memory and time cost to processes than user-level threads,

(Multiple Flows of Control in Migratable Parallel Programs 2006)

Why are user level threads faster than kernel level threads

Kernel-level threads require a context switch, which involves changing a large set of processor registers that define the current memory map and permissions. It also evicts some or all of the processor cache.

User-level threads just require a small amount of bookkeeping within one kernel thread or process.

However, the difference isn't big if your threads are predominantly doing I/O operations, as those have to go through the kernel in any case. It's most important if you're trying to implement some kind of simulation with a very large number of independant processes. In that case you need to pay careful attention to what thread synchronisation mechanisms you use, as some of them also go up to the kernel and trigger a context switch.

http://www.cs.rochester.edu/u/cli/research/switch.pdf "In general, the indirect cost of context switch ranges from several microseconds to more than one thousand microseconds for our workload."

Edit: user-level threads maintain a stack per-thread, and may or may not save the general-purpose registers depending on the architecture and the clobber rules of its calling convention. It can be as simple as dumping the registers to the stack, jumping to a new address, and popping a few registers, which may be in your cache if that thread was run recently.

Kernel-level context switches also change the memory map by writing to the TLB, and changing the security level (privilege level or "ring") of the processor. See "Performance Considerations"

Best Answer

Related Solutions

Number of kernel threads = cores

Why are user level threads faster than kernel level threads

Related Question