Does “ulimit -Su” limit the number of user processes created in interactive login subshells and tmux sessions created therein

bashkshprocesstmuxulimit

A few months ago, I wrote a set of bash scripts utilizing tmux to create a simple IDE on an AIX 7.1 server. There is a bug in one of my scripts that will sometimes generate user processes very rapidly up to the limit set by ulimit. This happens very infrequently (about once per month), and I have already spent several unsuccessful hours tracking down this bug, so I decided that for the time being, I could simply set my soft user process limit to something lower than the hard limit (e.g. 100 instead of 1024) so that when my bug demonstrates itself again, there won't be a noticeable performance hit for other users on the server. Unfortunately, "ulimit -Su 100" does not appear to work in bash on AIX 7.1, but it does work in ksh. I have performed the following workaround:

Made ksh the default shell:

$ chsh [username] /usr/bin/ksh

Wrote the following to ~/.kshrc:

ulimit -Su 100  # works in ksh, but not in bash
/bin/bash -il   # start bash as an interactive login shell
exit            # once bash exits, exit from ksh, too

So now, every time I create a shell, ksh sets the soft user process limit and starts bash as an interactive login shell (I still want ~/.bash_profile to get sourced). Now I have to wonder, are the user process limits set in ksh still going to be enforced in bash subshells? In the top-level bash subshell, I ran the following:

$ ulimit -Sa
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) unlimited
pipe size            (512 bytes, -p) 64
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited

As you can see, the user process limit is set to 1024.

Another major concern of mine is knowing whether or not the limit set in ksh will include processes created in tmux sessions in bash subshells.

Other detail:
Whenever I create a new pane in tmux, I am fairly certain that ksh is invoked, ~/.kshrc is sourced, and bash is started, just like normal. I believe this is the case because the title of each newly created tmux panes is "ksh" (the default title of a tmux pane is the name of the current process in the foreground), yet I am presented with a bash prompt instead of a ksh prompt.

This has rambled on a bit too much, so I suppose I'll omit further detail unless asked for it.

Edit 1: Strange Behavior

Look what happens when I try to get the user process limit with "ulimit -Su" (no args), with and without using truss:

$ truss ulimit -Su 2>| truss.out
100
$ ulimit -Su
1024

Maybe I'm using the tool wrong, but that looks plain strange. These commands were run inside tmux.

Edit 2: Additional Information

These commands were run from a regular bash prompt — no subshells or tmux.

$ truss ksh -c "ulimit -Su 100" 2>&1 | grep limit
getrlimit64(9, 0x2FF1B988)                      = 0
setrlimit64(9, 0x2FF1B988)                      = 0
$ truss bash -c "ulimit -Su 100" 2>&1 | grep limit
appulimit(1005, 0)                              = 0x2001C000
bash: line 0: ulimit: max user processes: cannot modify limit: A system call received a parameter that is not valid.

Best Answer

Introduction to resource limits

Resource limits in Unix-like systems are controlled by getrlimit() and setrlimit() system calls. These limits are configured per process and are inherited when a new process is spawn (e.g. by fork()). It means that if you want to set a limit from a shell the command must be built into the shell (not executed as a child process). Really ulimit is a builtin in many shells including ksh and bash.

The observed behaviour

The "builtin" nature of ulimit explains the different behaviour in ksh and bash.

The limit for number of user processes (ulimit -u) is set by setrlimit(RLIMIT_NPROC, ...) syscall. In old versions of AIX RLIMIT_NPROC was not supported.<1> The support was added in AIX 6.1<2 section 5.4.4 Implemented changes> so the ulimit in ksh uses setrlimit64() correctly. The bash was probably compiled to be compatible with older version of AIX and is not able to control this limit.

Conclusion

You can use the ulimit builtin from ksh and all the child processes will inherit the configured limits. The shells and processes in general have nothing to do with enforcing and keeping the resource limits unless they explicitly call setrlimit().

Alternative

In AIX there is also an alternative which should work in older versions of AIX too:

chdev -l sys0 -a maxuproc=100

See: 3, 4