Linux – How to tell which user limit I am running into

limitlinux

Multiple sessions of the same user. When one of them gets to the point that it can no longer run new programs, none of them can, not even a new login of that user. Other users can still run new programs just fine, including new logins.

Normally user limits are in limits.conf, but its documentation says "please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session."

I'm nowhere close to running out of ram (44GB available), but I can't figure out what else to look at. What limits exist that would have a global effect on all sessions using the same UID, but not other UIDs?

Edited on 6/12/16 at 8:45p to add:

While writing the below I realized that the problem could be X11 related. This user account on this box is used nearly exclusively for GUI applications. Is there a good text based program I can try to run from bash that will use lots of resources and give good error messages? The box does not get to the point where it cannot even run ls.

Unfortunately, the GUI programs this problem normally affects (Chrome and Firefox) do not do a good job of leaving error messages behind. Chrome tabs will start showing up blank or with the completely useless "Aw, Snap!" error. Firefox simply will refuse to start. The only even partially helpful error messages I managed to obtain came from trying to start Firefox from bash:

[pascal@firefox ~]$ firefox --display=:0 --safe-mode
Assertion failure: ((bool)(__builtin_expect(!!(!NS_FAILED_impl(rv)), 1))) && thread (Should successfully create image decoding threads), at /builddir/build/BUILD/firefox-45.2.0/firefox-45.2.0esr/image/DecodePool.cpp:359
#01: ???[/usr/lib64/firefox/libxul.so +0x10f2165]
#02: ???[/usr/lib64/firefox/libxul.so +0xa2dd2c]
#03: ???[/usr/lib64/firefox/libxul.so +0xa2ee29]
#04: ???[/usr/lib64/firefox/libxul.so +0xa2f4c1]
#05: ???[/usr/lib64/firefox/libxul.so +0xa3095d]
#06: ???[/usr/lib64/firefox/libxul.so +0xa52d44]
#07: ???[/usr/lib64/firefox/libxul.so +0xa4c051]
#08: ???[/usr/lib64/firefox/libxul.so +0x1096257]
#09: ???[/usr/lib64/firefox/libxul.so +0x1096342]
#10: ???[/usr/lib64/firefox/libxul.so +0x1dba68f]
#11: ???[/usr/lib64/firefox/libxul.so +0x1dba805]
#12: ???[/usr/lib64/firefox/libxul.so +0x1dba8b9]
#13: ???[/usr/lib64/firefox/libxul.so +0x1e3e6be]
#14: ???[/usr/lib64/firefox/libxul.so +0x1e48d1f]
#15: ???[/usr/lib64/firefox/libxul.so +0x1e48ddd]
#16: ???[/usr/lib64/firefox/libxul.so +0x20bf7bc]
#17: ???[/usr/lib64/firefox/libxul.so +0x20bfae6]
#18: ???[/usr/lib64/firefox/libxul.so +0x20bfe5b]
#19: ???[/usr/lib64/firefox/libxul.so +0x21087cd]
#20: ???[/usr/lib64/firefox/libxul.so +0x2108cd2]
#21: ???[/usr/lib64/firefox/libxul.so +0x210aef4]
#22: ???[/usr/lib64/firefox/libxul.so +0x22578b1]
#23: ???[/usr/lib64/firefox/libxul.so +0x228ba43]
#24: ???[/usr/lib64/firefox/libxul.so +0x228be1d]
#25: XRE_main[/usr/lib64/firefox/libxul.so +0x228c073]
#26: ???[/usr/lib64/firefox/firefox +0x4c1d]
#27: ???[/usr/lib64/firefox/firefox +0x436d]
#28: __libc_start_main[/lib64/libc.so.6 +0x21b15]
#29: ???[/usr/lib64/firefox/firefox +0x449d]
#30: ??? (???:???)
Segmentation fault
[pascal@firefox ~]$ firefox --display=:0 --safe-mode -g
1465632860286DeferredSave.extensions.jsonWARNWrite failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860287addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860288addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860289addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860289addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860290addons.xpi-utilsWARNFailed to save XPI database: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860358DeferredSave.addons.jsonWARNWrite failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
1465632860359addons.repositoryERRORSaveDBToDisk failed: Error: Could not create new thread! (resource://gre/modules/PromiseWorker.jsm:173:18) JS Stack trace: this.BasePromiseWorker.prototype._worker@PromiseWorker.jsm:173:18 < postMessage@PromiseWorker.jsm:292:9 < TaskImpl_run@Task.jsm:315:40 < Handler.prototype.process@Promise-backend.js:933:23 < this.PromiseWalker.walkerLoop@Promise-backend.js:812:7 < this.PromiseWalker.scheduleWalkerLoop/<@Promise-backend.js:746:1 < BG__finalUIStartup@nsBrowserGlue.js:770:1 < BG_observe@nsBrowserGlue.js:284:9
Segmentation fault
[pascal@firefox ~]$

[pascal@localhost ~]$ ulimit -aH
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 579483
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 579483
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[pascal@localhost ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 579483
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
[pascal@localhost ~]$ set /proc/*/task/*/cwd/.; echo $#
306
[pascal@localhost ~]$ prlimit
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited blocks
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited blocks
LOCKS      max number of file locks held      unlimited unlimited
MEMLOCK    max locked-in-memory address space     65536     65536 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0
NOFILE     max number of open files               32768     65536
NPROC      max number of processes                 4096    579483
RSS        max resident set size              unlimited unlimited pages
RTPRIO     max real-time priority                     0         0
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals         579483    579483
STACK      max stack size                       8388608 unlimited bytes

Edited on 6/13/16 at 10:24p to add:

Not a GUI problem. When I tried to su to the user today, that doesn't even work. Root is fine. I can ls, vi, create a new user, su to that user, everything works fine for that user, I exit and try to su to the problem user and no go. Bash kinda loaded the first time, but even exit didn't work. I had to reconnect to get back to root.

[root@firefox ~]# su - pascal
Last login: Sat Jun 11 03:08:47 CDT 2016 on pts/1
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ ls
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ exit
logout
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: retry: No child processes
-bash: fork: Resource temporarily unavailable
-bash-4.2$ 

[root@firefox ~]# ls -l /
total 126
lrwxrwxrwx.    1 root root   7 Jan 28 23:53 bin -> usr/bin
---- snip ----
drwxr-xr-x.   19 root root  23 May 27 18:03 var
[root@firefox ~]# vi /etc/rc.local
[root@firefox ~]# useradd test
[root@firefox ~]# su - test
[test@firefox ~]$ cd
[test@firefox ~]$ ls -l
total 0
[test@firefox ~]$ ls -l /
total 126
lrwxrwxrwx.    1 root root   7 Jan 28 23:53 bin -> usr/bin
---- snip ----
drwxr-xr-x.   19 root root  23 May 27 18:03 var
[test@firefox ~]$ vi /etc/rc.local
[test@firefox ~]$ exit
logout
[root@firefox ~]# su - pascal
Last login: Mon Jun 13 22:12:12 CDT 2016 on pts/1
su: failed to execute /bin/bash: Resource temporarily unavailable
[root@firefox ~]#

Best Answer

nproc was the problem:

[root@localhost ~]# ps -eLf | grep pascal | wc -l
4068
[root@localhost ~]# cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     4096
root       soft    nproc     unlimited
[root@localhost ~]#

man limits.conf states:

   Also, please note that all limit settings are set per login. They are
   not global, nor are they permanent; existing only for the duration of
   the session. One exception is the maxlogin option, this one is system
   wide. But there is a race, concurrent logins at the same time will not
   always be detected as such but only counted as one.

It appears to me that nproc is only enforced per login but counts globally. So a login with nproc 8192 and 5000 threads would have no problems, but a simultaneous login of the same UID with nproc 4096 and 50 threads would not be able to create more because the global count (5050) is above its nproc setting. 

[root@localhost ~]# ps -eLf | grep pascal | grep google/chrome | wc -l
3792
Related Question