DB2 – SQL2043N Unable to Start a Child Process or Thread on RedHat

db2db2-10.5redhat

I've seen the message SQL2043N Unable to start a child process or thread occur every once in a while since upgrading to rhel7 and db2 10.5. db2diag.log does not reveal much out of interest, beside:

FUNCTION: DB2 UDB, oper system services, 
sqlo_waitlist::initialize, probe:10
MESSAGE : SysV semget failed with errno:
DATA #1 : Hexdump, 4 bytes

The target machines are used for unit-testing and development and relevant kernel parameters seems appropriate:

]$ cat /proc/meminfo 
MemTotal:       12140640 kB

]$ ipcs -l
[...]
------ Semaphore Limits --------
max number of arrays = 3072
max semaphores per array = 250
max semaphores system wide = 256000
max ops per semop call = 32
semaphore max value = 32767

At the moment there are no databases on the machine, yet – what I believe – a huge number of semaphores is associated with db2fenc1:

]$ ipcs -s | grep db2fenc1 | wc -l
2800 [1]

[1] The number 2800 is not the exact number, but it was close to this

]$ db2stop force

has no effect on the number of semaphores in ipcs -s. To release the semaphores I had to do ipclean

The only thing I can think of that might be causing db2fenc1 to grab a lot of semaphores is a fenced java function:

CREATE FUNCTION NYA.REMOVE_DIACRITICS( S VARCHAR(50))
    RETURNS varchar(50)
    FENCED THREADSAFE
    DETERMINISTIC
    NO SQL
    LANGUAGE JAVA
    PARAMETER STYLE JAVA
    RETURNS NULL ON NULL INPUT
    EXTERNAL NAME
    'StringUtil:se.vhs.nya.framework.common.util.db.DiacriticUtil!removeDiacritics'
NO EXTERNAL ACTION @

The functions is used in an generated column:

CREATE TABLE ...
( [...]
, <COLNAME2> VARCHAR(25) NOT NULL
    GENERATED ALWAYS AS ( NYA.REMOVE_DIACRITICS( <COLNAME1> ) )
, [...] ) @

but there is no massive insert/updates against this table during testing.

Has anyone experienced something similar where semaphores are not released, and in that case, what was the solution?

PS. Not sure if it is relevant, but the JAVA_HEAP_SZ is:

(JAVA_HEAP_SZ) = 2048

Best Answer

Before reporting this as error, the max number of processes (ulimit -u) should be checked and increased for the instance owner. I'm sure they would ask it anyway.