Linux – realtime pthread created from non-realtime thread with init.d

init.dlinuxpriorityreal timescheduling

I have a piece of C++ code that runs just fine when I run it from a Linux terminal, but which throws an EPERM error when run from a SystemV (init.d) script on bootup. The error comes from a pthread_create with the following bit of attributes assigned to the thread attempting to be created:

pthread_t reading_thread;

pthread_attr_t read_attr;
struct sched_param read_param; 
pthread_attr_init(&read_attr);
pthread_attr_setschedpolicy(&read_attr, SCHED_FIFO);
pthread_attr_setinheritsched(&read_attr, PTHREAD_EXPLICIT_SCHED);
read_param.sched_priority = 30;
pthread_attr_setschedparam(&read_attr, &read_param);

k = pthread_create(&reading_thread, &read_attr, Reading_Thread_Function, 
            (void*) &variable_to_pass_to_Reading_Thread_Function); // Will return EPERM

This code works just fine when run from my terminal. It also runs just fine in the init.d script when I call "/etc/init.d/myinitdscript start". It also runs fine as "sudo service myinitdscript start". The init.d script contains the following:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          myinitdscript
# Required-Start:    $local_fs $remote_fs $syslog $network
# Required-Stop:     $local_fs $remote_fs $syslog $network
# Default-Start:     3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts my daemon
# Description:       Verbose explanation of starting my daemon
### END INIT INFO
PATH=/sbin:/usr/sbin:/bin:/usr/bin
LOG=/home/someusershome/initd.log
NAME=myinitdscript
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
[ -x "$DAEMON" ] || (echo "$DAEMON not found. Exiting $SCRIPTNAME." >> $LOG 2>&1 && exit 0)
USERTORUNAS=a_user_on_my_system
SOURCE_SCRIPT=/home/$USERTORUNAS/source_script
DAEMON_ARGS="some_args_for_script"

. /lib/init/vars.sh

. /lib/lsb/init-functions

# Source this script for environmental variables
[ -f $SOURCE_SCRIPT ] && . $SOURCE_SCRIPT

# This is called when called with 'start', I am skipping that for succintness
do_start()
{
    start-stop-daemon --start --make-pidfile --pidfile $PIDFILE --test --background --chuid $USERTORUNAS --startas /bin/bash -- -c "exec $DAEMON -- $DAEMON_ARGS >> $LOG 2>&1 " || return 1
    start-stop-daemon --start --make-pidfile --pidfile $PIDFILE --background --chuid $USERTORUNAS --startas /bin/bash -- -c "exec $DAEMON -- $DAEMON_ARGS >> $LOG 2>&1" || return 2
}

If I activate this init.d script using:

update-rc.d myinitdscript defaults 99

it will error on boot with an EPERM error thrown at the pthread_create call (k = 1, aka EPERM). I can run this using sudo service myinitdscript start, and it will run just fine. I can also call /etc/init.d/myinitdscript start, and it will run just fine. It is only when I let the system run this script on boot that it fails.

I find that if I add to my start-stop-daemon calls the option "-P fifo:99" I don't get the EPERM error and the code runs okay except at too high a priority so I won't call this a fix. The only part of the code that needs to run real-time is that pthread created from in the code. So I suppose this has to do with my permissions to create a real-time thread with priority 30 from within a normally scheduled thread.

Why does my script need special scheduling policies/priorities when run from boot versus when I manually start the init.d script or through service?

EDIT: Running on Ubuntu 12.04.

EDIT2: I tried adding a call to "ulimit -r" inside my code which the start-stop-daemon call starts, and I get unlimited, so as far as I can see, there shouldn't be any permissions issue going with SCHED_FIFO:30 there

EDIT3: It turns out I am running Upstart, and Upstart has an init script called rc-sysinit.conf which starts all the SystemV style scripts. So perhaps Upstart is screwing up my permissions.

Best Answer

The answer seems to have been to put the following in my init.d script, which I put just before the start-stop-daemon calls in do_start:

ulimit -r ## (where ## is a sufficiently high number; 99 works)

The way I was able to determine this was by making system calls to ulimit -a inside of a bash command inside of my code:

bash -c "ulimit -a"

The bash part is necessary, because ulimit -a is a shell builtin. ulimit -a on /bin/sh returns different information unrelated to real-time priority. For some reason, I found that my real-time priority was limited to 0 (no real-time priority) when my service is started at boot. When I run it with service or by calling the init.d script, it inherits my permissions which allow for real-time priority. But when the system calls it through the Upstart/SystemV backwards compatibility system, it doesn't get that elevated privilege. I suppose this might relate to posts I have seen that say Upstart doesn't read /etc/security/limits.conf which is where you would set system-wide real-time priority permissions for non-privileged users.

If anyone can verify or explain why this solution works, I would love to hear it.

Related Solutions

Bash – How to convert Ubuntu init script to Amazon Linux AMI init script

I updated my script to following code. Although it is not able to create pidfile but it is working as I want. If anybody has better script then kindly post your answer.

#!/bin/bash

# chkconfig: 2345 95 05
# description: MY Application

# Source function library.
. /etc/init.d/functions

RETVAL=0
prog="myserver"
PIDFILE=/var/run/$prog.pid
LOCKFILE=/var/lock/subsys/$prog
PATH=/sbin:/usr/sbin:/bin:/usr/bin
JAVA=/usr/bin/java
DESC="MY Application"
DAEMON=$JAVA
DAEMON_HOME="/home/ganesh/MyServer/"
JAR=$DAEMON_HOME/MyServer.jar
DAEMON_ARGS="-Xms512m -Xmx4112m -jar $JAR"
SCRIPTNAME=/etc/init.d/myserver
#the user that will run the script
USER=root

#echo "All value sets"

start() {
    if [ -f $LOCKFILE ];
    then
        echo "$DESC is already running. Exiting."
        exit 1
    else
        echo -n "Starting $prog:"
        cd $DAEMON_HOME
        daemon --user $USER  --pidfile $PIDFILE $DAEMON $DAEMON_ARGS >/dev/null 2>&1 &
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $LOCKFILE
    fi

    return $RETVAL
}

stop() {
        echo -n "Shutting down "$prog:
        kill -9  `ps -ef | grep "$JAR" | grep -v grep | awk '{ print $2 }'`        
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && rm -f $LOCKFILE
        return $RETVAL
}

check_status() {
        #echo -n "Checking $prog status: "
        status $prog
        RETVAL=$?
        return $RETVAL
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        check_status
        ;;
    restart)
        stop
        start
        ;;
    *)

        echo "Usage: $prog {start|stop|status|restart}"
        exit 1
        ;;
esac
exit $RETVAL

EDIT:

Another Version of above script where I am creating pid file and using it to get status and stop the application. I think the following script is more reliable than above.

#!/bin/bash

# chkconfig: 2345 95 05
# description: MY Application

# Source function library.
. /etc/init.d/functions

RETVAL=0
prog="myserver"
PIDFILE=/var/run/$prog.pid
LOCKFILE=/var/lock/subsys/$prog
PATH=/sbin:/usr/sbin:/bin:/usr/bin
JAVA=/usr/bin/java
DESC="MY Application"
DAEMON=$JAVA
DAEMON_HOME="/home/ganesh/MyServer/"
JAR=$DAEMON_HOME/MyServer.jar
DAEMON_ARGS="-Xms512m -Xmx4112m -jar $JAR"
SCRIPTNAME=/etc/init.d/myserver
#the user that will run the script
USER=root

#echo "All value sets"

start() {
    if [ -f $PIDFILE ]; then
        PID=`cat $PIDFILE`
        if [ -z "`pgrep $PID`" ] && [ "$PID" != "`ps aux|grep -vE 'grep|runuser|bash'|grep -w "$JAR"|awk '{print $2}'`" ]; then
            printf "%s\n" "Process dead but pidfile exists"
        else
            printf "$prog is already running!\n"
        fi
    else
        printf "%-50s" "Starting $prog ..."
        cd $DAEMON_HOME
        daemon --user $USER $DAEMON $DAEMON_ARGS >/dev/null 2>&1 &
        sleep 5
        PID=`ps aux|grep -vE 'grep|runuser|bash'|grep -w "$JAR"|awk '{print $2}'`
        if [ -z "$PID" ]; then
            printf "[ \e[31mFAIL\033[0m ]\n"
        else
            echo $PID > $PIDFILE
            printf "[ \e[32mOK\033[0m ]\n"
        fi
    fi
}

stop() {
    printf "%-50s" "Shutting down $prog:"
    if [ -f $PIDFILE ]; then
        PID=`cat $PIDFILE`
        kill -HUP $PID 2>/dev/null
        printf "[ \e[32mOK\033[0m ]\n"
        rm -f $PIDFILE
    else
        printf "[ \e[31mFAIL\033[0m ]\n" 
    fi
}

check_status() {
    printf "%-50s" "Checking $prog ..."
    if [ -f $PIDFILE ]; then
        PID=`cat $PIDFILE`
        if [ -z "`pgrep $PID`" ] && [ "$PID" != "`ps aux|grep -vE 'grep|runuser|bash'|grep -w "$JAR"|awk '{print $2}'`" ]; then
            printf "%s\n" "Process dead but pidfile exists"
        else
            printf "[ \e[32mRUNNING\033[0m ]\n"
        fi
    else
        printf "[ \e[31mSTOPPED\033[0m ]\n"
    fi
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    status)
        check_status
        ;;
    restart)
        stop
        start
        ;;
    *)
        echo "Usage: $prog {start|stop|status|restart}"
        exit 1
        ;;
esac
exit 1

Debian – Nginx init.d script

my nginx file in init.d:

… is entirely irrelevant. Look at the output of systemctl. It told you exactly what file to look at:

Loaded: loaded (/lib/systemd/system/nginx.service; enabled)

That is your service unit. And systemctl is telling you what that service unit has configured for preparing to start the service:

 Process: 14126 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=203/EXEC)

That's what you'll find in the ExecStartPre setting in the service unit. You'll notice from it that PATH is also entirely irrelevant (which is by design in systemd). The journal was even more explicit about what happened:

systemd[14126]: Failed at step EXEC spawning /usr/sbin/nginx: No such file or directory

Whatever you may think, the computer thinks that at the time that you tried to start that service /usr/sbin/nginx could not be run as a program because no file existed by that name.

My educated guess is that you have a RootDirectory setting in your service unit, and either you copied the file to the /usr/sbin that is not within the changed root environment or some ancillary shared libraries necessary for loading the program image are missing from the changed root environment.

Best Answer

Related Solutions

Bash – How to convert Ubuntu init script to Amazon Linux AMI init script

Debian – Nginx init.d script

Related Question