Centos – chkconfig bootup script failing, script runs good manually

centoschkconfiginit-scriptservices

I recently purchased a vps and I attempted to create a startup/restart script to start up my java program. I am running CentOS 6. The script works if I execute it manually with this command:

service policyserver start

But fails to run automatically, creates a file in the /tmp/test.log with the following error:

Starting Policy Server: failed

Here is the startup script inside /etc/init.d/

#!/bin/bash
#
#policyserver:          Startup script for CamSpark Policy Server Application.
#
#chkconfig: 3 80 05
#description:      Startup script for Cam Spark Policy Server Application.

CAMSPARK_POLICY_HOME=/tools/camsparkserver/Policy;
export CAMSPARK_POLICY_HOME

start() {
        echo -n "Starting Policy Server: "
        $CAMSPARK_POLICY_HOME/Policy.sh start >> /tmp/test.log
        sleep 2
        echo "done"
}

stop() {
        echo -n "Stopping Policy Server: "
        $CAMSPARK_POLICY_HOME/Policy.sh stop
        echo "done"
}

# See how we were called.
case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                stop
                start
                ;;
        *)
                echo $"Usage: policyserver {start|stop|restart}"
                exit
esac

Here is the main script (can't move this script)

#!/bin/bash
#
# chkconfig: 345 99 05 
# description: Java deamon script
#
# A non-SUSE Linux start/stop script for Java daemons.
#
# Set this to your Java installation
JAVA_HOME=/usr/java/latest

scriptFile=$(readlink -fn $(type -p $0))                   # the absolute, dereferenced path of this script file
scriptDir=$(dirname $scriptFile)                           # absolute path of the script directory
serviceNameLo="policyserver"                                  # service name with the first letter in lowercase
serviceName="PolicyServer"                                    # service name
serviceUser="root"                                      # OS user name for the service
serviceGroup="root"                                    # OS group name for the service
applDir="/tools/camsparkserver/Policy"                          # home directory of the service application
serviceUserHome="/home/$serviceUser"                       # home directory of the service user
serviceLogFile="/var/log/$serviceNameLo.log"               # log file for StdOut/StdErr
maxShutdownTime=15                                         # maximum number of seconds to wait for the daemon to terminate normally
pidFile="/var/run/$serviceNameLo.pid"                      # name of PID file (PID = process ID number)
javaCommand="java"                                         # name of the Java launcher without the path
javaExe="$javaCommand"                      # file name of the Java application launcher executable
javaArgs="PolicyServer"                     # arguments for Java launcher
javaCommandLine="$javaExe $javaArgs"                       # command line to start the Java service application
javaCommandLineKeyword="PolicyServer"                     # a keyword that occurs on the commandline, used to detect an already running service process and to distinguish it from others

# Makes the file $1 writable by the group $serviceGroup.
function makeFileWritable {
   local filename="$1"
   touch $filename || return 1
   chgrp $serviceGroup $filename || return 1
   chmod g+w $filename || return 1
   return 0; }

# Returns 0 if the process with PID $1 is running.
function checkProcessIsRunning {
   local pid="$1"
   if [ -z "$pid" -o "$pid" == " " ]; then return 1; fi
   if [ ! -e /proc/$pid ]; then return 1; fi
   return 0; }

# Returns 0 if the process with PID $1 is our Java service process.
function checkProcessIsOurService {
   local pid="$1"
   if [ "$(ps -p $pid --no-headers -o comm)" != "$javaCommand" ]; then return 1; fi
   grep -q --binary -F "$javaCommandLineKeyword" /proc/$pid/cmdline
   if [ $? -ne 0 ]; then return 1; fi
   return 0; }

# Returns 0 when the service is running and sets the variable $pid to the PID.
function getServicePID {
   if [ ! -f $pidFile ]; then return 1; fi
   pid="$(<$pidFile)"
   checkProcessIsRunning $pid || return 1
   checkProcessIsOurService $pid || return 1
   return 0; }

function startServiceProcess {
   cd $applDir || return 1
   rm -f $pidFile
   makeFileWritable $pidFile || return 1
   makeFileWritable $serviceLogFile || return 1
   cmd="nohup $javaCommandLine >>$serviceLogFile 2>&1 & echo \$! >$pidFile"
   # Don't forget to add -H so the HOME environment variable will be set correctly.
   sudo -u $serviceUser -H $SHELL -c "$cmd" || return 1
   sleep 0.1
   pid="$(<$pidFile)"
   if checkProcessIsRunning $pid; then :; else
      echo -ne "\n$serviceName start failed, see logfile."
      return 1
   fi
   return 0; }

function stopServiceProcess {
   kill $pid || return 1
   for ((i=0; i<maxShutdownTime*10; i++)); do
      checkProcessIsRunning $pid
      if [ $? -ne 0 ]; then
         rm -f $pidFile
         return 0
         fi
      sleep 0.1
      done
   echo -e "\n$serviceName did not terminate within $maxShutdownTime seconds, sending SIGKILL..."
   kill -s KILL $pid || return 1
   local killWaitTime=15
   for ((i=0; i<killWaitTime*10; i++)); do
      checkProcessIsRunning $pid
      if [ $? -ne 0 ]; then
         rm -f $pidFile
         return 0
         fi
      sleep 0.1
      done
   echo "Error: $serviceName could not be stopped within $maxShutdownTime+$killWaitTime seconds!"
   return 1; }

function startService {
   getServicePID
   if [ $? -eq 0 ]; then echo -n "$serviceName is already running"; RETVAL=0; return 0; fi
   echo -n "Starting $serviceName   "
   startServiceProcess
   if [ $? -ne 0 ]; then RETVAL=1; echo "failed"; return 1; fi
   echo "started PID=$pid"
   RETVAL=0
   return 0; }

function stopService {
   getServicePID
   if [ $? -ne 0 ]; then echo -n "$serviceName is not running"; RETVAL=0; echo ""; return 0; fi
   echo -n "Stopping $serviceName   "
   stopServiceProcess
   if [ $? -ne 0 ]; then RETVAL=1; echo "failed"; return 1; fi
   echo "stopped PID=$pid"
   RETVAL=0
   return 0; }

function checkServiceStatus {
   echo -n "Checking for $serviceName:   "
   if getServicePID; then
    echo "running PID=$pid"
    RETVAL=0
   else
    echo "stopped"
    RETVAL=3
   fi
   return 0; }

function main {
   RETVAL=0
   case "$1" in
      start)                                               # starts the Java program as a Linux service
         startService
         ;;
      stop)                                                # stops the Java program service
         stopService
         ;;
      restart)                                             # stops and restarts the service
         stopService && startService
         ;;
      status)                                              # displays the service status
         checkServiceStatus
         ;;
      *)
         echo "Usage: $0 {start|stop|restart|status}"
         exit 1
         ;;
      esac
   exit $RETVAL
}

main $1

Does anyone know what could be the problem?

EDIT: ran one of the answers posted here.

Here is what I get when I just run that command

[root@camspark Policy]# service policyserver stop
Stopping Policy Server: Stopping PolicyServer   stopped PID=683
done
[root@camspark Policy]# bash -x service policyserver start |& tee policyserver_startup.log
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ PROMPT=no
+++ AUTOSWAP=no
+++ ACTIVE_CONSOLES='/dev/tty[1-6]'
+++ SINGLE=/sbin/sushell
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
++ basename service
+ VERSION='service ver. 0.91'
++ basename service
+ USAGE='Usage: service < option > | --status-all | [ service_name [ command | --full-restart ] ]'
+ SERVICE=
+ SERVICEDIR=/etc/init.d
+ OPTIONS=
+ '[' 2 -eq 0 ']'
+ cd /
+ '[' 2 -gt 0 ']'
+ case "${1}" in
+ '[' -z '' -a 2 -eq 1 -a policyserver = --status-all ']'
+ '[' 2 -eq 2 -a start = --full-restart ']'
+ '[' -z '' ']'
+ SERVICE=policyserver
+ shift
+ '[' 1 -gt 0 ']'
+ case "${1}" in
+ '[' -z policyserver -a 1 -eq 1 -a start = --status-all ']'
+ '[' 1 -eq 2 -a '' = --full-restart ']'
+ '[' -z policyserver ']'
+ OPTIONS=' start'
+ shift
+ '[' 0 -gt 0 ']'
+ '[' -f /etc/init.d/policyserver ']'
+ env -i PATH=/sbin:/usr/sbin:/bin:/usr/bin TERM=xterm /etc/init.d/policyserver start
Starting Policy Server: done
[root@camspark Policy]#

Here is what is inside that log file

[root@camspark Policy]# head policyserver_startup.log
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
[root@camspark Policy]#

Best Answer

You can debug an issue like this by running the start script in a more verbose manner.

$ bash -x service policyserver start

You might want to capture this output to a log file too.

$ bash -x service policyserver start |& tee policyserver_startup.log

You can then go through the log file and see which command is failing.

Of course it does.

If they weren't started by the service management subsystem, they won't be tracked by the service management subsystem. And they won't be proper dæmons, really.

background on van Smoorenburg `rc` compatibility mechanisms

further reading: https://unix.stackexchange.com/a/233581/5132

The van Smoorenburg rc compatibility mechanism provided by systemd is a generator. It ensures that there is a generated abc.service service that runs /etc/init.d/abc start at service start and /etc/init.d/abc stop at service stop.

Note, at this point, that the existence of /usr/lib/systemd/system/abc.service completely prevents the generation of an abc.service by systemd's generator.

This is the entire extent of van Smoorenburg rc compatibility in vanilla systemd. The ability for the superuser to directly invoke /etc/init.d/abc in various ways, and have that connect up to systemd is provided as an augmentation to vanilla systemd by the operating system's individual developers.

The Debian and Ubuntu people, for example, provide a hook in their own /lib/lsb/init-functions.d/ subsystem that acts as follows:

If the hook detects that the init.d script is being invoked as the ExecStart/ExecStop of a generated systemd service, it does nothing special and just runs the rest of the script as it stands.
If the hook detects that the init.d script is being invoked directly, not as part of a generated systemd service, it transforms /etc/init.d/name verb into systemctl verb name and switches to that, not running the rest of the script. (Some verbs it passes through, but you are talking of start and stop, which are handled in the way explained here.)

in the absence of the compatibility mechanisms

Not all operating systems have such a van Smoorenburg rc compatibility mechanism that translates the direct invocation of /etc/init.d/name verb into systemd's way of doing things. Some operating systems (such as the one run by the person at "Why does `init 0` result in "Excess Arguments" on Arch install?", for example) provide no van Smoorenburg rc compatibility at all, not providing a hook like Debian's/Ubuntu's and even outright disabling the compatibility mechanism that comes with vanilla systemd.

On such an operating system, running the van Smoorenburg rc script directly just runs that script as-is.

Such a script does not start a service under service management. It does things such as double-forking and whatnot, in vain and in most cases fruitless attempts to run in the same environment that actual service dæmons run in. (Much of this so-called "dæmonization" stuff does not work and has not worked since the 1980s; this being the reason that proper dæmon management systems were invented in the early 1990s in the first place.) But as far as service management is concerned it is just the superuser in an interactive login session forking stuff.

Indeed, systemd will consider such directly invoked van Smoorenburg rc scripts and all of the vainly "dæmonized" programs that they fork off into the background to be running as part of the user's interactive session scope inside the user's slice, not as services that run outwith user sessions in the system slice.

Worse, it ends up using the highly flawed mechanisms of the van Smoorenburg rc system, such as killing all processes running anywhere that match the service name at service stop, instead of just the specific service processes that the service manager started and is tracking. This is why /etc/init.d/name stop is appearing to work for you. The script is killing all processes that match a name, which just happens to also include the processes that run under the service manager. This indiscriminate killing of everything is a bug, though, not a feature. This is only the appearance of proper functionality, and it will bite you down the road, as it has bitten so many system administrators in the past several decades.

the correct thing to do

If you lack such compatibility mechanisms, then do not invoke van Smoorenburg rc scripts directly. It is as simple as that. Use the service or systemctl commands to communicate with systemd's service management; but do not run /etc/init.d/anything directly for stopping, starting, and obtaining the statuses of services.

A subordinate point is that you should not be mucking around with /usr/lib/systemd/system/abc.service just to try to bodge the van Smoorenburg rc script into sort-of working. Type=forking is almost certainly wrong for your service. (It does not match almost all actual services in the wild.) And if the people who came up with /usr/lib/systemd/system/abc.service managed to get rid of the well-known-to-be-broken PID file mechanism that is wholly unnecessary for true service management, it is outright daft to be putting it back in again.

Best Answer

Related Solutions

Debian – Migrate socat init script to systemd

Systemctl – Fix Inaccurate Daemon Status Reporting

Of course it does.

background on van Smoorenburg rc compatibility mechanisms

in the absence of the compatibility mechanisms

the correct thing to do

Further reading

Related Question

background on van Smoorenburg `rc` compatibility mechanisms