Mongodb 3.0.3 Ubuntu 14.04.2 AWS m3.medium Upstart PID mismatch

mongodb-3.0Ubuntu

(Cross posted from stackoverflow, seems like this belongs here but 10gen link to stackoverflow explicitly from their site…)

Is it possible to run mongod via upstart and keep track of the PID via start-stop-daemon or otherwise?

After following these instructions on the mongodb docs page for ubuntu installation:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu "$(lsb_release -sc)"/mongodb-org/3.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org

All remnants of the ubuntu package have been removed prior to this.

I now have a running instance of mongod on boot, via Upstart. But for some reason Upstart and initctl do not know about it. It starts up fine, but initctl thinks it's in stop/waiting state.

To wit:

My /etc/mongod.conf.yml:

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true

systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
  logRotate: rename
  component:
    accessControl:
      verbosity: 2

net:
   bindIp: 127.0.0.1
   port: 27017

processManagement:
  fork: true

setParameter:
   enableLocalhostAuthBypass: false

security:
  authorization: disabled

My /etc/init/mongod.conf upstart script (renamed mongodb.pid to mongod.pid):

# Ubuntu upstart file at /etc/init/mongod.conf

# Recommended ulimit values for mongod or mongos
# See http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings
#
limit fsize unlimited unlimited
limit cpu unlimited unlimited
limit as unlimited unlimited
limit nofile 64000 64000
limit rss unlimited unlimited
limit nproc 32000 32000

kill timeout 300 # wait 300s between SIGTERM and SIGKILL.

pre-start script
  DAEMONUSER=${DAEMONUSER:-mongodb}
  if [ ! -d /var/lib/mongod ]; then
    mkdir -p /var/lib/mongodb && chown mongodb:mongodb /var/lib/mongodb
  fi
  if [ ! -d /var/log/mongod ]; then
    mkdir -p /var/log/mongodb && chown mongodb:mongodb /var/log/mongodb
  fi

  touch /var/run/mongod.pid

  chown $DAEMONUSER /var/run/mongod.pid;

  if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
    echo never > /sys/kernel/mm/transparent_hugepage/enabled
  fi

  if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
    echo never > /sys/kernel/mm/transparent_hugepage/defrag
  fi

end script

start on runlevel [2345]
stop on runlevel [06]

script
  ENABLE_MONGOD="yes"
  CONF=/etc/mongod.conf.yml
  DAEMON=/usr/bin/mongod
  DAEMONUSER=${DAEMONUSER:-mongodb}

  if [ -f /etc/default/mongod ]; then . /etc/default/mongod; fi

  # Handle NUMA access to CPUs (SERVER-3574)
  # This verifies the existence of numactl as well as testing that the command works
  NUMACTL_ARGS="--interleave=all"
  if which numactl >/dev/null 2>/dev/null && numactl $NUMACTL_ARGS ls / >/dev/null 2>/dev/null
  then
    NUMACTL="$(which numactl) -- $NUMACTL_ARGS"
    DAEMON_OPTS=${DAEMON_OPTS:-"--config $CONF"}
  else
    NUMACTL=""
    DAEMON_OPTS="-- "${DAEMON_OPTS:-"--config $CONF"}
  fi

  if [ "x$ENABLE_MONGOD" = "xyes" ]
  then
    exec start-stop-daemon --start \
        --chuid $DAEMONUSER \
        --pidfile /var/run/mongod.pid \
        --make-pidfile \
        --exec $NUMACTL $DAEMON $DAEMON_OPTS
  fi
end script

After a reboot I see this:

$ ps aux | grep mongo
mongodb   1085  0.2  1.1 363764 46704 ?        Sl   11:57   0:06 /usr/bin/mongod --config /etc/mongod.conf.yml

And everything appears to be fine. But the mongod.pid file does not store the same pid as the process:

$ cat /var/run/mongod.pid
985

Should be 1085.

What is the best way to fix this so Upstart has access to the actual PID?

UPDATE: tried to add expect daemon and expect fork with some change in behavior: initctl now sees a PID and denotes that mongod is running, but has the wrong PID. This means any subsequent command like sudo stop mongod or sudo start mongod will hang. Neither fork or daemon seems to fix this; what am I missing?

UPDATE 2: I ran strace as prescribed in the Upstart Cookbook to find out the number of forks mongod makes and got 12. Is that right? Would make sense that start-stop-daemon is required if that's accurate, but why does it still not track the right PID?

Maybe forego start-stop-daemon and use --fork on mongod itself? Will start messing with pre-stop and post-stop upstart stanzas as well, maybe grep for the mongod --config /etc/mongod.conf.yml line or something to find the process to halt?

Best Answer

Ok so a bit of egg on my face - I was overlooking the fact that my shiny new /etc/mongod.conf.yml contained processManagement.fork: true. Setting this to false allows start-stop-daemon to capture the appropriate PID.

Related Question