Ubuntu – “respawning too fast, stopped”. How to make UPSTART disable for 5 mins

job-controlupstart

In the old days, the init daemon used to disable tasks that were respawning too fast.

I discovered to my peril that UPSTART just stops them. forever.

I have found the COUNT and INTERVAL controls for when to stop a job that is respawning fast, but I cannot find how to restart that job after a specified waiting time.

This is for a remote server that must connect via ssh in order for me to tunnel back into it. But it regularly has connection problems and the job that runs the ssh connection is dying! I foolishly expected upstart to keep the job running.

Is there a way, or do I need to do something like run a cron job to check if it is running and if not restart it?

Best Answer

Upstart allows a post-stop sleep in order to delay the restarting of a job. From http://upstart.ubuntu.com/cookbook/#delay-respawn-of-a-job

post-stop exec sleep 5

Now when the job fails, it will wait 5 seconds before attempting to connect again. Combine that with setting the respawn to something like 10 times in 30 seconds and it should try to respawn forever (it would only try 7 times in 30 seconds, so it shouldn't get kicked out of respawn)

Related Solutions

Ubuntu – Upstart kill timeout not respected

That's not what kill timeout does; it specifies how long to wait after sending a polite "terminate" signal for the process to exit before giving up and launching nukes.

If you want to set a process time limit in the upstart way, the easiest way is probably to have something periodically issue a timer event, and use stop on timer. Alternately (and perhaps more generally), create a wrapper program which forks a small process which sends an event after a time period, and stop on that event.

For more information and best practices, see The Upstart Cookbook.

Ubuntu – Why isn’t the upstart service starting on system boot

I would recommend increasing the verbosity of the job, e.g. by using pre-start/post-start entries.

pre-start script
  logger "pre-start for myprog"
end script

post-start script
  logger "post-start for myprog"
end script

# and for PMS itself:
script
  logger "just before executing PMS"
  exec /home/administrator/pms-current/PMS.sh
end script

Further information at http://upstart.ubuntu.com/cookbook/

Also have a look at http://upstart.ubuntu.com/wiki/Debugging

Best Answer

Related Solutions

Ubuntu – Upstart kill timeout not respected

Ubuntu – Why isn’t the upstart service starting on system boot

Related Question