I have a cronjob that runs php5 wp-cron.php every minute to update my website.
However something happened and i had 30+ instance of it (31 is marked on this one dump of ps aux). It ate up my ram, caused additional instance to terminate do to lack of memory and caused me not to be able to ssh into the box.
I can't understand why instances were living >30mins, one usually takes a few seconds. The day it happened i had no jobs planned (although maybe wp cache usesd it? but i never had a problem before)
What can i do to prevent a cronjob from spamming and destroying my memory? Is there a way i can say do not start if an instance is alive? and if an instance is alive for more then 5mins kill it?
Is there a way i can protect myself from something similar from happening?
Best Answer
For the goal to prevent multiple copies from running, use flock (Linux), lockf (FreeBSD), shlock (provided with some systems, less reliable). This won't limit execution time but ensures only one process is running. Then, if it hangs, you can analyze its state on the fly.
You can limit CPU time of the spawned process using ulimit shell builtin.
To limit wall time, you can write script which waits for a process termination and kills it after timeout. It's easier in Python/Perl/etc. but shell also allows this (using trap and/on background children).
Sometimes it's useful to provide fixed time between invocations (i.e. from end of previous one to start of next one) instead of invocation starts, as cron does. Usual kinds of cron doesn't allow this, you should run special script.