Shell – Prevent a script exhausing system resources and crashing entire system

killprocessresourcesSecurityshell-script

I was thinking if there is "canonical" way to have this?

Background & description

I have to install some program on live server. Although I do trust the vendor (FOSS, Github, multiple authors…) I would rather to ensure avoiding not entirely impossible scenario of script falling in some trouble and exhausting system resources and leaving server unresponsive. I had the case of installing amavis which was started right after and because of some messy configuration it have produced loadavg of >4 and system barely responsive.

My first taught was nicenice -n 19 thatscript.sh. This may and may not help, but I was thinking that it would be best that I write and activate script that would do following:

  • run as daemon, on (example) 500ms-2s

  • check for labeled process with ps and grep

  • if labeled process(es) (or any other process) takes too much CPU (yet to be defined) – kill them with SIGKILL

My second taught was – it would not be the first time that I'm reinventing the wheel.

So, is there any good way to "jail" the program and processes produced by it into some predefined limited amount of system resources or automated kill if some threshold was reached by them?

Best Answer

Alternative #1: Monitor your process with monit

Install M/Monit and create a configuration file based on this template:

check process myprogram
matching "myprogram.*"
start program = "/usr/bin/myprogram" with timeout 10 seconds
stop program = "/usr/bin/pkill thatscript"
if cpu > 99% for 2 cycles then stop
if loadavg (5min) > 80 for 10 cycles then stop

Alternative #2: Limit process CPU usage with cgroups

The most native Linux specific solution of them all. Offers a lot of options and complexity.

Example:

sudo cgcreate -g cpu:/cpulimited
sudo cgset -r cpu.shares=512 cpulimited
sudo cgexec -g cpu:cpulimited /usr/bin/myprogram > /dev/null &

I encourage you to read more at:

DigitalOcean - How-to: Limit resources using cgroups on CentOS 6

RedHat - Resource Management Guide

Oracle - List of cgroups subsystems

Alternative #3: Limit process CPU usage with cpulimit

Get latest version of cpulimit from your package manager of choice, or by getting the source available at GitHub.

Limit the CPU usage to 90%: cpulimit -l 90 /usr/bin/myprogram > /dev/null &

Sidenote:

You can also pin a certain process to use certain CPU core(s) to ensure that you always have some free CPU power.

Related Question