Linux – How to monitor memory usage for alarming purpose

linuxmemory

We have embedded Linux system without swap.

Currenly we must raise alarm when memory usage % increases over a threashold.
And reboot when memory usage % increases over a (higher) threshold.

Why we want to do that: If some program leaks, we can do safety reboot, before kernel start killing our processes (which may lead to data corruption or unavailability).

But we have a problem:

How to count memory usage-% which can be used for our purpose?

We tried to count memory usage by using values of /proc/meminfo:

/ # cat /proc/meminfo
MemTotal:       126744 kB
MemFree:         58256 kB
Buffers:         16740 kB
Cached:          31308 kB
SwapCached:          0 kB
Active:          37580 kB
Inactive:        24000 kB

Without success:

(MemTotal - MemFree) is not usable, because it contains for example caches.

(MemTotal - MemFree - Buffers - Cached) did ignore effect of Inactive. So it also gives too big memory usage values.

(MemTotal - MemFree - Buffers - Cached - Inactive) is unusable, because result can be negative.

Best Answer

Monitor system via free

[root@localhost ~]# free
          total       used       free     shared    buffers     cached
Mem:    2058240    1776788     281452          0      89780    1335840
-/+ buffers/cache:  351168    1707072
Swap:   4095992        100    4095892

Look at the -/+ buffers/cache line used and free

Monitor each process via /proc

I used this python script and /proc/pid/stat to monitor the memory of a process:

http://phacker.org/2009/02/20/monitoring-virtual-memory-usage-with-python/

you would probably like to translate something like this to c.

Limit resource for each process

or use ulimit / setrlimit

https://stackoverflow.com/questions/4983120/limit-memory-usage-for-a-single-linux-process

Related Question