Ubuntu – General way to monitor ALL memory usages

freememory usagezfs

I have Kubuntu 16.04 and am using ZFS.

As you can see on superuser, I just had a long journey in tracking down where my memory was vanishing to.

It turned out to be the ARC of ZFS that was eating it up without showing up anywhere in the tools/files other than its own special file /proc/spl/kstat/zfs/arcstats

Is there a generic way to see how much memory is used by the kernel and if possible, by which kernel modules?

Example:

free -h was showing me this: 

              total        used        free      shared  buff/cache   available                                           
Mem:            31G         19G        1.8G        406M         10G        5.2G                                           
Swap:          8.0G          0B        8.0G

So 19GB in use. But TOP only showed me about 8GB of used memory. So I went in search of the other 11GB.
6GB turned out to be slab which can be seen in /proc/memstat and with nmon. But 5GB were still missing.

In the end I found that it had been consumed by the ARC of ZFS. But it was only listed in its own special info file. /proc/spl/kstat/zfs/arcstats

Neither /proc/memstat nor top, htop, nmon, free, or atop could tell me who is using those 5GB.

Is there a file/tool to see the memory used by such special modules, without knowing/understanding those modules? Or do I have to account for every existing module if I would want to write a monitoring tool that can tell me this?

From my perspective, memory info under linux is a total mess. Dozen of number that all don't add up to total memory. I bet free calculates the "in use" parameter by substracting free/cache/buffers from the total.

I would like to have/write a tool/info where I can see

Total memory: 32GB
  in use: 19GB
    processes: 8GB
    slab: 5GB
    mapper: 1Gb
    kernel modules: 5Gb
       kernel: 0.3GB
       spl: 4.7GB
  cache: 6GB
  buffers: 2GB
  free: 5GB
Additional stats:
  Active memory: 8.3GB
    Active(anon): 7.1GB
    Active(file): 1.2GB
  Inactive memory: 2.4GB
  ...

You know, where each subsection actually addes up to its parent section and each byte of memory is accounted for 😉

So is such a tool possible under Linux? Does it exist already?

Update 1

The tool smem as suggested below indeed shows a colum "Dynamic kernel memory" which sums up to 100% of memory and accounts for ARC.
However according to this answer

BTW: I already checked smem sources, it basically does (memtotal – userspace – free – cache).

So this is only achieved by simply substracting all known figures from /proc/memstat and labeling what is missing as "noncache Dynamic Kernel Memory" I find this disturbing and really hope the linux kernel has more to offer than that.

Best Answer

You might try smem for at least a goodly portion of what you are looking for.

It's available in the Universe repository and can be installed with sudo apt-get install smem

There are a number of useful switches that are covered on the man page man smem

Some examples are as follows: Note I'm using the -t switch for totals in all examples for brevity. you can leave it out if you want more verbose output.

PSS is the interesting column in these examples as it takes shared memory into account.
Unlike RSS it's meaningful to add it up.

System Wide

me@zippy-64bit:~$ smem -tw
Area                           Used      Cache   Noncache 
firmware/hardware                 0          0          0 
kernel image                      0          0          0 
kernel dynamic memory        989656     860076     129580 
userspace memory             936032     180020     756012 
free memory                 5692716    5692716          0 
----------------------------------------------------------
                            7618404    6732812     885592

By user

me@zippy-64bit:~$ smem -tu
User     Count     Swap      USS      PSS      RSS 
me          67        0   861452   898915  1245592 
---------------------------------------------------
            67        0   861452   898915  1245592

Also of note are smemcap for capture of data for future analysis with smem and smemstat which reports the physical memory usage taking into consideration shared memory. The tool can either report a current snapshot of memory usage or periodically dump out any changes in memory.

There's also a list at the bottom of the page of similar packages.

Sources:

Alternative 1: You might also consider parsing the output of cat /proc/meminfo or if you want it by the page less /proc/meminfo

Much of the information here is used by free, top and ps commands. In fact, the output of the free command is similar in appearance to the contents and structure of /proc/meminfo. But by looking directly at /proc/meminfo, many more details are revealed including : 

$ cat /proc/meminfo | grep "Slab"
Slab:             267680 kB

Alternative Source: https://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-proc-topfiles.html

Alternative 2: If you are truly committed to a better parsing of kernel RAM you might roll your own kstat. While kstat appears to be a Solaris only command. the link above will take you to sources for for an open source attempt at kstat that was last modified in 2013 (as of this edit.) A clever C programmer might be able to modify the code for their specific purpose.

Alternative 2 source:

Related Question