Different kernels report different amounts of total memory on the same machine

configurationkernelmeminfo

I have two x86_64 kernels compiled on the same machine against the same code (4.15.0 in Linus' source tree).

The config files were produced by running make localmodconfig against that source, using different, larger original config files coming from different distros: Arch and Slackware respectively. I'll nickname them

arch config

and

slk config

for that reason.

The issue: running cat /proc/meminfo consistently reports about 55-60 MB more in the MemTotal field for arch than for slk:

MemTotal: 32600808 kB for arch

MemTotal: 32544992 kB for slk

I say 'consistently' because I've tried the experiment with the same config files against earlier versions of the source (a bunch of the 4.15-rc kernels, 4.14 before that, etc., rolling over from one source to the next with make oldconfig).

This is reflected in the figures reported by htop, with slk reporting ~60MB less usage on bootup than arch. This is consistent with the htop dev's explanation of how htop's used memory figures are based on MemTotal.

My question is: any suggestions for which config options I should look at that would make the difference?

I of course don't mind the 60MB (the machine the kernels run on has 32 GB..), but it's an interesting puzzle to me and I'd like to use it as a learning opportunity.

Memory reporting on Linux is discussed heavily on these forums and outside in general, but searching for this specific type of issue (different kernels / same machine => different outcome in memory reporting) has not produced anything I found relevant.

Edit

As per the suggestions in the post linked by @ErikF, I had a look at the output of

journalctl --boot=#

where # stands for 0 or -1, for the current and previous boots respectively (corresponding to the two kernels). These lines do seem to reflect the difference, so it is now a little clearer to me where it stems from:

arch (the one reporting larger MemTotal):

Memory: 32587752K/33472072K available (10252K kernel code, 1157K rwdata, 2760K rodata, 1364K init, 988K bss, 884320K reserved, 0K cma-reserved)

slk (the one reporting smaller MemTotal):

Memory: 32533996K/33472072K available (14348K kernel code, 1674K rwdata, 3976K rodata, 1616K init, 784K bss, 938076K reserved, 0K cma-reserved)

That's a difference of ~55 MB, as expected!

I know the slk kernel is larger, as verified by comparing the sizes of the two vmlinuz files in my /boot/ folder, but the brunt of the difference seems to come from how much memory the two respective kernels reserve.

I'd like to better understand what in the config files affects that to the extent that it does, but this certainly sheds some light.

Second edit

Answering the questions in the comment by @Tim Kennedy.

Do you have a dedicated GPU, or use shared video memory

No dedicated GPU; it's a laptop with on-board Intel graphics.

and do both kernels load the same graphics driver?

Yes, i915.

Also, compare the output of dmesg | grep BIOS-e820 | grep reserved

As you expected, does not change. In all cases it's 12 lines, identical in every respect (memory addresses and all).

(Final?) edit

I believe it may just be as simple as this: the kernel reporting less MemTotal has much more of the driver suite built in; I just hadn't realized it would make such a noticeable difference..

I compared:

du -sh /lib/modules/<smaller-kernel>/modules.builtin

returns 4K, while

du -sh /lib/modules/<larger-kernel>/modules.builtin

returns 16K.

So in the end, I believe I was barking up the wrong tree: it won't be a single config option (or a handful), but rather the accumulated effect of many more built-in drivers.

Best Answer

I believe the picture is as in my last edit above; I just did not know enough about the process of reserving memory to realize when I asked the question: it all boiled down to the larger kernel having more of the drivers built in rather than modularized.

Incidentally, I did want to put this to some (remotely) practical use: I wanted to have as slim a kernel as I could, but still boot it without an initrd. I knew that

the smaller kernel (moniker arch above) is very lightweight and fast to boot, but not without an initramfs;
the larger kernel (slk) will boot without an initramfs, but takes longer.

So I made a compromise that I think I'll stick with for now. I took the two config files, call them large and small, and made sure that every unset config option in small is reflected in large.

First,

awk '/^# CO/ {print} small > staging'

grabs all lines in the small config file that are of the form

# CONFIG_BLAH is not set

and dumps them in a staging text file. Then,

for i in $(awk '{print $2}' staging) ; do sed -i "s/^$i=./# $i is not set/" large ; done

takes all lines from the config file large that set (via =y or =m) an option contained in staging and unsets them.

I then ran make oldconfig on the resulting file large in the kernel source directory and compiled. It all went through all right:

the new kernel boots some 3 seconds faster
without an initramfs
is much smaller in the sense that its /lib/modules/<kernel>/modules.builtin got cut down in half, from 16K to 8K.

Those are not much to write home about, but as I said this was puzzling me.. I think I'm all set on the issue now.

Presumably the more straightforward thing to do would have been to figure out once and for all precisely which drivers I need on this machine in order to boot without an initramfs, but I'll leave that for some other day. Besides, playing off one config against the other was a fun exercise in its own right.

Related Solutions

Linux – Compare two kernel config files from the menuconfig perspective

Try using scripts/diffconfig from the Linux kernel source tree.

Usage:

diffconfig config-a config-b

Shows a list of lines removed, modified and added to the config-b file with respect to config-a.

Linux – How to Get Available Memory Across Distributions

MemAvailable is included in /proc/meminfo since version 3.14 of the kernel; it was added by commit 34e431b0a. That's the determining factor in the output variations you show. The commit message indicates how to estimate available memory without MemAvailable:

Currently, the amount of memory that is available for a new workload, without pushing the system into swap, can be estimated from MemFree, Active(file), Inactive(file), and SReclaimable, as well as the "low" watermarks from /proc/zoneinfo.

The low watermarks are the level beneath which the system will swap. So in the absence of MemAvailable you can at least add up the values given for MemFree, Active(file), Inactive(file) and SReclaimable (whichever are present in /proc/meminfo), and subtract the low watermarks from /proc/zoneinfo. The latter also lists the number of free pages per zone, that might be useful as a comparison...

The complete algorithm is given in the patch to meminfo.c and seems reasonably easy to adapt:

sum the low watermarks across all zones;
take the identified free memory (MemFree);
subtract the low watermark (we need to avoid touching that to avoid swapping);
add the amount of memory we can use from the page cache (sum of Active(file) and Inactive(file)): that's the amount of memory used by the page cache, minus either half the page cache, or the low watermark, whichever is smaller;
add the amount of memory we can reclaim (SReclaimable), following the same algorithm.

So, putting all this together, you can get the memory available for a new process with:

awk -v low=$(grep low /proc/zoneinfo | awk '{k+=$2}END{print k}') \
 '{a[$1]=$2}
  END{ 
   print a["MemFree:"]+a["Active(file):"]+a["Inactive(file):"]+a["SReclaimable:"]-(12*low); 
  }' /proc/meminfo

Best Answer

Related Solutions

Linux – Compare two kernel config files from the menuconfig perspective

Linux – How to Get Available Memory Across Distributions

Related Question