This problem might be caused by an incorrect sizing of the maximum size of the connection tracking table and the hash table. The Linux kernel tries to allocate contiguous pages to track the connection tables for the iptables nf_conntrack module. As you don't have enough physical memory, conntrack fails back to vmalloc.
This table is not dynamically created based on established connections but, rather, fully allocated based on some kernel parameters.
Some additional symptoms might be finding a large number of nf_conntrack: falling back to vmalloc. messages in the /var/log/messages (or /var/log/kern.log, or in both).
This is easily solvable by just fine tuning your connection track table and sizing it down. Proper sizing has to be done based on the system usage. The connection track table needs to be high if you are running a dedicated network firewall in this system, but can be much lower if you are just using iptables to protect it from network intrusions.
For more information on connection tracking tuning please refer to https://wiki.khnet.info/index.php/Conntrack_tuning
To fine tune the values for your system, you can first evaluate the number of connections your system keeps open by running conntrack -L (or
/sbin/sysctl net.netfilter.nf_conntrack_count). Better yet, keep a statistic of tracked connections over time (munin does this nicely) and use the maximum number of tracked connections as a baseline. Based on this information you can configure /etc/sysctl.conf accordingly.
When fine tuning make sure you also review how much time do you keep connections in the tracking table. Sometimes conntrack tables contain spurious data connections due to network misconfiguration or errors. For example, when the server receives SYN connections that are never closed or when client disconnects abruptly and leave open sockets for a long time.
Second, check if your conntrack entries make sense. Sometimes conntrack tables are filled with rubbish because of some network or firewall mis-configuration. Usually those are entries for connections which were never fully established. That may happen e.g. when the server gets incoming connection SYN packets, but the server replies are always lost somewhere on the network.
When fine tuning these values running sysctl -a | grep conntrack | grep timeout might be provide some insight. The default values are quite conservative: 600 (10 minutes) for generic timeouts and 432000 (5 days) for an established TCP connection. Depending on the system purpose and network behaviour those might need to be fined tuned to reduce the number of active connections in the conntrack table. Which will help define a lower value to it.
Make sure, however, that you do not size the conntrack table down too much as you can have the opposite effect: connections being dropped by iptables because they cannot be tracked and you will start having messages such as this in your log files: 'kernel: ip_conntrack: table full, dropping packet.'
In order to confirm if that is the problem please provide the output of the following:
cat /proc/sys/net/ipv4/ip_conntrack_max
cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
Best Answer
First off, a 32 bit system has
0xffffffff(4'294'967'295) linear addresses to access a physical location ontop of the RAM.The kernel divides these addresses into user and kernel space.
User space (high memory) can be accessed by the user and, if necessary, also by the kernel.
The address range in hex and dec notation:
Kernel space (low memory) can only be accessed by the kernel.
The address range in hex and dec notation:
Like this:
Thus, the memory layout you saw in
dmesgcorresponds to the mapping of linear addresses in kernel space.First, the .text, .data and .init sequences which provide the initialization of the kernel's own page tables (translate linear into physical addresses).
The range where the kernel code resides.
The range where the kernel data segments reside.
The range where the kernel's initial page tables reside.
(and another 128 kB for some dynamic data structures.)
This minimal address space is just large enough to install the kernel in the RAM and to initialize its core data structures.
Their used size is shown inside the parenthesis, take for example the kernel code:
In decimal notation, that's
3'255'914(3179 kB).Second, the usage of kernel space after initialization
The lowmem range can be used by the kernel to directly access physical addresses.
This is not the full 1 GB, because the kernel always requires at least 128 MB of linear addresses to implement noncontiguous memory allocation and fix-mapped linear addresses.
Virtual memory allocation can allocate page frames based on a noncontiguous scheme. The main advantage of this schema is to avoid external fragmentation, this is used for swap areas, kernel modules or allocation of buffers to some I/O devices.
The permanent kernel mapping allows the kernel to establish long-lasting mappings of high-memory page frames into the kernel address space. When a HIGHMEM page is mapped using kmap(), virtual addresses are assigned from here.
These are fix-mapped linear addresses which can refer to any physical address in the RAM, not just the last 1 GB like the lowmem addresses. Fix-mapped linear addresses are a bit more efficient than their lowmem and pkmap colleagues. There are dedicated page table descriptors assigned for fixed mapping, and mappings of HIGHMEM pages using kmap_atomic are allocated from here.
If you want to dive deeper into the rabbit hole:
Understanding the Linux Kernel