Disk space and inodes appear to consume RAM in a Docker container

containerdockerfilesystemsmemoryoverlayfs

I'm trying to determine why a web server running in a Dockerized environment is consuming more memory than I expect it to. While investigating this problem, I discovered the following behavior which doesn't make sense to me:

Create a Docker container, docker run -d --name test ubuntu:22.04 tail -f /dev/null
Check docker stats, memory usage is reported as 400KiB / 62.68GiB
Check docker top test, only one process running (tail -f /dev/null)
Now create a large number of files and directories in the running container, for example docker exec -it test sh -c 'apt update && apt install -y git && git clone https://git.savannah.gnu.org/git/emacs.git'
Check docker stats again, memory usage is now reported as 494.9MiB / 62.68GiB
Check docker top test, verify that still only one process is running (tail -f /dev/null)

So what is using this almost 500MB of memory in my Docker container? I was under the impression that memory was consumed by processes, yet there is only one process in the container and it cannot be responsible.

Is the memory usage from the filesystem? The host system is running btrfs. I use the default storage driver for Docker, overlay2, and here is information on the overlay filesystem:

% docker inspect test | jq '.[].GraphDriver.Data.MergedDir' -r
/var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/merged

% mount | grep overlay
overlay on /var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/JADZ5UYJDWI5EDAAGALCRWUM3B:/var/lib/docker/overlay2/l/JK3H4JAYNUAUSN2W3V2DQ6TB6C,upperdir=/var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/diff,workdir=/var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/work)

% sudo ls -lAd /var/lib/docker/overlay2/l/JADZ5UYJDWI5EDAAGALCRWUM3B/emacs                                           
ls: cannot access '/var/lib/docker/overlay2/l/JADZ5UYJDWI5EDAAGALCRWUM3B/emacs': No such file or directory

% sudo ls -lAd /var/lib/docker/overlay2/l/JK3H4JAYNUAUSN2W3V2DQ6TB6C/emacs
ls: cannot access '/var/lib/docker/overlay2/l/JK3H4JAYNUAUSN2W3V2DQ6TB6C/emacs': No such file or directory

% sudo ls -lAd /var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/diff/emacs
drwxr-xr-x 1 root root 602 Feb 25 16:08 /var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/diff/emacs

% sudo ls -lAd /var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/merged/emacs
drwxr-xr-x 1 root root 602 Feb 25 16:08 /var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/merged/emacs

Since my base filesystem is btrfs, I would expect that the cloned Git repository would be residing on disk in btrfs (under /var/lib/docker/overlay2/559d41c89d074c45a1ae89109ebec145e8fd4d929151819f08aaae26f73f7bda/diff), which is not within any special volume mount, and thus would not be consuming memory.

If I remove the added files and directories, e.g. with docker exec test rm -rf emacs, then the memory usage drops instantly to 68.16 MiB / 62.68GiB.

Questions:

What is consuming this memory? What misunderstanding do I have about Docker containers or the overlay2 filesystem that leads me to assume adding files and directories to the working layer should not consume memory?
Is there a way I can reduce the amount of memory that is consumed in a Docker container from sources other than process RAM, by changing configuration either with the Docker daemon or with the containerized workload?

I reviewed existing questions on UNIX Stack Exchange under the [docker] tag and wasn't able to find any that helped my understanding. I also read the official Docker documentation on overlayfs and briefly reviewed the Linux kernel documentation for overlayfs, but didn't find any information that would explain the memory usage.

My end goal is to reduce resource consumption for an idle Dockerized workload running in production, since I am billed based on CPU and memory usage. Based on production metrics I observe that my workloads do not release as much memory as they should when they become idle, even though the processes that were using that memory have been terminated.

Tests above were performed on Pop!_OS 22.04 with Docker CE 23.0.1 and Linux kernel 6.0.12 using btrfs.

System parameters for my btrfs

My /etc/fstab showing the mount options for the filesystem – basically, the defaults for everything:

UUID=ee43bc7e-1d9d-4300-a46a-5d2d772a1e0f / btrfs defaults,subvol=@ 0 0
UUID=ee43bc7e-1d9d-4300-a46a-5d2d772a1e0f /.snapshots btrfs defaults,subvol=@snapshots 0 0
UUID=127d5704-324b-4a50-97bf-9f5f4646ddd5 /home/raxod502 btrfs defaults,subvol=@ 0 0
UUID=127d5704-324b-4a50-97bf-9f5f4646ddd5 /home/raxod502/.snapshots btrfs defaults,subvol=@snapshots 0 0

I take automated snapshots using snapper, about 100 in total kept by my retention policy:

% sudo snapper -c system list | wc -l
41
% sudo snapper -c home list | wc -l  
44

I have never explicitly used reflinks, nor software that to my knowledge creates them automatically. The most advanced btrfs usage I have is snapper, as mentioned above.

Here is the output of sudo slaptop -o: https://gist.github.com/raxod502/843a6580dc6fa0f9949d2f4dc03b5c23

Happy to provide any further system configuration details that may be helpful.

Comparison for ext4

I ran the same test on ext4 on another Linux system, got roughly the same results. Memory usage before clone was 328KiB / 969.4MiB. Memory usage after clone was 233.5MiB / 969.4MiB. Memory usage after deleting cloned repository was 16.84MiB / 969.4MiB.

Best Answer

This doesn't seem to be filesystem related but a file cache memory thing. To reproduce this exact case I first cloned the Emacs repo to the /usr/src/emacs.

$ docker run --mount type=bind,source=/usr/src/emacs/.git,destination=/mnt/emacs.git,ro -d --rm --name test debian tail -f /dev/null
$ docker exec -it test sh -c 'apt update && apt install -y git && git clone /mnt/emacs.git'
$ docker stats --no-stream test
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
c468a203b9d2   test      0.00%     46.91MiB / 15.52GiB   0.30%     30.5MB / 667kB   0B / 786MB   1

In my case I didn't get ~500MiB memory usage, I'm guessing this has something to do with the filesystem, but the point is the resulting memory usage is about the same as when cloning from the network. Next I deleted the resulting /emacs directory and the memory usage dropped.

$ docker exec -it test rm -rf /emacs
$ docker stats --no-stream test
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O    PIDS
c468a203b9d2   test      0.00%     15.84MiB / 15.52GiB   0.10%     30.5MB / 667kB   0B / 786MB   1

To test whether this was filesystem related I skipped the cloning part and tried mving the emacs directory from another directory located in the same mount. Assume /usr/src/emacs and the overlay upper directory of the container are in the same block device mount.

$ docker run -d --rm --name test debian tail -f /dev/null
$ TEST_UPPER=$(docker container inspect --format '{{.GraphDriver.Data.UpperDir}}' test)
$ mv -v /usr/src/emacs "$TEST_UPPER/emacs"
renamed '/usr/src/emacs' -> '/var/lib/docker/overlay2/dc228a74029510c61ce454549248132bec806686c4da9eac8909d89595ff5a32/diff/emacs'
$ docker stats --no-stream test
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
e700998acb86   test      0.00%     400KiB / 15.52GiB   0.00%     3.79kB / 866B   0B / 0B     1

Again, but docker cp-ing to a container directory in the root:

$ docker run -d --rm --name test debian tail -f /dev/null
$ docker exec -it test mkdir /emacs
$ tar -C /usr/src/emacs -c . | docker cp - test:/emacs
$ docker exec -i ls -la /emacs | wc -l
42
$ docker stats --no-stream test
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O         BLOCK I/O   PIDS
4ea520193998   test      0.00%     444KiB / 15.52GiB   0.00%     3.79kB / 866B   0B / 0B     1

According to this, it seems that all that memory usage is cache memory used by apt and git. The recommendation is simple, don't modify your container during runtime.

Related Solutions

Do docker container cause slow Disk/OS performance

I'll answer this from the theoretical standpoint, I don't have the experience with Kafka on Docker.

Docker is largely an isolation technology NOT a virtual machine. This means that it is much lighter weight than you might expect. A large portion of it is built around namespaces and mounts including bind mounts.

It will depend a little on what you are asking Docker to do:

If you use a bind mount or a Docker volume then these are stored directly as files on the host system. Their performance overhead should be no more than that of a Linux bind mount because that's exactly what you get. This performance overhead is near zero.
Other storage volumes such as those backed by amazon S3 can come with an overhead.

In short the result on disk should generally be very similar to that of running on the host system. Docker just creates a neat sandbox and calls it a container.

Remaining disk space on docker overlay filesystem

/dev/mmcblk0p2 is the root filesystem where your Linux distribution is installed.

Where 2.2GB is in use and 55GB is available.

/var/lib/docker/overlay2 directory is the location where Docker stores its images and containers.

Docker uses a copy-on-write file system for the storage, which creates a new layer on top of the existing file system.

This is the Docker overlay filesystem and the location you see with df -h

overlay          58G  2.2G   55G   4% /var/lib/docker/overlay2/b28da5a318945ac7ae1d17d26a635edb9a662c6116dea37fb4f6c13e1c76d7d2/merged

You have 55GB of storage available on your SD card to store files, install packages, and use Docker, etc..

You can change this path, to another directory, storage or disk if you want.

Docker storage drivers

Use the OverlayFS storage driver

The legacy overlay driver was used for kernels that did not support the “multiple-lowerdir” feature required for overlay2 All currently supported Linux distributions now provide support for this, and it is therefore deprecated.

Overlay in Docker

Docker uses the overlay filesystem to create images as well as to position the container layer on top of the image layers. When an image is downloaded, its layers are located inside the /var/lib/docker/overlay2 folder.

The merged folders are overlay filesystems, they take no disk space themselves, instead df reports the disk usage of the underlying filesysem, which in your case is /

/var/lib/docker/overlay2/*/merged take too much space than it should be