Linux – Determine Cause of Increasing Disk Space Usage

I have just done a fresh install of Raspian Buster Lite on a Raspberry Pi 3B, and installed Prometheus along with Node Exporter to just scrape a few simple diagnostics – particularly the available disk space.

The problem is that the available disk space appears to be continuously decreasing. I can check the bytes availble by using df -B1, which returns the following (13.08 GB from the "16 GB" SD card used):

pi@raspberrypi:/home $ df -B1

Filesystem       1B-blocks       Used   Available Use% Mounted on
/dev/root      15383740416 1647845376 13078351872  12% /
devtmpfs         480808960          0   480808960   0% /dev
tmpfs            485527552          0   485527552   0% /dev/shm
tmpfs            485527552    6459392   479068160   2% /run
tmpfs              5242880       4096     5238784   1% /run/lock
tmpfs            485527552          0   485527552   0% /sys/fs/cgroup
/dev/mmcblk0p1   264289280   54748672   209540608  21% /boot
tmpfs             97103872          0    97103872   0% /run/user/1000

If I examine this periodically, it just reduces. Checking it a few minutes later shows the following:

Filesystem       1B-blocks       Used   Available Use% Mounted on
/dev/root      15383740416 1648463872 13077733376  12% /
devtmpfs         480808960          0   480808960   0% /dev
tmpfs            485527552          0   485527552   0% /dev/shm
tmpfs            485527552    6459392   479068160   2% /run
tmpfs              5242880       4096     5238784   1% /run/lock
tmpfs            485527552          0   485527552   0% /sys/fs/cgroup
/dev/mmcblk0p1   264289280   54748672   209540608  21% /boot
tmpfs             97103872          0    97103872   0% /run/user/1000

I can examine this metric in the Prometheus database using Grafana:

It may well just be some logs or something, but I want to understand it a little better. Based on this graph, it seems the space is reducing at a rate of around 2.9 MB per hour, which seems like a lot to me.

Ideally I suppose I am looking for a way to view only files that have changed in size, let's say in the last minute or something. What would be a good way to track down the source of this? Thanks for your patience – I am brand new with linux.

Edit

In case anyone is interested, I discovered that almost all of the 2.9 MB/hour can be attributed to the write ahead logs (WAL) in Prometheus – you can read about how it stores data here. I was able to observe the WAL directory building up by using the command

watch -d "sudo du -s -B1 /home/pi/prometheus/data/*"

which highlights nicely the changing files:

As Richie Frame hinted at, every so often (I believe it is supposed to be every 2 hours in Prometheus by default, but I need to check that) the data is compacted from the WAL and is put into more permenant storage, which is more efficient in terms of required disk space. You can see on Grafana the compaction happening (this was taken overnight whilst the Pi was otherwise just sitting there idle), and some of the disk space being released:

I am not sure why the disk space seems to jump by a large amount only every 4 hours, instead of the expected 2 hours, but that is a task for another day. Thanks for everyone for your help!