Ubuntu – Very high Xorg memory usage with Ubuntu 18.04

According to df, your /dev filesystem is full. This is a tmpfs, and so is stored in shared memory. Normally it should only contain a very small amount of data, mainly device nodes that provide a file interface to kernel drivers, so something is incorrectly filling it up. You can dig around with du to see what files are taking up all of the space.

Based on your comments, you say cache usage doesn't noticeably drop when you try to echo 3 > /proc/sys/vm/drop_caches

This can only happen if this is a cache for writing. If you write 5 GB to some files, the data immediately lands in cache and your program continues. The cache is actually written to storage in the background as fast as possible. In your case the storage seems dramatically slow and you accumulate the unwritten cache until it drains all of your RAM and starts pushing everything out to swap.

Kernel will never write cache to swap partition. It keeps it in RAM until it is safely written to destination.

Kernel will never drop unwritten cache, because it would be a data loss (you've saved a file, so you expect the data to land on the permanent storage).

You can only solve it by speeding up the storage. This issue is often seen on storage mounted via network (check your mount for types cifs, nfs, sshfs, etc.) or slow USB1 devices.

You could also make issue much less dramatic to the system by capping the dirty cache with sysctl vm.dirty_ratio=10 before it grows too much.

dirty_ratio

Contains, as a percentage of total available memory that contains free pages and reclaimable pages, the number of pages at which a process which is generating disk writes will itself start writing out dirty data.

The total available memory is not equal to total system memory.

If that's a correct diagnosis, you will see that cache can be easily dropped (at least 90% of it) and that the process that writes these gigabytes becomes very slow. The rest of system will become more responsive.