I just discovered that my sistem doesn't have any swap space
If this is true, then it's probably contributing to the problem.
By using multiple VMs with large memory allocations, an amount of RAM that would normally be more than enough for normal use (8GB) starts becoming tight, which means your system won't be able to use as much cache, and will want to swap. Now, if you don't have swap, there will be even more pressure on memory and you may even get tasks killed due to running out of memory.
Swap tries to improve overall stability and performance in situations like the one you have described, by re-allocating some allocated but seldom-used data in memory to disk, freeing memory for data you are using. Adding swap, if you indeed have none, will help. However, in performance terms you'll make much more improvement if you also buy more RAM, or use less of it.
On a system where you're running VMs but you're also using a regular desktop on the VM host, you don't want your VMs to take up all your RAM, in fact I'd save half your RAM (up to 4GB) for the host, given you're running software on the host and want it to be responsive.
Note too that VMs have inherent inefficiencies not just because their RAM is totally separate they don't share any cache or buffers, but they are running a whole separate OS under a number of layers of abstraction. So you can always get better efficiency if you decide to run something on the host rather than in a VM - though if your purpose for using VMs is to use other operating systems this won't be possible.
Is there a way to ensure that the UI remains responsive when my system is thrashing?
When the amount of RAM you have is a problem, anything you can do is only a hack - the real solution is to buy more RAM (although in your case as I mentioned you may be able to just allocate less to the VMs).
Your question is put in simple terms - you want the UI to be able to continue to operate smoothly even if IO is congested. However, it's complicated to answer. Your system will already be doing this to the best of its ability: operations that don't require disk access won't be being delayed. If your "UI is stuttering" it's because the particular process responsible for that UI needs disk access. In a low memory situation and where the disk is thrashing, even something which will normally be trivial - such as reading some config from your home directory to find out what your display preferences are - can be slow due to depletion of disk cache.
I could imagine something like a "nice" for memory (so that I could nice Chrome and my VM)
There isn't really a "nice for memory". Memory access is so fast that you can usually practically ignore it. When you have thrashing, it's not memory, it's disk, that's the problem. There is an "ionice" because of this. Though that's probably not what I'd recommend for your issue.
Another rough equivalent of "nice for memory" is swap - you can't control how swappy each process is individually but the system does a pretty good job of working out which processes you are actively using and therefore are a higher priority to be kept in RAM.
But as I said and as is a golden rule, if you're running out of RAM, then buying more RAM is the solution with the least headaches and drawbacks.
Be conscious that some system monitors don't factor in cache and buffering. The is where the system transparently uses RAM to store things from disk, to make their second access faster. The cache is automagically reduced in size if you need more RAM. See my free output:
total used free shared buffers cached
Mem: 24110 19949 4161 366 900 14299
-/+ buffers/cache: 4749 19361
Swap: 0 0 0
At a glance it might look like I'm using almost 20GB of RAM, but 14GB of that is cache. It's the second line of values that show the actual system usage 4.5 used and 19.5 free.
Best Answer
Probably your problem is caused by the system "thrashing" – moving many pages of memory to and from swap space at once, and not leaving time for real processes to execute.
If you want processes that use too much memory to be killed rather than making your whole system run slowly, you can disable swap. Running
sudo swapoff -awill achieve this until a reboot; to disable swap permanently, you need to edit/etc/fstabto remove the swapfile/swap partition, by removing or commenting out the line withswapin the third column. For example, my/etc/fstablooks a bit like this:To disable swap, I would comment out the last line (by putting a hash
#at the beginning of the line), then reboot. Make sure you don't modify any of the rest of the file, otherwise you might make your system unable to boot.Caveat: If you disable swap and don't have enough physical memory for basic system services, the OOM killer may decide to kill one of those and lead the system to crash or become otherwise unusable (which the question says you want to avoid).