I just discovered that my sistem doesn't have any swap space
If this is true, then it's probably contributing to the problem.
By using multiple VMs with large memory allocations, an amount of RAM that would normally be more than enough for normal use (8GB) starts becoming tight, which means your system won't be able to use as much cache, and will want to swap. Now, if you don't have swap, there will be even more pressure on memory and you may even get tasks killed due to running out of memory.
Swap tries to improve overall stability and performance in situations like the one you have described, by re-allocating some allocated but seldom-used data in memory to disk, freeing memory for data you are using. Adding swap, if you indeed have none, will help. However, in performance terms you'll make much more improvement if you also buy more RAM, or use less of it.
On a system where you're running VMs but you're also using a regular desktop on the VM host, you don't want your VMs to take up all your RAM, in fact I'd save half your RAM (up to 4GB) for the host, given you're running software on the host and want it to be responsive.
Note too that VMs have inherent inefficiencies not just because their RAM is totally separate they don't share any cache or buffers, but they are running a whole separate OS under a number of layers of abstraction. So you can always get better efficiency if you decide to run something on the host rather than in a VM - though if your purpose for using VMs is to use other operating systems this won't be possible.
Is there a way to ensure that the UI remains responsive when my system is thrashing?
When the amount of RAM you have is a problem, anything you can do is only a hack - the real solution is to buy more RAM (although in your case as I mentioned you may be able to just allocate less to the VMs).
Your question is put in simple terms - you want the UI to be able to continue to operate smoothly even if IO is congested. However, it's complicated to answer. Your system will already be doing this to the best of its ability: operations that don't require disk access won't be being delayed. If your "UI is stuttering" it's because the particular process responsible for that UI needs disk access. In a low memory situation and where the disk is thrashing, even something which will normally be trivial - such as reading some config from your home directory to find out what your display preferences are - can be slow due to depletion of disk cache.
I could imagine something like a "nice" for memory (so that I could nice Chrome and my VM)
There isn't really a "nice for memory". Memory access is so fast that you can usually practically ignore it. When you have thrashing, it's not memory, it's disk, that's the problem. There is an "ionice" because of this. Though that's probably not what I'd recommend for your issue.
Another rough equivalent of "nice for memory" is swap - you can't control how swappy each process is individually but the system does a pretty good job of working out which processes you are actively using and therefore are a higher priority to be kept in RAM.
But as I said and as is a golden rule, if you're running out of RAM, then buying more RAM is the solution with the least headaches and drawbacks.
Best Answer
You're experiencing thrashing due to exhaustion of physical memory, so the system is constantly swapping memory pages in and out of disk.
http://en.wikipedia.org/wiki/Thrashing_%28computer_science%29
Three possible solutions:
As to how to do a proper shutdown, you can try the REISUB sequence (http://kember.net/articles/reisub-the-gentle-linux-restart/). Failing that, you'll probably just have to risk power-cycling the system.