Turns out the simple solution sometimes is the proper one after all! It seems /private/var/vm/ should always exist (even if it is empty, ie on restart), so however it got deleted the answer was to just sudo mkdir /private/var/vm/. Seemingly non-robust coding on Apple's part, but at least it's fixed. Never mind!
In my experience, this occurs when my main system hard drive is running low on free space. Operating Systems use the hard drive for extra memory storage, called "virtual memory". (I've definitely always wished that the OS could just reserve enough space for itself, but it just can't predict how many applications we'll be running).
On top of that, it's worth noting that regular web use now requires far more memory than it did in the past. In activity monitor, you'll notice that every single tab & window (every open web page) is it's own process, taking up a significant chunk of memory. On top of that, account for all the ads, movies, flash, scripts, plugins and 360 videos etc. that we expect to run smoothly. New OS's and new web pages just use a lot of memory to provide us with the services we expect to "just work" (eg. syncing across devices, notifications, automatic updating etc. etc.).
In short, in my experience there usually isn't a single process that's suddenly taking up a huge amount of memory (although a leaky program could indeed be a culprit - Sketchup 2016 does this to me, for example). More commonly, it's the additional functionality we expect of many programs/web plugins.
I believe restarting the computer always alleviates this problem for a short while - primarily by unloading all the webpages and apps we had launched over time. But if our expectation of the computer and hardware constraints stay the same (and we run the same number of processes without changing anything else), eventually we'll run into the problem again.
Two solutions that work for me:
1) Open fewer tabs/pages and fewer programs at one time. Close some web-pages/programs before opening the hefty apps, such as MS Office, Parallels, 3D CAD, Adobe programs etc.
2) Free up more space on the system hard drive (eg. move all your music and photos to another drive), to allow the system to handle your typical virtual memory needs. For me, this means my 1TB OS drive needs >20% free space (200GB)! Your requirements may be different.
If you're on an older Apple laptop or iMac or Mini, the OWC Data-Doubler is a really fantastic way to accomplish this.
Method (1) is my temporary fix, so that when I eventually enact method (2) I will have restored the snappy performance I expect while running many heavy-duty programs simultaneously.
Best Answer
You most probably do not want to avoid macOS from suspending apps when the system runs out of RAM when you know that the only alternative is to kill the program. I.e. stop the program from running now and here without saving its data or anything like that.
However, you do not have to restart the programs that are paused as in "close the program and start it over". Instead you can actually unpause the program when you have sufficient free memory on the system. You can do this from the terminal by running:
where should be replaced with the process ID of the process, you want to unpause.
You can find process ID using Activity Monitor or the "ps" command in the Terminal (in this case, look for processes with status T as these are the paused/stopped ones).
An easy way of unpausing and finding the ID in one go is:
where myprogram is the name of the program you want to unpause.
The second part of your question where you describe the ideal scenario is very hard to accomplish with a standard program. In general programs can request memory from the operating system, but the operating system cannot request it back afterwards. It is possible, but it requires very specialized programs which is generally not the case.
This means that once macOS have granted your program for example 1 GB of RAM, the operating system cannot just take it back when other programs want more memory. If you take away memory from programs that are not specially crafted to allow that, they will effectively crash as soon as they attempt to use the lost memory.