When does swap start to be used? When does a process get an out of memory error and be killed

No, it’s a bad idea.

You shouldn’t think of swap as a mechanism by which you can expand memory; it’s a storage area for parts of memory which don’t have to remain in physical memory, and whose contents don’t exist anywhere else. See Why does Linux need swap space in a VM? for details.

If the processes running inside your VMs are running out of memory, you need to determine what their real working set is, both in nominal operation and in the worst case. Then, assuming you can’t reduce their memory usage, you need to configure their memory setups to suit: RAM allocation, swap, and kernel configuration (swappiness etc.). The RAM allocation will have a direct impact on the number of VMs you can run per host, and that should really be your main adjustment variable if you can’t add more memory to your hosts. (That doesn’t help with the cost aspect of course...) Depending on what you need VMs for, another strategy could be to use containers instead since that will allow you to reduce the overhead.

Operating systems typically start using swap when they need to allocate memory and they’ve run out of available physical memory, and the least used memory pages currently in physical memory don’t have what’s called a backing store (or rather, their backing store is swap). When a program needs more memory, the kernel will first look for some free memory; then it will look through a hierarchical list of things it can get rid of — cache, buffers, mapped executables, etc. Note that swap can be used even in the absence of “visible” memory pressure: there are always pieces of data stored in memory which aren’t actually used, and are better stored in swap.

In /proc/meminfo you find:

CommitLimit:    1551056920 kB
Committed_AS:   1549560424 kB

So you are at the commit limit.

If you have disabled overcommiting of memory (to avoid the OOM-killer) by:

echo 2 > /proc/sys/vm/overcommit_memory

Then the commit limit is computed as:

2   -   Don't overcommit. The total address space commit
        for the system is not permitted to exceed swap + a
        configurable amount (default is 50%) of physical RAM.
        Depending on the amount you use, in most situations
        this means a process will not be killed while accessing
        pages but will receive errors on memory allocation as
        appropriate.

(From: https://www.kernel.org/doc/Documentation/vm/overcommit-accounting)

You can use the full memory by:

echo 100 > /proc/sys/vm/overcommit_ratio

Then you will get out-of-memory when physical RAM and swap is all reserved.

The name overcommit_ratio is in this case a bit misleading: You are not overcommitting anything.

Even with this setup you may see out-of-memory before swap is exhausted. malloc.c:

#include <stdio.h>
#include <malloc.h>
#include <stdlib.h>
#include <unistd.h>

void main(int argc, char **argv) {
  long bytes, sleep_sec;
  if(argc != 3) {
    printf("Usage: malloc bytes sleep_sec\n");
    exit(1);
  }
  sscanf(argv[1],"%ld",&bytes);
  sscanf(argv[2],"%ld",&sleep_sec);
  printf("Bytes: %ld Sleep: %ld\n",bytes,sleep_sec);
  if(malloc(bytes)) {
    sleep(sleep_sec);
  } else {
    printf("Out of memory\n");
    exit(1);
  }
}

Compile as:

gcc -o malloc malloc.c

Run as (reserve 1 GB for 10 seconds):

./malloc 1073741824 10

If you run this you may see OOM even though there is swap free:

# Plenty of ram+swap free before we start
$ free -m
              total        used        free      shared  buff/cache   available
Mem:         515966        2824      512361          16         780      511234
Swap:       1256720           0     1256720

# Reserve 1.8 TB
$ ./malloc 1800000000000 100 &
Bytes: 1800000000000 Sleep: 100

# It looks as if there is plenty of ram+swap free
$ free -m
              total        used        free      shared  buff/cache   available
Mem:         515966        2824      512361          16         780      511234
Swap:       1256720           0     1256720

# But there isn't: It is all reserved (just not used yet)
$ cat /proc/meminfo |grep omm
CommitLimit:    1815231560 kB
Committed_AS:   1761680484 kB

# Thus this fails (as you would expect)
$ ./malloc 180000000000 100
Bytes: 180000000000 Sleep: 100
Out of memory

So while free in practice often will do The Right Thing, looking at CommitLimit and Committed_AS seems to be more bullet-proof.