Machine freezes once it hits swap space under heavy load

I recently had the same problem, and created this to clean the swap out quickly:

#Single-proc function to core-dump swapped ranges > 1M to /dev/null
unswap(){ (awk -F'[ \t-]+' '/^[a-f0-9]*-[a-f0-9]* /{recent="0x"$1" 0x"$2}/Swap:/&&$2>1000{print recent}' /proc/$1/smaps | while read astart aend; do gdb --batch --pid $1 -ex "dump memory /dev/null $astart $aend" &>/dev/null; done&)2>/dev/null;};

#Loop to run unswap on the top 20 swap-consuming processes
grep VmSwap /proc/*/status 2>/dev/null | sort -nk2 | tail -n20 | cut -d/ -f3 | while read line; do unswap $line; done;

#Observe the number of core dumps currently running, along with free swap, over time. 
echo "Dumps Free(m)"; rcount=10; while [[ $rcount -gt 0 ]]; do rcount=$(ps fauxww | grep "dump memory" | grep -v grep | wc -l); echo "$rcount        $(free -m | awk '/Swap/{print $4}')"; sleep 1; done 

If this doesn't work in your specific environment, the process is as follows:

1. Get a list of the processes consuming the most swap.

In my case, I'm checking /proc/$pid/status, with the swap use in the VmSwap row.

# grep VmSwap /proc/*/status 2>/dev/null | sort -nk2 | tail -n5
/proc/22457/status:VmSwap:      3780 kB
/proc/22684/status:VmSwap:      4260 kB
/proc/7408/status:VmSwap:       4396 kB
/proc/31992/status:VmSwap:      9176 kB
/proc/2967/status:VmSwap:      60840 kB

2. For the high-swap procs, I find the memory address ranges which are contributing the most swap use.

I get this from /proc/$pid/smaps:

7f2fd1bc4000-7f2fd1d24000 rw-p 00000000 00:00 0 <<< Address range
Size:               1408 kB
Rss:                 900 kB
Pss:                 900 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:       900 kB
Referenced:            4 kB
Anonymous:           900 kB
AnonHugePages:         0 kB
Swap:                508 kB    << Swap used
KernelPageSize:        4 kB
MMUPageSize:           4 kB

3. I use gdb to do core dumps of those address ranges straight to /dev/null

This forces the system to access that memory and pull it out of swap. Going to /dev/null avoids unnecessary IO.

gdb --batch --pid $pid -ex "dump memory /dev/null $astart $aend"

The process I mentioned originally omits any memory regions using less than 1M of swap in order to get the majority of the swap with considerably fewer core dumps, but this is not mandatory.

You have got two problems here, apparently.

The REISUB sequence not working could be caused by magic SysRq not being activated - check out

cat /proc/sys/kernel/sysrq

# List of possible values in /proc/sys/kernel/sysrq:

#  0 - disable sysrq completely
#  1 - enable all functions of sysrq
#    >1 - bitmask of allowed sysrq functions (see below for detailed function description):
#      2 - enable control of console logging level
#      4 - enable control of keyboard (SAK, unraw)
#      8 - enable debugging dumps of processes etc.
#     16 - enable sync command
#     32 - enable remount read-only
#     64 - enable signalling of processes (term, kill, oom-kill)
#    128 - allow reboot/poweroff
#    256 - allow nicing of all RT tasks

You want 4 (R) + 64 (E,I) + 16 (S) + 32 (U) + 128 (B) = 244 for classic REISUB (the effect is the same as the skinny elephant version; as a mnemonic, it's BUSIER spelled backwards).

For the swappiness of the system: it's working as expected. The cause of the behaviour is that you have lots of memory used - that could be the 95% you're getting - and then something else is requesting a whole lot of new RAM, forcing most of that 95% onto swap and fighting all the way.

The best thing you could do is install more RAM. Otherwise, see whether you can tame either the small processes building up to that 95%, or the big one eating a further 40-50%. The top utility might help you there.

As a fancy solution you could increase apparent swappiness and keep the small fry "pared down" with some process that monitored the system and, if the big hog was not running and the memory footprint was too large, started allocating and releasing large swaths of memory - say, all remaining memory plus 64M, to force 64M of small fry onto the cache, then +128M, and so on, until the allocation delay grew beyond a given time threshold or the big hog started.

This answer could give you a good start if you really want to do that.