On Mac OS X Yosemite 10.10.5, when I try to run a calculation that needs to allocate and use 128 GB of memory (it is a command line program written in C), the kernel kills my process with extreme prejudice. This console log entry is an example of one instance:
9/25/15 7:08:40.000 PM kernel[0]: low swap: killing pid 6202 (huffgrp)
The calculation works fine and in a reasonable amount of time when it allocates and uses 64 GB of memory. My Mac has 32 GB of RAM and beaucoup space on the hard drive. I also tried this on another Mac with 8 GB of RAM, on which the 64 GB calculation runs fine as well, taking longer of course, but the 128 GB calculation gets killed by the kernel in the same way.
By the way, malloc() never returns an error, no matter how much space I ask for. The kernel will only kill the process once too much of that memory is actually being used by the process, resulting in lots of swapping to the hard drive.
So there appears to be a secret swap space limit somewhere between 64 GB and 128 GB.
My question is: how do I reconfigure the kernel to permit more swap space? I found a promising looking file, /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist, but I don't see the secret number in there. The man page for dynamic_pager says that all it does is set the name and location of the swap files. There is an older version of the same man page that documents a -S option to set the size of the swapfiles created. I tried that, requesting 160 GB swapfiles, but it had no effect. The swapfiles were still 1 GB each, and the process was still killed by the kernel.
Best Answer
Not the answer you asked for, but if you create your own file of an appropriate size, mmap it into your process, and then run your calculation in this address space it should have the same effect as a swap file and you're guaranteed to have the space, as opposed to competing with other processes for available RAM/swap.
It might also be slower, depending on how often you overwrite the data, but should be much more portable.