OK, quite a list you have there. Let me respond inline
- How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?.
I don't like the concept of this script at all. That you have an external harddrive that you're trying to use as swap is just a bad idea. If you really intend to do this on a regular basis then resize your partitions to put a proper swap partition in, add a swap file, or just buy a bigger internal disk.
- How to put warnings if the matrix size exceeds the swap size?
Just do do the math. If you know the size of the matrix before the program begins then compute the size in MiB and compare it to the available swap.
- How to have a progress bar in computing your huge matrix in Matlab?
matlab has an API right? I don't think this is the right forum for that question. Even if you had an API, you'll be blocking on IO via swap so it'll just be a jerky progress bar that doesn't actually reflect reality.
- How to kill busy progress and/or swapon -s/swapoff in iteration (2)?
You don't. just because you're done with the computation doesn't mean the operating system is done with the resources you allocated. When it's done writing out to swap, it'll free up. You've consumed so much memory that lots of applications can't get the memory they need so they're using swap too. Just leave it on and let the operating system do it's thing. Before you perform your next run clear the caches.
echo 3 > /proc/sys/vm/drop_caches
There's probably more to it than that, I'm not a Linux VM expert. It would be worthwhile to investigate how the SLAB/SLUB allocator works and how to tune it for your large memory requirements. You may be able to MLOCK matlab into memory. That forces the OS to reserve memory for you, or it just doesn't start, you also have to unlock it when you're done. I can do this with the C API just fine but I not sure how you would do that outside of a process I can't recompile, that would require some research.
Finally, this is the sort of stuff EC2 was made for. It looks like 16G is what you need, a m4.4xlarge has 64G ram @ $0.958 per Hour. That's less than a cup coffee. Script your install of matlab using a juju charm or similar and turn the whole thing into a computation as a service.
Is 16G 16 GB?
- Yes, normally when we leave off the suffix we mean base2 numbers in units of bytes. If you want to concise you would write 16GiB.
"I need matrices that are > 100 GB. I do not know if you can do it with EC2."
Should you clear your caches also by echo 3 > /proc/sys/vm/drop_caches?
- Yes, it doesn't hurt to always do that. See Documentation/sysctl/vm.txt
in the linux kernel.
How can you MLOCK Matlab into memory?
man mlock. Though I goofed when I quoted that. This call assures that you can allocate all the memory you want and keep in from being swapped out, it'll never use virtual memory. That's not what you want.
I think you can bind C API to Matlab. - - Do you have any idea of turning off swap if there is any failure in the processes?
- I'm going to be frank here, the concept of micromanaging swap files in the manner you propose is ridiculous. The operating system's job is to manage resources and hand them out in a fair & consistent manner. Once you give it more resources, it's going to use them as it sees fit. You don't get to tell it when you're done and yank resources out from underneath it, the OS tells you when it's done.
When I ask the OS for an memory address space, sometimes it doesn't always succeed, that doesn't mean I can't try again. That matlab can't figure out to call malloc twice is matlab's problem.
So, to affect the change you want, if that 100G of space is really at a premium then you need to figure out how to tell the operating system to trim it's memory footprint (by clearing caches for starters) so the memory manager doesn't feel the need to use the additional swap space that it was provided. Then and only then can you ask the memory manager to release the swap file.
It's easy to grow things like memory and disks, it's a lot harder to shrink them. Shrinking forces a re-balancing of every user who has resources allocated in that space. If I instead said "I have a 100TB storage array but now I only need 60TB, why is it when I remove 40TB of disk that the array stops working?" Well, the answer would be obvious right?
So here are your options as I see it.
investigate the matlab C API to see if you can get better control over how memory is allocated for these massive working sets.
refactor your computation to compute what you have now using sub-matrices or some other sparse data representation.
write your own program in C/C++ using the plethora of linear algebra libraries out there to perform the computation and use malloc or mmap anonymous to allocate the address space you need.
It doesn't look like Linux supports dynamic swap file sizing (at any rate, I couldn't get it to detect that a swap file had changed size without a swapoff/swapon).
Presumably it is easier to assume that a swap file doesn't change file size dynamically when writing code for supporting it. I don't see any great use for it either. Since you can use multiple swap files, nothing's preventing you from creating more swap files as needed.
Also note this paragraph from the swapon manpage:
The swap file implementation in the kernel expects to be able to write
to the file directly, without the assistance of the filesystem. This
is a problem on preallocated files (e.g. fallocate(1)) on filesystems
like XFS or ext4, and on copy-on-write filesystems like btrfs.
I'd think the same problem would apply to a dynamically-sized swap file.
Best Answer
Well you can try of course mentioned approaches. One more simple thing is to install Ubuntu just without swap partition at all. Linux will not block you from this, it can work without swap.