Ubuntu – How toncrease Ubuntu’s Virtual Memory and/or Swap for Matlab

16.04MATLABramswap

Situation: fix Out of Memory errors in Ubuntu's Matlab
Goal: allocate some virtual memory and/or Swap on external HDD/SSD; the read/write decrease from 20 GBps to 0.1 GBps, which is ok!
Terminology: swap and virtual memory here

No, swap and virtual memory are completely different. For example, memory mapping a 1GB file uses an additional 1GB of virtual memory but has no change in the use of swap. Swap is a form of backing store. Many uses of virtual memory don't have anything to do with backing store. (And there have been systems with virtual memory and no swap as well as systems with swap but no virtual memory.)

Consumer hardware is limited so I need to use more virtual memory and/or swap with my external HDDs. Matlab says about the swap memory (TODO excluding virtual memory?)

Linux Systems — Change your swap space by using the mkswap and swapon commands.

Characteristics of the system

  • You can see how much you have it by swapon -s 

    Filename                Type        Size    Used    Priority
/dev/sda3               partition   8326140 0       -1

  • Matlab's configurations

    % https://stackoverflow.com/a/35971040/54964
com.mathworks.services.Prefs.setIntegerPref('JavaMemHeapMax', 2048); % MB

% TODO cannot find ways how to put Matlab use /dev/sda3

  • So you see that my Matlab is not using it. I receive Out of Memory errors with big matrices in Matlab. I have been unsuccessful in reshape'ing matrices to vectors and writing parallel code.
    So I want to use virtual memory because I need to get work done; the rate does not matter.

Pseudocode

  1. Shell script that creates a swap, start MATLAB, and delete the swap when MATLAB exits. (MichaelHooreman)
  2. Enable swap on external HDD. How to use sudo swapon -a here?
  3. Start Matlab.
  4. Put Matlab use the swap.
  5. Delete swap when Matlab exits.

Temporary Swap, Running Client and Closing/Removing Swap in Micheal's script

Situation: cannot control errors in setting up the environment (1), running Matlab (2) and closing the environment (3)
Script

#!/usr/bin/env bash

# https://stackoverflow.com/a/69808/54964
set -e 
# TODO How to do swapoff if any error?

SWAP_FILE="/media/masi/SamiSwapVirtual/.swap_file_20.7.2016"
SIZE_MB=16000
TO_RUN="matlab"

dd if="/dev/zero" of=${SWAP_FILE} bs="1M" count=${SIZE_MB} status="progress"
mkswap ${SWAP_FILE}
chmod 0600 ${SWAP_FILE}
sudo chown 0.0 ${SWAP_FILE} # https://unix.stackexchange.com/a/297153/16920
sudo swapon -v ${SWAP_FILE}
echo "Swap enabled. Press enter to continue"; read
${TO_RUN}
echo "I will remove the swap. Press enter to continue"; read
sudo swapoff -v ${SWAP_FILE}
rm -vf ${SWAP_FILE}

Iteration 1 with Transcend 25M3 1 TB with little use berofe where filesystem ext4

  1. Logs after starting the script

    sh start_matlab_with_swap.sh 
16000+0 records in
16000+0 records out
16777216000 bytes (17 GB, 16 GiB) copied, 134.489 s, 125 MB/s
Setting up swapspace version 1, size = 15.6 GiB (16777211904 bytes)
no label, UUID=48c2835b-4499-4534-aa49-0648e15bd5d9
[sudo] password for masi: 
swapon /media/masi/SamiWeek/tmp/swap_file_18.7.2016
swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: insecure file owner 1000, 0 (root) suggested.
swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: found swap signature: version 1d, page-size 4, same byte order
swapon: /media/masi/SamiWeek/tmp/swap_file_18.7.2016: pagesize=4096, swapsize=16777216000, devsize=16777216000
Swap enabled. Press enter to continue
start_matlab_with_swap.sh: 11: read: arg count

  2. Running the client

    • Command history gone in the first start of Matlab in the swap memory (ticket #02075943), with the error There was a problem reading your command history – –. Just restart your Matlab and and the problem is solved if you have default settings. Command prefdir gives /home/masi/.matlab/R2016a which is the default location (/home/{username}/.matlab/R2016a. File /home/masi/.matlab/R2016a/matlab.prf exists after the restart, here.

    • … [other errors] …

  3. Closing Matlab and typing password again in Terminal 

    [sudo] password for masi: 
swapoff /media/masi/SamiWeek/tmp/swap_file_18.7.2016
[ bugs here! ]

Open: How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?

Permanent Swap = Separate Swap Setup from Running Client

Setting up Swap 

# https://unix.stackexchange.com/q/297767/16920
masi@masi:~$ sudo fallocate -l 20G /mnt/.swapfile

masi@masi:~$ sudo mkswap /mnt/.swapfile 
Setting up swapspace version 1, size = 20 GiB (21474832384 bytes)
no label, UUID=45df9e48-1760-47e8-84d7-7a14f56bbd72

masi@masi:~$ sudo swapon /mnt/.swapfile
swapon: /mnt/.swapfile: insecure permissions 0644, 0600 suggested.

masi@masi:~$ sudo chmod 600 /mnt/.swapfile

masi@masi:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7925        1494         175         196        6255        5892
Swap:         28610           0       28610

Put the following in the end of /etc/fstab for the permanent change

# https://unix.stackexchange.com/a/298212/16920
# https://unix.stackexchange.com/a/298543/16920

# If swap is on SSD, trim blocks each time at startup.
#/mnt/.swapfile  none    swap    defaults,discard      0        0

# If swap on External HDD, just use sw.
/media/masi/SamiWeek/.swapfile  none    swap    sw      0        0

System: Linux Ubuntu 16.04 64 bit
Linux kernel: 4.6
Linux kernel options: wl
Matlab: 2016a
Official Matlab docs: Resolve "Out of Memory" Errors
External HDD: Transcend 1 TB StoreJet 25M3 review, Transcend 2 TB StoreJet 25M3
External HDD filesystem: ext4
External HDD buffer: 8 MB
Related threads: How to increase MATLAB memory limitation in ubuntu? (how to use mkswap, swapon for MATLAB?), How to Reduce Physical memory increase in Matlab?, How to Resolve out of memory error in Matlab?, How to Fix Out of Memory error in Matlab for 10800×10800 matrix?, How do I increase memory limit (contiguous as well as overall) in Matlab r2012b?, How to Increase Array Block and Resolve Out of Memory Error in Matlab 2009b?, How to Resolve this Out of Memory Issue for a Small Variable in Matlab?, 'Out of memory' in Matlab. A slow but a permanent solution?

Best Answer

OK, quite a list you have there. Let me respond inline

  1. How to Apply the Better Error-handling of Error trapping here? See my script for the example in the source. Thread How to Do Error-trapping and Swapoff if Error/Warning?.

I don't like the concept of this script at all. That you have an external harddrive that you're trying to use as swap is just a bad idea. If you really intend to do this on a regular basis then resize your partitions to put a proper swap partition in, add a swap file, or just buy a bigger internal disk.

  1. How to put warnings if the matrix size exceeds the swap size?

Just do do the math. If you know the size of the matrix before the program begins then compute the size in MiB and compare it to the available swap.

  1. How to have a progress bar in computing your huge matrix in Matlab?

matlab has an API right? I don't think this is the right forum for that question. Even if you had an API, you'll be blocking on IO via swap so it'll just be a jerky progress bar that doesn't actually reflect reality.

  1. How to kill busy progress and/or swapon -s/swapoff in iteration (2)?

You don't. just because you're done with the computation doesn't mean the operating system is done with the resources you allocated. When it's done writing out to swap, it'll free up. You've consumed so much memory that lots of applications can't get the memory they need so they're using swap too. Just leave it on and let the operating system do it's thing. Before you perform your next run clear the caches.

echo 3 > /proc/sys/vm/drop_caches

There's probably more to it than that, I'm not a Linux VM expert. It would be worthwhile to investigate how the SLAB/SLUB allocator works and how to tune it for your large memory requirements. You may be able to MLOCK matlab into memory. That forces the OS to reserve memory for you, or it just doesn't start, you also have to unlock it when you're done. I can do this with the C API just fine but I not sure how you would do that outside of a process I can't recompile, that would require some research.

Finally, this is the sort of stuff EC2 was made for. It looks like 16G is what you need, a m4.4xlarge has 64G ram @ $0.958 per Hour. That's less than a cup coffee. Script your install of matlab using a juju charm or similar and turn the whole thing into a computation as a service.

Is 16G 16 GB?

  • Yes, normally when we leave off the suffix we mean base2 numbers in units of bytes. If you want to concise you would write 16GiB.

"I need matrices that are > 100 GB. I do not know if you can do it with EC2."

Should you clear your caches also by echo 3 > /proc/sys/vm/drop_caches?

  • Yes, it doesn't hurt to always do that. See Documentation/sysctl/vm.txt in the linux kernel.

How can you MLOCK Matlab into memory?

  • man mlock. Though I goofed when I quoted that. This call assures that you can allocate all the memory you want and keep in from being swapped out, it'll never use virtual memory. That's not what you want.

I think you can bind C API to Matlab. - - Do you have any idea of turning off swap if there is any failure in the processes?

  • I'm going to be frank here, the concept of micromanaging swap files in the manner you propose is ridiculous. The operating system's job is to manage resources and hand them out in a fair & consistent manner. Once you give it more resources, it's going to use them as it sees fit. You don't get to tell it when you're done and yank resources out from underneath it, the OS tells you when it's done.

When I ask the OS for an memory address space, sometimes it doesn't always succeed, that doesn't mean I can't try again. That matlab can't figure out to call malloc twice is matlab's problem.

So, to affect the change you want, if that 100G of space is really at a premium then you need to figure out how to tell the operating system to trim it's memory footprint (by clearing caches for starters) so the memory manager doesn't feel the need to use the additional swap space that it was provided. Then and only then can you ask the memory manager to release the swap file.

It's easy to grow things like memory and disks, it's a lot harder to shrink them. Shrinking forces a re-balancing of every user who has resources allocated in that space. If I instead said "I have a 100TB storage array but now I only need 60TB, why is it when I remove 40TB of disk that the array stops working?" Well, the answer would be obvious right?

So here are your options as I see it.

  1. investigate the matlab C API to see if you can get better control over how memory is allocated for these massive working sets.

  2. refactor your computation to compute what you have now using sub-matrices or some other sparse data representation.

  3. write your own program in C/C++ using the plethora of linear algebra libraries out there to perform the computation and use malloc or mmap anonymous to allocate the address space you need.

Related Question