Linux – Creating a ram disk on Linux

Quite generally speaking, all operations happen in RAM first - file systems are cached. There are exceptions to this rule, but these rather special cases usually arise from quite specific requirements. Hence until you start hitting the cache flushing, you won't be able to tell the difference.

Another thing is, that the performance depends a lot on the exact file system - some are targeting easier access to huge amounts of small files, some are efficient on real-time data transfers to and from big files (multimedia capturing/streaming), some emphasise data coherency and others can be designed to have small memory/code footprint.

Back to your use case: in just one loop pass you spawn about 20 new processes, most of which just create one directory/file (note that () creates a sub-shell and find spawns cat for every single match) - the bottleneck indeed isn't the file system (and if your system uses ASLR and you don't have a good fast source of entropy your system's randomness pool gets depleted quite fast too). The same goes for FUSE written in Perl - it's not the right tool for the job.

I just went ahead and see what happens, when you write too many files to /dev/shm.

OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/fastssd/bot_subreddit_recom/write_user_docs.py", line 85, in <module>
    f.write(subreddits)
OSError: [Errno 28] No space left on device
inFile: RC_2018-01-02
outDir: tmp
outputToScreenOnly: 0
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/fastssd/bot_subreddit_recom/write_user_docs.py", line 85, in <module>
    f.write(subreddits)
OSError: [Errno 28] No space left on device

It “ran out of space” after 33 days of input files, as shown above.

>>> 36.*33/12
99.0

So an estimated 99GB -- which is very close to 100GB -- of files, is where the “no space left” error happened. This is exactly the size of the /dev/shm device which I had set to be 100G. This means swap was not really able to be used when it was needed. This has demonstrated that unfortunately, /dev/shm simply runs out, and does not let you start to use swap as virtual memory to keep going, when there’s a shortage. So now we know thanks to this experiment. You just need real physical RAM for /dev/shm to work, and virtual RAM aka swap is not helpful for /dev/shm at all.

I have posted this answer to share with others who may benefit from the information gained.