MongoDB Insufficient free space for journal files

The only time that moving the journal is an absolute recommendation is if you have to use a direct NFS mount - NFS is not recommended for MongoDB in general but in particular it does not play well with the journal.

In general, the journal will have quite a different access pattern to the rest of your data (sequential versus random access). Hence it is often a good idea to separate the journal and the data from a performance perspective. Note that this is a very broad generalization and will vary depending on your usage of MongoDB, but will generally be true.

Since your questions are largely about EC2 and EBS then NFS will not play a part here, but I thought it worth mentioning in a broader context, onto specifics.

For EC2/EBS, you will generally only need to separate the journal out if you are seeing write IO contention (or overall IO contention given the nature of EBS) - it will move IO to a different disk and free up some capacity on the data disk. Of course, with EBS your IO is also dependent on the network IO available on the instances and that is dependent on your instance size (and whether you have opted for P-IOPS), hence a lot of variables.

If you are seeing high IOWait times and your disk looks write bound (see IOStat), this is something you should consider, but only as a temporary measure, because it will only be a small tweak compared to increasing available IO. There are plenty of options available there depending on your starting point, like adding more EBS drives to the RAID or by taking advantage of P-IOPS provisioning. You can also occasionally get better performance by changing instance type so that you have less network contention. Each of these may be more effective than moving the journal and have less headaches.

To explain, there are other considerations here - snapshots for one. Thanks to the journal you no longer have to fsync and lock the database to get a consistent snapshot (be it EBS or LVM or other). However the journal has to be included in the snapshot for that to be the case. Hence whatever node you use for backing up, if you intend to snapshot without taking downtime for that node, then you need to make sure the journal is included.

Finally, one of the uses of the journal is to facilitate recovery from an unclean shutdown, such as an OS level crash/reboot. If the journal is on the ephemeral disk in EC2, it will be blown away by such a reboot and hence not be useful in that context. Any such crash/reboot with such a configuration would therefore require a resync from scratch or a restore from backup/snapshot should it occur.

Overall, like most configuration decisions, you have to weigh the pros and cons and pick the solution most relevant for your use case. Hopefully this will give you enough information to make an informed choice.

You have to create a symbolic link that points to the new mountpoint:

If your server runs out of disk space for journal files, the server process will exit. By default, mongod creates journal files in a sub-directory of dbpath named journal. You may elect to put the journal files on another storage device using a filesystem mount or a symlink.

Note

If you place the journal files on a separate storage device you will not be able to use a file system snapshot tool to capture a valid snapshot of your data files and journal files.

Updated link about: Journal files