I have seen and heard from multiple sources that it can be a good idea, performance-wise, to write your db journal and your data files to separate disks.
What's the recommended way to do this (e.g. on EC2)?
- If we write our data files to RAID (e.g. of EBS drives), where would most people write the journal to?
- Should I just use completely separate (RAIDed) set of drives
- Should I use a single non-RAID drive?
- Should I just leave data and journal on the same RAID?
Best Answer
The only time that moving the journal is an absolute recommendation is if you have to use a direct NFS mount - NFS is not recommended for MongoDB in general but in particular it does not play well with the journal.
In general, the journal will have quite a different access pattern to the rest of your data (sequential versus random access). Hence it is often a good idea to separate the journal and the data from a performance perspective. Note that this is a very broad generalization and will vary depending on your usage of MongoDB, but will generally be true.
Since your questions are largely about EC2 and EBS then NFS will not play a part here, but I thought it worth mentioning in a broader context, onto specifics.
For EC2/EBS, you will generally only need to separate the journal out if you are seeing write IO contention (or overall IO contention given the nature of EBS) - it will move IO to a different disk and free up some capacity on the data disk. Of course, with EBS your IO is also dependent on the network IO available on the instances and that is dependent on your instance size (and whether you have opted for P-IOPS), hence a lot of variables.
If you are seeing high IOWait times and your disk looks write bound (see IOStat), this is something you should consider, but only as a temporary measure, because it will only be a small tweak compared to increasing available IO. There are plenty of options available there depending on your starting point, like adding more EBS drives to the RAID or by taking advantage of P-IOPS provisioning. You can also occasionally get better performance by changing instance type so that you have less network contention. Each of these may be more effective than moving the journal and have less headaches.
To explain, there are other considerations here - snapshots for one. Thanks to the journal you no longer have to fsync and lock the database to get a consistent snapshot (be it EBS or LVM or other). However the journal has to be included in the snapshot for that to be the case. Hence whatever node you use for backing up, if you intend to snapshot without taking downtime for that node, then you need to make sure the journal is included.
Finally, one of the uses of the journal is to facilitate recovery from an unclean shutdown, such as an OS level crash/reboot. If the journal is on the ephemeral disk in EC2, it will be blown away by such a reboot and hence not be useful in that context. Any such crash/reboot with such a configuration would therefore require a resync from scratch or a restore from backup/snapshot should it occur.
Overall, like most configuration decisions, you have to weigh the pros and cons and pick the solution most relevant for your use case. Hopefully this will give you enough information to make an informed choice.