MongoDB – Is It Always Faster to Create Indexes After Loading Data?

The delay you are seeing when loading from a snapshot is not cause by how indexes are laid out on disk, it's far more likely that you are seeing the delay because when you start an instance from a snapshot, the data is loaded only on first use, and will be significantly slower than subsequent uses - that is a basic limitation of using snapshots in this way and really has little to do with the application that is trying to access disk. That's why you will see guides on "how to warm up an EBS volume" and the like (there are penalties on writing for the first time too). If you do that (warm up the disk with another application like dd for example) and the performance issue goes away, then you have pretty decent proof that the layout of data has nothing to do with the issue.

Along those lines, MongoDB has the touch command that will allow you to warm up the data before you use it in anger (you can touch data, data and indexes or just indexes). Again, after you initially attach the volume, it will be slow, and touch is going to take a while, but at least after that warming up phase, your results should be somewhat consistent.

In terms of how things are stored on disk, you have the basics correct in terms of file allocation but there is a logical structure within the files, extents, that are the real units of storage. That and far more is covered in detail by this presentation by Mathias Stearn - one of the kernel developers at MongoDB.

Indexes are just another (structured) form of data in MongoDB and they are stored in linked extents throughout the file. Fragmentation can become an issue (that's what the compact command is for) as can disk space used (the repair command is used to reclaim) but you haven't described a workload that would immediately make me think you are hitting a fragmentation issue which is why I suspect something else (like the first use penalty) is your root cause.

The first thing to look at is db.serverStatus().ft. This has a bunch of metrics that may be helpful, to figure out where you're spending time. These are documented here: http://docs.tokutek.com/tokumx/tokumx-server-status.html

Usually the way to improve query time is to make sure you have the right index for your query. You might be doing a query on {field: value} and have an index on {field: 1}, but if you're not projecting in your query, you'll still be doing lots of point queries back to the primary key index to get the full document, which is very inefficient. You may want to add other fields to the index to make it covering, and then project to those fields, e.g. index on {field: 1, other: 1, fields: 1} and then make sure you project in your query like db.foo.find({field: "val"}, {field: 1, other: 1, fields: 1}).

Alternatively, you can use a custom primary key of {field: 1, _id: 1} if these are your most common queries, or instead just make the secondary key on {field: 1} a clustering key.

If you want more help, it would be great if you could provide your server configuration, collection schema (db.foo.getIndexes() is fine) and stats (db.foo.stats()), realistic sample queries and cardinality, and output of db.serverStatus().ft.