Mongodb – How to correct Failed global > initialization: FileNotOpen

A few patterns are evident in your data:

• The primary activity is updates.
• Updates come for several seconds after query/inserts.
• The criteria fields in finds is indexed.
• Even so, the 'index miss' rate is very high.
• Background flush is very low (as it should be with SSD) so you're not getting I/O contention.
• Lock-average is extremely high.

Based on this, I'm suspecting that your document sizes are created small and continually appended to, and you get some significant variation in final document sizes which renders Mongo's padding factor optimization less useful. If that's happening, you're running into a particular write penalty that goes like this:

1. Document is INSERTed, Mongo allocates extra free space due to padding factor. (write: 512B)
2. Document is updated. (write 1KB)
3. Document is updated. (write 1.5KB)
4. Document is updated, but there isn't any adjacent free-space left.
5. Mongo moves the Document to a new spot with free space.
   1. Mongo writes the whole, appended document in a new free block (write 2KB) and marks the old block as available.
   2. Mongo reindexes every indexable field on the document (depending on what kind of indexes you have, this could be a big write hit).
6. Document is updated. (write 512B)

Stub records that are continually appended to are a bit of an anti-pattern with MongoDB due to the penalty of growing beyond the padding factor. You can get away with it if your documents end up about the same size, as the padding factor can compensate.

However, if you have to continually append data to records and can't rely on the padding-factor, you'll have to manually pad at insert time. When you create the record, add enough junk fields to make it close to your average document size, and delete/unset the junk field on your first insert. This will reduce the instance of moves like the above, and should bring your lock-average down.

I also suspect you're running into full table-scans to return records, as that's the 'idx miss' column in the mongostat output. Update calls do run finds, that's how the system locates the record to update. An index miss invokes a double-read of the system; the first in the index to find it, and a second full table-scan to find the record. Typically, this is caused by there not being enough RAM to hold the index, but may also be caused if an UPDATE modifies an indexed field.

There was a full set of instructions for the 2.6 upgrade that you don't seem to indicated that you have followed.

In particular are the instructions that indicate you need to process the --upgrade option by launching a single mongos instance while the balancer is switched off:

Start a single 2.6 mongos instance with the configDB pointing to the cluster’s config servers and with the --upgrade option.

To run a mongos with the --upgrade option, you can upgrade an existing mongos instance to 2.6, or if you need to avoid reconfiguring a production mongos instance, you can use a new 2.6 mongos that can reach all the config servers.

mongos --configdb <config servers> --upgrade

As noted there, on completion the following messages should be emitted:

upgrade of config server to v5 successful
Config database is at version v5

So there is an intended upgrade process for the config servers that it would appear you have not followed yet. The full process is in the link at the top of this answer.

Also have a look at the compatibility changes list which has some additional considerations with replica sets and sharding that might possibly apply in your situation.