Your mongo stat shows higher number of updates vs inserts. One thing that could cause high write lock issues is if your updates typically are increasing the document size and causing the document to move in the data file. We ran into this ourselves, but we were working with mongo support at the time to figure out so I don't remember what metric or stat would tell you this is the case. This would likely only be an issue if your document sizes were very large. We ended up splitting out a sub array that was always being added to into its own collection so that we were just adding new documents instead of modifying an existing one.
The usePowerOf2Sizes flag on the collection can also help alleviate this by giving the documents room for growth. This is apparently the default now on 2.6, but you would need to turn it on if you're not on 2.6 yet. Setting that is described here: http://docs.mongodb.org/manual/reference/command/collMod/
SQL Server has JSON support though it seems to be a variation on it's XML support.
PostGres has had JSON support since 9.2.
Teradata has had it since 2014.
This makes them hybrids. Obviously PostGres is the open-source one.
It really depends on what you need from JSON support. If it is to return the document on a key then that has always been possible without explicit JSON support.
If you want to index certain parts of the JSON document then that does require specific JSON support. There is the question of how sparse the attribute is that you want to index. If it is present in the vast majority of cases (or even a mandatory attribute) then I would break it out of the JSON and have it as an explicit attribute in a hybrid store.
If by parallelize or die you mean support a distributed dataset or die then I'm not sure that I agree. For OLTP work you would have to be in a Times top 100 company to approach the limits of the traditional RDBMS. By definition most companies just don't generate as much useful data as they would like to think.
Companies like Facebook, Twitter and NetFlix are dealing with data on an order of magnitude far beyond what most people will see.
For high end web analytics work then yes, you might want a NOSQL product as a collector. Cassandra is useful in that respect, plus it has tunable consistency.
The gotchas in distributed systems is Brewers CAP theorem. You can have any two of Consistency, Availability or Partition tolerance. It's a bit more blurred than that but the 2 of 3 rule is generally true.
There's the challenge of how you handle referential integrity and in some cases how do you honour primary key constraints. Your data might be splashed across many servers so honouring a foreign key concept might require a lookup between one server and another. Even if this were a feature it would seriously impact performance.
If people want to interact with an API that only talks JSON then I see no problem with that providing there is an underlying DAL that obfuscates whether a data attribute is present in a JSON document or explicit attribute in a hybrid design.
Best Answer
It's unclear what you mean by comparing map-reduce to sharding. But the short answer is: sharding.
Generally speaking you design-out map-reduce queries, you do not want 100s of map-reduce queries being executed at once - you'd just overload mongo since that essentially means 100s of full collection scans all being run at the same time.
If you have an example of one of your existing map-reduce queries - please add it to your question.
Regarding sharding, it all comes down to what you use for the sharding key.
If you shard a users collection on username for example,
will cause mongos to send the query to all shards and merge the result sets together (intelligently). Adding the sharding key to the query:
would give mongos the option to talk to less mongod at a time.
a better example, if you query for:
mongos will send the query to the shards whose shard keys indicate that they could contain answers, most likely that is one server only, resulting in a fast query and no extra load on the part of mongos.
Maybe the above examples are not-news to you.
The queries you send to mongo now are same syntax you'd use to send to a sharded database. The only thing you'd do differently is (previously) analysing on what keys to shard so that you can, where necessary, modify your queries to incorporate the sharding key and thereby enable mongos to act like a proxy instead of an agregator.
A poor sharding key, or simply not taking advantage of sharding in the queries you are generating, would result in mongos needing to query all mongod servers for all queries resulting in high load and poor performance.