MongoDB Using String-Type Shard Key for Zone Ranges

mongodbshardingspatial

Mongo lets any field type be used to set ranges for Shard Zones, but doesn't specify how analyses are conducted.

My goal is to point queries and writes at the exact right server, WITHOUT using a compound shard key (for various execution complexities).

I want to embed a unique user alpha-numeric ID, plus a latitude and longitude value into a single string, and use this as my shard key.

Ex: (ID.lat.lon) "4kjaj29.48.-89"

Would fall within the range:

"0000000.45.-100" to "zzzzzzz.50.-80"

Mongo uses utf8 strings, https://www.utf8-chartable.de/unicode-utf8-table.pl, in which for alpha-numeric characters 0 is the "least" and z is the "greatest".

So I have to imagine this is how range inclusion is computed? Does anyone know off-hand if this approach is correct? And if not, can you point me in the right direction to accomplish this?

There is definitely a way to accomplish this, since some kind of range inclusions calculation is occurring with string types, but just requires knowing the analysis strategy.

Best Answer

As per MongoDB documentation here In sharded clusters, you can create zones of sharded data based on the shard key. You can associate each zone with one or more shards in the cluster. A shard can associate with any number of zones. In a balanced cluster, MongoDB migrates chunks covered by a zone only to those shards associated with the zone.

Some common deployment patterns where zones can be applied are as follows:

Isolate a specific subset of data on a specific set of shards. Ensure that the most relevant data reside on shards that are geographically closest to the application servers. Route data to shards based on the hardware / performance of the shard hardware.

For example

In the above mention figure a sharded cluster with three shards and two zones. The A zone represents a range with a lower boundary of 1 and an upper bound of 10. The B zone represents a range with a lower boundary of 10 and an upper boundary of 20. Shards Alpha and Beta have the A zone. Shard Beta also has the B zone. Shard Charlie has no zones associated with it. The cluster is in a steady state and no chunks violate any of the zones.

After configuring a zone with a shard key range and associating it with a shard or shards, the cluster may take some time to migrate the affected data. This depends on the division of chunks and the current distribution of data in the cluster. When balancing is complete, reads and writes for documents in a given zone are routed only to the shard or shards inside that zone.

Shard Key

You must use fields contained in the shard key when defining a new range for a zone to cover. If using a compound shard key, the range must include the prefix of the shard key.

For example, given a shard key { a : 1, b : 2, c : 3 }, creating or updating a zone to cover values of b requires including a as the prefix. Creating or updating a zone to covers values of c requires including a and b as the prefix.

You cannot create zones using fields not included in the shard key. For example, if you wanted to use zones to partition data based on geographic location, the shard key would need at least one field that contained geographic data.

When choosing a shard key for a collection, consider what fields you might want to use for configuring zones. After sharding, you cannot change the shard key. See Choosing a Shard Key for considerations in choosing a shard key.

Hashed Shard Keys and Zones

When using zones on a hashed shard key, each zone covers the hashed shard key values. Given a shard key { a : 1 } and a zone alpha with a lower bound of 1 and an upper bound of 5, the bounds represent the hashed value of a, and not the actual value. Therefore, there is no guarantee that MongoDB routes documents where a has a value of 1 to 5 to zone alpha. MongoDB routes any document where the hashed shard key value falls within the range of 1 or 5 to a shard inside zone alpha.

In general, a zone covering a sequential range of hashed shard key values may exhibit unexpected behavior.

It is possible create a zone which covers the entire range of shard key values using minkey and maxkey to guarantee that MongoDB restricts all the data for a specific collection to the shard or shards in that zone.

For your further ref here, here, here and here

Related Solutions

MongoDB – Presplitting Chunks for Compound Shard Key

Just a slight issue with how you are passing the $minKey values in, try this instead:

db.adminCommand( { split : "mydb.mycollection" , middle : { "region" : "region1", "foo" : MinKey , "bar" : MinKey } } );
db.adminCommand( { split : "mydb.mycollection" , middle : { "region" : "region2", "foo" : MinKey , "bar" : MinKey } } );

This got me the following layout:

sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "version" : 4,
    "minCompatibleVersion" : 4,
    "currentVersion" : 5,
    "clusterId" : ObjectId("53a2cd9d98b4ace818666544")
}
  shards:
    {  "_id" : "shard0000",  "host" : "localhost:30000" }
    {  "_id" : "shard0001",  "host" : "localhost:30001" }
    {  "_id" : "shard0002",  "host" : "localhost:30002" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "mydb",  "partitioned" : true,  "primary" : "shard0001" }
        mydb.mycollection
            shard key: { "region" : 1, "foo" : 1, "bar" : 1 }
            chunks:
                shard0000   1
                shard0001   2
            {
    "region" : { "$minKey" : 1 },
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} -->> {
    "region" : "region1",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} on : shard0000 Timestamp(2, 0) 
            {
    "region" : "region1",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} -->> {
    "region" : "region2",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} on : shard0001 Timestamp(2, 2) 
            {
    "region" : "region2",
    "foo" : { "$minKey" : 1 },
    "bar" : { "$minKey" : 1 }
} -->> {
    "region" : { "$maxKey" : 1 },
    "foo" : { "$maxKey" : 1 },
    "bar" : { "$maxKey" : 1 }
} on : shard0001 Timestamp(2, 3)

The use of the $minKey (MinKey) and $maxKey (MaxKey) values is a bit tough to tease out (they are rarely used except internally), but there is a decent, and illustrative example here in the docs.

MongoDB – How to Choose a Randomly Distributed Shard Key

While it's hard to say for certain without full context, I'm assuming its referring to the need to keep the working set in memory.

A randomly distributed shard key would distribute the workload across an entire index, meaning that the entire index would need to be fit in memory to efficiently handle the workload. Performance would deteriorate once the size of this index on a shard grows larger than RAM, as the index on the shard key would need to be page faulting data in and out of memory.

In contrast, a non-random shard key may have a "hot" subset that handles most of the working set. For example, consider a website where only newer "posts" by users are frequently accessed and older "posts" are rarely accessed. While the indexes on "posts" may be larger than available memory, only subsets of the indexes may need to fit in memory, reducing memory pressure and the potential of page faults.

Best Answer

Related Solutions

MongoDB – Presplitting Chunks for Compound Shard Key

MongoDB – How to Choose a Randomly Distributed Shard Key

Related Question