I know this question is similar to other ones. But…
Only that we are switching from 2 very large collections (15Gb in size, constantly growing).
Basically those 2 large collections keep different data for our clients. I'm thinking of creating 2 collections for each client. In terms of size, they will be from 20MB to a few GB each (depending on client size).
Now the pros of having these is that when we loose clients, we need to remove their data. And will be much easier using dropCollection.
The other one is the index, as we keep the client Id on each document in the large collection. And this is not needed, we can remove this index.
About sharding. Now those 2 large ones are sharded across 3 servers. With the small ones, what would be best:
- automatically create and shard them when a client comes in (we create collections, add the shard key, all automatically).
- only shard the large clients, manually.
What happens if there are lots of small collections (like 10.000, each of 30mb let's say) vs a very large one?
Best Answer
Multiple small collections will take more space than few big collections. Depending on amount of memory, big collections indexes are more effective than large amount of indexes of small collections.