Mongodb – Query MongoDB Collection Compression Settings

In the future, ask to have the question migrated here instead of double posting.

From the MongoDB Documentation,

To calculate how much RAM you need, you must calculate your working set size, or the portion of your data that clients use most often. This depends on your access patterns, what indexes you have, and the size of your documents. Because MongoDB uses a thread per connection model, each database connection also will need up to 1MB of RAM, whether active or idle.

If the query is not completely covered by an index then the entire document is loaded into RAM by MongoDB. This is the same regardless of the type of query.

As for whether or not you should split your collection, that's really a design decision that requires a lot more background to understand and weigh in on. You'll need to perform tests to see if the gains from it based on the query patterns are worth it compared to the loss of atomicity (a business driven decision potentially).

I probably wouldn't recommend this approach though and would assess what your true SLA's are and whether or not you're using the right amount of hardware (RAM) for the desired results. Your real question here is "Do I have enough RAM to satisfy my query performance requirements when it comes to cached data?" If the answer is no, figure out what design works best with your constraints.

What does "page written failed to compress" indicate?

WiredTiger's general strategy is to only compress pages on-disk where there is a storage benefit.

The "page written failed to compress" counter is incremented when compression was attempted but didn't result in an on-disk storage saving (so a page was stored uncompressed).

Another metric for pages that are stored uncompressed is "page written was too small to compress". In this case, the size of a page was considered too small to attempt compression.

The commonly expected case is that pages can be stored compressed, which increments the "compressed pages written" metric.

Is it an error I should be concerned about with regards to data integrity and why does it happen?

These storage metrics are informational counters and expected behaviour (i.e. unrelated to data integrity).

There isn't any obvious actionable outcome based on these metrics, but if a collection wasn't getting much overall benefit from compression the counters might provide insight into how often pages aren't compressed (as compared to compressible pages that don't have a significant level of compression).