MongoDB Bulk Find and Remove – How to Reduce Time

bulk-insertmongodb

I'm just making my hands dirty(handson) with MongoDB. First I did bulk insert of 1 million documents.

// Bulk insert query
var bulk = db.bulktest.initializeUnorderedBulkOp();
for(var i=0;i<1000000;i++){bulk.insert({a:i});}
bulk.execute();

It took 10 seconds to insert all the document which is comparatively good. None of the documents were missed.

However, when it comes to bulk find and removal, the performance has drastically came down and is really bad as it is removing each document in the order of 10ms.

// find and removal bulk query
var bulk = db.bulktest.initializeUnorderedBulkOp();
for(var i=0;i<1000000;i++){bulk.find({a:i}).remove();}
bulk.execute();

Has anyone done tweaking around bulk removal ?

I've only one option to drop the complete collection and I don't want to do this.

Best Answer

This shouldn't have anything to do with bulk operations. The find operation must read through the collection, which takes time on 1million documents.

Adding an index on a should improve the speed dramatically.

Related Solutions

Mongodb – How should I store time series in mongodb

I suggest a single time series entry per document. There are some problems with storing multiple entries per document:

a single document is limited to a certain size (currently 16 MB); this limits how many entries can be stored in a single document
as more entries are added to a document, the entire document (and time series) will needlessly be deleted and reallocated to a larger piece of memory
queries on sub-documents are limited compared to queries on regular documents
documents with very flat structures (like one sub-document for each second) are not performant
the built-in map-reduce does not work as well on sub-documents

Also note a timestamp is built-in to the default MongoDB ObjectId. You can use this if the time series precision is less than one second.

Here is an example BSON document from an event logging library that uses MongoDB:

Example format of generated bson document:
{
    'thread': -1216977216,
    'level': 'ERROR',
    'timestamp': Timestamp(1290895671, 63),
    'message': 'test message',
    'fileName': '/var/projects/python/log4mongo-python/tests/test_mongo_handler.py',
    'lineNumber': 38,
    'method': 'test_emit_exception',
    'loggerName':  'testLogger',
    'exception': {
        'stackTrace': 'Traceback (most recent call last):
                       File "/var/projects/python/log4mongo-python/tests/test_mongo_handler.py", line 36, in test_emit_exception
                       raise Exception(\'exc1\')
                       Exception: exc1',
        'message': 'exc1',
        'code': 0
    }
}

Since an event log is similar to a time series, it may be worth studying the rest of the code. There are versions in Java, C#, PHP, and Python.

Here is another similar open source project: Zarkov

[update] In response to @RockScience's comment, I've adding some more references:

Very verbose case study of how MongoDB was used to store time series. Interesting parts:
- Slide 25: how much data? 20 4-byte data points @ 1 minute intervals for 10 years (not apparent if these are actual statistics or just projected estimates)
- Slide 66: shows sample MongoDB document used to store time series data
Best practices for storing log data, straight from the creators of MongoDB

Mongodb – Check and remove iteratively without an index

The current version does not allow for specifying different priorities for different operations. It is likely that maintaining an index on the collection to allow this periodic process to execute much quicker than a full collection scan will give you better overall performance.

Best Answer

Related Solutions

Mongodb – How should I store time series in mongodb

Mongodb – Check and remove iteratively without an index

Related Question