I'm in charge of building a nodejs module that dumps varied amount of data from a mongodb into a file and then put it into S3 (sometimes a single document, sometimes thousands, like a whole collection).
The project where this module will be already uses "mongodb" driver for different use cases.
I was told that for larger dumps, using mongoexport could be better in performance, than running the query within nodejs, but this solution would imply spawning a new process and depending on a different tool (mongoexport), it certainly adds complexity to the solution.
My concern isn't the performance in terms of speed, but in terms of overhead to the DB.
Is it true that mongoexport will dump a whole collection faster (and with less "effort") than simply querying and writing the result with a fs stream?
are there guidelines or best practices to do things like this?
Best Answer
Mongoexport connects to the MongoDB server using the Go driver.
Any operation it performs can also be performed by any other user.
If there is no filter provided, it will hint to use
{$natural:1}to read the documents in the order they appear on disk, which is more efficient than using an index when reading the entire collection.You should be able to discover the exact queries and options used by any particular run of mongoexport by setting
slowmsto 0, or by increasing the logging verbosity of the query component.Once you know what queries it uses for your use cases, you can compare that with what you could do with your driver to determine which is more efficient.