Mongodb – Transfer image files from MongoDB to S3

mongodb

I have a NodeJS app that stores all of its data, even images, to MongoDB. Now I want to use an S3 (DigitalOcean Space) to store my app's files (images, 43gb) and so, I need to transfer there all files that I have in my mongo.

I found the s3cmd tool that can transfer my files to the S3 instance (DigitalOcean space), but I can't find a way to actually access those files. I know they are stored in the fs.files and fs.chunks collections, but I can't figure out how to actually use them as input to s3cmd!

Any help will be much appreciated!!!

EDIT: This is the format the images are being stored with:

fs.files

{
  "_id" : ObjectId("5ae97922c1dabec8d2d0bdb0"),
  "filename" : "2b57455f3878d11dabc9c984da7de314_postImage.jpeg",
  "contentType" : "binary/octet-stream",
  "length" : 2291623,
  "chunkSize" : 261120,
  "uploadDate" : ISODate("2018-05-02T08:38:58.549Z"),
  "aliases" : null,
  "metadata" : null,
  "md5" : "9ad420eaa7c28a73e449199430627802"
}

fs.chunks.findOne()

{
  "_id" : ObjectId("5ae2d77f6616b4a9d93cb4b1"),
  "files_id" : ObjectId("5ae2d77f6616b4a9d93cb4b0"),
  "n" : 0,
  "data" : BinData(0,"iVBORw0KGgoAAAANSUhEUgAAAuAAAAJvCAYAAAA6OGQEAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAABWWlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRher38tcAAAAASUVORK5CYII=") }

Best Answer

In MongoDB, this is called GridFS, but s3cmd can only upload files from your operational system filesystem. You need to make a script to query GridFS, create a temporary file, then use scm3 to upload it.

Read this article describing how to use GridFS on Node.JS: https://mongodb.github.io/node-mongodb-native/api-articles/nodekoarticle2.html

If you want to understand how to make a GridFS to S3 script, see this Perl script: https://github.com/mediaprojects/gridfs-to-s3

And if you have Perl installed, you can just run the script.

Related Question