I am developing a product that, as a part of it's operation, must track a large number of files/directories. The idea is to store stat information in a database then, on boot, create watches for each file. Files that change will be queued (in the database) for a group sync to a remote database. They will be synced in order of priority, a number between 1-10.
Information about database:
- < 100,000 entries of stat info
- Entire database read at boot, only the file path is necessary
- Queued files will have a priority field (nothing else needs be searched by)
- Insertions can be slow
I've found a couple databases that I think will work, but I'm not sure which would be the best:
- Redis– store file-path as key, stat data as value; queue would be a list
- MongoDB– more query options than Redis, but still fast
I'm thinking a NoSQL database would be the best solution here, as there isn't too much relational logic going on, and the total data size isn't too large (something like < 100 mb, closer to < 30 mb). I did look at SQLite because it seems to be simple enough to embed in an installable application.
Since this is a distributed application for end-users and not a high-load server, the database doesn't have to support many simultaneous users. The main priority here is to find a database whose model makes the most sense.
So the question, which database would be most applicable for this situation?
Also, are there any other databases that would make more sense for an application like this?
Best Answer
The first thing that comes to mind is a particular RDBMS that's familiar to me. I recognize, however, that it may not be the best for this application.
So, my advice is to go with a database that are familiar to you. If you're familiar with Redis or MongoDB, then go with one of those. If you're more familiar with SQLite, then chose that.
On a database of this size, it's all going to be pretty quick. Even databases that are more disk-heavy will use some sort of caching so that disk speed isn't too much of a concern.