It's unclear what you mean by comparing map-reduce to sharding. But the short answer is: sharding.
Generally speaking you design-out map-reduce queries, you do not want 100s of map-reduce queries being executed at once - you'd just overload mongo since that essentially means 100s of full collection scans all being run at the same time.
If you have an example of one of your existing map-reduce queries - please add it to your question.
Regarding sharding, it all comes down to what you use for the sharding key.
If you shard a users collection on username for example,
db.users.find()
will cause mongos to send the query to all shards and merge the result sets together (intelligently). Adding the sharding key to the query:
db.users.find().sort({username: 1}).limit(100);
would give mongos the option to talk to less mongod at a time.
a better example, if you query for:
db.users.find({username: /^bob/})
mongos will send the query to the shards whose shard keys indicate that they could contain answers, most likely that is one server only, resulting in a fast query and no extra load on the part of mongos.
Maybe the above examples are not-news to you.
The queries you send to mongo now are same syntax you'd use to send to a sharded database. The only thing you'd do differently is (previously) analysing on what keys to shard so that you can, where necessary, modify your queries to incorporate the sharding key and thereby enable mongos to act like a proxy instead of an agregator.
A poor sharding key, or simply not taking advantage of sharding in the queries you are generating, would result in mongos needing to query all mongod servers for all queries resulting in high load and poor performance.
If the files are sorted you can just find the id or timestamp of the last inserted document in the database, then only insert the records in the files which have ids that come after that id.
If they are sorted by _id or timestamp then doing this should be trivial.
The solutions you came up with are great as well.
However, be aware that creating entirely new collections will be slower than what i proposed, and upserting will be even slower than that.
Best Answer
You can do it either way.
There isn't much benefit to starting all of the overhead for sharding now, unless you simply want to try it out to see how it works. It is important to pick the correct shard key, so you would want to make sure your schema and insert/update usage is firm enough to know what to pick.
Kristina Chodorow has a great blog entry about how to pick a shard key: http://www.snailinaturtleneck.com/blog/2011/01/04/how-to-choose-a-shard-key-the-card-game/
For general information about how sharding works: http://www.snailinaturtleneck.com/blog/2010/03/30/sharding-with-the-fishes/
For some instructions on how to set up clustering on a single machine, see here: http://www.mongodb.org/display/DOCS/A+Sample+Configuration+Session and here: http://www.snailinaturtleneck.com/blog/2010/08/30/return-of-the-mongo-mailbag/