What are some effective ways to perform programmatic performance testing on database operations, especially in environments where the databases themselves do not offer dedicated tools?
For example, in Google App Engine, entire page-loads are evaluated as one operation which may include specific database operations. This problem is also likely present in SQLite and other integrated DBs. As it is difficult to completely abstract the (equivalent of) selects and inserts that need to be tested, are there any recommended database tools to perform more thorough diagnostics on these sorts of queries?
Best Answer
It seems to me your problem is that you are trying to test performance metrics that are not well supported in the underlying db. This makes it very difficult to compare performance across systems because the underlying approaches are very different. I don't think it is possible to do apples to apples comparisons just as I don't think you can do an apples to apples comparison of ORDBMS type approaches to RDBMS type approaches. The performance concerns are just too different and if Stonebraker is right that optimizing an ORDBMS for TPC-C tests misses the point, then for systems that are even further apart it is going to be impossible. (I think he is right there, however, only where ORDBMS functionality comes into play.)
I think what you need to be honest is to look at how you would use each system and build a benchmark tool based on the approach you'd take with each. Then you can say at least for that workflow that the benchmark shows something specific. I don't see how you can generalize however. You could further run it in a profiler in order to get additional information about where the time is being spent in the test run on various engines.
However db benchmarking is very difficult to make meaningful in the best of circumstances and when you are comparing dissimilar systems it becomes impossible to generalize.