Selasa, 08 Juli 2014

Performance Unit Testing to Reduce Friction in Software Evolution

Dr. Linas Vepstas' concern regarding premature distribution architecture in OpenCog AtomSpace that hinders performance optimization:

Anyway, a few years ago, the AtomSpace used the proxy pattern and it was a performance disaster. I removed it, which was a lot of hard, painful work, because it should never have been added in the first place.  Atomspace addnode/link operations got 3x faster, getting and setting truth values got 30x faster, getting outgoing sets got 30x faster.  You can read about it in opencog/benchmark/diary.txt


With regard to performance, I think it's probably useful to have a performance-unit-test for each project. Just like unit tests but for timing iterations of operations with known data, instead of correctness.
Doesn't have to be lots of data, just a few curated set of input data that's somewhat representative.

This way development (and development branches) will have a history of simple performance metrics, on a regular basis.

The intention is not to encourage premature optimization, but to give early warning when a code restructure/refactoring might impact performance negatively.
With that out of the way, the developer is free-er to explore software design options, test feature branches, so that internal structures & implementation stay "sane".

(it would imply that the tests themselves would have maintenance cost and this would be alleviated by having a relatively-stable public API that the tests can rely on to.)