Testing, Evaluation and Performance of Optimization and Learning Systems

Abstract

Benchmarks and test suites are widely used to evaluate optimization and learning systems. The advantage is that these test problems provide an objective means of comparing systems. The potential disadvantage is that systems can become overfitted to work well on benchmarks and therefore that good performance on benchmarks does not generalize to real world problems. The meaning and significance of benchmarks is examined in light of theoretical results such as “No Free Lunch.” The “structure” of common benchmarks is also explored.