Improving experimental methods on success rates in evolutionary computation

Abstract

Due to the complexity of theoretical approaches in evolutionary computation (EC), research has being largely performed on experimental basis. One popular measure used by the EC community is the success rate (SR), which is used alone or as part of more complex measures such as Koza’s computational effort in genetic programming. A common practice in EC is to report just a punctual estimation of the SR, without additional information about its associated uncertainty. We aim to motivate EC researchers to adopt more rigorous practices when working with SRs. In particular, we introduce the importance of correctly reporting this measure and highlight its binomial nature. Unfortunately, this fact is usually overlooked in the literature. Considering the binomiality of the SR opens the whole corpus of binomial statistics to EA research and practice. In particular, we focus on studying several methods to compute SR confidence intervals, the factors that determine their quality in terms of coverage probability and interval length. Due to its practical interest, we also briefly discuss the number of required runs to build confidence intervals with a certain quality, providing a sound method to set the number of runs, one of the most important experimental settings in EC. Evidence suggests that Wilson is, on average, a reliable and simple method to bound an estimation of SR with confidence intervals, while the standard method, which is quite popular because of its conceptual simplicity, should be avoided in any case. However, other methods can also be of interest under certain circumstances. We encourage to report the number of trials and successes, as well as the interval, to ease further comparability of the results.