The field of Search Based Software Engineering (SBSE) has grown around the need to find new ways of heuristically selecting solutions for software engineering problems. As software systems increase in size and complexity, we face an increasing number of tasks that are intractable to perform through manual or even automated exhaustive means. In essence, the potential solution spaces for these problems are very large and often exponential in nature. This includes tasks such as generating test cases to cover specific branches or lines of code, finding optimal sequences of program refactorings, and reverse engineering a program’s module structure. In SBSE, heuristic algorithms such as hill climbing, or meta-heuristic algorithms such as simulated annealing, tabu search, genetic algorithms, and ant colony optimisation are used to efficiently explore the solution space. They are guided by a f itness function, which is a measure of the quality of an individual solution. The idea is to converge on a solution that may not be optimal, but that is good enough. SBSE has been rapidly growing in its presence in the software engineering literature. According to the online SBSE repository,1 the annual number of publications in SBSE more than doubled between 2006 and 2010 (from 60 to almost 160). While this jump in the use of SBSE is an exciting development, the stochastic nature of these algorithms means that individual runs of an algorithm may not be indicative of its effectiveness. In order to obtain reliable data, and to be able to extrapolate results to valid and meaningful conclusions, it is essential to create carefully designed empirical evaluations. Experiments must include a well thought
Read full abstract