Towards Improving Experimentation in Software Engineering

Abstract

[Background:] Experimentation in Software Engineering plays a central role on sharing and verifying scientific findings. As experiments have increased significantly in Software Engineering area, we observe that most of them fail to provide a way to be repeated, replicated or reproduced, thus jeopardizing or delaying the evolution of the Software Engineering area. [Aims:] In this vision paper, we present and discuss techniques and infrastructure to continuously improve experiments towards repeatability, replicability, and reproducibility. [Method:] We define these techniques and infrastructure based on experiences of our research groups and existing literature. Furthermore, we follow Open Science principles. [Results:] We provide incipient results and foresee a central infrastructure composed of two repositories and two recommendation systems to support techniques for: reporting experiments; developing ontologies for experiments and open educational resources; mining and recommending experiments; specifying data management plans, identifying families of experiments; and teaching and learning experimentation. [Conclusions:] Our techniques and infrastructure will prospectively motivate and benefit Software Engineering evolution by improving the conduction and further reproducibility of experiments.