Resources for Reproducibility of Experiments in Empirical Software Engineering: Topics Derived From a Secondary Study

Abstract

Background: Replication is a recurrent issue in empirical software engineering (ESE). Although it is a foundation of science, replication is hard to execute despite the many supporting tools meant to facilitate reproducibility. For example, in an experiment, which is the most used method in ESE, the number of replications is not enough compared to other sciences. Objective: In this study, we aim to identify tools that maximize reproducibility in software engineering experiments and how they are applied. Methods: We performed a Systematic Mapping Study and complementary strategies to analyze replication from three concerns (communication, knowledge management, and motivation). We analyzed more than 2,600 studies to get 40 primary studies, using a qualitative analytical tool (Atlas.ti) to create semantic maps for synthesizing our results. Result: We found that tools and practices depend on the experiment domain. Human-oriented experiments tend to use an informal mechanism that is costly and time-consuming. On the other hand, technology-oriented experiments are automated, domain-centric, and specialized so they require a learning process and are not transferable to other domains. Conclusion: Tools and practices still lack acceptation and usability among the ESE research community. Therefore, reproducibility is mostly relegated to internal replication, at which time and costs can be assumed within research groups. A focus on new alternatives should be considered to broaden replication.