Assessing and Evolving a Domain Specific Language for Formalizing Software Engineering Experiments: An Empirical Study

Abstract

The research about the formalization and conduction of controlled experiments in software engineering has reported important insights and guidelines for their organization. However, the computational support to formalize and execute controlled experiments still requires deeper investigation. In this context, this paper presents an empirical study that evaluates a domain-specific language (DSL) proposed to formalize controlled experiments in software engineering. The language is part of a model-driven approach that allows the generation of executable workflows for the experiment participants, according to the statistical design of the experiment. Our study involves the modeling of 16 software engineering experiments to analyze the completeness and expressiveness of the investigated DSL when specifying different controlled experiments. The results highlight several limitations of the DSL that affect the formalization and execution of experiments. These outcomes were used to extend and improve the evaluated DSL. Finally, the improved version of the language was used to model the same experiments in order to illustrate the benefits of the proposed improvements.