Empirical studies omit reporting necessary details: A systematic literature review of reporting quality in model based testing

Abstract

ContextEmpirical studies are essential in evaluating the effectiveness of Model-based Testing (MBT) research and should be reported properly to ensure their replication and to highlight the strengths and limitations of the MBT techniques being evaluated. Researchers have proposed guidelines detailing what information should be reported when presenting empirical studies and what should be the structure of such primary studies. There is a need to evaluate the reporting quality of the empirical studies in MBT literature. ObjectiveTo evaluate the reporting quality of empirical studies in the model based testing domain; identifying where the reported studies fail to follow the proposed guidelines and finding frequently omitted details. As an auxiliary goal we aim to quantify the percentage of empirical studies conducted in industrial context. MethodWe evaluate the reporting quality and the execution contexts of MBT empirical studies reported in literature. For our study we consider the MBT papers published in top ten software engineering journals over the last eighteen years. We evaluate the published primary studies using the empirical study reporting guidelines. ResultsWe found 87 empirical in MBT that met our selection criteria. Initial results showed that the existing guidelines were not only too strict (for example they demand presence of specific sections rather than simply having the details present in the paper), they also did not adequately cover MBT specific details. Therefore, we propose modified the guidelines for reporting empirical studies in MBT and re-evaluated the selected studies. Results show that while only a few empirical studies follow the exact structure proposed by the guidelines, approximately half the papers contain at least 50% of the required details. Most of the papers omit details related to process and analysis leading to presented results. We found a positive trend of improving reporting quality of empirical studies in MBT over the last Eighteen years. Another important finding from the review is that few reported studies were conducted in real industrial context. ConclusionsModel based testing community needs to be more aware of the reporting guidelines and more effort should be spent on reporting the necessary details. Furthermore, we found that only few studies that are conducted in industrial context and hence more focus should be given to empirical case studies in real industry context. However, the reporting quality of research papers presenting empirical evaluations is gradually improving.