Statement frequency coverage: A code coverage criterion for assessing test suite effectiveness

Abstract

Abstract Context: Software testing is a pivotal activity in the development of high-quality software. As software is evolving through its life cycle, the need for a fault-revealing criterion assessing the effectiveness of the test suite grows. Over the years, researchers have proposed coverage-based criteria, including statement and branch coverage, to solve this issue. In literature, the effectiveness of such criteria is attested in terms of their correlations with the mutation score. Objective: In this paper, we aim at proposing a coverage-based criterion named statement frequency coverage, which outperforms statement and branch coverage in terms of correlation with mutation score. Method: To this end, we incorporated the frequency of executed statements into statement coverage and created a coverage-based criterion for assessing test suite effectiveness. Statement frequency coverage assigns a continuous value to a statement whose value is proportional to the number of times executed during test execution. We evaluated our approach on 22 real-world Python projects with more than 118 000 source lines of code (without blank lines, comments, and test cases) and 21 000 test cases through measuring the correlation between statement frequency coverage and corresponding mutation score. Results: The results show that statement frequency coverage outperforms statement and branch coverage criteria. The correlation between it and the corresponding mutation score is higher than the correlation of statement and branch coverage with their mutation score. The results also show that unlike statement and branch coverage, there is no statistical difference between statement frequency coverage and mutation score. Conclusion: Statement frequency coverage is a better choice compared to statement and branch coverage in assessing test suite effectiveness in the real-world setting. Furthermore, we demonstrate that although statement frequency coverage subsumes statement coverage, it is incomparable to branch coverage under the adequate test suite condition.