Test case prioritization

Abstract

Test case prioritization (TCP) techniques aim to reorder test cases execution according to a goal. One common goal is fault detection, in which test cases that have a higher chance of detecting a fault are executed first than the remaining test cases. The goal of this study is to investigate TCP empirical studies in order to synthesize reported effectiveness results and provide a basis for future research. We conducted a systematic literature mapping to characterize TCP empirical studies and a systematic literature review to analyze reported TCP techniques effectiveness results. Among selected studies from 1999 to 2016, we found that there is a high amount of empirical studies evaluating many TCP techniques. However, when we applied our quality assessment criteria, most of them were discarded, indicating that they might have methodological problems. Analyzed studies reported results of coverage-based TCP techniques. Furthermore, we found that some context factors regarding faults, test cases of the application being tested and coverage granularity considered by TCP techniques may significantly affect the effectiveness of their execution. These results suggest that more rigorous empirical methodology is needed when evaluating TCP techniques and also, authors need to compare effectiveness results of their proposed TCP techniques with well established techniques to generate more evidences Furthermore, our analysis of significant factors on TCP techniques effectiveness can guide researchers when planning empirical evaluations and help them choosing features to compose new techniques.