On the Use of Fault Abstractions for Assessing System Test Case Prioritization Techniques

Abstract

Empirical studies often evaluate Test Case Prioritization (TCP) techniques by measuring their ability to uncover faults as early as possible. In this context, techniques reorder test cases using different sources of information. These prioritizations are evaluated considering faults detected by failure occurrences during testing execution. While this accounts for validity, results can be biased. For instance, conclusions regarding a TCP technique's behavior may be affected by the number of failures, faults and, in turn, the failure-fault relationship. We propose the use of fault abstractions, which relate faults to the related test cases that fail, to exercise techniques on preliminary empirical studies of system level TCP techniques. Rather than modeling specific classes of faults, our strategy is based on identifying cliques of test cases that are related according to a distance function as a fault abstraction, assuming that the test cases can fail due to a number of faults in common. Our strategy calculate these fault abstractions based on the set of maximal cliques obtained from test case distance graphs along with a set of essential test cases. Therefore, one can sample fault abstractions respecting different failure-fault relationships and apply on empirical studies. In our studies, even though our strategy is not predictive, a significant amount of fault abstractions calculated by the proposed technique corresponded to actual reported faults. Therefore Fault abstractions may represent the relationship between faults and failures to assess TCP techniques when none or a small amount of artifacts is available to perform a preliminary empirical evaluation.