A Theory of Pending Schemas in Combinatorial Testing

Xintao Niu,Changhai Nie,Xiaoyin Wang,Huayao Wu,Yu Lei

doi:10.1109/tse.2021.3113920

Abstract

Combinatorial Testing (CT) is an effective testing technique for detecting failures which are triggered by the interactions of various factors that influence the behaviour of a system. Although many studies in CT have designed elaborate test suites (called covering arrays) to systemically check each possible factor interaction, they provide weak support to locate the concrete failure-inducing interactions, i.e., the Minimal Failure-causing Schemas (MFS). To this end, a variety of MFS identification approaches have been proposed. However, as this study reveals, these approaches suffer from various issues such as cannot identify multiple overlapping MFSs, cannot handle MFSs with high degrees, cannot be applied to systems with large number of parameters, etc. These issues are essentially caused by the exponential computing complexity of checking every interaction in the test cases. Therefore, they can only focus on a subset of all the possible interactions, resulting in many interactions unnoticed. Ignoring these unnoticed interactions could potentially cause failures that have never been systematically checked. Hence, it is beneficial for MFS identification approaches to identify these interactions. In order to account for these unnoticed interactions in CT, this study introduces the notion of pending schema, based on which a theoretical framework of CT schemas is established. In particular, we formally define the determinability of a schema in CT with respect to given information; as such, the yet-to-be determined schemas are exactly the pending schemas. The relationships between the different schemas (faulty, healthy, and pending) and test cases are also theoretically analyzed. Based on which, we further propose three formulas, along with three corresponding algorithms, for the identification of the pending schemas in failing test cases, and formally prove their correctness. As a result, we reduce the complexity of obtaining pending schemas with respect to the number of factors that may have influences on the software.

Highlights

T HE behavior of modern software is affected by many factors, such as input parameters, configuration options, and communication events
The specific suspicious value of a schema is computed by summing up the suspicious values of every parameter value in it, where the suspicious value of each parameter value is about the ratio of the times it appears in the failing test cases to the times it appears in all the test cases
This computing formula is based on the intuition that the more a parameter value appears in a failing test case and the less it appears in the passing test case, the more it would be a failure-inducing factor

Summary

Introduction

T HE behavior of modern software is affected by many factors, such as input parameters, configuration options, and communication events. To test such a software system is challenging, as, in theory, we should test all the possible interactions of these factors to ensure the correctness of the System Under Test (SUT) [1], [2]. Instead of testing all the possible interactions in a system, it focuses on checking those interactions with the number of involved factors no more than a predefined constant. Many studies in CT focus on designing an elaborate test suite (called covering array) to reveal such failures. Covering arrays are effective and efficient as test suites, they provide weak support to distinguish the failure-inducing

Objectives

Methods

Discussion

Conclusion