Chapter 4 - Testing of Configurable Systems

Abstract

Abstract Configurable software system allows users to customize its applications in various ways, and is becoming increasingly prevalent. Testing configurable software requires extra effort over testing traditional software because there is evidence that running the same test case under different configurations may detect different faults. Differentiating test cases and configurations as two independent factors for testing, we must consider not just which test case to utilize, but also which configurations to test. Ideally, an exhaustive testing approach would combine every test case with every possible configuration. But since the full configuration space of most software systems is huge, it is infeasible to test all possible configurations with all test cases. Instead, selection techniques are necessary to select configurations for testing a software system, and to select test cases for the different configurations under test. Despite successful selection techniques, sometimes it is still costly to run only selected configurations and test cases. In particular, the cost is magnified when new features and functionality are added as a system evolves, and the new version is regression tested. Regression testing is an important but expensive way to build confidence that software changes do not introduce new faults as the software evolves, and many efforts have been made to improve its performance given limited resources. Test case prioritization has been extensively researched to determine which test cases should be run first, but has rarely been considered for configurations. In this chapter we introduce issues relevant to testing configurable software systems, we then present techniques for both selection and prioritization of these systems.