Controlled Experiments Comparing Black-box Testing Strategies for Software Product Lines

Abstract

SPL testing has been considered a challenging task, mainly due to the diversity of products that might be generated from an SPL. To deal with this problem, techniques for specifying and deriving product specific functional test cases have been proposed. However, there is little empirical evidence of the benefits and drawbacks of these techniques. To provide this kind of evidence, in a previous work we conducted an empirical study that compared two design techniques for black-box manual testing, a generic technique that we have observed in an industrial test execution environment, and a product specific technique whose functional test cases could be derived using any SPL approach that considers variations in functional tests. Besides revisiting the first study, here we present a second study that reinforce our findings and brings new insights to our investigation. Both studies indicate that specific test cases improve test execution productivity and quality.