Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc., Building Technologies Division

Abstract

The goal of software testing is to find defects before the product is released to the customer. Siemens has a Zero Defect Culture which strives to continuously improve our software testing methods. The current methods use automated testing and manual testing. This paper examines the use of mixed-strength covering arrays, resulting in requiring fewer tests for higher strength coverage. Using the Combinatorial Test Coverage Measurement command line version (CCM) developed by the National Institute of Standards and Technology (NIST), we found practical ways to measure the coverage percentage and the strength of covering arrays, as well as any given test suite. This enabled us to show where we are and the direction we should take with minimal cost improvements. We investigated combinatorial methods, the use of covering arrays for test inputs and test configurations, and methods to combine the two in various ways during the product life cycle. The goal was to craft the leanest and most impactful formal testing and devise optimal configurations for the system under test. In particular, we consider the problem of testing a large number of setup configurations where each configuration requires many tests. Tradeoffs for producing practical test sets are analyzed, with overall and minimum combinatorial coverage measured, demonstrating significant combinatorial coverage improvements while keeping test set size tractable.