Specify and measure, cover and reveal: A unified framework for automated test generation

Abstract

• Automatic test input generation is a major topic in software engineering. • We define a new coverage specification mechanism, called labels. • We propose innovative testing techniques for labels. • Label coverage provides a rich framework to handle test objectives. Automatic test input generation (ATG) is a major topic in software engineering, analysis and security. In this paper, we bridge the gap between state-of-the-art white-box ATG techniques, especially Dynamic Symbolic Execution, and the diversity of test objectives that they may be used to cover in practice, including many of those defined by common source-code coverage criteria. We define a new coverage specification mechanism, called labels, for specifying test objectives, and prove it to be both expressive and amenable to efficient automation. We present an efficient approach for detecting – revealing – infeasible (i.e. uncoverable) test objectives expressed as labels. We demonstrate that measuring the achieved coverage can be efficiently performed for labels. Finally, we propose an innovative extension of DSE resulting in an efficient support for label coverage , while the existing naive approach induces an exponential blow-up of the search space. Experiments show that our ATG technique yields very significant savings and confirm the interest of infeasible label detection, enabling to lift DSE to label coverage with only a slight overhead. Overall, we show that label coverage provides the basis of a rich framework allowing one to express and handle test objectives from various contexts in an efficient and generic manner. To illustrate this framework, we describe LTest , an all-in-one testing toolset based on labels and used in the industry, which offers automatic program annotation, ATG, coverage measurement and detection of infeasible test objectives.