Test case generation, selection and coverage from natural language

Abstract

In Model-based Testing (MBT), test cases are automatically generated from a formal model of the system. A disadvantage of MBT is that developers must deal with formal notations. This limitation is addressed in this paper, where use cases are used to model the system. In previous work, we have proposed an automatic strategy for generating test cases from use cases written in a Controlled Natural Language (CNL), which is an English textual notation with a well-defined grammar. Due to its precise syntax, it can be processed and translated into a formal representation for the purpose of automatic test case generation. This paper extends our previous work by proposing a state-based CNL for describing use case control flows enriched with state and data operations. We translate state-based use case descriptions into CSP processes from which test cases can be automatically generated. In addition, we show how a similar notation can be used to specify test selection via the definition of state-based test purposes, which are also translated into CSP processes. Test generation and selection are mechanised by refinement checking using the CSP tool FDR. Despite the fact that we work at a purely process algebraic level to define a test generation strategy, we are able to address model coverage criteria. Particularly, by using FDR, it is possible to have access to the underlying LTS models; we then implemented algorithms for covering events or transitions, possibly combined with selection using test purposes. We also discuss several ways of improving the efficiency of the test generation strategy. As far as we are aware, this integration between an algebraic approach to test case generation with an operational approach for coverage criteria is an original and promising insight. All steps of the strategy are integrated into a tool that provides a GUI for authoring use cases and test purposes described in the proposed CNL, so the formal CSP notation is completely hidden from the test designer. We illustrate our tool and techniques with a running example and a more elaborate case study taken from an industrial setting. • A process algebraic approach to test generation from use cases in natural language. • Integration of coverage criteria into a purely algebraic test generation approach. • Test generation considering coverage of events or transitions of the model. • Complexity analysis and a provably sound optimisation of the test generation approach. • An industrial case study with metrics on the efficiency of the test generation strategy.