Structural Coverage of LTL Requirements for Learning-based Testing

Abstract

Learning-based testing (LBT) is an innovative variant of black-box testing in which test cases are derived by making use of automaton learning and model checking algorithms along with providing Linear Temporal Logic (LTL) requirements of the System Under Test (SUT). There is a scarcity of test coverage metrics for black-box testing in general and LBT in particular. Structural coverage of an LTL requirement is a mechanism that gauges how well a test suite has exercised the structure of the LTL formula. In contrast to the code-driven or model-driven coverage metrics, this coverage provides implementation-independent coverage corresponding to an LTL requirement. This has been defined and used in the literature for black-box testing; however, not for LBT. This paper analyzes and implements the structural coverage criteria for the LTL requirements for evaluating the LBT-generated test suite. We evaluate the structural coverage metrics using the Cruise Control System (CCS) and the ATM systems. The results show that the LBT test suite provides complete structural coverage of the safety LTL requirements in terms of Requirement Coverage (RC), Antecedent Coverage (AC), and Unique First Cause Coverage (UFCC). In the case of liveness LTL requirements, relatively less structural coverage is achieved by the LBT tests, possibly because of the involvement of loops in tests.