Structural Coverage Metrics Research Articles

An overview of structural coverage metrics for testing neural networks

Deep neural network (DNN) models, including those used in safety-critical domains, need to be thoroughly tested to ensure that they can reliably perform well in different scenarios. In this article, we provide an overview of structural coverage metrics for testing DNN models, including neuron coverage, k-multisection neuron coverage, top-k neuron coverage, neuron boundary coverage, strong neuron activation coverage and modified condition/decision coverage. We evaluate the metrics on realistic DNN models used for perception tasks (LeNet-1, LeNet-4, LeNet-5, ResNet20) including networks used in autonomy (TaxiNet). We also provide a tool, DNNCov, which can measure the testing coverage for all these metrics. DNNCov outputs an informative coverage report to enable researchers and practitioners to assess the adequacy of DNN testing, to compare different coverage measures, and to more conveniently inspect the model’s internals during testing.

An integration test coverage metric for Java programs

A large number of test coverage metrics have been proposed in the context of unit and system testing of object-oriented programs. However, structural coverage metrics for integration testing have scarcely been reported. On the other hand, interface bugs are known to be one of the most insidious. In this context, we propose a suite of integration test coverage metrics based on the coverage of data and control dependency edges of the JSysDG (Java System Dependency Graph). We also define a metric that additionally uses the state machine models of the given classes to be integrated. We present our experimental results and a comparison of our metrics with related metrics.

Proposal for A Structural Integration Test Coverage Metric for Object-Oriented Programs

Though a large number of test coverage metrics have been proposed in the context of unit and system testing of object oriented programs, structural coverage metrics for integration testing have scarcely been reported. In this context, we propose an integration test coverage metric based on the coverage of the data and control dependency edges of the JSysDG (Java System Dependency Graph).

Bidirectional Symbolic Analysis for Effective Branch Testing

Structural coverage metrics, and in particular branch coverage, are popular approaches to measure the thoroughness of test suites. Unfortunately, the presence of elements that are not executable in the program under test and the difficulty of generating test cases for rare conditions impact on the effectiveness of the coverage obtained with current approaches. In this paper, we propose a new approach that combines symbolic execution and symbolic reachability analysis to improve the effectiveness of branch testing. Our approach embraces the ideal definition of branch coverage as the percentage of executable branches traversed with the test suite, and proposes a new bidirectional symbolic analysis for both testing rare execution conditions and eliminating infeasible branches from the set of test objectives. The approach is centered on a model of the analyzed execution space. The model identifies the frontier between symbolic execution and symbolic reachability analysis, to guide the alternation and the progress of bidirectional analysis towards the coverage targets. The experimental results presented in the paper indicate that the proposed approach can both find test inputs that exercise rare execution conditions that are not identified with state-of-the-art approaches and eliminate many infeasible branches from the coverage measurement. It can thus produce a modified branch coverage metric that indicates the amount of feasible branches covered during testing, and helps team leaders and developers in estimating the amount of not-yet-covered feasible branches. The approach proposed in this paper suffers less than the other approaches from particular cases that may trap the analysis in unbounded loops.

Optimizing compilation with preservation of structural code coverage metrics to support software testing

Code-coverage-based testing is a widely-used testing strategy with the aim of providing a meaningful decision criterion for the adequacy of a test suite. Code-coverage-based testing is also mandated for the development of safety-critical applications; for example, the DO178b document requires the application of the modified condition/decision coverage. One critical issue of code-coverage testing is that structural code coverage criteria are typically applied to source code whereas the generated machine code may result in a different code structure because of code optimizations performed by a compiler. In this work, we present the automatic calculation of coverage profiles describing which structural code-coverage criteria are preserved by which code optimization, independently of the concrete test suite. These coverage profiles allow to easily extend compilers with the feature of preserving any given code-coverage criteria by enabling only those code optimizations that preserve it. Furthermore, we describe the integration of these coverage profile into the compiler GCC. With these coverage profiles, we answer the question of how much code optimization is possible without compromising the error-detection likelihood of a given test suite. Experimental results conclude that the performance cost to achieve preservation of structural code coverage in GCC is rather low. Copyright © 2012 John Wiley & Sons, Ltd.

Enhancing IP cores specifications using hierarchical composition and set theory

During the functional verification, complex interactions between multiple blocks that compose an Intellectual Property (IP) core can reveal hard-to-find bugs. Functional verification specifications must be precise to assure these interactions occur during the simulation. In this work, we are proposing a technique for improving the functional verification specification of individual blocks, preserving the occurrence of these interaction scenarios in the composition phase. Our approach was implemented for the VeriSC methodology, a SystemC-based functional verification methodology. After each block that composes the IP core was stand-alone verified, we exploit the composition phase using set theory to increase the coverage numbers and to justify why some of these numbers cannot, or need not, reach 100%. By applying our approach in a MPEG 4 video decoder design, we show how our work can save functional verification time during the hierarchical composition. Using mutation based-tests, we demonstrate that our work can contribute to error detection. Furthermore, we demonstrate the effectiveness of our approach with regard to traditional structural coverage metrics, such as line coverage and branch coverage.

Comparison of Structural Test Coverage Metrics

Empirically comparing structural test coverage metrics reveals that test sets that satisfy one metric are likely to satisfy metric as we.