Test Coverage Metrics Research Articles

Test Coverage in Microservice Systems: An Automated Approach to E2E and API Test Coverage Metrics

Test coverage is a critical aspect of the software development process, aiming for overall confidence in the product. When considering cloud-native systems, testing becomes complex, as it becomes necessary to deal with multiple distributed microservices that are developed by different teams and may change quite rapidly. In such a dynamic environment, it is important to track test coverage. This is especially relevant for end-to-end (E2E) and API testing, as these might be developed by teams distinct from microservice developers. Moreover, indirection exists in E2E, where the testers may see the user interface but not know how comprehensive the test suits are. To ensure confidence in health checks in the system, mechanisms and instruments are needed to indicate the test coverage level. Unfortunately, there is a lack of such mechanisms for cloud-native systems. This manuscript introduces test coverage metrics for evaluating the extent of E2E and API test suite coverage for microservice endpoints. It elaborates on automating the calculation of these metrics with access to microservice codebases and system testing traces, delves into the process, and offers feedback with a visual perspective, emphasizing test coverage across microservices. To demonstrate the viability of the proposed approach, we implement a proof-of-concept tool and perform a case study on a well-established system benchmark assessing existing E2E and API test suites with regard to test coverage using the proposed endpoint metrics. The results of endpoint coverage reflect the diverse perspectives of both testing approaches. API testing achieved 91.98% coverage in the benchmark, whereas E2E testing achieved 45.42%. Combining both coverage results yielded a slight increase to approximately 92.36%, attributed to a few endpoints tested exclusively through one testing approach, not covered by the other.

An Experimental Evaluation of Path-Based Product Line Integration Testing and Test Coverage Metrics

Product line testing is significant because any faults in a product line platform can lead to widespread impacts on multiple products configured from that platform within a product line. Due to the shared platform, certain testing can be repeatedly performed across different products, leading to unnecessary costs. To enhance quality and reduce costs in product line testing, it is essential to minimize redundant testing of the products in a product line. Because test coverage provides a way to explicitly state the extent to which a software item has been tested, having a clear understanding of test coverage helps avoid unnecessary repetition of tests and ensures that the testing efforts are focused on areas that require attention, ultimately leading to more efficient and effective product line testing. It is necessary to define appropriate test coverage metrics of product line testing that enable testers to identify redundancies in their testing efforts. Path-based integration testing has been proven to be an effective approach to product line integration testing. This paper defines coverage metrics for path-based product line integration testing and demonstrates their effectiveness in preventing redundant testing between platform testing and testing for individual products, while also effectively detecting faults. The experiment results highlight the coverage metrics’ effectiveness in avoiding redundant testing, reducing costs, and covering interfacing across different modules.

Revisiting deep neural network test coverage from the test effectiveness perspective

AbstractMany test coverage metrics have been proposed to measure the deep neural network (DNN) testing effectiveness, including structural coverage and nonstructural coverage. These test coverage metrics are proposed based on the fundamental assumption: They are correlated with test effectiveness. However, the fundamental assumption is still not validated sufficiently and reasonably, which brings question on the usefulness of DNN test coverage. This paper conducted a revisiting study on the existing DNN test coverage from the test effectiveness perspective, to effectively validate the fundamental assumption. Here, we carefully considered the diversity of subjects, three test effectiveness criteria, and both typical and state‐of‐the‐art test coverage metrics. Different from all the existing studies that deliver negative conclusions on the usefulness of existing DNN test coverage, we identified some positive conclusions on their usefulness from the test effectiveness perspective. In particular, we found the complementary relationship between structural and nonstructural coverage and identified the practical usage scenarios and promising research directions for these existing test coverage metrics.

Structural Test Coverage Criteria for Deep Neural Networks

Deep neural networks (DNNs) have a wide range of applications, and software employing them must be thoroughly tested, especially in safety-critical domains. However, traditional software test coverage metrics cannot be applied directly to DNNs. In this paper, inspired by the MC/DC coverage criterion, we propose a family of four novel test coverage criteria that are tailored to structural features of DNNs and their semantics. We validate the criteria by demonstrating that test inputs that are generated with guidance by our proposed coverage criteria are able to capture undesired behaviours in a DNN. Test cases are generated using a symbolic approach and a gradient-based heuristic search. By comparing them with existing methods, we show that our criteria achieve a balance between their ability to find bugs (proxied using adversarial examples and correlation with functional coverage) and the computational cost of test input generation. Our experiments are conducted on state-of-the-art DNNs obtained using popular open source datasets, including MNIST, CIFAR-10 and ImageNet.

FinalFilter: Asserting Security Properties of a Processor at Runtime

In an ideal world, it would be possible to build a provably correct and secure processor. However, the complexity of today's processors puts this ideal out of reach. The complete verification of a modern processor remains intractable. Statically verifying even a simple security property -for example, privilege escalation never occurs -remains beyond the state of the art in formal verification. Testing can complement formal verification methods, yet testing is incomplete and bugs in the hardware that leave it vulnerable continue to elude test suites. Further, a crafty malicious actor can evade typical testing coverage metrics. Recent efforts, including that of three of the authors, have explored the use of static analysis on the design files (e.g., hardware description level source code or gate-level netlists) to find suspicious circuitry. These techniques rely on heuristics to define patterns that indicate a likely trojan and then search for instances in the design that match the pattern. However, malicious circuitry that does not match the pattern will be missed, as will inadvertent bugs that open vulnerabilities. By the time the weakness is uncovered, the hardware is already in the end user's hands and vulnerable to attack. In the absence of a full proof of correctness, what is needed is a final filter: a runtime verification technique that works-postdeployment-to detect and respond to security property violations as they occur during execution. In this article, we make the case for final filters using our tool, FinalFilter, as a case study.

An integration test coverage metric for Java programs

A large number of test coverage metrics have been proposed in the context of unit and system testing of object-oriented programs. However, structural coverage metrics for integration testing have scarcely been reported. On the other hand, interface bugs are known to be one of the most insidious. In this context, we propose a suite of integration test coverage metrics based on the coverage of data and control dependency edges of the JSysDG (Java System Dependency Graph). We also define a metric that additionally uses the state machine models of the given classes to be integrated. We present our experimental results and a comparison of our metrics with related metrics.

Proposal for A Structural Integration Test Coverage Metric for Object-Oriented Programs

Though a large number of test coverage metrics have been proposed in the context of unit and system testing of object oriented programs, structural coverage metrics for integration testing have scarcely been reported. In this context, we propose an integration test coverage metric based on the coverage of the data and control dependency edges of the JSysDG (Java System Dependency Graph).

Agile deployment and code coverage testing metrics of the boot software on-board Solar Orbiter's Energetic Particle Detector

In this day and age, successful embedded critical software needs agile and continuous development and testing procedures. This paper presents the overall testing and code coverage metrics obtained during the unit testing procedure carried out to verify the correctness of the boot software that will run in the Instrument Control Unit (ICU) of the Energetic Particle Detector (EPD) on-board Solar Orbiter. The ICU boot software is a critical part of the project so its verification should be addressed at an early development stage, so any test case missed in this process may affect the quality of the overall on-board software. According to the European Cooperation for Space Standardization ESA standards, testing this kind of critical software must cover 100% of the source code statement and decision paths. This leads to the complete testing of fault tolerance and recovery mechanisms that have to resolve every possible memory corruption or communication error brought about by the space environment. The introduced procedure enables fault injection from the beginning of the development process and enables to fulfill the exigent code coverage demands on the boot software.

A Time-Optimized Scheme Towards Analysis of Channel-Shorts in on-Chip Networks

With the continuous growth in technology, the role of nano-electronic systems is rapidly expanding in every facet of modern life. Subsequently, the demand of high performance computations and communications by many applications in order to interact with environment and users has become a compulsion. Traditional integrated circuit-based systems, e.g., bus-based systems-on-chip (SoCs), as a consequence are turned into networks- on-chips (NoCs) because of the communication bottleneck by the SoCs. Though, the NoCs visibly move the concern and meet the requirements by the applications but fabricating such NoCs without any defect in channels or logics has eventually evolved into a major challenge. Indeed, one of the major demand nowadays for such systems is to design a time-efficient mechanism for the pre- and post-manufacturing testing of NoC functional components, in order to guarantee outgoing quality while not sacrificing yield and reliability. This paper presents a cost effective and distributed on-line test mechanism that addresses channel-shorts in NoC-based systems. The proposed mechanism detects and diagnoses both intra- and inter-channel short faults. The test mechanism additionally targets transient and stuck-at faults in channels. A convenient test scheduling and energy model are presented. The scheduling scheme offers constant test time with little hardware area and performance overheads on the NoCs irrespective of their topologies and sizes. The proposed test solution is evaluated with fault injection campaign in channels of a set of NoC architectures. Fault simulation detects all modeled faults in channels resulting 100% test and fault coverage metrics. On-line evaluation of the proposed solution reveals insights of the effect of channel-shorts on various performance metrics at large amount of traffic. Furthermore, the proposed solution improves various quality metrics on the set of NoCs, for instance, the test area overhead is reduced up to 28% while the proposed model becomes 16 × faster. Also, improvement on the performance overhead is noticed. The packet latency for instance, is improved by 14.98---50.67% while packet flit energy consumption is reduced by 6.83---43.89%. Such improvements are found to grow with NoC size.

Simulation environment based on the Universal Verification Methodology

Universal Verification Methodology (UVM) is a standardized approach of verifying integrated circuit designs, targeting a Coverage-Driven Verification (CDV). It combines automatic test generation, self-checking testbenches, and coverage metrics to indicate progress in the design verification. The flow of the CDV differs from the traditional directed-testing approach. With the CDV, a testbench developer, by setting the verification goals, starts with an structured plan. Those goals are targeted further by a developed testbench, which generates legal stimuli and sends them to a device under test (DUT). The progress is measured by coverage monitors added to the simulation environment. In this way, the non-exercised functionality can be identified. Moreover, the additional scoreboards indicate undesired DUT behaviour. Such verification environments were developed for three recent ASIC and FPGA projects which have successfully implemented the new work-flow: (1) the CLICpix2 65 nm CMOS hybrid pixel readout ASIC design; (2) the C3PD 180 nm HV-CMOS active sensor ASIC design; (3) the FPGA-based DAQ system of the CLICpix chip. This paper, based on the experience from the above projects, introduces briefly UVM and presents a set of tips and advices applicable at different stages of the verification process-cycle.

Business Process Quality Measurement using Advances in Static Code Analysis

Business process models play an important role in the life of a company. Resemblances between software programs and business processes inspired several researchers to adapt software metrics from the field of static code analysis to help designers to build more effective and understandable processes. This paper aims to add recent advances in software quality measurement such as benchmarking and ISO/IEC 25010 standard based quality models to business process quality measurement. These techniques were proved to be very useful in software engineering both for managers and developers; moreover, they can be easily adopted to business process workflows. We focused on a specific type of flowchart called event-driven process chain (EPC), because in an EPC the activities are very often managed by software systems and our assumption is that the quality of these software systems affects the quality of the EPC itself. The presented business process quality model also uses the quality and test coverage metrics of these software systems besides business process metrics.

USING RULE STRUCTURE TO EVALUATE THE COMPLETENESS OF RULE-BASED SYSTEM TESTING: A CASE STUDY

Rule-based systems are typically tested using a set of inputs which will produce known outputs. However, one does not know how thoroughly the software has been exercised. Traditional test-coverage metrics do not account for the dynamic data-driven flow of control in rule-based systems. Our literature review found that there has been little prior work on coverage metrics for rule-based systems. This paper proposes test-coverage metrics for rule-based systems derived from metrics defined by prior work, and presents an industrial scale case study. We conducted a case study to evaluate the practicality and usefulness of the proposed metrics. The case study applied the metrics to a system for computational fluid-dynamics models based on a rule-based application framework. These models were tested using a regression-test suite. The data-flow structure built by the application framework, along with the regression-test suite, provided case-study data. The test suite was evaluated against three kinds of coverage. The measurements indicated that complete coverage was not achieved, even at the lowest level definition. Lists of rules not covered provided insight into how to improve the test suite. The case study illustrated that structural coverage measures can be utilized to measure the completeness of rule-based system testing.

C0 COVERAGE-MEASURE AND TESTING-DOMAIN METRICS BASED ON A SOFTWARE RELIABILITY GROWTH MODEL

Generally, a software development manager is performing the testing-progress management by using various CASE tools. These CASE tools support testing activities of Windows applications or embedded software systems. Then, by using various testing coverage-measures, the testing-progress management can be recognized visually. However, in the system-testing phase, it is impossible to measure them of software systems which have been becoming large-scale and complicated recently. Therefore, the manager has to determine the testing termination time in consideration of the convergence situation of the cumulative number of detected faults and the prespecified delivery time to the users. These determination method have ambiguity based on the manager's experience or intuition. In this paper, we investigate the relationship between the testing-domain rate derived from a testing-domain dependent software reliability growth model and the testing-path coverage. Then, we show that the testing-domain rate, which is defined as the increasing ratio of the testing-paths in the modules and functions in the software system to be influenced by executed test-cases, is useful as an alternative measure of testing-coverage metrics. Further, this model is applied to the fault data observed in an actual software project, and the comparisons of goodness-of-fit with the conventional software reliability growth models are performed. Finally, in numerical illustrations, we discuss the software reliability analysis and the usefulness of the testing-domain rate, and propose a new testing-path coverage rate.

Orthogonal software testing: Taguchi methods in software unit and subsystem testing

Suggests that, in order to detect and correct software defects as early as possible, identifying and generating more defect‐sensitive test cases for software unit and subsystem testing is one solution. Proposes an orthogonal software testing approach based on the quality optimization techniques, Taguchi methods. This orthogonal approach treats the input parameters of a software unit or subsystem as design factors in an orthogonal arrays, and stratifies input parameter domains into equivalent classes to form levels of factors. Describes how test cases are generated statistically for each trial of factorial orthogonal experiments. The adequacy of the generated test cases can be validated by examining testing coverage metrics. The results of test case executions can be analysed in order to find the sensibility of test cases for detecting defects, to generate more effective test cases in further testing, and to help locate and correct defects in the early stage of testing.

Systems integration throughout the early life cycle

This paper presents various methodologies and approaches for performing systems integration and testing throughout the early and middle stages of the development life cycle. It discusses techniques for estimation and management of the integration early life cycle. Prevention and early detection of errors significantly reduces the cost of failure in a system. The systematic use of quality gates throughout the life cycle reduces the propagation of problems. The appropriate use of documentation standards ensures information and knowledge is disseminated to all those who require it. These and other techniques for effective control of integration and test are discussed, together with techniques for measuring the effectiveness of systems integration, such as test coverage analysis, metrics, and post-delivery assessment.

Implementation-based analysis and testing of Prolog programs

In this paper, we describe the PROTest II (PROlog Test Environment, Version 2) system to test logic programs in an interactive support environment. Logic programs are augmented with declarative information about the types and modes of the arguments of a predicate. Modes correspond to in, out, and in-out parameters. With this information PROTest II statically checks the types of Prolog programs, generates test cases, executes Prolog programs, and produces reports summarizing results including information about new test coverage metrics. Thus, PROTest II enables both static analysis and dynamic testing uniformly using a Prolog-based test language DTL/1. The strength of PROTest II stems from its idea of defining coverage in real logic programming terms, rather than adapting imperative programming ideas.

Ct coverage ? initial results

Software testing techniques based upon automated analysis of control flow are useful for improving software reliability. The fundamental types of control flow analysis, in ascending order of effectiveness, are statement, branch, and path coverage. Automated tools that perform branch coverage analysis are now accepted practice and researchers are exploring the area that exists between branch and path analysis. Path testing is complicated by the huge number of paths in ordinary programs and by anomalies, such as infeasible paths. The Ct testing strategy is a method for obtaining a manageable set of path classes by specifying a minimum iteration count k. The Ct test coverage metric is a measurement of the proportion of Ct path classes that are exercised within a program. This paper describes an initial experiment in which a special, automated tool was used to evaluate the Ct coverage of tests for a C program of significant size. The results of the study showed that high values of branch coverage may not necessarily imply high path coverage, that approximately 15% of the functions analysed were ‘too complex’ to analyse effectively, and that the majority of the remaining functions achieved on the order of 11% Ct k=1 coverage. Experimental work in the attainment of high Ct coverage suggests the development of a new, more efficient software testing strategy, which combines Ct path and data flow analysis.

Comparison of Structural Test Coverage Metrics

Empirically comparing structural test coverage metrics reveals that test sets that satisfy one metric are likely to satisfy metric as we.