Modified Condition/Decision Coverage Research Articles

Automated SC-MCC Test Case Generation using Bounded Model Checking for Safety-Critical Applications

Modified Condition/Decision Coverage (MC/DC) is an important criterion to test the safety-critical applications because it generates test cases in a linear manner from N+1 to 2N, where N is the number of Atomic Conditions in a given predicate. It is desirable in comparison to the exponential test cases produced for Multiple Condition Coverage (MCC), i.e., 2N. However, MCC with Short-Circuit (SC-MCC) is recommended since most of the safety-critical applications are built on the high-level languages that use Short-Circuit evaluation property. Our goal is to demonstrate that, despite the added overhead, the SC-MCC coverage-based test cases have a high error-detection probability compared to that of MC/DC. In this work, we have considered the CBMC tool to generate both the SC-MCC and MC/DC test cases for 80 RERS benchmark programs. Then, by optimizing the traditional Mutation Testing (using the GCOV tool), we computed the Mutation Score (%) to assess the quality of the generated test cases. As per the Mutation analysis, the proposed SC-MCC outperformed MC/DC for over 75% of the 80 RERS programs considered. Additionally, as per the Execution time analysis, the average total time taken to evaluate the MC/DC part of the proposed framework is 3065.98 s, whereas SC-MCC framework evaluation took 2167.70 s. This proves the efficiency of the proposed (SC-MCC) work over the traditional criterion (MC/DC) based work.

Automated Test Data Generation for Embedded System Models Using Combinatorial Testing

Abstract— Embedded systems have been playing very important roles in modern society. They have appeared in every aspect of life from automotive to avionics industries to home appliances, etc. These embedded systems therefore must satisfy high quality requirements. As a consequence, quality assurance for these kinds of systems has attracted much attention and investment from both academic research and industry communities. In developing embedded systems, testing often requires high coverage with different measures with respect to international standards like Decision Coverage (DC), Condition Coverage (CC), Modified Condition/Decision Coverage (MC/DC) respected to ISO 26262. However, it is difficult to generate test cases with high coverage due to the complexity of the embedded system, the larger number of inputs, and the complex continuous signals of inputs. In order to have good test cases with high DC, CC, MC/DC coverage measures, this paper proposes a method to automatically generate test-cases by applying Combinatorial testing technique. The requirements of the inputs of the embedded model will be encoded to the input of Combinatorial testing technique in order to generate test cases that cover all pair values of any two inputs. Experiments on case studies showed that proposed method has coverage result better than that of the random testing method.

BOOMPizer: Minimization and prioritization of CONCOLIC based boosted MC/DC test cases

Recent research evidence indicates that the powerful testing tools, even though generate test inputs automatically for coverage measures, but not up to satisfaction. These tools sometimes achieve high structural coverage, which do not guarantee to have high fault detection ability. These findings lead us to a decisive point that code coverage is merely one factor towards effective test data generation. Thus, we discuss our findings and proposed work on Modified Condition/ Decision Coverage (MC/DC) test case generation and prioritization techniques. This work aims to generate, minimize, and prioritize MC/DC test cases obtained through concolic testing process. This work presents three technical contributions. The first contribution is to propose a greedy algorithm to increase the number of effective test cases for improving MC/DC scores. The second contribution is to minimize the updated test suite size to have only the optimal number of contributing test cases towards forming MC/DC pairs. The third contribution is to prioritize these test cases by considering both their Contribution Index (CI) values and Fault Exposing Potential (FEP) values. The proposed approach is validated by experimenting on eighteen Java programs and achieved on an average ×1.67 times increase in the number of effective test cases that lead to an average increase in MC/DC score by 41.08%. We also achieved on an average 49.00% reduction rate to minimize the test suite size and finally prioritized the test cases, based on their prioritization index values.

A new disjunctive literal insertion fault detection strategy in boolean specifications

AbstractIn fault‐based Boolean expression testing, the main challenge is to generate effective test cases that can detect faults within expressions. Previous studies show that it is hard to detect literal insertion faults, more specifically Disjunctive literal insertion fault (LIF[+]) compared with other faults in Boolean expressions. Researchers have been using different strategies such as multiple near false point coverage (MNFP) and modified condition decision coverage (MCDC) to detect LIF[+] faults. However, these strategies have their own limitations. For example, MNFP can only be applied when the expression is in irredundant disjunctive normal form (IDNF), and MCDC detects a low percentage of LIF[+] faults. In this paper, we propose an abstract syntax tree (AST)‐based test case generation strategy for LIF[+] fault detection that overcomes these limitations. Furthermore, our experimental results indicate that, on average, the test suites satisfying the proposed strategy can detect approximately 97.3% of LIF[+] faults for general form expressions and 89.7% of LIF[+] faults for IDNF expressions, which are 15.6% and 13.8% improvement, respectively, compared to the MCDC test suites. Moreover, the size of the required test suite is smaller than that of MCDC test suite.

Modified condition/decision coverage (MC/DC) oriented compiler optimization for symbolic execution

Symbolic execution is an effective way of systematically exploring the search space of a program, and is often used for automatic software testing and bug finding. The program to be analyzed is usually compiled into a binary or an intermediate representation, on which symbolic execution is carried out. During this process, compiler optimizations influence the effectiveness and efficiency of symbolic execution. However, to the best of our knowledge, there exists no work on compiler optimization recommendation for symbolic execution with respect to (w.r.t.) modified condition/decision coverage (MC/DC), which is an important testing coverage criterion widely used for mission-critical software. This study describes our use of a state-of-the-art symbolic execution tool to carry out extensive experiments to study the impact of compiler optimizations on symbolic execution w.r.t. MC/DC. The results indicate that instruction combining (IC) optimization is the important and dominant optimization for symbolic execution w.r.t. MC/DC. We designed and implemented a support vector machine based optimization recommendation method w.r.t. IC (denoted as auto). The experiments on two standard benchmarks (Coreutils and NECLA) showed that auto achieves the best MC/DC on 67.47% of Coreutils programs and 78.26% of NECLA programs.

Verification and testing of safety-critical airborne systems: A model-based methodology

In this paper, we address the issues of safety-critical software verification and testing that are key requirements for achieving DO-178C and DO- 331 regulatory compliance for airborne systems. Formal verification and testing are considered two different activities within airborne standards and they belong to two different levels in the avionics software development cycle. The objective is to integrate model-based verification and model-based testing within a single framework and to capture the benefits of their cross-fertilization. This is achieved by proposing a new methodology for the verification and testing of parallel communicating agents based on formal models. In this work, properties are extracted from requirements and formally verified at the design level, while the verified properties are propagated to the implementation level and checked via testing. The contributions of this paper are a methodology that integrates verification and testing, formal verification of some safety critical software properties, and a testing method for Modified Condition/Decision Coverage (MC/DC). The results of formal verification and testing can be used as evidence for avionics software certification.

Investigation into the effectiveness of white‐box T‐way testing

An unduly large number of test cases are required for effective testing of programmes containing complex decision statements. In this context, modified condition/decision coverage (MC/DC) testing has been acknowledged to provide effective testing using a test suite whose size is linear in the number of the clauses present in a predicate. MC/DC testing is well-accepted and is mandated by several testing standards. T-way testing is another prominent testing technique that helps to limit the combinatorial explosion of black-box test cases. Its application to white-box testing promises to provide effective testing with a comparatively small number of test cases. The authors empirically investigate the effectiveness of MC/DC testing vis-a-vis white-box pairwise, 3-way and 4-way testing.

Modified condition decision coverage criteria for test suite prioritization using particle swarm optimization

PurposeTest suite prioritization technique is the process of modifying the order in which tests run to meet certain objectives. Early fault detection and maximum coverage of source code are the main objectives of testing. There are several test suite prioritization approaches that have been proposed at the maintenance phase of software development life cycle. A few works are done on prioritizing test suites that satisfy modified condition decision coverage (MC/DC) criteria which are derived for safety-critical systems. The authors know that it is mandatory to do MC/DC testing for Level A type software according to RTCA/DO178C standards. The paper aims to discuss this issue.Design/methodology/approachThis paper provides a novel method to prioritize the test suites for a system that includes MC/DC criteria along with other important criteria that ensure adequate testing.FindingsIn this approach, the authors generate test suites from the input Java program using concolic testing. These test suites are utilized to measure MC/DC% by using the coverage calculator algorithm. Now, use MC/DC% and the execution time of these test suites in the basic particle swarm optimization technique with a modified objective function to prioritize the generated test suites.Originality/valueThe proposed approach maximizes MC/DC% and minimizes the execution time of the test suites. The effectiveness of this approach is validated by experiments on 20 moderate-sized Java programs using average percentage of fault detected metric.

Validating object-oriented software at design phase by achieving MC/DC

This paper deals with a new technique for validating object-oriented software at design phase of project development. There are several modeling diagrams used at design phase of Software Development Life Cycle. But in this paper, we focus on UML Activity Diagram. In our work, first we construct the UML activity diagram for the given system using ArgoUML. Then, the XML (“EXtensible Markup Language”) code is generated for the constructed activity diagram. Next, this XML code is translated to XSD (“XML Schema Definition”) code. This XSD code is given as input to JAXB (“Java Architecture for XML Binding”), which generates the Java template. Then, this Java template is instrumented to a complete Java program with minimal manual effort. Next, we carryout concolic testing of this Java code using jCUTE. This tool generates test cases by taking the Java program as input. Then, the obtained test suite and generated Java source code are inputed into our in-house developed tool named COPECA (COverage PErcentage CAlculator) to calculate MC/DC (Modified Condition/Decision Coverage) score. We have achieved 56.31% MC/DC on doing experiment with fourteen activity diagrams, which is a fair (moderate) achievement compared to the existing work.

Reduced energy consumption for MC/DC testing

After deployment, testing and validation techniques have become an important requirement for the reliability of software. However, these techniques do not measure energy that they consume, which is also an important parameter for software systems with short energy budgets. In this paper, we propose a new energy conservation technique. This novel technique does not affect the coverage percentage, but reduces the energy consumption. Energy conservation is based on test case minimisation technique according to modified condition/decision coverage (MC/DC) requirements. Our experimental study shows that, as compared to existing work, our proposed approach achieves less energy consumption. On an average of five programs, we save 40.8% energy consumption.

Reduced energy consumption for MC/DC testing

After deployment, testing and validation techniques have become an important requirement for the reliability of software. However, these techniques do not measure energy that they consume, which is also an important parameter for software systems with short energy budgets. In this paper, we propose a new energy conservation technique. This novel technique does not affect the coverage percentage, but reduces the energy consumption. Energy conservation is based on test case minimisation technique according to modified condition/decision coverage (MC/DC) requirements. Our experimental study shows that, as compared to existing work, our proposed approach achieves less energy consumption. On an average of five programs, we save 40.8% energy consumption.

Prioritizing MCDC test cases by spectral analysis of Boolean functions

SummaryTest case prioritization aims at scheduling test cases in an order that improves some performance goal. One performance goal is a measure of how quickly faults are detected. Such prioritization can be performed by exploiting the fault exposing potential (FEP) parameters associated to the test cases. The FEP is usually approximated by mutation analysis under certain fault assumptions. Although this technique is effective, it could be relatively expensive compared to the other prioritization techniques. This study proposes a cost‐effective FEP approximation for prioritizing modified condition decision coverage (MCDC) test cases. A strict negative correlation between the FEP of an MCDC test case and the influence value of the associated input condition allows to order the test cases easily without the need of an extensive mutation analysis. The method is entirely based on mathematics and it provides useful insight into how spectral analysis of Boolean functions can benefit software testing.

GECOJAP: A novel source-code preprocessing technique to improve code coverage

Safety critical standards such as DO178B/DO178C/ RTCA (Radio Technical Commission for Aeronautics) mandates coverage based testing in Aerospace applications. These standards mandate Level A certification for Modified Condition/Decision Coverage (MC/DC). To perform exhaustive and rigorous testing, concolic testing is used in the testing phase of the software development life cycle. But, still some concolic testers need to improve their performance, so that they can achieve higher coverage. We present an automated Java code transformation technique that can be used as a front-end to concolic testing tool for achieving high coverage. We have developed our tool using four modules. The tool named GEaring COverage for JAva Program (GECOJAP) for implementation of our approach. The first module shows a source code preprocessing technique called JEX-NCT (Java Exclusive-NOR Code Transformer) that inserts dummy branches according to Modified Condition / Decision Coverage (MC/DC) criterion. The second module represents a concolic tester named jCUTE (an open source tool) we used to generate test cases. The third module presents computation of MC/DC% using the generated test cases and original program. The fourth module shows the speed calculator that measures speed of test case generation, GECOJAP is more powerful and efficient in comparison to the existing techniques in terms of code transformation. Using GECOJAP one can, achieve higher code coverage. Also, GECOJAP results in the time and speed of the test case generation process. Our experimentation on ten Java programs for thirty executions shows that our approach achieves higher Branch Coverage and MC/DC over traditional concolic testers by 13.79% and 19% respectively.

Design verification enhancement of field programmable gate array-based safety-critical I&C system of nuclear power plant

Safety-critical instrumentation and control (I&C) system in nuclear power plant (NPP) implemented on programmable logic controllers (PLCs) plays a vital role in safe operation of the plant. The challenges such as fast obsolescence, the vulnerability to cyber-attack, and other related issues of software systems have currently led to the consideration of field programmable gate arrays (FPGAs) as an alternative to PLCs because of their advantages and hardware related benefits. However, safety analysis for FPGA-based I&C systems, and verification and validation (V&V) assessments still remain important issues to be resolved, which are now become a global research point of interests. In this work, we proposed a systematic design and verification strategies from start to ready-to-use in form of model-based approaches for FPGA-based reactor protection system (RPS) that can lead to the enhancement of the design verification and validation processes. The proposed methodology stages are requirement analysis, enhanced functional flow block diagram (EFFBD) models, finite state machine with data path (FSMD) models, hardware description language (HDL) code development, and design verifications. The design verification stage includes unit test – Very high speed integrated circuit Hardware Description Language (VHDL) test and modified condition decision coverage (MC/DC) test, module test – MATLAB/Simulink Co-simulation test, and integration test – FPGA hardware test beds. To prove the adequacy of the proposed approaches, the design architect that focused on the RPS bistable trip logics which are the safety-critical functions of RPS are designed, analyzed, verified and discussed, using bistable fixed setpoint trip logic algorithms as case study. The results showed that the proposed approaches can enhance the design verification processes alongside the reduction in rigorous V&V tasks of FPGA-based safety-critical I&C system for NPP.

J3 Model: A novel framework for improved Modified Condition/Decision Coverage analysis

In the real-time systems and safety critical domains, software quality assurance adheres to protocols such as DO-178C standard. Regarding these issues, concolic testing generates test cases that can attain high coverage using an augmented approach based on Modified Condition/Decision Coverage (MC/DC). In this paper, we propose a framework to compute MC/DC percentage for test case generation. To achieve an increase in MC/DC, we transform the input Java program, J, into its transformed version, J′, using Java Program Code Transformer (JPCT). Then, we use JCUTE tool to generate test cases. At last, we use Java Coverage Analyzer (JCA) to compute MC/DC percentage. The Java program code transformer adds additional empty nested if-else conditional statements for each decision that causes variation in MC/DC percentage. In later step, these extra conditional statements get stripped-off. This approach resolves some of the bottleneck issues associated with traditional concolic testers. In our experimental study, we have experimented with forty Java programs. We have computed the difference of MC/DC%, for both the scenarios (i.e. with code transformation and without code transformation). Our approach (i.e. with code transformation achieves) 24.09% average increase in MC/DC% over the traditional approach (i.e. without code transformation).

An improved distributed concolic testing approach

Summary Distributed concolic testing (DCT) for complex programs takes a remarkable computational time. Also, the achieved modified condition/decision coverage (MC/DC) for such programs is often inadequate. We propose an improved DCT approach that reduces the computational time and simultaneously enhanced the MC/DC. We have named our approach SMCDCT (scalable MC/DC percentage calculator using DCT). Our experimental study on forty-five C programs indicates 6.62% of average increase in MC/DC coverage. Copyright © 2016 John Wiley & Sons, Ltd.

What to expect of predicates: An empirical analysis of predicates in real world programs

One source of complexity in programs is logic expressions, i.e., predicates. Predicates define much of the functional behavior of the software. Many logic-based test criteria have been developed, including the active clause coverage (ACC) criteria and the modified condition/decision coverage (MCDC). The MCDC/ACC criteria is viewed as being expensive, which motivated us to evaluate the cost of applying these criteria using a basic proxy: the number of clauses. We looked at the frequency and percentage of predicates in 63 Java programs. Moreover, we also compared these Java programs with three programs in the safety-critical domain, in which logic-basic testing is often used. Although around 99% of the predicates within Java programs contain at most three clauses, there is a positive linear correlation between overall measures of size and the number of predicates that have more than three clauses. Furthermore, safety-critical C/C++ programs have more complex predicates than non-safety-critical programs. However, similar to the predicates in non-safety-critical programs, most predicates in safety-critical programs have up to three clauses. We conclude that non-safety-critical and safety-critical programs do not have many complex predicates. Thus, MCDC/ACC is only needed on a small fraction of the predicates.

Model-Based Test-Suite Minimization using Modified Condition/Decision Coverage (MC/DC)

Testing is very expensive for high-assurance software, like commercial aircraft systems, weapon research, weather forecast, earthquake forecast, and software used for safety critical system. A small and simple flaw in the end product can be enough for destroying the entire effort of the developer with a huge unrecoverable damage to the society. For this reason, Federal Aviation Administration’s requirement is that, the test-suites should be comprises of Modified Condition/Decision Coverage (MC/DC) adequate. By using logic coverage criteria lots of flaws can be removed for safety critical software. MC/DC was proposed by NASA, and had been widely accepted in the field of testing. MC/DC is an effective verification technique, and helps to uncover safety faults. It is a challenge to minimize the number of test-suites when there is a partial change in the software. This can be achieved by using models. Unified Modeling Language (UML) not only helps to design software but also plays a vital role in detecting the faults early phase of design and in minimizing the test-suite. Existing test-suite minimization techniques investigated by different researchers may not be effective in minimizing MC/DC-adequate test-suites because they do not consider the complexity of the present software. A new approach for test-suite minimization is presented in this work, using dissimilarity matrix, which can be well fitted with MC/DC. We also present the results generated out of a case study of the test-suite minimization.

Coverage-Based Testing for Service Level Agreements

Service level agreements (SLAs) are typically used to specify rules regarding the consumption of services that are agreed between the providers of the service-based applications (SBAs) and their consumers. An SLA includes a list of terms that contain the guarantees that must be fulfilled during the provisioning and consumption of the services. Since the violation of such guarantees may lead to the application of potential penalties, it is important to assure that the SBA behaves as expected. In this paper, we propose a proactive approach to test SLA-aware SBAs by means of identifying test requirements, which represent situations that are relevant to be tested. To address this issue, we define a four-valued logic that allows evaluating both the individual guarantee terms and their logical relationships. Grounded in this logic, we devise a test criterion based on the modified condition decision coverage (MCDC) in order to obtain a cost-effective set of test requirements from the structure of the SLA. Furthermore by analyzing the syntax and semantics of the agreement, we define specific rules to avoid non-feasible test requirements. The whole approach has been automated and applied over an eHealth case study.

Reasonability of MC/DC for safety-relevant software implemented in programming languages with short-circuit evaluation

Modified condition/decision coverage (MC/DC) is a structural code coverage metric, originally defined in the standard DO-178B, intended to be an efficient coverage metric for the evaluation of the testing process of software incorporating decisions with complex Boolean expressions. The upcoming standard ISO 26262 for safety-relevant automotive systems prescribes MC/DC for ASIL D as a highly recommended coverage metric. One assumed benefit of MC/DC is that it requires a much smaller number of test cases in comparison to multiple condition coverage (MCC), while sustaining a quite high error-detection probability. Programming languages like C, commonly used for implementing software for the automotive domain, are using short-circuit evaluation. For short-circuit evaluation the number of test cases for MCC is much smaller than in a non-short-circuit environment because many redundant test cases occur. We evaluated the trade-off between the number of test cases for MCC and MC/DC for a case study from the automotive domain and observed a very low overhead (only 5 %) for the number of test cases necessary for MCC compared to MC/DC. This motivated an analysis of programs containing decisions where the number and structure of the referring Boolean expressions vary. Our results show that the overhead for a test suite for MCC is on the average only about 35 % compared to MC/DC and the maximum overhead is approximately 100 % (for decisions with up to 5 conditions). This means that a test set for MCC is in the worst case around twice as big as a test set for MC/DC for a program with short-circuit evaluation with maximum 5 conditions. Considering the lower error-detection effectiveness of MC/DC compared to MCC, we conclude with the strong recommendation to use MCC as a coverage metric for testing safety-relevant software (with a limited number of conditions) implemented in programming languages with short-circuit evaluation.