Structural Code Coverage Research Articles

Random or evolutionary search for object‐oriented test suite generation?

SummaryAn important aim in software testing is constructing a test suite with high structural code coverage, that is, ensuring that most if not all of the code under test have been executed by the test cases comprising the test suite. Several search‐based techniques have proved successful at automatically generating tests that achieve high coverage. However, despite the well‐established arguments behind using evolutionary search algorithms (eg, genetic algorithms) in preference to random search, it remains an open question whether the benefits can actually be observed in practice when generating unit test suites for object‐oriented classes. In this paper, we report an empirical study on the effects of using evolutionary algorithms (including a genetic algorithm and chemical reaction optimization) to generate test suites, compared with generating test suites incrementally with random search. We apply the EVOSUITE unit test suite generator to 1000 classes randomly selected from the SF110 corpus of open‐source projects. Surprisingly, the results show that the difference is much smaller than one might expect: While evolutionary search covers more branches of the type where standard fitness functions provide guidance, we observed that, in practice, the vast majority of branches do not provide any guidance to the search. These results suggest that, although evolutionary algorithms are more effective at covering complex branches, a random search may suffice to achieve high coverage of most object‐oriented classes.

Automatic Generation of Coverage tests for System Programs

Software testing is one of the critical activity for the organizations who spend lot of time and cost for the improvement of software quality. Programming testing is characterized as the way toward executing a project with the plan of discovering mistakes. Testing would ensure the correctness and produces reliable software. In order to achieve the quality software, sufficient number of test cases should be designed and tested.To calculate the integrity of test cases and identify that there are no unexpected functionalities, the structural code coverage should be measured like, statement or decision coverage.

Reasonability of MC/DC for safety-relevant software implemented in programming languages with short-circuit evaluation

Modified condition/decision coverage (MC/DC) is a structural code coverage metric, originally defined in the standard DO-178B, intended to be an efficient coverage metric for the evaluation of the testing process of software incorporating decisions with complex Boolean expressions. The upcoming standard ISO 26262 for safety-relevant automotive systems prescribes MC/DC for ASIL D as a highly recommended coverage metric. One assumed benefit of MC/DC is that it requires a much smaller number of test cases in comparison to multiple condition coverage (MCC), while sustaining a quite high error-detection probability. Programming languages like C, commonly used for implementing software for the automotive domain, are using short-circuit evaluation. For short-circuit evaluation the number of test cases for MCC is much smaller than in a non-short-circuit environment because many redundant test cases occur. We evaluated the trade-off between the number of test cases for MCC and MC/DC for a case study from the automotive domain and observed a very low overhead (only 5 %) for the number of test cases necessary for MCC compared to MC/DC. This motivated an analysis of programs containing decisions where the number and structure of the referring Boolean expressions vary. Our results show that the overhead for a test suite for MCC is on the average only about 35 % compared to MC/DC and the maximum overhead is approximately 100 % (for decisions with up to 5 conditions). This means that a test set for MCC is in the worst case around twice as big as a test set for MC/DC for a program with short-circuit evaluation with maximum 5 conditions. Considering the lower error-detection effectiveness of MC/DC compared to MCC, we conclude with the strong recommendation to use MCC as a coverage metric for testing safety-relevant software (with a limited number of conditions) implemented in programming languages with short-circuit evaluation.

A Memetic Algorithm for whole test suite generation

The generation of unit-level test cases for structural code coverage is a task well-suited to Genetic Algorithms. Method call sequences must be created that construct objects, put them into the right state and then execute uncovered code. However, the generation of primitive values, such as integers and doubles, characters that appear in strings, and arrays of primitive values, are not so straightforward. Often, small local changes are required to drive the value toward the one needed to execute some target structure. However, global searches like Genetic Algorithms tend to make larger changes that are not concentrated on any particular aspect of a test case. In this paper, we extend the Genetic Algorithm behind the EvoSuite test generation tool into a Memetic Algorithm, by equipping it with several local search operators. These operators are designed to efficiently optimize primitive values and other aspects of a test suite that allow the search for test cases to function more effectively. We evaluate our operators using a rigorous experimental methodology on over 12,000 Java classes, comprising open source classes of various different kinds, including numerical applications and text processors. Our study shows that increases in branch coverage of up to 53% are possible for an individual class in practice.

Optimizing compilation with preservation of structural code coverage metrics to support software testing

Code-coverage-based testing is a widely-used testing strategy with the aim of providing a meaningful decision criterion for the adequacy of a test suite. Code-coverage-based testing is also mandated for the development of safety-critical applications; for example, the DO178b document requires the application of the modified condition/decision coverage. One critical issue of code-coverage testing is that structural code coverage criteria are typically applied to source code whereas the generated machine code may result in a different code structure because of code optimizations performed by a compiler. In this work, we present the automatic calculation of coverage profiles describing which structural code-coverage criteria are preserved by which code optimization, independently of the concrete test suite. These coverage profiles allow to easily extend compilers with the feature of preserving any given code-coverage criteria by enabling only those code optimizations that preserve it. Furthermore, we describe the integration of these coverage profile into the compiler GCC. With these coverage profiles, we answer the question of how much code optimization is possible without compromising the error-detection likelihood of a given test suite. Experimental results conclude that the performance cost to achieve preservation of structural code coverage in GCC is rather low. Copyright © 2012 John Wiley & Sons, Ltd.

Enhancing structural software coverage by incrementally computing branch executability

Structural code coverage criteria have been studied since the early seventies, and now they are well supported by commercial and open-source tools and are commonly embedded in several advanced industrial processes. Most industrial applications still refer to simple criteria, like statement and branch coverage, and consider complex criteria, like modified condition decision coverage, only rarely and often driven by the requirements of certification agencies. The industrial value of structural criteria is limited by the difficulty of achieving high coverage, due to both the complexity of deriving test cases that execute specific uncovered elements and the presence of many infeasible elements in the code. In this paper, we propose a technique that both generates test cases that execute yet uncovered branches and identifies infeasible branches that can be eliminated from the computation of the branch coverage. In this way, we can increase branch coverage to closely approximate full coverage, thus improving its industrial value. The algorithm combines symbolic analysis, abstraction refinement, and a novel technique named coarsening, to execute unexplored branches, identify infeasible ones, and mitigate the state space explosion problem. In the paper, we present the technique and illustrate its effectiveness through a set of experimental results obtained with a prototype implementation.

Automated Regression Testing using Symbolic Execution

The aim of this paper is to describe a way to construct tests which validate that changes made during software evolution did not introduce regression faults. Developers usually run a new version of the program against the same set of tests. In order to achieve this goal, symbolic execution was used for test input generation and full structural code coverage. Moreover, the extension of symbolic execution was developed to increase the quality of tests. As a result, regression faults were detected in the program. The concept of the technique and an example model are presented. Ill. 7, bibl. 18 (in English; abstracts in English and Lithuanian). http://dx.doi.org/10.5755/j01.eee.112.6.457

Towards Preserving Model Coverage and Structural Code Coverage

Embedded systems are often used in safety-critical environments. Thus, thorough testing of them is mandatory. To achieve a required structural code-coverage criteria it is beneficial to derive the test data at a higher program-representation level than machine code. Higher program-representation levels include, beside the source-code level, languages of domain-specific modeling environments with automatic code generation. For a testing framework with automatic generation of test data this will enable high retargetability of the framework. In this article we address the challenge of ensuring that the structural code coverage achieved at a higher program representation level is preserved during the code generations and code transformations down to machine code. We define the formal properties that have to be fullfilled by a code transformation to guarantee preservation of structural code coverage. Based on these properties we discuss how to preserve code coverage achieved at source-code level. Additionally, we discuss how structural code coverage at model level could be preserved. The results presented in this article are aimed toward the integration of support for preserving structural code coverage into compilers and code generators.

Model Based Testing in Practice at Microsoft

As part of Microsofts Trustworthy Computing [Trustworthy Computing : Reliability URL http://www.microsoft.com/mscorp/twc/reliability/default.mspx] initiative the company has sought many ways to increase reliability. One approach being extensively investigated and used is Model Based Testing. With a Finite State Machine modeling tool (TMT) successfully deployed and in use by many test groups, a need for more powerful and exible modeling has arisen. Several product groups are exploring the use of the Abstract State Machine Language (AsmL) and its associated test tool (AsmL/T). Results from both approaches have shown an increased ability to nd defects earlier, including in the specification and design stages, as well as achieve higher structural code coverage on the actual systems under test.