Test Suite Reduction Research Articles

Test Suite Minimization using Hybrid Algorithm for GA generated Test Cases

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible. As the software evolves the size of test suites also grows. When the no of test cases generated are more, obviously size of the test suite will also be more. So the testing time is to be minimized by reducing the execution time of the algorithm used for test data generation and also by introducing minimization procedure for test suite reduction. Due to limited resources and timing constraints for testing, test suite minimization techniques are needed to eliminate redundant test cases as possible. By considering multiple objectives rather than the coverage alone, the test cases are being generated which satisfies the testing requirements. Most of the existing techniques are code-based. In this article we present an approach by modifying an existing heuristic for test suite minimization. Genetic algorithm has been used for random test data generation and the output of GA is given to the minimization procedure for reducing the total no of generated test cases, collectively named as Hybrid Algorithm (HA). The results are satisfactory and show significant improvements in reducing test suite size with minimum execution time. Experiments have been done for simple to medium complexity java programs taken from SIR and execution time is reduced to 5,685ms for a test set. The results are compared with existing method Mutant Gene Algorithm and size of test suite is minimized upto 13.6% using Hybrid Algorithm.

소프트웨어 취약성 평가를 위한 길이기반 파일 퍼징 테스트 슈트 축약 알고리즘

ABSTRACT Recently, automated software testing methods such as fuzzing ha ve been researched to find software vulnerabilities. The purpose of fuzzing is to disclose software vulnerabilities by p roviding a software with malformed data. In order to increase t he probability of vulnerability discovery by fuzzing, we must solv e the test suite reduction problem because the probability depends on the test case quality. In this paper, we propose a n ew method to solve the test suite reduction problem which is suitable for the long test case such as file. First, we suggested the length of test case as a measure in addition to old measures such as coverage and redundancy. Next we designed a test suite reduction algorithm using the new measure. In the experimental results, the proposed algorithm showed better perf ormance in the size and length reduction ratio of the test suit e than previous studies. Finally, results from an empirical study suggested the viability of our proposed measure and algorithm for file fuzzing.Keywords: Test Suite Reduction, File Fuzzing, Software Vulnerability접수일(2013년 3월 6일), 게재확정일(2013년 3월 18일)†주저자, mirr1004@ensec.re.kr‡교신저자, ymkim@jnu.ac.kr

A test-suite reduction approach to improving fault-localization effectiveness

In order to improve the effectiveness of fault localization, researchers are interested in test-suite reduction to provide suitable test-suite inputs. Different test-suite reduction approaches have been proposed. However, the results are usually not ideal. Reducing the test-suite improperly or excessively can even negatively affect fault-localization effectiveness. In this paper, we propose a two-step test-suite reduction approach to remove the test cases which have little or no effect on fault localization, and improve the distribution evenness of concrete execution paths of test cases. This approach consists of coverage matrix based reduction and path vector based reduction, so it analyzes not only the test cases coverage but also the concrete path information. We design and implement experiments to verify the effect of our approach. The experimental results show that our reduced test-suite can improve fault-localization effectiveness. On average, our approach can reduce the size of a test-suite in 47.87% (for Siemens programs) and 23.03% (for space program). At the same time, on average our approach can improve the fault-localization effectiveness, 2.12 on Siemens programs and 0.13 on space program by Tarantula approach.

Efficient Path Selection Strategy Based on Static Analysis for Regression Testing

the changes made to the software system for enhancement. The most common approach to this problem is simply execute the existing test cases in the test suite; i.e. retest all approach, a systematically selected subset is chosen to be reuse, then substantial resources will be saved, due to the limited size of the selected test data (5), (6). There are distinctions between classes of techniques of regression testing: Test suites minimization techniques can be used to reduce test suite by eliminating redundant test suites. In safe approaches, every test case is selected that exercised any program element which could be affected by a program modification. In coverage approaches, test cases are selected which met some structural criteria. In this approach, select a single test case satisfying each coverage requirement introduced by the criterion, minimization techniques work like coverage approaches (7).

A Test-Suite Reduction Based on DU-Chain Requirements Optimization

A Test-Suite Reduction Based on DU-Chain Requirements Optimization

Regression test suite selection using dependence analysis

SUMMARYDependence analysis on an Extended Finite State Machine representation of the requirements of a system under test identifies various types of control and data dependencies between transitions caused by a set of modifications on the requirements. These particular types of dependencies capture the effects of the modifications, that is, their direct effects on the changed parts of the system and their side effects on the unchanged parts of the system. Recent work on model‐based regression testing shows that dependencies capturing direct effects and side effects of the changes made on the requirements can be used for regression test suite (RTS) reduction (reducing the size of a given test suite by eliminating redundancies), for RTS prioritization (ordering test cases in a given test suite for early fault detection), or for RTS generation (designing a test suite covering the identified dependencies). This paper proposes an additional use of such dependencies, namely, RTS selection, which is the process of selecting a subset of a given test suite to form an RTS by considering the coverage of dependencies related to the effects of the modifications. The dependencies marked during this process as uncovered provide a basis for augmenting an (incomplete) RTS with test cases covering uncovered dependencies. Copyright © 2012 John Wiley & Sons, Ltd.

Improving the effectiveness of test suite reduction for user-session-based testing of web applications

ContextTest suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a reduced test suite can be time consuming. ObjectiveWe propose ordering the tests in a reduced suite to increase its rate of fault detection. The ordered reduced test suite can be executed in time constrained situations, where, even if test execution is stopped early, the best test cases from the reduced suite will already be executed. MethodIn this paper, we present several approaches to order reduced test suites using experimentally verified prioritization criteria for the domain of web applications. We conduct an empirical study with three subject applications and user-session-based test cases to demonstrate how ordered reduced test suites often make a practical contribution. To enable comparison between test suites of different sizes, we develop Mod_APFD_C, a modification of the traditional prioritization effectiveness measure. ResultsWe find that by ordering the reduced suites, we create test suites that are more effective than unordered reduced suites. In each of our subject applications, there is at least one ordered reduced suite that outperforms the best unordered reduced suite and the best prioritized original suite. ConclusionsOur results show that when a tester does not have enough time to execute the entire reduced suite, executing an ordered reduced suite often improves the rate of fault detection. By coupling the underlying system’s characteristics with observations from our study on the criteria that produce the best ordered reduced suites, a tester can order their reduced test suites to obtain increased testing effectiveness.

A Test Suite Reduction Technique for Partial Coverage of Test Requirements

During software system development,one has to face with frequent local updates and partial fault corrections,where selective-form regression testing is required.To save its test effort,one needs to solve the partial coverage test suite reduction problem.Under selective-form regression testing,this paper formally defines the multi-objective test suite reduction problem,and develops the HATS algorithm to solve the problem.The HATS conforms to the heuristic greedy search framework,and defines the weighting factor α to balance the two objectives: one is to reduce the test suite size;and the other is to avoid coverage of irrelevant test requirements.The experiment results show that,compared with current test suite reduction techniques and algorithms,with proper setting of the factor α,the HATS can reduce the test suite size,cut down the coverage of irrelevant test requirements,while less compromise the fault detection ability.Original contributions of this paper include two aspects: First defined the multi-objective test suite reduction problem under selective-form regression testing;Second designed the HATS algorithm which used fewer test cases to maintain the fault detection effectiveness.

An automated approach to reducing test suites for testing retargeted C compilers for embedded systems

Abstract With widespread use of embedded processors, there is an increasing need to develop compilers for them in a timely manner. Retargeting has been an effective approach to constructing new compilers by modifying the back-end of existing compilers. An efficient testing method for retargeted compilers using intermediate codes is proposed in this paper. The concepts of the intermediate-code-based approach are described, and it is demonstrated that the proposed approach is efficient and effective enough. We have developed an automated tool, PLOOSE, which generates test suites based on source-code coverage criteria and then reduces them based on intermediate-code coverage. In addition, case studies are presented which reveal that a test suite based on source codes can be considerably reduced using intermediate codes. Moreover, by mutation analysis, it was found that the size of the test suite has been reduced by over 90% on average, but the fault detection capability of the reduced test suites is still approximately 80% of that of the original test suites. In particular, the proposed reduction method can be useful for testing compilers in the early stage of the development of retargeted compilers.

A Constraint-based Test Suite Reduction Method for Conservative Regression Testing

In regression testing, an important problem is how to select a smaller size of test set for execution. We present a novel constraint-oriented test suite reduction method for conservative regression testing by which we mean that all bugs discovered must be revealed by the reduced test suite. A test constraint for a bug is Boolean formulas defined over the input variables of program under test. The reduced test constraints for a pool of bugs are constructed using the subsumption relationship between test constraint conditions. Test case selection is based on the reduced test constraint set. A test case is selected into the test suite when and only when it satisfies one or more test constraints. The selection process is completed when all test constraint conditions are satisfied by the selected test cases. An empirical study is conducted and the experimental results show that our method can significantly save efforts for the conservative regression testing.

Incremental Model-based Test Suite Reduction with Formal Concept Analysis

Abstract —Test scenarios can be derived based on some system models for requirements validation purposes. Model-based test suite reduction aims to provide a smaller set of test scenarios which can preserve the original test coverage with respect to some testing criteria. We are proposing to apply Formal Concept Analysis (FCA) in analyzing the association between a set of test scenarios and a set of transitions specified in a state machine model. By utilizing the properties of concept lattice, we are able to determine incrementally a minimal set of test scenarios with adequate test coverage. Keywords —Test Suite Reduction, Model-based Testing, State Machine Model, Formal Concept Analysis 1. I NTRODUCTION Test scenarios can be derived based on some system models for requirements validation pur-poses. Model-based testing [1, 16] refers to deriving a suite of test scenarios from a model that represents the behavior of a software system. In particular, state machine model has been widely used for this purpose in testing event-driven, reactive systems, and embedded software systems [1, 2]. State machine model can be used to specify the dynamic perspective of a system and its interactions with the users through sequences of transitions. The sequences of transitions can form a set of test scenarios for validation of functional requirements by test engineers and end users. However, since cycles in the state machine model may lead to an infinite number of test scenarios, exhaustive testing is usually not possible. Moreover, many test scenarios are part of some other test scenarios and thus lead to redundancy in the test suite. Model-based test suite reduction can be applied in this situation and derive a smaller set of test scenarios which still preserves the original test coverage with respect to some testing criteria. A default criterion of adequate testing with a state machine model is all-transition coverage criterion [1, 11, 16], which means each transition specified in the state machine model should be triggered at least once by executing the test scenarios. In this paper, we shall describe an incremental approach for reducing model-based test suite using Formal Concept Analysis (FCA) [6]. FCA is a mathematical technique for formulating concepts in terms of a set of formal objects and their associated formal attributes, and providing

A usable selection range standard based on test suite reduction algorithms

In this paper, we analyze the features and distinctions of 6 classical algorithms: greedy algorithm (G), greedy evolution algorithm (GE), heuristics algorithm (H), greedy heuristic G (GRE), integer linear programming algorithm (ILP) and genetic algorithm (GA) to ensure the main influencing factors—the performance of algorithms and the running time of algorithms. What’s more, we would not only present a research design that aims at gaining deeper understanding about the algorithm classification and its function as well as their distinction, but also make an empirical study in order to obtain a practical range standard that can guide the selection of reduction algorithms. When the size of a test object (product of test requirements and test cases) is smaller than 2000×2000, G algorithm is the commonly recommended algorithm. With the growth of test size, the usage of GE and GRE becomes more general.

A Strategy for Automatic Quality Signing and Verification Processes for Hardware and Software Testing

We propose a novel strategy to optimize the test suite required for testing both hardware and software in a production line. Here, the strategy is based on two processes: Quality Signing Process and Quality Verification Process, respectively. Unlike earlier work, the proposed strategy is based on integration of black box and white box techniques in order to derive an optimum test suite during the Quality Signing Process. In this case, the generated optimal test suite significantly improves the Quality Verification Process. Considering both processes, the novelty of the proposed strategy is the fact that the optimization and reduction of test suite is performed by selecting only mutant killing test cases from cumulating t-way test cases. As such, the proposed strategy can potentially enhance the quality of product with minimal cost in terms of overall resource usage and time execution. As a case study, this paper describes the step-by-step application of the strategy for testing a 4-bit Magnitude Comparator Integrated Circuits in a production line. Comparatively, our result demonstrates that the proposed strategy outperforms the traditional block partitioning strategy with the mutant score of 100% to 90%, respectively, with the same number of test cases.

A Tester-Assisted Methodology for Test Redundancy Detection

Test redundancy detection reduces test maintenance costs and also ensures the integrity of test suites. One of the most widely used approaches for this purpose is based on coverage information. In a recent work, we have shown that although this information can be useful in detecting redundant tests, it may suffer from large number of false-positive errors, that is, a test case being identified as redundant while it is really not. In this paper, we propose a semiautomated methodology to derive a reduced test suite from a given test suite, while keeping the fault detection effectiveness unchanged. To evaluate the methodology, we apply the mutation analysis technique to measure the fault detection effectiveness of the reduced test suite of a real Java project. The results confirm that the proposed manual interactive inspection process leads to a reduced test suite with the same fault detection ability as the original test suite.

Regression test suite reduction based on SDL models of system requirements

AbstractThis paper proposes a model‐based regression test suite reduction method. The proposed method considers an SDL model representing the requirements of a system under test and a set of modifications on this model, applies dependence analysis to identify interaction patterns related to each type of modifications, i.e., adding, deleting, and changing transitions in the SDL model, and reduces the size of a given regression test suite by examining interaction patterns covered by each test case in the test suite. Results of empirical studies are reported. Copyright © 2009 John Wiley & Sons, Ltd.

A method of test-suite reduction for regression integration testing

This paper describes a method of test-suite reduction for regression testing of the interaction between two software modules based on a model of module interaction on a test suite and subsequent filtering of the suite using the constructed model. The interaction model is constructed in terms of sequences of interface functions of the modules being integrated called during the software run.

재겨냥성 C 컴파일러를 위한 테스트 집합 생성 시스템

ABSTRACT With the increasing adoption of embedded processors, the need of developing compilers for the embedded processors with timely manner is also growing. Retargeting has been adopted as a viable approach to constructing new compilers by modifying the back-end of an existing compiler. This paper proposes a test suite generation system for testing retargetable C compilers. The proposed system generates the test suite using the grammar coverage concept. Generally, the size of the test suite satisfying the grammar coverage of the source language is very large. Hence, the proposed system also provides the facility to reduce the size of the test suite. According to the experimental result, the reduced test suite can detect 75% of the compiler faults detected by the original test suite though the size of the reduced test suite is only 10% of that of the original test suite in average. This result indicates that the reduction technique proposed in this paper can be effectively used in the prior phase of the development procedure of the embedded compilers. Keywords:Compiler Testing, Retargetable C Compiler, Embedded System, Intermediate Code

Analysis of test suite reduction with enhanced tie-breaking techniques

Test suite minimization techniques try to remove redundant test cases of a test suite. However, reducing the size of a test suite might reduce its ability to reveal faults. In this paper, we present a novel approach for test suite reduction that uses an additional testing criterion to break the ties in the minimization process. We integrated the proposed approach with two existing algorithms and conducted experiments for evaluation. The experiment results show that our approach can improve the fault detection effectiveness of reduced suites with a negligible increase in the size of the suites. Besides, under specific conditions, the proposed approach can also accelerate the process of minimization.

Analysing the effectiveness of rule-coverage as a reduction criterion for test suites of grammar-based software

The term grammar-based software describes software whose input can be specified by a context-free grammar. This grammar may occur explicitly in the software, in the form of an input specification to a parser generator, or implicitly, in the form of a hand-written parser. Grammar-based software includes not only programming language compilers, but also tools for program analysis, reverse engineering, software metrics and documentation generation. Hence, ensuring their completeness and correctness is a vital prerequisite for their use. In this paper we propose a strategy for the construction of test suites for grammar based software, and illustrate this strategy using the ISO C + + grammar. We use the concept of grammar-rule coverage as a pivot for the reduction of an implementation-based test suite, and demonstrate a significant decrease in the size of this suite. The effectiveness of this reduced test suite is compared to the original test suite with respect to code coverage and more importantly, fault detection. This work greatly expands upon previous work in this area and utilises large scale mutation testing to compare the effectiveness of grammar-rule coverage to that of statement coverage as a reduction criterion for test suites of grammar-based software. This work finds that when grammar rule coverage is used as the sole criterion for reducing test suites of grammar based software, the fault detection capability of that reduced test suite is greatly diminished when compared to other coverage criteria such as statement coverage.

Call-Stack Coverage for GUI Test Suite Reduction

Graphical user interfaces (GUIs) are used as front ends to most of today's software applications. The event-driven nature of GUIs presents new challenges for testing. One important challenge is test suite reduction. Conventional reduction techniques/tools based on static analysis are not easily applicable due to the increased use of multilanguage GUI implementations, callbacks for event handlers, virtual function calls, reflection, and multithreading. Moreover, many existing techniques ignore code in libraries and fail to consider the context in which event handlers execute. Consequently, they yield GUI test suites with seriously impaired fault-detection abilities. This paper presents a reduction technique based on the call-stack coverage criterion. Call stacks may be collected for any executing program with very little overhead. Empirical studies in this paper compare reduction based on call-stack coverage to reduction based on line, method, and event coverage, including variations that control for the size and optional consideration of library methods. These studies show that call-stack-based reduction provides unique trade-offs between the reduction in test suite size and the loss of fault detection effectiveness, which may be valuable in practice. Additionally, an analysis of the relationship between coverage requirements and fault-revealing test cases is presented.