Regression Test Selection Approach Research Articles

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

Regression test selection (RTS) approaches reduce the cost of regression testing of evolving software systems. Existing RTS approaches based on UML models use behavioral diagrams or a combination of structural and behavioral diagrams. However, in practice, behavioral diagrams are incomplete or not used. In previous work, we proposed a fuzzy logic based RTS approach called FLiRTS that uses UML sequence and activity diagrams. In this work, we introduce FLiRTS 2, which drops the need for behavioral diagrams and relies on system models that only use UML class diagrams, which are the most widely used UML diagrams in practice. FLiRTS 2 addresses the unavailability of behavioral diagrams by classifying test cases using fuzzy logic after analyzing the information commonly provided in class diagrams. We evaluated FLiRTS 2 on UML class diagrams extracted from 3331 revisions of 13 open-source software systems, and compared the results with those of code-based dynamic (Ekstazi) and static (STARTS) RTS approaches. The average test suite reduction using FLiRTS 2 was 82.06%. The average safety violations of FLiRTS 2 with respect to Ekstazi and STARTS were 18.88% and 16.53%, respectively. FLiRTS 2 selected on average about 82% of the test cases that were selected by Ekstazi and STARTS. The average precision violations of FLiRTS 2 with respect to Ekstazi and STARTS were 13.27% and 9.01%, respectively. The average mutation score of the full test suites was 18.90%; the standard deviation of the reduced test suites from the average deviation of the mutation score for each subject was 1.78% for FLiRTS 2, 1.11% for Ekstazi, and 1.43% for STARTS. Our experiment demonstrated that the performance of FLiRTS 2 is close to the state-of-art tools for code-based RTS but requires less information and performs the selection in less time.

A Framework for the Regression Testing of Model-to-Model Transformations

Background: Model transformations play a key role in Model-Driven Engineering (MDE). Testing model transformation is an important activity to ensure the quality and correctness of the generated models. However, during the evolution and maintenance of these model transformation programs, frequently testing them by running a large number of test cases can be costly. Regression test selection is a form of testing, which selects tests from an existing test suite to test a modified program. Aim: The aim of the paper is to present a test selection approach for the regression testing of model transformations. The selected test case suite should be smaller in size than the full test suite, thereby reducing the testing overhead, while at the same time the fault detection capability of the full test suite should not be compromised. Method: approach is based on the use of a traceability mapping of test cases with their corresponding rules to select the affected test items. The approach is complemented with a tool that automates the proposed process. Results: Our experiments show that the proposed approach succeeds in reducing the size of the selected test case suite, and hence its execution time, while not compromising the fault detection capability of the full test suite. Conclusion: The experimental results confirm that our regression test selection approach is cost-effective compared to a retest strategy.

SPIRITuS: a SimPle Information Retrieval regressIon Test Selection approach

Context:Regression Test case Selection (RTS) approaches aim at selecting only those test cases of a test suite that exercise changed parts of the System Under Test (SUT) or parts affected by changes.Objective:We present SPIRITuS (SimPle Information Retrieval regressIon Test Selection approach). It uses method code coverage information and a Vector Space Model to select test cases to be run. In a nutshell, the extent of a lexical modification to a method is used to decide if a test case has to be selected. The main design goals of SPIRITuS are to be: (i) easy to adapt to different programming languages and (ii) tunable via an easy to understand threshold.Method:To assess SPIRITuS, we conducted a large experiment on 389 faulty versions of 14 open-source programs implemented in Java. We were mainly interested in investigating the tradeoff between the number of selected test cases from the original test suite and fault detection effectiveness. We also compared SPIRITuS against well-known RTS approaches.Results:SPIRITuS selects a number of test cases significantly smaller than the number of test cases the other approaches select at the price of a slight reduction in fault detection capability.Conclusions:SPIRITuS can be considered a viable competitor of existing test case selection approaches especially when the average number of test cases covering a modified method increases (such information can be easily derived before test case selection takes place).

Supporting inheritance hierarchy changes in model-based regression test selection

Models can be used to ease and manage the development, evolution, and runtime adaptation of a software system. When models are adapted, the resulting models must be rigorously tested. Apart from adding new test cases, it is also important to perform regression testing to ensure that the evolution or adaptation did not break existing functionality. Since regression testing is performed with limited resources and under time constraints, regression test selection (RTS) techniques are needed to reduce the cost of regression testing. Applying model-level RTS for model-based evolution and adaptation is more convenient than using code-level RTS because the test selection process happens at the same level of abstraction as that of evolution and adaptation. In earlier work, we proposed a model-based RTS approach called MaRTS to be used with a fine-grained model-based adaptation framework that targets applications implemented in Java. MaRTS uses UML models consisting of class and activity diagrams. It classifies test cases as obsolete, reusable, or retestable based on changes made to UML class and activity diagrams of the system being adapted. However, MaRTS did not take into account the changes made to the inheritance hierarchy in the class diagram and the impact of these changes on the selection of test cases. This paper extends MaRTS to support such changes and demonstrates that the extended approach performs as well as or better than code-based RTS approaches in safely selecting regression test cases. While MaRTS can generally be used during any model-driven development or model-based evolution activity, we have developed it in the context of runtime adaptation. We evaluated the extended MaRTS on a set of applications and compared the results with code-based RTS approaches that also support changes to the inheritance hierarchy. The results showed that the extended MaRTS selected all the test cases relevant to the inheritance hierarchy changes and that the fault detection ability of the selected test cases was never lower than that of the baseline test cases. The extended MaRTS achieved comparable results to a graph-walk code-based RTS approach (DejaVu) and showed a higher reduction in the number of selected test cases when compared with a static analysis code-based RTS approach (ChEOPSJ).

Optimal control based regression test selection for service-oriented workflow applications

Regression test selection problem is treated as an optimal control issue.Three rules are given to transform BPEL application into an universal BPEL form.Our approach can select fewer test cases to test BPEL applications effectively.We prove our optimal control strategy is safe under controlled regression testing. Regression test selection, which is well known as an effective technology to ensure the quality of modified BPEL applications, is regarded as an optimal control issue. The BPEL applications under test serves as a controlled object and the regression test selection strategy functions as the corresponding controller. The performance index is to select fewest test cases to test modified BPEL applications. In addition, a promising controller (regression test selection approach) should be safe, which means that it can select all test cases in which faults might be exposed in modified versions under controlled regression testing from the original test suite. However, existing safe controllers may rerun some test cases without exposing fault. In addition, the unique features (e.g., dead path elimination semantics, communication mechanism, multi-assignment etc.) of BPEL applications also raise enormous problems in regression test selection. To address these issues, we present in this paper a safe optimal controller for BPEL applications. Firstly, to handle the unique features mentioned above, we transform BPEL applications and their modified versions into universal BPEL forms. Secondly, For our optimal controller, BPEL program dependence graphs corresponding to the two universal BPEL forms are established. Finally, guided by behavioral differences between the two versions, we construct an optimal controller and select test cases to be rerun. By contrast with the previous approaches, our approach can eliminate some unnecessary test cases to be selected. We conducted experiments with 8 BPEL applications to compare our approach with other typical approaches. Experimental results show that the test cases selected using our approach are fewer than other approaches.

State-Model-Based Regression Test Reduction for Component-Based Software

We present a novel regression test selection approach based on analysis of state and dependence models of components. Our technique targets to select a smaller regression test suite compared to the pure dependence-based RTS approaches while maintaining the fault revealing effectiveness. In our approach, after a modification, control and data dependencies are analyzed to identify the potentially affected statements. Subsequently, the state model of the component is analyzed to compute a precise publishable change information to support efficient regression test selection by the application developers.

Test coverage optimization for large code problems

Abstract Software developers frequently conduct regression testing on a series of major, minor, or bug-fix software or firmware releases. However, retesting all test cases for each release is time-consuming. For example, it takes about 36 test-bed-days to thoroughly exercise a test suite made up of 2320 test cases for the MPLS testing area that contains 57,758 functions in Cisco IOS. The cost is infeasible for a series of regression testing on the MPLS area. Thus, the test suite needs to be reduced intelligently, not just randomly, and its fault detection capability must be kept as much as possible. The mode of safe regression test selection approach is adopted for seeking a subset of modification-traversing test cases to substitute for fault-revealing test cases. The algorithms, CW-NumMin, CW-CostMin, and CW-CostCov-B, apply the safe-mode approach in selecting test cases for achieving full-modified function coverage. It is assumed that modified functions are fault-prone, and the fault distribution of the testing area is Pareto-like. Moreover, we also assume that once a subject program is getting more mature, its fault concentration will become stronger. Only function coverage criterion is adopted because of the scalability of a software system with large code. The metrics of test’s function reachability and function’s test intensity are defined in this study for algorithms. Both CW-CovMax and CW-CostMin algorithms are not safe-mode, but the approaches they use still attempt to obtain a test suite with a maximal amount of function coverage under certain constraints, i.e. the effective-confidence level and time restriction. We conclude that the most effective algorithm in this study can significantly reduce the cost (time) of regression testing on the MPLS testing area to 1.10%, on the average. Approaches proposed here can be effectively and efficiently applied to the regression testing on bug-fix releases of a software system with large code, especially to the releases having very few modified functions with low test intensities.

A model-based regression test selection approach for embedded applications

Regression test selection techniques for embedded programs have scarcely been reported in the literature. In this paper, we propose a model-based regression test selection technique for embedded programs. Our proposed model, in addition to capturing the data and control dependence aspects, also represents several additional program features that are important for regression test case selection of embedded programs. These features include control flow, exception handling, message paths, task priorities, state information and object relations. We select a regression test suite based on slicing our proposed graph model. We also propose a genetic algorithm-based technique to select an optimal subset of test cases from the set of regression test cases selected after slicing our proposed model.