Regression Test Selection Techniques Research Articles

Speeding Up Genetic Improvement via Regression Test Selection

Genetic Improvement (GI) uses search-based optimisation algorithms to automatically improve software with respect to both functional and non-functional properties. Our previous work showed that Regression Test Selection (RTS) can help speed up the use of GI and enhance the overall results while not affecting the software system’s validity. This article expands upon our investigation by answering further questions about safety and applying a GI algorithm based on Local Search (LS) in addition to the previously explored Genetic Programming (GP) approach. Further, we extend the number of subjects to 12 by analysing five larger real-world open-source programs. We empirically compare two state-of-the-art RTS techniques combined with GP and LS for these 12 programs. The results show that both RTS techniques are safe to use and can reduce the cost of GI by up to 80% and by 31% on average across programs. We also observe that both search-based algorithms impact the effectiveness gains of GI differently, and that various RTS strategies achieve differing gains in terms of efficiency. These results serve as further evidence that RTS must be used as a core component of the GI search process to maximise its effectiveness and efficiency.

An information retrieval-based regression test selection technique

An information retrieval-based regression test selection technique

A review paper: optimal test cases for regression testing using artificial intelligent techniques

<span lang="EN-US">The goal of the testing process is to find errors and defects in the software being developed so that they can be fixed and corrected before they are delivered to the customer. Regression testing is an essential quality testing technique during the maintenance phase of the program as it is performed to ensure the integrity of the program after modifications have been made. With the development of the software, the test suite becomes too large to be fully implemented within the given test cost in terms of budget and time. Therefore, the cost of regression testing using different techniques should be reduced, here we dealt many methods such as retest all technique, regression test selection technique (RTS) and test case prioritization technique (TCP). The efficiency of these techniques is evaluated through the use of many metrics such as average percentage of fault detected (APFD), average percentage block coverage (APBC) and average percentage decision coverage (APDC). In this paper we dealt with these different techniques used in test case selection and test case prioritization and the metrics used to evaluate their efficiency by using different techniques of artificial intelligent and describe the best of all.</span>

Using goal–question–metric to compare research and practice perspectives on regression testing

AbstractRegression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large‐scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal–question–metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.

An empirical comparison of four Java-based regression test selection techniques

Regression testing is a critical but expensive activity that ensures previously tested functionality is not broken by changes made to the code. Regression test selection (RTS) techniques aim to select and run only those test cases impacted by code changes. The techniques possess different characteristics related to their selection accuracy, test suite size reduction, time to select and run the test cases, and the fault detection ability of the selected test cases. This paper presents an empirical comparison of four Java-based RTS techniques (Ekstazi, HyRTS, OpenClover and STARTS) using multiple revisions from five open source projects.The results show that STARTS selects more test cases than Ekstazi and HyRTS. OpenClover selects the most test cases. Safety and precision violations measure to what extent a technique misses test cases that should be selected and selects only the test cases that are impacted. Using HyRTS as the baseline, OpenClover had significantly worse safety violations compared to STARTS and Ekstazi, and significantly worse precision violations compared to Ekstazi. While STARTS and Ekstazi did not differ on safety violations, Ekstazi had significantly fewer precision violations than STARTS. The average fault detection ability of the RTS techniques was 8.75% lower than the original test suite.

Overlap‐aware rapid type analysis for constructing one‐to‐one matched call graphs in regression test selection

Regression testing is an important but costly activity for verifying a programme with the changed code. Regression test selection (RTS) aims to reduce this cost by selecting only the test cases affected by the changes. Among the several ways of selecting such affected test cases, call graphs have been statically constructed to select the test cases at the method-level granularity. However, RTS techniques will reduce the cost of regression testing less than expected unless the call graphs are efficiently one-to-one matched with the test cases. In this study, the authors propose overlap-aware rapid type analysis (ORTA). ORTA is designed to minimise the redundant cost of creating the matched call graphs using rapid type analysis (RTA). The one-to-one matching and ORTA were evaluated on 1487 commits selected from 30 Java projects. RTA-based RTS with the one-to-one matching selected 46.90% fewer test cases with 2.76% longer end-to-end time of regression testing than without the one-to-one matching. The time increased with the one-to-one matching was reduced by 22.58% when ORTA substituted for RTA. ORTA achieved the cost reduction while removing 82.77% of the duplicate edges that RTA created on 993 commits.

Reflection-aware static regression test selection

Regression test selection (RTS) aims to speed up regression testing by rerunning only tests that are affected by code changes. RTS can be performed using static or dynamic analysis techniques. Our prior study showed that static and dynamic RTS perform similarly for medium-sized Java projects. However, the results of that prior study also showed that static RTS can be unsafe, missing to select tests that dynamic RTS selects, and that reflection was the only cause of unsafety observed among the evaluated projects. In this paper, we investigate five techniques—three purely static techniques and two hybrid static-dynamic techniques—that aim to make static RTS safe with respect to reflection. We implement these reflection-aware (RA) techniques by extending the reflection-unaware (RU) class-level static RTS technique in a tool called STARTS. To evaluate these RA techniques, we compare their end-to-end times with RU, and with RetestAll, which reruns all tests after every code change. We also compare safety and precision of the RA techniques with Ekstazi, a state-of-the-art dynamic RTS technique; precision is a measure of unaffected tests selected. Our evaluation on 1173 versions of 24 open-source Java projects shows negative results. The RA techniques improve the safety of RU but at very high costs. The purely static techniques are safe in our experiments but decrease the precision of RU, with end-to-end time at best 85.8% of RetestAll time, versus 69.1% for RU. One hybrid static-dynamic technique improves the safety of RU but at high cost, with end-to-end time that is 91.2% of RetestAll. The other hybrid static-dynamic technique provides better precision, is safer than RU, and incurs lower end-to-end time—75.8% of RetestAll, but it can still be unsafe in the presence of test-order dependencies. Our study highlights the challenges involved in making static RTS safe with respect to reflection.

Regression test selection model: a comparison between ReTSE and pythia

As software systems change and evolve over time regression tests have to be run to validate these changes. Regression testing is an expensive but essential activity in software maintenance. The purpose of this paper is to compare a new regression test selection model called ReTSE with Pythia. The ReTSE model uses decomposition slicing in order to identify the relevant regression tests. Decomposition slicing provides a technique that is capable of identifying the unchanged parts of a system. Pythia is a regression test selection technique based on textual differencing. Both techniques are compare using a Power program taken from Vokolos and Frankl’s paper. The analysis of this comparison has shown promising results in reducing the number of tests to be run after changes are introduced.

Source-Code Level Regression Test Selection: the Model-Driven Way.

In order to ensure that existing functionalities have not been impacted by recent program changes, test cases are regularly executed during regression testing (RT) phases. The RT time becomes problematic as the number of test cases is growing. Regression test selection (RTS) aims at running only the test cases that have been impacted by recent changes. RTS reduces the duration of regression testing and hence its cost. In this paper, we present a model-driven approach for RTS. Execution traces are gathered at runtime, and injected in a static source-code model. We use this resulting model to identify and select all the test cases that have been impacted by changes between two revisions of the program. Our MDE approach allows modularity in the granularity of changes considered. In addition, it offers better reusability than existing RTS techniques: the trace model is persistent and standardised. Furthermore, it enables more interoperability with other model-driven tools, enabling further analysis at different levels of abstraction (e.g. energy consumption). We evaluate our approach by applying it to four well-known open-source projects. Results show that our MDE proposal can effectively reduce the execution time of RT, by up to 32 % in our case studies. The overhead induced by the model building makes our approach slower than dedicated RTS tools, but the reuse of trace models for further analysis is overtaking this time difference.

Multi-objective regression test selection in practice: An empirical study in the defense software industry

ContextExecuting an entire regression test-suite after every code change is often costly in large software projects. To cope with this challenge, researchers have proposed various regression test-selection techniques. ObjectiveThis paper was motivated by a real industrial need to improve regression-testing practices in the context of a safety-critical industrial software in the defence domain in Turkey. To address our objective, we set up and conducted an “action-research” collaborative project between industry and academia. MethodAfter a careful literature review, we selected a conceptual multi-objective regression-test selection framework (called MORTO) and adopted it to our industrial context by developing a custom-built genetic algorithm (GA) based on that conceptual framework. GA is able to provide full coverage of the affected (changed) requirements while considering multiple cost and benefit factors of regression testing. e.g., minimizing the number of test cases, and maximizing cumulative number of detected faults by each test suite. ResultsThe empirical results of applying the approach on the Software Under Test (SUT) demonstrate that this approach yields a more efficient test suite (in terms of costs and benefits) compared to the old (manual) test-selection approach, used in the company, and another applicable approach chosen from the literature. With this new approach, regression selection process in the project under study is not ad-hoc anymore. Furthermore, we have been able to eliminate the subjectivity of regression testing and its dependency on expert opinions. ConclusionSince the proposed approach has been beneficial in saving the costs of regression testing, it is currently in active use in the company. We believe that other practitioners can apply our approach in their regression-testing contexts too, when applicable. Furthermore, this paper contributes to the body of evidence in regression testing by offering a success story of successful implementation and application of multi-objective regression testing in practice.

Supporting inheritance hierarchy changes in model-based regression test selection

Models can be used to ease and manage the development, evolution, and runtime adaptation of a software system. When models are adapted, the resulting models must be rigorously tested. Apart from adding new test cases, it is also important to perform regression testing to ensure that the evolution or adaptation did not break existing functionality. Since regression testing is performed with limited resources and under time constraints, regression test selection (RTS) techniques are needed to reduce the cost of regression testing. Applying model-level RTS for model-based evolution and adaptation is more convenient than using code-level RTS because the test selection process happens at the same level of abstraction as that of evolution and adaptation. In earlier work, we proposed a model-based RTS approach called MaRTS to be used with a fine-grained model-based adaptation framework that targets applications implemented in Java. MaRTS uses UML models consisting of class and activity diagrams. It classifies test cases as obsolete, reusable, or retestable based on changes made to UML class and activity diagrams of the system being adapted. However, MaRTS did not take into account the changes made to the inheritance hierarchy in the class diagram and the impact of these changes on the selection of test cases. This paper extends MaRTS to support such changes and demonstrates that the extended approach performs as well as or better than code-based RTS approaches in safely selecting regression test cases. While MaRTS can generally be used during any model-driven development or model-based evolution activity, we have developed it in the context of runtime adaptation. We evaluated the extended MaRTS on a set of applications and compared the results with code-based RTS approaches that also support changes to the inheritance hierarchy. The results showed that the extended MaRTS selected all the test cases relevant to the inheritance hierarchy changes and that the fault detection ability of the selected test cases was never lower than that of the baseline test cases. The extended MaRTS achieved comparable results to a graph-walk code-based RTS approach (DejaVu) and showed a higher reduction in the number of selected test cases when compared with a static analysis code-based RTS approach (ChEOPSJ).

A regression test selection technique for embedded software

The current approaches for regression test selection of embedded programs are usually based on data- and control-dependency analyses, often augmented with human reasoning. Existing techniques do not take into account additional execution dependencies which may exist among code elements in such programs due to features such as tasks, task deadlines, task precedences, and intertask communications. In this context, we propose a model-based regression test selection technique for such programs. Our technique first constructs a graph model of the program; the proposed graph model has been designed to capture several characteristics of embedded programs, such as task precedence order, priority, intertask communication, timers, exceptions and interrupt handlers, which we consider important for regression-test selection. Our regression test selection technique selects test cases based on an analysis of the constructed graph model. We have implemented our technique to realize a prototype tool. The experimental results obtained using this tool show that, on average, our approach selects about 28.33% more regression test cases than those selected by a traditional approach. We observed that, on average, 36.36% of the fault-revealing test cases were overlooked by the existing regression test selection technique.

An Optimal Technique for Reducing the Effort of Regression Test

Regression test selection techniques are proposed often but are many times inaccurate when used with larger systems. The proposed new selection technique will be safer, more precise, and can handle the object-oriented features even in larger systems through its phases. Selecting the subset of the test case from the existing test suite is an important problem in regression testing and is addressed in the regression test selection technique. Safe regression test selection technique selects and identifies the program parts that are affected by the change. The test selection is performed by matching the identified change information with the coverage information. A tool is implemented that reduces the testing effort efficiently and the result shows that it can achieve considerable savings in the regression testing time.

An Optimal Technique for Reducing the Effort of Regression Test

Regression test selection techniques are proposed often but are many times inaccurate when used with larger systems. The proposed new selection technique will be safer, more precise, and can handle the object-oriented features even in larger systems through its phases. Selecting the subset of the test case from the existing test suite is an important problem in regression testing and is addressed in the regression test selection technique. Safe regression test selection technique selects and identifies the program parts that are affected by the change. The test selection is performed by matching the identified change information with the coverage information. A tool is implemented that reduces the testing effort efficiently and the result shows that it can achieve considerable savings in the regression testing time.

A Survey on Regression Test Selection Techniques on Aspect-Oriented Programming

An aspect oriented programming is gaining a lot of popularity these days, there is a growing interest because of crosscutting concerns in object oriented systems. When the aspect oriented features are added into object oriented features the new program needs to be regression tested, and, to reduce the cost, Selection technique is used which eliminates the redundant test cases and thus makes them cost effective. Unfortunately the already existing approaches of object oriented programming does not work out for aspect oriented programming because of the following new features of aspects such as join points ,crosscutting concerns ,aspect weaving, etc. Therefore, this paper proposes the techniques used for object oriented programs and for aspect oriented programming.

A Regression Test Selection and Prioritization Technique

Regression testing is a very costly process performed primarily as a software maintenance activity. It is the process of retesting the modified parts of the software and ensuring that no new errors have been introduced into previously tested source code due to these modifications. A regression test selection technique selects an appropriate number of test cases from a test suite that might expose a fault in the modified program. In this paper, we propose both a regression test selection and prioritization technique. We implemented our regression test selection technique and demonstrated in two case studies that our technique is effective regarding selecting and prioritizing test cases. The results show that our technique may significantly reduce the number of test cases and thus the cost and resources for performing regression testing on modified software.

Regression Test Selection Technique for Multi-Programming Language

Regression testing is a maintenance activity applied to modified software to provide confidence that the changed parts are correct and that the unchanged parts have not been adversely affected by the modifications. Regression test selection techniques reduce the cost of regression testing, by selecting a subset of an existing test suite to use in retesting modified programs. This paper presents the first general regression-test-selection technique, which based on code and allows selecting test cases for any programs written in any programming language. Then it handles incomplete program. We also describe RTSDiff, a regression-test-selection system that implements the proposed technique. The results of the empirical studied that performed in four programming languages java, C#, C++ and Visual basic show that the efficiency and effective in reducing the size of test suit.

A model-based regression test selection approach for embedded applications

Regression test selection techniques for embedded programs have scarcely been reported in the literature. In this paper, we propose a model-based regression test selection technique for embedded programs. Our proposed model, in addition to capturing the data and control dependence aspects, also represents several additional program features that are important for regression test case selection of embedded programs. These features include control flow, exception handling, message paths, task priorities, state information and object relations. We select a regression test suite based on slicing our proposed graph model. We also propose a genetic algorithm-based technique to select an optimal subset of test cases from the set of regression test cases selected after slicing our proposed model.

Regression Test Selection for C# Programs

We present a regression test selection technique for C# programs. C# is fairly new and is often used within the Microsoft .Net framework to give programmers a solid base to develop a variety of applications. Regression testing is done after modifying a program. Regression test selection refers to selecting a suitable subset of test cases from the original test suite in order to be rerun. It aims to provide confidence that the modifications are correct and did not affect other unmodified parts of the program. The regression test selection technique presented in this paper accounts for C#.Net specific features. Our technique is based on three phases; the first phase builds an Affected Class Diagram consisting of classes that are affected by the change in the source code. The second phase builds a C# Interclass Graph (CIG) from the affected class diagram based on C# specific features. In this phase, we reduce the number of selected test cases. The third phase involves further reduction and a new metric for assigning weights to test cases for prioritizing the selected test cases. We have empirically validated the proposed technique by using case studies. The empirical results show the usefulness of the proposed regression testing technique for C#.Net programs.

IMPROVING CLASS FIREWALL REGRESSION TEST SELECTION BY REMOVING THE CLASS FIREWALL

One regression test selection technique proposed for object-oriented programs is the Class firewall regression test selection technique. The selection technique selects test cases for regression test, which test changed classes and classes depending on changed classes. However, in empirical studies of the application of the technique, we observed that another technique found the same defects, selected fewer tests and required a simpler, less costly, analysis. The technique, which we refer to as the Change-based regression test selection technique, is basically the Class firewall technique, but with the class firewall removed. In this paper we formulate a hypothesis stating that these empirical observations are not incidental, but an inherent property of the Class firewall technique. We prove that the hypothesis holds for Java in a stable testing environment, and conclude that the effectiveness of the Class firewall regression testing technique can be improved without sacrificing the defect detection capability of the technique, by removing the class firewall.