Regression Test Selection Research Articles

Speeding up Genetic Improvement via Regression Test Selection

Genetic Improvement (GI) uses search-based optimisation algorithms to automatically improve software with respect to both functional and non-functional properties. Our previous work showed that Regression Test Selection (RTS) can help speed up the use of GI and enhance the overall results while not affecting the software system’s validity. This article expands upon our investigation by answering further questions about safety and applying a GI algorithm based on Local Search (LS) in addition to the previously explored Genetic Programming (GP) approach. Further, we extend the number of subjects to 12 by analysing five larger real-world open-source programs. We empirically compare two state-of-the-art RTS techniques combined with GP and LS for these 12 programs. The results show that both RTS techniques are safe to use and can reduce the cost of GI by up to 80% and by 31% on average across programs. We also observe that both search-based algorithms impact the effectiveness gains of GI differently, and that various RTS strategies achieve differing gains in terms of efficiency. These results serve as further evidence that RTS must be used as a core component of the GI search process to maximise its effectiveness and efficiency.

Regression test selection in test-driven development

Regression test selection in test-driven development

Comparison of Mutation Testing and Manual Seeded Testing in Regression Test Selection

Comparison of Mutation Testing and Manual Seeded Testing in Regression Test Selection

Multi-Objective Fault-Coverage Based Regression Test Selection and Prioritization Using Enhanced ACO_TCSP

Regression testing of the software during its maintenance phase, requires test case prioritization and selection due to the dearth of the allotted time. The resources and the time in this phase are very limited, thus testers tend to use regression testing methods such as test case prioritization and selection. The current study evaluates the effectiveness of testing with two major goals: (1) Least running time and (2) Maximum fault coverage possible. Ant Colony Optimization (ACO) is a well-known soft computing technique that draws its inspiration from nature and has been widely researched, implemented, analyzed, and validated for regression test prioritization and selection. Many versions of ACO approaches have been prolifically applied to find solutions to many non-polynomial time-solvable problems. Hence, an attempt has been made to enhance the performance of the existing ACO_TCSP algorithm without affecting its time complexity. There have been efforts to enhance the exploration space of various paths in each iteration and with elite exploitation, reducing the total number of iterations required to converge to an optimal path. Counterbalancing enhanced exploration with intelligent exploitation implies that the run time is not adversely affected, the same has also been empirically validated. The enhanced algorithm has been compared with the existing ACO algorithm and with the traditional approaches. The approach has also been validated on four benchmark programs to empirically evaluate the proposed Enhanced ACO_TCSP algorithm. The experiment revealed the increased cost-effectiveness and correctness of the algorithm. The same has also been validated using the statistical test (independent t-test). The results obtained by evaluating the proposed approach against other reference techniques using Average Percentage of Faults Detected (APFD) metrics indicate a near-optimal solution. The multiple objectives of the highest fault coverage and least running time were fruitfully attained using the Enhanced ACO_TCSP approach without compromising the complexity of the algorithm.

An information retrieval-based regression test selection technique

An information retrieval-based regression test selection technique

A review paper: optimal test cases for regression testing using artificial intelligent techniques

<span lang="EN-US">The goal of the testing process is to find errors and defects in the software being developed so that they can be fixed and corrected before they are delivered to the customer. Regression testing is an essential quality testing technique during the maintenance phase of the program as it is performed to ensure the integrity of the program after modifications have been made. With the development of the software, the test suite becomes too large to be fully implemented within the given test cost in terms of budget and time. Therefore, the cost of regression testing using different techniques should be reduced, here we dealt many methods such as retest all technique, regression test selection technique (RTS) and test case prioritization technique (TCP). The efficiency of these techniques is evaluated through the use of many metrics such as average percentage of fault detected (APFD), average percentage block coverage (APBC) and average percentage decision coverage (APDC). In this paper we dealt with these different techniques used in test case selection and test case prioritization and the metrics used to evaluate their efficiency by using different techniques of artificial intelligent and describe the best of all.</span>

Generic and Industrial Scale Many-Criteria Regression Test Selection

Generic and Industrial Scale Many-Criteria Regression Test Selection

Analyzing the impact of API changes on Android apps

The continuous evolution of Android mobile operating system leads to regular updates to its APIs, which may compromise the functionality of Android apps. Given the high frequency of Android API updates, analyzing the impact of API changes is vital to ensure the high reliability of Android apps. This paper introduces Apicia, a novel approach to analyzing the impact of API changes on Android apps. Apicia investigates the impact of changing the target API and identifies the affected program elements (i.e., classes, methods, and statements), the affected tests whose executions may exhibit changed behaviors as a result of the API update, as well as the app code that is not covered by the existing tests. We evaluate Apicia on 219 real-world Android apps. According to the results, API changes impact 46.30% of tests per app on average, and regression test selection based on Apicia can be cost effective. Moreover, many affected statements are not covered by existing tests, which indicates Apicia can help with test suite augmentation to achieve better coverage. These findings suggest that Apicia is a promising approach for assisting Android developers with understanding, testing, and debugging Android apps that are subject to rapid API updates.

A microservice regression testing selection approach based on belief propagation

Regression testing is required to assure the quality of each iteration of microservice systems. Test case selection is one of main techniques to optimize regression testing. Existing techniques mainly involve artifacts acquisition, processing and maintenance, thus hard to apply in microservice regression testing since it is difficult to obtain and process required artifacts from multiple development teams, which is normal in cases of microservice systems. This paper proposes a novel approach, namely MRTS-BP, which takes API gateway logs instead of artifacts as inputs. By mining service dependencies from API gateway logs, MRTS-BP analyzes service change impacts based on a propagation calculation, and selects test cases affected by changes based on impact degree values. To evaluate the effectiveness of MRTS-BP, empirical studies based on four real deployed systems are presented. Retest-all strategy and a regression testing selection approach based on control flow graphs called RTS-CFG are compared with MRTS-BP. The results show that, MRTS-BP can significantly reduce both the number of test cases and overall time cost while maintaining the fault detection capability of selected test suite, and that MRTS-BP can save more time cost than RTS-CFG with the similar safety and precision.

Fault Coverage-Based Test Case Prioritization and Selection Using African Buffalo Optimization

Software needs modifications and requires revisions regularly. Owing to these revisions, retesting software becomes essential to ensure that the enhancements made, have not affected its bug-free functioning. The time and cost incurred in this process, need to be reduced by the method of test case selection and prioritization. It is observed that many nature-inspired techniques are applied in this area. African Buffalo Optimization is one such approach, applied to regression test selection and prioritization. In this paper, the proposed work explains and proves the applicability of the African Buffalo Optimization approach to test case selection and prioritization. The proposed algorithm converges in polynomial time (O(n2)). In this paper, the empirical evaluation of applying African Buffalo Optimization for test case prioritization is done on sample data set with multiple iterations. An astounding 62.5% drop in size and a 48.57% drop in the runtime of the original test suite were recorded. The obtained results are compared with Ant Colony Optimization. The comparative analysis indicates that African Buffalo Optimization and Ant Colony Optimization exhibit similar fault detection capabilities (80%), and a reduction in the overall execution time and size of the resultant test suite. The results and analysis, hence, advocate and encourages the use of African Buffalo Optimization in the area of test case selection and prioritization.

Using goal–question–metric to compare research and practice perspectives on regression testing

AbstractRegression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large‐scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal–question–metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.

Generic and Industrial Scale Many-Criteria Regression Test Selection

Generic and Industrial Scale Many-Criteria Regression Test Selection

An empirical comparison of four Java-based regression test selection techniques

Regression testing is a critical but expensive activity that ensures previously tested functionality is not broken by changes made to the code. Regression test selection (RTS) techniques aim to select and run only those test cases impacted by code changes. The techniques possess different characteristics related to their selection accuracy, test suite size reduction, time to select and run the test cases, and the fault detection ability of the selected test cases. This paper presents an empirical comparison of four Java-based RTS techniques (Ekstazi, HyRTS, OpenClover and STARTS) using multiple revisions from five open source projects.The results show that STARTS selects more test cases than Ekstazi and HyRTS. OpenClover selects the most test cases. Safety and precision violations measure to what extent a technique misses test cases that should be selected and selects only the test cases that are impacted. Using HyRTS as the baseline, OpenClover had significantly worse safety violations compared to STARTS and Ekstazi, and significantly worse precision violations compared to Ekstazi. While STARTS and Ekstazi did not differ on safety violations, Ekstazi had significantly fewer precision violations than STARTS. The average fault detection ability of the RTS techniques was 8.75% lower than the original test suite.

A Systematic Analysis of Regression Test Case Selection: A Multi-Criteria-Based Approach

In applied software engineering, the algorithms for selecting the appropriate test cases are used to perform regression testing. The key objective of this activity is to make sure that modification in the system under test (SUT) has no impact on the overall functioning of the updated software. It is concluded from the literature that the efficacy of the test case selection solely depends on the following metrics, namely, the execution cost of the test case, the lines of the code covered in unit time also known as the code coverage, the ability to capture the potential faults, and the code modifications. Furthermore, it is also observed that the approaches for the regression testing developed so far generated results by focusing on one or two parameters. In this paper, our key objectives are twofold: one is to explore the importance of the role of each metric in detail. The secondary objective is to study the combined effect of these metrics in test case selection task that is capable of achieving more than one objective. In this paper, a detailed and comprehensive review of the work related to regression testing is provided in a very distinct and principled way. This survey will be useful for the researchers contributing to the field of regression testing. It is noteworthy that our systematic literature review (SLR) included the noteworthy work published from 2007 to 2020. Our study observed that about 52 relevant studies focused on all of the four metrics to perform their respective tasks. The results also revealed that about 30% of the different categories of regression test case reported the results using metaheuristic regression test selection (RTS). Similarly, about 31% of the literature reported results using the generic regression test case selection techniques. Most of the researchers focus on the datasets, namely, Software-Artefact Infrastructure Repository (SIR), JodaTime, TreeDataStructure, and Apache Software Foundation. For validation purpose, following parameters were focused, namely, the inclusiveness, precision, recall, and retest-all.

Case Study for Regression Testing using Genetic Algorithm

For quality-assured information engineering software assessment has become critical. Software quality is defined as all the features and proprieties of a software system that can meet certain criteria to test a software product to know how good it is. Software testing is a necessary process for software development, and automating data production helps reduce time and expense. The design of software tests in today’s industry is mostly focused on the experience of the experts, while the tools for test automation can be used only for pre-planned tests. The new investigation focuses on the safe, accurate and effective discriminatory regression testing. A selective algorithm for regression testing is considered secure and sound if the disclosing of test cases is not ignored. Precision that tests the sum of the algorithm moving through the test cases without alteration. The successful selection of these cases is one of the key issues in regression testing. Due to the utility of selective regression testing, an efficient algorithm is important due to the speed of test case selection. We use a formal language approach in this article, to explain the problems and to give tentative solutions. For the test case selection problem, we find the genetic algorithm. We demonstrate that this genetic algorithm that satisfies both secure and reliable standards take time and time in general exponentially. In addition, a secure algorithm requires the most severe case of O (n2) time and Ont) (location, in which n is as many statements as possible, and t is as many test cases as possible.

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

Regression test selection (RTS) approaches reduce the cost of regression testing of evolving software systems. Existing RTS approaches based on UML models use behavioral diagrams or a combination of structural and behavioral diagrams. However, in practice, behavioral diagrams are incomplete or not used. In previous work, we proposed a fuzzy logic based RTS approach called FLiRTS that uses UML sequence and activity diagrams. In this work, we introduce FLiRTS 2, which drops the need for behavioral diagrams and relies on system models that only use UML class diagrams, which are the most widely used UML diagrams in practice. FLiRTS 2 addresses the unavailability of behavioral diagrams by classifying test cases using fuzzy logic after analyzing the information commonly provided in class diagrams. We evaluated FLiRTS 2 on UML class diagrams extracted from 3331 revisions of 13 open-source software systems, and compared the results with those of code-based dynamic (Ekstazi) and static (STARTS) RTS approaches. The average test suite reduction using FLiRTS 2 was 82.06%. The average safety violations of FLiRTS 2 with respect to Ekstazi and STARTS were 18.88% and 16.53%, respectively. FLiRTS 2 selected on average about 82% of the test cases that were selected by Ekstazi and STARTS. The average precision violations of FLiRTS 2 with respect to Ekstazi and STARTS were 13.27% and 9.01%, respectively. The average mutation score of the full test suites was 18.90%; the standard deviation of the reduced test suites from the average deviation of the mutation score for each subject was 1.78% for FLiRTS 2, 1.11% for Ekstazi, and 1.43% for STARTS. Our experiment demonstrated that the performance of FLiRTS 2 is close to the state-of-art tools for code-based RTS but requires less information and performs the selection in less time.

PERBANDINGAN KEMAMPUAN REGRESSION TESTING TOOL PADA REGRESSION TEST SELECTION: STARTS DAN EKSTAZI

Regression testing as an essential activity in software development that has changed requirements. In practice, regression testing requires a lot of time so that an optimal strategy is needed. One approach that can be used to speed up execution time is the Regression Test Selection (RTS) approach. Currently, practitioners and academics have started to think about developing tools to optimize the process of implementing regression testing. Among them, STARTS and Ekstazi are the most popular regression testing tools among academics in running test case selection algorithms. This article discusses the comparison of the capabilities of the STARTS and Ekstazi features by using feature parameter evaluation. Both tools were tested with the same input data in the form of System Under Test (SUT) and test cases. The parameters used in the tool comparisons are platform technology, test case selection, functionality, usability and performance efficiency, the advantages, and disadvantages of the tool. he results of the trial show the differences and similarities between the features of STARTS and Ekstazi, so that it can be used by practitioners to take advantage of tools in the implementation of regression testing that suit their needs. In addition, experimental results show that the use of Ekstazi is more precise in sorting out important test cases and is more efficient, when compared to STARTS and regression testing with retest all.

A Framework for the Regression Testing of Model-to-Model Transformations

Background: Model transformations play a key role in Model-Driven Engineering (MDE). Testing model transformation is an important activity to ensure the quality and correctness of the generated models. However, during the evolution and maintenance of these model transformation programs, frequently testing them by running a large number of test cases can be costly. Regression test selection is a form of testing, which selects tests from an existing test suite to test a modified program. Aim: The aim of the paper is to present a test selection approach for the regression testing of model transformations. The selected test case suite should be smaller in size than the full test suite, thereby reducing the testing overhead, while at the same time the fault detection capability of the full test suite should not be compromised. Method: approach is based on the use of a traceability mapping of test cases with their corresponding rules to select the affected test items. The approach is complemented with a tool that automates the proposed process. Results: Our experiments show that the proposed approach succeeds in reducing the size of the selected test case suite, and hence its execution time, while not compromising the fault detection capability of the full test suite. Conclusion: The experimental results confirm that our regression test selection approach is cost-effective compared to a retest strategy.

Automation of Selection of a Pool of Graphical Interface Regression Tests for Multi Module Information Systems

Features of using the regression test selection method for automated testing of the graphical user interface in the development of information systems that consist of a set of modules are considered. The source of the need to create additional test environments required in the development of multi-module information systems that are using databases is specified. The three most popular approaches to organizing test environments – Copying, Scaling, and Scaling with synthetic data generation – are considered. The positive and negative sides are considered in terms of implementation, using, and resources spent on creating and maintaining resources, as well as in terms of the reliability of the results obtained in the process of testing models created using these approaches. The positive aspects of checking the quality of complex multi-module information systems from the point of view of the graphical user interface by various testing methods and, in particular, in the process of performing regression testing are presented. The positive aspects of using regression testing automation in conditions of lack of resources using various software platforms are indicated. The advantages of using the dynamic selection method for regression tests for automated testing are also given, as well as recommendations for implementing the selection method in existing and beginning projects.

Efficient Regression Testing of Software Product Lines by Reducing Redundant Test Executions

Regression testing for software product lines (SPLs) is challenging because it must ensure that all the products of a product family work correctly whenever changes are made. One approach to reducing the cost of regression testing is the regression test selection (RTS), which selects a subset of regression test cases. However, even when RTS is applied, SPL regression testing can still be expensive because, in the product line context, each test case can be executed on more than one product that reuses the test case, which would typically result in a large number of test executions. A promising direction is to eliminate redundant test executions of test cases. We propose a method that, given a test case, identifies a set of products, on which the test case will cover the same sequence of source code statements and produce the same testing results, and then excludes these products from products to apply the test case to. The evaluation results showed that when the full selection approach and the approach of repetitively applying an RTS method for a single software system are used for test selection, our method reduced, respectively, 59.3% and 40.0% of the numbers of test executions of the approaches.