Test Suite Reduction Approaches Research Articles

A systematic review on search‐based test suite reduction: State‐of‐the‐art, taxonomy, and future directions

AbstractRegression testing remains a promising research area for the last few decades. It is a type of testing that aims at ensuring that recent modifications have not adversely affected the software product. After the introduction of a new change in the system under test, the number of test cases significantly increases to handle the modification. Consequently, it becomes prohibitively expensive to execute all of the generated test cases within the allocated testing time and budget. To address this situation, the test suite reduction (TSR) technique is widely used that focusses on finding a representative test suite without compromising its effectiveness such as fault‐detection capability. In this work, a systematic review study is conducted that intends to provide an unbiased viewpoint about TSR based on various types of search algorithms. The study's main objective is to examine and classify the current state‐of‐the‐art approaches used in search‐based TSR contexts. To achieve this, a systematic review protocol is adopted and, the most relevant primary studies (57 out of 210) published between 2007 and 2022 are selected. Existing search‐based TSR approaches are classified into five main categories, including evolutionary‐based, swarm intelligence‐based, human‐based, physics‐based, and hybrid, grounded on the type of employed search algorithm. Moreover, the current work reports the parameter settings according to their category, the type of considered operator(s), and the probabilistic rate that significantly impacts on the quality of the obtained solution. Furthermore, this study describes the comparison baseline techniques that support the empirical comparison regarding the cost‐effectiveness of a search‐based TSR approach. Finally, it isconcluded that search‐based TSR has great potential to optimally solve the TSR problem. In this regard, several potential research directions are outlined as useful for future researchers interested in conducting research in the TSR domain.

GASSER: A Multi-Objective Evolutionary Approach for Test Suite Reduction

Regression testing is a practice that ensures a System Under Test (SUT) still works as expected after changes have been implemented. The simplest approach for regression testing is Retest-all, which consists of re-executing the entire Test Suite (TS) on the changed version of the SUT. Retest-all could be expensive in case a SUT and its TS grow in size and, if resources are insufficient, its application could be impracticable. A Test Suite Reduction (TSR) approach aims to overcome these issues by reducing the size of TSs, while preserving their fault-detection capability. In this paper, we introduce and validate an approach for TSR based on a multi-objective evolutionary algorithm, namely, Non-dominated Sorting Genetic Algorithm II (NSGA-II). This approach seeks to reduce TSs by maximizing both statement coverage and diversity of test cases of the reduced TSs, while minimizing the size of the reduced TSs. We named this approach Genetic Algorithm for teSt SuitE Reduction (GASSER). To assess GASSER, we conducted an experiment on 19 versions of four software systems from a public dataset—i.e. Software-artifact Infrastructure Repository (SIR). We compared GASSER with nine baseline approaches. The comparison was based on the size of the reduced TSs and their fault-detection capability. The most important take-away result is that GASSER, as compared with the baseline approaches, reduces more the size of the TSs with a non-significant effect on their fault-detection capability. The results of our empirical assessment suggest that the application of multi-objective evolutionary algorithms and, in particular, NSGA-II might represent a viable means to deal with TSR.

A Multi-Objective Approach for Test Suite Reduction During Testing of Web Applications

During the development and maintenance phases of evolving software, new test cases would be needed for the verification of the accuracy of the modifications as well as for new functionalities leading to an increase in the size of the test suite. Various related objectives are to be kept in mind while reducing the original test suite by removing redundancy and generating a practical representative set of the unique test cases, some of which may need to be maximized and the remaining ones minimized. This paper presents a multi-objective approach for the test suite reduction problem in which one objective is to be minimized and the remaining two maximized. In this study, experiments were performed on diverse versions of four web applications. Various state-of-the-art algorithms and their updated versions were compared with non-dominated sorting genetic algorithm-II (NSGA-II) for performance evaluation. Based on experimental findings, it was concluded that NSGA-II outperforms all other algorithms; moreover, the algorithm attempts to satisfy all the objectives without compromising coverage.

Adequate vs. inadequate test suite reduction approaches

Abstract Context: Regression testing is an important activity that allows ensuring the correct behavior of a system after changes. As the system grows, the time and resources to perform regression testing increase. Test Suite Reduction (TSR) approaches aim to speed up regression testing by removing obsolete or redundant test cases. These approaches can be classified as adequate or inadequate. Adequate TSR approaches reduce test suites and completely preserve test requirements (e.g., covered statements) of the original test suites. Inadequate TSR approaches do not preserve test requirements. The percentage of satisfied test requirements indicates the inadequacy level. Objective: We compare some state-of-the-art adequate and inadequate TSR approaches with respect to the size of reduced test suites and their fault-detection capability. We aim to increase our body of knowledge on TSR approaches by comparing: (i) well-known traditional adequate TSR approaches; (ii) their inadequate variants; and (iii) several variants of a novel Clustering-Based (CB) approach for (adequate and inadequate) TSR. Method: We conducted an experiment to compare adequate and inadequate TSR approaches. This comparison is founded on a public dataset containing information on real faults. Results: The most important findings from our experiment can be summarized as follows: (i) there is not an inadequate TSR approach that outperforms the others; (ii) some inadequate variants of the CB approach, and few traditional inadequate approaches, outperform the adequate ones in terms of reduction in test suite size with a negligible effect on fault-detection capability; and (iii) the CB approach is less sensitive than the other inadequate approaches, that is, variations in the inadequacy level have small effect on reduction in test suite size and on loss in fault-detection capability. Conclusions: These findings imply that inadequate TSR approaches and especially the CB approach might be appealing because they lead to a greater reduction in test suite size (with respect to the adequate ones) at the expense of a small loss in fault-detection capability.

Combining Code and Requirements Coverage with Execution Cost for Test Suite Reduction

Test suites tend to become large and complex after software evolution iterations, thus increasing effort and cost to execute regression testing. In this context, test suite reduction approaches could be applied to identify subsets of original test suites that preserve the capability of satisfying testing requirements and revealing faults. In this paper, we propose Multi-Objective test suites REduction (named MORE+): a three-dimension approach for test suite reduction. The first dimension is the structural one and concerns the information on how test cases in a suite exercise the under-test application. The second dimension is functional and concerns how test cases exercise business application requirements. The third dimension is the cost and concerns the time to execute test cases. We define MORE+ as a multi-objective approach that reduces test suites so maximizing their capability in revealing faults according to the three considered dimensions. We have compared MORE+ with seven baseline approaches on 20 Java applications. Results showed, in particular, the effectiveness of MORE+ in reducing test suites with respect to these baselines, i.e., significantly more faults are revealed with test suites reduced by applying MORE+.

Test case selection using multi-criteria optimization for effective fault localization

As spectra-based fault localization techniques report suspicious statements by analyzing the coverage of test cases, the effectiveness of the results is highly dependent on the composition of test suites. This paper proposes an approach for selecting a subset of the passed test suite when a failure revealed by a failed test case. The goal is to obtain a more effective fault localization using a minimal number of test cases than using the originally given large number of test cases. A novelty is that a prioritization criterion and a selection criterion are defined. Different from previous studies, the failed trace is fully considered. The prioritization criterion partitions statements in the failed trace into more suspicious and less suspicious, and then ranks passed test cases by their ability in distinguishing the more suspicious statements from the less suspicious ones. The selection criterion selects the minimal passed test suite which can maximize the number of coverage equivalent classes in the failed trace, so as to distinguish the suspicious statements and meanwhile reduce the size of the test suite. Another novelty is that our approach turns the test case selection into a multi-criteria optimization to make the prioritization and the selection criteria complement each other. This approach was evaluated with 5 fault localization techniques, 8 subject programs and 35,392 test cases. The results show that the fault localization effectiveness can be significantly improved with less than 5% passed test cases. Our approach has advantages over the statement- based and vector-based test suite reduction approaches in both fault localization effectiveness and test suite reduction rate.

A Systematic Review on Test Suite Reduction: Approaches, Experiment’s Quality Evaluation, and Guidelines

Regression testing aims at testing a system under test (SUT) in the presence of changes. As a SUT changes, the number of test cases increases to handle the modifications, and ultimately, it becomes practically impossible to execute all of them within limited testing budget. Test suite reduction (TSR) approaches are widely used to improve the regression testing costs by selecting representative test suite without compromising effectiveness, such as fault-detection capability, within allowed time budget. The aim of this systematic review is to identify state-of-the-art TSR approaches categories, assess the quality of experiments reported on this subject, and provide a set of guidelines for conducting future experiments in this area of research. After applying a two-facet study selection procedure, we finalized 113 most relevant studies from an initial pool of 4230 papers published in the field of TSR between 1993 and 2016. The TSR approaches are broadly classified into four main categories based on the literature including greedy, clustering, search, and hybrid approaches. It is noted that majority of the experiments in TSR do not follow any specific guidelines for planning, conducting, and reporting the experiments, which may pose validity threats related to their results. Thus, we recommend conducting experiments that are better designed for the future. In this direction, an initial set of recommendations is provided that are useful for performing well-designed experiments in the field of TSR. Furthermore, we provide a number of future research directions based on current trends in this field of research.

Probability Model-Based Test Suite Reduction

The paper presents a test suite reduction approach based on probability models in regression testing. First, we model the process of software testing by using the joint probability of both test cases and program faults, and propose two probability models for the software's old and new versions, respectively. Then, a theorem combining set operations and probability models is given to implement our approach. Different from the traditional coverage-based test suite reduction methods, the reduced test suite constructed by our approach does not need to cover all test requirements, but also remains the same fault detection capability of the original test suite.The paper presents a test suite reduction approach based on probability models in regression testing. First, we model the process of software testing by using the joint probability of both test cases and program faults, and propose two probability models for the software's old and new versions, respectively. Then, a theorem combining set operations and probability models is given to implement our approach. Different from the traditional coverage-based test suite reduction methods, the reduced test suite constructed by our approach does not need to cover all test requirements, but also remains the same fault detection capability of the original test suite.

A Test Suite Reduction Approach for Software Unit Testing

A Test Suite Reduction Approach for Software Unit Testing

An Efficient Approach for Test Suite Reduction using Density based Clustering Technique

Density based clustering technique groups similar data objects based on density. In this paper, a methodology has been proposed based on density based clustering technique in order to remove redundant test cases so that time wasted in testing unnecessary test cases can be reduced. Experiments show that how our technique can significantly reduced test suite.

A Test-Suite Reduction Approach for Improving Path-Based Fault-Localization Effectiveness

Path-based fault-localization technique is proved to be more effective than coverage-based fault localization because of analyzing the semantics of a program. Most previous work focused on investigating coverage-based test-suite reduction approach for fault localization. But it is not suitable for path-based fault localization, because it only analyzes coverage information, and does not take the execution path information of test cases into account. In this paper, we propose a test-suite reduction approach for path-based fault localization. This approach analyzes the concrete execution path of test cases. The experimental results show that the proposed method works well in improving the effectiveness of path-based fault localization.

A test-suite reduction approach to improving fault-localization effectiveness

In order to improve the effectiveness of fault localization, researchers are interested in test-suite reduction to provide suitable test-suite inputs. Different test-suite reduction approaches have been proposed. However, the results are usually not ideal. Reducing the test-suite improperly or excessively can even negatively affect fault-localization effectiveness. In this paper, we propose a two-step test-suite reduction approach to remove the test cases which have little or no effect on fault localization, and improve the distribution evenness of concrete execution paths of test cases. This approach consists of coverage matrix based reduction and path vector based reduction, so it analyzes not only the test cases coverage but also the concrete path information. We design and implement experiments to verify the effect of our approach. The experimental results show that our reduced test-suite can improve fault-localization effectiveness. On average, our approach can reduce the size of a test-suite in 47.87% (for Siemens programs) and 23.03% (for space program). At the same time, on average our approach can improve the fault-localization effectiveness, 2.12 on Siemens programs and 0.13 on space program by Tarantula approach.

Analysis of test suite reduction with enhanced tie-breaking techniques

Test suite minimization techniques try to remove redundant test cases of a test suite. However, reducing the size of a test suite might reduce its ability to reveal faults. In this paper, we present a novel approach for test suite reduction that uses an additional testing criterion to break the ties in the minimization process. We integrated the proposed approach with two existing algorithms and conducted experiments for evaluation. The experiment results show that our approach can improve the fault detection effectiveness of reduced suites with a negligible increase in the size of the suites. Besides, under specific conditions, the proposed approach can also accelerate the process of minimization.

Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction