Test Suite Reduction Research Articles

On the suitability of checked coverage and genetic parameter tuning in test suite reduction

AbstractAs software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser‐known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open‐source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power.

Test suite reduction techniques for software product line testing: a survey

Test suite reduction techniques for software product line testing: a survey

Test case simplification based on coupling metrics in software bug location

Software test cases are one of the most critical aspects of software testing in the product development process. As software products are updated several times, the same test requirement may be covered by multiple test cases, so this aspect is often redundant, yet the approximate test case set has an impact on its error detection rate. This study proposes the idea of using redundant test cases in software error location, introduces a coupling metric, analyses its program slicing and establishes a second coverage criterion in order to balance the relationship between the reduced test suite and the false detection rate the test case set. The results show that the size of test set and the number of error detection by the Ruby On Rails (ROR) method used in this study are larger than those of other commonly used reduction algorithms. The test suite has the lowest error detection loss rate, with an average of 17.96 % across the six test case sets. The highest error detection capability of individual test cases was found in the reduced test set, with a mean value of 90.63 % in the test set. The method also has the highest average reduction efficiency of 91.05 %. Compared with other simplification methods, the research method has a better balance between the size and false detection rate of the reduced test suite and the advantages of simplification.

A systematic review on search‐based test suite reduction: State‐of‐the‐art, taxonomy, and future directions

AbstractRegression testing remains a promising research area for the last few decades. It is a type of testing that aims at ensuring that recent modifications have not adversely affected the software product. After the introduction of a new change in the system under test, the number of test cases significantly increases to handle the modification. Consequently, it becomes prohibitively expensive to execute all of the generated test cases within the allocated testing time and budget. To address this situation, the test suite reduction (TSR) technique is widely used that focusses on finding a representative test suite without compromising its effectiveness such as fault‐detection capability. In this work, a systematic review study is conducted that intends to provide an unbiased viewpoint about TSR based on various types of search algorithms. The study's main objective is to examine and classify the current state‐of‐the‐art approaches used in search‐based TSR contexts. To achieve this, a systematic review protocol is adopted and, the most relevant primary studies (57 out of 210) published between 2007 and 2022 are selected. Existing search‐based TSR approaches are classified into five main categories, including evolutionary‐based, swarm intelligence‐based, human‐based, physics‐based, and hybrid, grounded on the type of employed search algorithm. Moreover, the current work reports the parameter settings according to their category, the type of considered operator(s), and the probabilistic rate that significantly impacts on the quality of the obtained solution. Furthermore, this study describes the comparison baseline techniques that support the empirical comparison regarding the cost‐effectiveness of a search‐based TSR approach. Finally, it isconcluded that search‐based TSR has great potential to optimally solve the TSR problem. In this regard, several potential research directions are outlined as useful for future researchers interested in conducting research in the TSR domain.

SPIDER: Speeding up Side-Channel Vulnerability Detection via Test Suite Reduction

Side-channel attacks allow adversaries to infer sensitive information, such as cryptographic keys or private user data, by monitoring unintentional information leaks of running programs. Prior side-channel detection methods can identify numerous potential vulnerabilities in cryptographic implementations with a small amount of execution traces due to the high diffusion of secret inputs in crypto primitives. However, because non-cryptographic programs cover different paths under various sensitive inputs, extending existing tools for identifying information leaks to non-cryptographic applications suffers from either insufficient path coverage or redundant testing. To address these limitations, we propose a new dynamic analysis framework named SPIDER that uses fuzzing, execution profiling, and clustering for a high path coverage and test suite reduction, and then speeds up the dynamic analysis of side-channel vulnerability detection in non-cryptographic programs. We analyze eight non-cryptographic programs and ten cryptographic algorithms under SPIDER in a fully automated way, and our results confirm the effectiveness of test suite reduction and the vulnerability detection accuracy of the whole framework.

Value-Based Test Case Prioritization for Regression Testing Using Genetic Algorithms

Test Case Prioritization (TCP) techniques perform better than other regression test optimization techniques including Test Suite Reduction (TSR) and Test Case Selection (TCS). Many TCP techniques are available, and their performance is usually measured through a metric Average Percentage of Fault Detection (APFD). This metric is value-neutral because it only works well when all test cases have the same cost, and all faults have the same severity. Using APFD for performance evaluation of test case orders where test cases cost or faults severity varies is prone to produce false results. Therefore, using the right metric for performance evaluation of TCP techniques is very important to get reliable and correct results. In this paper, two value-based TCP techniques have been introduced using Genetic Algorithm (GA) including Value-Cognizant Fault Detection-Based TCP (VCFDB-TCP) and Value-Cognizant Requirements Coverage-Based TCP (VCRCB-TCP). Two novel value-based performance evaluation metrics are also introduced for value-based TCP including Average Percentage of Fault Detection per value (APFDv) and Average Percentage of Requirements Coverage per value (APRCv). Two case studies are performed to validate proposed techniques and performance evaluation metrics. The proposed GA-based techniques outperformed the existing state-of-the-art TCP techniques including Original Order (OO), Reverse Order (REV-O), Random Order (RO), and Greedy algorithm.

Developing a Modified Fuzzy-GE Algorithm for Enhanced Test Suite Reduction Effectiveness

Developing a Modified Fuzzy-GE Algorithm for Enhanced Test Suite Reduction Effectiveness

A Study of Wolf Pack Algorithm for Test Suite Reduction

A Study of Wolf Pack Algorithm for Test Suite Reduction

Test case prioritization using test case diversification and fault-proneness estimations

Regression testing activities greatly reduce the risk of faulty software release. However, the size of the test suites grows throughout the development process, resulting in time-consuming execution of the test suite and delayed feedback to the software development team. This has urged the need for approaches such as test case prioritization (TCP) and test-suite reduction to reach better results in case of limited resources. In this regard, proposing approaches that use auxiliary sources of data such as bug history can be interesting. We aim to propose an approach for TCP that takes into account test case coverage data, bug history, and test case diversification. To evaluate this approach we study its performance on real-world open-source projects. The bug history is used to estimate the fault-proneness of source code areas. The diversification of test cases is preserved by incorporating fault-proneness on a clustering-based approach scheme. The proposed methods are evaluated on datasets collected from the development history of five real-world projects including 357 versions in total. The experiments show that the proposed methods are superior to coverage-based TCP methods. The proposed approach shows that improvement of coverage-based and fault-proneness-based methods is possible by using a combination of diversification and fault-proneness incorporation.

Optimum Test Suite Using Fault-Type Coverage-Based Ant Colony Optimization Algorithm

Software Product Lines(SPLs) covers a mixture of features for testing Software Application Program(SPA). Testing cost reduction is a major metric of software testing. In combinatorial testing(CT), maximization of fault type coverage and test suite reduction plays a key role to reduce the testing cost of SPA. Metaheuristic Genetic Algorithm(GA) do not offer best outcome for test suite optimization problem due to mutation operation and required more computational time. So, Fault-Type Coverage Based Ant Colony Optimization(FTCBACO) algorithm is offered for test suite reduction in CT. FTCBACO algorithm starts with test cases in test suite and assign separate ant to each test case. Ants elect best test cases by updating of pheromone trails and selection of higher probability trails. Best test case path of ant with least time are taken as optimal solution for performing CT. Hence, FTCBACO Technique enriches reduction rate of test suite and minimizes computational time of reducing test cases efficiently for CT.

A Study of Test Suite Reduction Based on Ant Lion Optimizer

A Study of Test Suite Reduction Based on Ant Lion Optimizer

GASSER: A Multi-Objective Evolutionary Approach for Test Suite Reduction

Regression testing is a practice that ensures a System Under Test (SUT) still works as expected after changes have been implemented. The simplest approach for regression testing is Retest-all, which consists of re-executing the entire Test Suite (TS) on the changed version of the SUT. Retest-all could be expensive in case a SUT and its TS grow in size and, if resources are insufficient, its application could be impracticable. A Test Suite Reduction (TSR) approach aims to overcome these issues by reducing the size of TSs, while preserving their fault-detection capability. In this paper, we introduce and validate an approach for TSR based on a multi-objective evolutionary algorithm, namely, Non-dominated Sorting Genetic Algorithm II (NSGA-II). This approach seeks to reduce TSs by maximizing both statement coverage and diversity of test cases of the reduced TSs, while minimizing the size of the reduced TSs. We named this approach Genetic Algorithm for teSt SuitE Reduction (GASSER). To assess GASSER, we conducted an experiment on 19 versions of four software systems from a public dataset—i.e. Software-artifact Infrastructure Repository (SIR). We compared GASSER with nine baseline approaches. The comparison was based on the size of the reduced TSs and their fault-detection capability. The most important take-away result is that GASSER, as compared with the baseline approaches, reduces more the size of the TSs with a non-significant effect on their fault-detection capability. The results of our empirical assessment suggest that the application of multi-objective evolutionary algorithms and, in particular, NSGA-II might represent a viable means to deal with TSR.

Multi-Objective Adapted Binary Bat for Test Suite Reduction

Regression testing is an essential quality test technique during the maintenance phase of the software. It is executed to ensure the validity of the software after any modification. As software evolves, the test suite expands and may become too large to be executed entirely within a limited testing budget and/or time. So, to reduce the cost of regression testing, it is mandatory to reduce the size of the test suite by discarding the redundant test cases and selecting the most representative ones that do not compromise the effectiveness of the test suite in terms of some predefined criteria such as its fault-detection capability. This problem is known as test suite reduction (TSR); and it is known to be as nondeterministic polynomial-time complete (NP-complete) problem. This paper formulated the TSR problem as a multi-objective optimization problem; and adapted the heuristic binary bat algorithm (BBA) to resolve it. The BBA algorithm was adapted in order to enhance its exploration capabilities during the search for Pareto-optimal solutions. The effectiveness of the proposed multi-objective adapted binary bat algorithm (MO-ABBA) was evaluated using 8 test suites of different sizes, in addition to twelve benchmark functions. Experimental results showed that, for the same fault discovery rate, the MO-ABBA is capable of reducing the test suite size more than each of the multi-objective original binary bat (MO-BBA) and the multi-objective binary particle swarm optimization (MO-BPSO) algorithms. Moreover, MO-ABBA converges to the best solutions faster than each of the MO-BBA and the MO-BPSO.

Applying Cluster-based Approach to Improve the Effectiveness of Test Suite Reduction

Applying Cluster-based Approach to Improve the Effectiveness of Test Suite Reduction

Analysis and assessment of weighted combinatorial criterion for test suite reduction

AbstractSoftware plays an important role in modern society. New software features have to be completed and tested rapidly due to business and user needs and goals. To ensure the quality of modified software components, new test cases must be generated and included in the current test pool. Furthermore, test pool size often grows too large to test all cases. A considerably large test pool typically consumes an exorbitant quantity of time, leading to inefficient regression testing. Test suite reduction is often used to solve this problem by removing redundant test cases. Thus, time and resource constraints make it necessary to minimize the test suite. That is, test pool size should be efficiently reduced, while the remaining test cases should be adequate to provide the same coverage as the pool. The main purpose of this paper is to propose four weighted combinatorial algorithms and a three‐objective binary integer linear programming (ILP) model to improve fault detection effectiveness. We first consider both the control flow and data flow as our testing criteria and combine them by using a weighted means with four existing test suite reduction algorithms. A genetic algorithm is used to find the best combination of weighting factors to assign to the two testing criteria. Experiments based on real subject programs are presented. Experimental results demonstrate that 77.5% of our proposed weighted combinatorial algorithms have an even or better performance in suite size reduction, 83.75% of our approaches perform even or better in fault detection effectiveness, and 82.5% of our approaches have even or better effectiveness in fault‐to‐test ratio. For the three‐objective binary ILP model, test suite size is significantly reduced without any accompanying loss in fault detection ability.

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

Regression test selection (RTS) approaches reduce the cost of regression testing of evolving software systems. Existing RTS approaches based on UML models use behavioral diagrams or a combination of structural and behavioral diagrams. However, in practice, behavioral diagrams are incomplete or not used. In previous work, we proposed a fuzzy logic based RTS approach called FLiRTS that uses UML sequence and activity diagrams. In this work, we introduce FLiRTS 2, which drops the need for behavioral diagrams and relies on system models that only use UML class diagrams, which are the most widely used UML diagrams in practice. FLiRTS 2 addresses the unavailability of behavioral diagrams by classifying test cases using fuzzy logic after analyzing the information commonly provided in class diagrams. We evaluated FLiRTS 2 on UML class diagrams extracted from 3331 revisions of 13 open-source software systems, and compared the results with those of code-based dynamic (Ekstazi) and static (STARTS) RTS approaches. The average test suite reduction using FLiRTS 2 was 82.06%. The average safety violations of FLiRTS 2 with respect to Ekstazi and STARTS were 18.88% and 16.53%, respectively. FLiRTS 2 selected on average about 82% of the test cases that were selected by Ekstazi and STARTS. The average precision violations of FLiRTS 2 with respect to Ekstazi and STARTS were 13.27% and 9.01%, respectively. The average mutation score of the full test suites was 18.90%; the standard deviation of the reduced test suites from the average deviation of the mutation score for each subject was 1.78% for FLiRTS 2, 1.11% for Ekstazi, and 1.43% for STARTS. Our experiment demonstrated that the performance of FLiRTS 2 is close to the state-of-art tools for code-based RTS but requires less information and performs the selection in less time.

A Multi-Objective Approach for Test Suite Reduction During Testing of Web Applications

During the development and maintenance phases of evolving software, new test cases would be needed for the verification of the accuracy of the modifications as well as for new functionalities leading to an increase in the size of the test suite. Various related objectives are to be kept in mind while reducing the original test suite by removing redundancy and generating a practical representative set of the unique test cases, some of which may need to be maximized and the remaining ones minimized. This paper presents a multi-objective approach for the test suite reduction problem in which one objective is to be minimized and the remaining two maximized. In this study, experiments were performed on diverse versions of four web applications. Various state-of-the-art algorithms and their updated versions were compared with non-dominated sorting genetic algorithm-II (NSGA-II) for performance evaluation. Based on experimental findings, it was concluded that NSGA-II outperforms all other algorithms; moreover, the algorithm attempts to satisfy all the objectives without compromising coverage.

A Hybrid Technique for Selection and Minimization of Test Cases in Regression Testing

Background: In today’s era, modifications in a software is a common requirement by customers. When changes are made to existing software, re-testing of all the test cases is required to ensure that the newly introduced changes do not have any unwanted effect on the behavior of the software. However, retesting of all the test cases would not only be time consuming but also expensive. Therefore, there is a need for a technique that reduces the number of tests to be performed. Regression testing is one of the ways to reduce the number of test cases. Selection technique is one such method which seeks to identify the test cases that are relevant to some set of recent changes. Objective: It is evident that most of the studies have used different selection techniques and have focused only on one parameter for achieving reduced test suite size without compromising the performance of regression testing. However, to the best of our knowledge, no study has taken two or more parameters of coverage, and/or execution time in a single testing. This paper presents a hybrid technique that combines both regression test selection using slicing technique and minimization of test cases using modified firefly algorithm with combination of parameters coverage and execution time in a single testing. Methods: A hybrid technique has been described that combines both selection and minimization. Selection of test cases is based upon slicing technique while minimization is done using firefly algorithm. Hybrid technique selects and minimizes the test suite using information on statement coverage and execution time. Results: The proposed technique gives 43.33% much superior result as compared to the other hybrid approach in terms of significantly reduced number of test cases. It shows that the resultant test cases were effective enough to cover 100% of the statements, for all the programs. The proposed technique was also tested on four different programs namely Quadratic, Triangle, Next day, Commission respectively for test suite selection and minimization which gave comparatively superior result in terms of reduction (%) in number of test cases required for testing. Conclusion: The combination of parameters used in slicing based approach, reduces the number of test cases making software testing an economical, feasible and time saving option without any fault in the source code. This proposed technique can be used by software practitioners/experts to reduce time, efforts and resources for selection and minimization of test cases.

Test Suite Reduction and Prioritization Framework in Regression Testing

Most of the test cases minimization reduced test cases during regression testing to generate new test suite to cover the same software requirements.The objective of this paper is to present new framework that integrate the idea of minimization and prioritization.Hence, reduction and prioritization able to reduce test cases based on the statements covered by the previous test cases to avoid redundancy.Beginning from the reduction of the test cases, followed by weighted prioritizationaccording to their usefulness.The framework was tested using sample test suite and the results obtained shown increases on the average percentage of faults detection (APFD). Future plan is to test on the larger size of test suite.

Search-Based Regression Testing Optimization

Regression testing is one of the essential activities during the maintenance phase of software projects. It is executed to ensure the validity of an altered software. However, as the software evolves, regression testing becomes prohibitively expensive. In order to reduce the cost of regression testing, it is mandatory to reduce the size of the test suite by selecting the most representative test cases that do not compromise the effectiveness of the regression testing in terms of fault-detection capability. This problem is known as test suite reduction (TSR) problem, and it is known to be an NP-complete. The paper proposes a multi-objective adapted binary bat algorithm (ABBA) to solve the TSR problem. The original binary bat (OBBA) algorithm was adapted to enhance its exploration capabilities during the search for a Pareto-optimal surface. The effectiveness of the ABBA was evaluated using six Java programs with different sizes. Experimental results showed that for the same fault discovery rate, the ABBA is capable of reducing the test suite size more than the OBBA and the BPSO.