Test Suite Reduction Method Research Articles

Literature Review on Software Testing Techniques and Test Suite Reduction (TSR) Frameworks/tools

Testers scrutinise the software application with respect to user and technical requirements. Software testing tends to become a chaotic process when it is performed on a daily basis. The test case generation is an employment-intensive task and it is unachievable to accomplish all the test cases. In this paper various test suite reduction (TSR) methods are discussed to solve the test suite size problem heuristically. Over the past few years, numerous test suite reduction (TSR) frameworks/tools are formulated to minimise the testing effort and to test with least execution time within the limited resource. The main motive of this paper is to elaborate various testing techniques and to emphasise on TSR related work to highlight the critical aspects. The study discloses that the generic TSR framework independent of any programming field and TSR framework for agile environment should be formulated. Finally, the research challenges and issues are presented for future exploration.

Literature review on software testing techniques and test suite reduction frameworks/tools

Testers scrutinise the software application with respect to user and technical requirements. Software testing tends to become a chaotic process when it is performed on a daily basis. The test case generation is an employment-intensive task and it is unachievable to accomplish all the test cases. In this paper various test suite reduction (TSR) methods are discussed to solve the test suite size problem heuristically. Over the past few years, numerous test suite reduction (TSR) frameworks/tools are formulated to minimise the testing effort and to test with least execution time within the limited resource. The main motive of this paper is to elaborate various testing techniques and to emphasise on TSR related work to highlight the critical aspects. The study discloses that the generic TSR framework independent of any programming field and TSR framework for agile environment should be formulated. Finally, the research challenges and issues are presented for future exploration.

Fault coverage-based test suite optimization method for regression testing: learning from mistakes-based approach

This paper presents a novel method referred as fault coverage-based test suite optimization (FCBTSO) for regression test suite optimization. FCBTSO is proposed based on Harrolds–Gupta–Soffa (HGS) test suite reduction method, and it follows the phenomenon: “learning from mistakes”. We conducted computational experiments on 12 versions of benchmarked programs retrieved from software artefact infrastructure repository and dummy fault matrix test. The performance of the proposed FCBTSO is measured against the traditional test suite reduction methods (Greedy method, Additional Greedy, HGS, and Enhanced HGS) by following the performance measures: fault coverage, execution time and reduced optimized test suite size. Rigorous statistical tests are conducted to determine the performance significance, which indicates that FCBTSO outperforms other approaches implemented with respect to the execution time that includes the execution time of the proposed approach to find the optimized test suite and the execution time of test cases in the optimized test suite.

Probability Model-Based Test Suite Reduction

The paper presents a test suite reduction approach based on probability models in regression testing. First, we model the process of software testing by using the joint probability of both test cases and program faults, and propose two probability models for the software's old and new versions, respectively. Then, a theorem combining set operations and probability models is given to implement our approach. Different from the traditional coverage-based test suite reduction methods, the reduced test suite constructed by our approach does not need to cover all test requirements, but also remains the same fault detection capability of the original test suite.The paper presents a test suite reduction approach based on probability models in regression testing. First, we model the process of software testing by using the joint probability of both test cases and program faults, and propose two probability models for the software's old and new versions, respectively. Then, a theorem combining set operations and probability models is given to implement our approach. Different from the traditional coverage-based test suite reduction methods, the reduced test suite constructed by our approach does not need to cover all test requirements, but also remains the same fault detection capability of the original test suite.

A new test suite reduction method for wearable embedded software

The verification of wearable embedded software trustability plays a prominent role in the research of wearable embedded technology. Software testing is an alternative way by which to improve the trustability of embedded software systems. The test suites developed are often reused because of the frequent version updates of wearable embedded software; as a result, many test cases in the test suite become redundant. In this paper, a new Boolean subtraction operation is proposed, one that does not alter the coverage of test requirements by the test suites. Redundant requirements and cases are reduced by a column–row transformation operation through the newly defined operation, and the interrelation among the testing requirements and test cases is taken into consideration. The results of experiments on some wearable embedded software test suite reduction indicate that the proposed method results in improved properties compared with H, GE, GRE, and can generate a minimal test suite with which all test requirements can be sufficiently tested.

Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity

Test suite reduction is an important approach that decreases the cost of regression testing. A test suite reduction technique operates based on the relationship between the test cases in the regression test suite and the test requirements in the program under test. Thus, its effectiveness should be closely related to the complexity of a regression test suite – the product of the number of test cases and the number of test requirements. Our previous work has shown that cost-aware techniques (i.e., the test suite reduction techniques that aim to decrease the regression test suite's execution cost) generally outperform the others in terms of decreasing the cost of running the regression test suite. However, the previous empirical studies that evaluated cost-aware techniques did not take into account test suite complexity. That is, prior experiments do not reveal if the cost-aware techniques scale and work effectively on test suites with more test cases and more test requirements. This means that prior experiments do not appropriately shed light on how well test suite reduction methods work with large programs or test suites. Therefore, this paper focuses on the Greedy-based techniques and empirically evaluates the additional Greedy and two cost-aware Greedy techniques – at different levels of test suite complexity – from various standpoints including the cost taken to run the regression test suite, the time taken to reduce the test suites, the total regression testing costs, the fault detection capability, the fault detection efficiency, and the common rates of the representative sets. To the best of our knowledge, none of the previous empirical studies classify a considerable number of test suites according to their complexity. Nor do any prior experiments evaluate the test suite reduction techniques, in terms of the aforementioned criteria, at different levels of test suite complexity. This paper represents the first such attempt to carry out this important task. Based on the empirical results, we confirm the strengths and weaknesses of the cost-aware techniques and develop insights into how the cost-aware techniques' effectiveness varies as the test suite complexity increases. • We evaluate test reduction methods at various levels of test suite complexity. • Few previous studies evaluated the test reduction methods in the way that we did. • The cost-aware methods generally attain the lowest total regression testing costs. • The cost-aware methods generally realize higher fault detection efficiency. • The benefits of using cost-aware methods increase as test suite complexity grows.

Multi Objective Test Suite Reduction for GUI Based Software Using NSGA-II

Regression Testing is a performed to ensure modified code does not have any unintended side effect on the software. If regression testing is performed with retest-all method it will be very t ime consuming as testing activity. Therefore test suite reduction methods are used to reduce the size of original test suite. Object ive of test suite reduction is to reduce those test cases which are redundant or less important in their fau lt revealing capability. Test suite reduction can only be used when time is critical to run all test cases and selective testing can only be done. Various methods exist in the literature related to test suite reduction of traditional software. Most of the methods are based of single objective optimization. In case of mult i ob jective optimization o f test suite, usually researchers assign different weight values to different objectives and co mbine them as single object ive. However in test suite reduction mu ltiple Pareto-optimal solutions are present, it is difficult to select one test case over other. Since GUI based software is our concern there exist very few reduction techniques and none of them consider mult iple objective based reduction. In this work we propose a new test suite reduction technique based on two objectives, event weight and number of faults identified by test case. We evaluated our results for 2 different applications and we achieved 20% reduction in test suite size for both applications. In Terp Paint 3.0 application co mpro mise 15.6% fau lt revealing capability and for Notepad 11.1% fau lt revealing capability is reduced.

Evaluation and analysis of incorporating Fuzzy Expert System approach into test suite reduction

ContextSoftware has become increasingly important in our modern society. However, when new features are developed due to user requests, such requests could make the sizes of test-case pools bigger. Many techniques are proposed to solve this problem, such as test suite reduction. However, the ability to expose faults may be weakened when reducing the sizes of the test suites. In this paper, we propose some methods using fuzzy logic in order to improve existing test-suite reduction techniques. ObjectiveThe main purpose of this research is to use a Fuzzy Expert System approach in order to enhance the effectiveness of fault detection during software testing. MethodIncorporating a Fuzzy Expert System into traditional test suite reduction techniques is presented and studied. More objective criteria are used in order to compare the performance of our proposed and selected test suite reduction methods. Some important measures (and metrics) will also be obtained and discussed. Application of the Fuzzy Expert System approach for test case prioritization is also discussed. ResultsThe experiments in three improved test-suite reduction techniques show that the modified algorithms can reduce the size of test suites, which have improved the fault detection quality. ConclusionDuring software testing, test data are generally classified with Boolean logic. This method can classify data into groups easily. However, there may be ambiguity in classifications due to similar properties for certain data. Ambiguous data can be classified in each group. In this study, Boolean logic will be replaced by fuzzy logic. Incorporating the Fuzzy Expert System approach into three traditional test suite reduction techniques (i.e., HGS, GRE, and Greedy) is presented and evaluated. The experiments, based on nine real subject programs ranging from 173 LOCs to 35,545 LOCs, have demonstrated that our proposed Fuzzy-HGS, Fuzzy-GRE, and Fuzzy-Greedy algorithms can significantly reduce the sizes of test suites while also improving fault detection effectiveness. For instance, Fuzzy-HGS, Fuzzy-GRE, and Fuzzy-Greedy algorithms have almost the same reduction capability of test suite as traditional HGS, GRE, and Greedy algorithms. But in terms of the percentage of fault detection effectiveness loss (FDE loss), both Fuzzy-HGS and Fuzzy-GRE algorithms are averagely decreased by 21% and 5%, respectively. Additionally, Fuzzy-Greedy algorithm still provide the lower FDE loss for large subject programs compared to traditional Greedy algorithm. Based upon the integrated theoretical foundation, the approaches presented in this paper offer an efficient, useful software testing scheme in the testing and debugging phases.

Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests

ContextIn software development and maintenance, a software system may frequently be updated to meet rapidly changing user requirements. New test cases will be designed to ensure the correctness of new or modified functions, thus gradually increasing the test suite’s size. Test suite reduction techniques aim to decrease the cost of regression testing by removing the redundant test cases from the test suite and then obtaining a representative set of test cases that still yield a high level of code coverage. ObjectiveMost of the existing reduction algorithms focus on decreasing the test suite’s size. Yet, the differences in execution costs among test cases are usually significant and it may take a lot of execution time to run a test suite consisting of a few long-running test cases. This paper presents and empirically evaluates cost-aware algorithms that can produce the representative sets with lower execution costs. MethodWe first use a cost-aware test case metric, called Irreplaceability, and its enhanced version, called EIrreplaceability, to evaluate the possibility that each test case can be replaced by others during test suite reduction. Furthermore, we construct a cost-aware framework that incorporates the concept of test irreplaceability into some well-known test suite reduction algorithms. ResultsThe effectiveness of the cost-aware framework is evaluated via the subject programs and test suites collected from the Software-artifact Infrastructure Repository — frequently chosen benchmarks for experimentally evaluating test suite reduction methods. The empirical results reveal that the presented algorithms produce representative sets that normally incur a low cost to yield a high level of test coverage. ConclusionThe presented techniques indeed enhance the capability of the traditional reduction algorithms to reduce the execution cost of a test suite. Especially for the additional Greedy algorithm, the presented techniques decrease the costs of the representative sets by 8.10–46.57%.

A Test Suite Reduction Method based on Test Requirement Partition

Test suite reduction aims at improving the effectiveness of testing and cutting down the test cost with the least test cases under the condition of satisfying all testing objectives. This paper proposes a new method for test suite reduction with test requirement partition. First, it gives a partition to the set of all the available test cases based on the test requirements. After that, a test suite by the partition is generated and then a smaller test suite is obtained by further reduction according to the relationship between test requirements and test suites. At last, the experimental result shows that the proposed approach is helpful to generate the reduced test suite, which is used to test all the test requirements sufficiently by comparing with the existed methods, at the cost of a moderate loss in fault detection capability.

A Constraint-based Test Suite Reduction Method for Conservative Regression Testing

In regression testing, an important problem is how to select a smaller size of test set for execution. We present a novel constraint-oriented test suite reduction method for conservative regression testing by which we mean that all bugs discovered must be revealed by the reduced test suite. A test constraint for a bug is Boolean formulas defined over the input variables of program under test. The reduced test constraints for a pool of bugs are constructed using the subsumption relationship between test constraint conditions. Test case selection is based on the reduced test constraint set. A test case is selected into the test suite when and only when it satisfies one or more test constraints. The selection process is completed when all test constraint conditions are satisfied by the selected test cases. An empirical study is conducted and the experimental results show that our method can significantly save efforts for the conservative regression testing.

Regression test suite reduction based on SDL models of system requirements

AbstractThis paper proposes a model‐based regression test suite reduction method. The proposed method considers an SDL model representing the requirements of a system under test and a set of modifications on this model, applies dependence analysis to identify interaction patterns related to each type of modifications, i.e., adding, deleting, and changing transitions in the SDL model, and reduces the size of a given regression test suite by examining interaction patterns covered by each test case in the test suite. Results of empirical studies are reported. Copyright © 2009 John Wiley & Sons, Ltd.