Test Suite Reduction Research Articles

Applying Concept Analysis to User-Session-Based Testing of Web Applications

The continuous use of the web for daily operations by businesses, consumers, and the government has created a great demand for reliable web applications. One promising approach to testing the functionality of web applications leverages user-session data collected by web servers. User-session-based testing automatically generates test cases based on real user profiles. The key contribution of this paper is the application of concept analysis for clustering user sessions and a set of heuristics for test case selection. Existing incremental concept analysis algorithms are exploited to avoid collecting and maintaining large user-session data sets and thus to provide scalability. We have completely automated the process from user session collection and test suite reduction through test case replay. Our incremental test suite update algorithm coupled with our experimental study indicate that concept analysis provides a promising means for incrementally updating reduced test suites in response to newly captured user sessions with little loss in fault detection capability and program coverage.

An experimental study of four typical test suite reduction techniques

In software development, developers often rely on testing to reveal bugs. Typically, a test suite should be prepared before initial testing, and new test cases may be added to the test suite during the whole testing process. This may usually cause the test suite to contain more or less redundancy. In other words, a subset of the test suite (called the representative set) may still satisfy all the test objectives. As the redundancy can increase the cost of executing the test suite, quite a few test suite reduction techniques have been brought out in spite of the NP-completeness of the general problem of finding the optimal representative set of a test suite. In the literature, there have been some experimental studies of test suite reduction techniques, but the limitations of the these experimental studies are quite obvious. Recently proposed techniques are not experimentally compared against each other, and reported experiments are mainly based on small programs or even simulation data. This paper presents a new experimental study of the four typical test suite reduction techniques, including Harrold et al.’s heuristic, and three other recently proposed techniques such as Chen and Lau’s GRE heuristic, Mansour and El-Fakin’s genetic algorithm-based approach, and Black et al.’s ILP-based approach. Based on the results of this experimental study, we also provide a guideline for choosing the appropriate test suite reduction technique and some insights into why the techniques vary in effectiveness and efficiency.

On the effect of test-suite reduction on automatically generated model-based tests

Model checking techniques can be successfully employed as a test-case generation technique to generate tests from formal models. The number of tests-cases produced, however, is typically large for complex coverage criteria such as MC/DC. Test-suite reduction can provide us with a smaller set of test-cases that preserve the original coverage--often a dramatically smaller set. Nevertheless, one potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing implementations derived from formal models of software we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results indicate that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications.

Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction

A concept analysis inspired greedy algorithm for test suite minimization

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible and to ensure that changes to existing software do not break the software. Test suites once developed are reused and updated frequently as the software evolves. As a result, some test cases in the test suite may become redundant as the software is modified over time since the requirements covered by them are also covered by other test cases. Due to the resource and time constraints for re-executing large test suites, it is important to develop techniques to minimize available test suites by removing redundant test cases. In general, the test suite minimization problem is NP complete. In this paper, we present a new greedy heuristic algorithm for selecting a minimal subset of a test suite T that covers all the requirements covered by T. We show how our algorithm was inspired by the concept analysis framework. We conducted experiments to measure the extent of test suite reduction obtained by our algorithm and prior heuristics for test suite minimization. In our experiments, our algorithm always selected same size or smaller size test suite than that selected by prior heuristics and had comparable time performance.

Analyzing clusters of web application user sessions

User sessions provide valuable insight into the dynamic behavior of web applications. They also play a key role in user-session-based testing, which gathers user sessions in the field and replays selected sessions to test an evolving application. To decrease the testing and analysis effort, testers reduce the set of collected user sessions by either clustering user sessions by their shared URL attributes or by program coverage requirements-based reduction techniques. Clustering URL attributes can be considerably less expensive; however, the tradeoff may be that clustering is not representative of dynamic behavior similarities. This paper describes our analysis of user session data to reveal correlations between sessions clustered on the sessions' attributes and the relative dynamic behavior of the program for those sessions. The results of our analysis also motivate other clustering and test suite reduction techniques. Our results can also be used to learn more about how clusters of web application use cases are related in terms of the underlying user session attributes, program coverage, and fault detection.

On test suite composition and cost-effective regression testing

Regression testing is an expensive testing process used to revalidate software as it evolves. Various methodologies for improving regression testing processes have been explored, but the cost-effectiveness of these methodologies has been shown to vary with characteristics of regression test suites. One such characteristic involves the way in which test inputs are composed into test cases within a test suite. This article reports the results of controlled experiments examining the effects of two factors in test suite composition---test suite granularity and test input grouping---on the costs and benefits of several regression-testing-related methodologies: retest-all, regression test selection, test suite reduction, and test case prioritization. These experiments consider the application of several specific techniques, from each of these methodologies, across ten releases each of two substantial software systems, using seven levels of test suite granularity and two types of test input grouping. The effects of granularity, technique, and grouping on the cost and fault-detection effectiveness of regression testing under the given methodologies are analyzed. This analysis shows that test suite granularity significantly affects several cost-benefit factors for the methodologies considered, while test input grouping has limited effects. Further, the results expose essential tradeoffs affecting the relationship between test suite design and regression testing cost-effectiveness, with several implications for practice.

Test-suite reduction and prioritization for modified condition/decision coverage

Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be modified condition/decision coverage (MC/DC) adequate. Despite its cost, there is evidence that MC/DC is an effective verification technique and can help to uncover safety faults. As the software is modified and new test cases are added to the test suite, the test suite grows and the cost of regression testing increases. To address the test-suite size problem, researchers have investigated the use of test-suite reduction algorithms, which identify a reduced test suite that provides the same coverage of the software according to some criterion as the original test suite, and test-suite prioritization algorithms, which identify an ordering of the test cases in the test suite according to some criteria or goals. Existing test-suite reduction and prioritization techniques, however, may not be effective in reducing or prioritizing MC/DC-adequate test suites because they do not consider the complexity of the criterion. This paper presents new algorithms for test-suite reduction and prioritization that can be tailored effectively for use with MC/DC. The paper also presents the results of empirical studies of these algorithms.

On the divide-and-conquer approach towards test suite reduction

When testing a program, testers need to generate a set of test cases that satisfies a testing objective. This set of test cases is referred to as a test suite. Usually, the generated test suite may contain redundancy, that is, some test cases may be removed from the test suite without rendering the satisfaction of the testing objective. A representative set is a subset of a test suite that satisfies the same testing objective, and test suite reduction is aimed at finding the representative sets. This paper studies the properties of representative sets related to the divide-and-conquer approach towards test suite reduction. The results provide us a better understanding of the structural properties of representative sets and how to apply the divide-and-conquer approach towards test suite reduction.

Empirical studies of test‐suite reduction

AbstractTest‐suite reduction techniques attempt to reduce the costs of saving and reusing test cases during software maintenance by eliminating redundant test cases from test suites. A potential drawback of these techniques is that reducing the size of a test suite might reduce its ability to reveal faults in the software. Previous studies have suggested that test‐suite reduction techniques can reduce test‐suite size without significantly reducing the fault‐detection capabilities of test suites. These studies, however, involved particular programs and types of test suites, and to begin to generalize their results, further work is needed. This paper reports on the design and execution of additional studies, examining the costs and benefits of test‐suite reduction, and the factors that influence these costs and benefits. In contrast to previous studies, results of these studies reveal that the fault‐detection capabilities of test suites can be severely compromised by test‐suite reduction. Copyright © 2002 John Wiley & Sons, Ltd.

On the Completeness of a Test Suite Reduction Strategy

On the Completeness of a Test Suite Reduction Strategy

A simulation study on some heuristics for test suite reduction

In order to test a program, software testers must first define a testing objective. From this a test suite is then constructed. In fact, some subsets of the constructed test suite may still satisfy the same testing objective. Such subsets are referred to as representative sets of the test suite. Finding the optimal representative sets of the test suite reduction problem is known to be NP-complete. Different heuristics, including G, GE, grE and H, have been proposed by different researchers. However, it has been demonstrated that none of these four heuristics are guaranteed to deliver smaller sized representative sets. This paper presents the result of a simulation study of these four heuristics. The aim of the study is to provide application guidelines for choosing the most appropriate heuristic for test suite reduction.

A new heuristic for test suite reduction

A testing objective has to be defined in testing a program. A test suite is then constructed to satisfy the testing objective. The constructed test suite contains redundancy when some of its proper subsets can still satisfy the same testing objective. Since the costs of executing test cases and maintaining a test suite for regression testing may be expensive, the problem of test suite reduction arises. This paper proposes a heuristic towards the optimization of a test suite.

Comments on “a reduced test suite for protocol conformance testing”

A previous ACM TOSEM article of Ph. Bernhard (“A Reduced Test Suite of Protocol Conformance Testing,” ACM Transactions on Software Engineering and Methodology, Vol. 3, No. 3, July 1994, pages 201-220) describes three new versions of the so-called W-method for solving the protocol-testing problem, i.e., solving the Mealy machine equivalence problem. The author claims that these versions all have the same fault detection capability as the original W-method. In this correspondence we prove that the results of that article are incorrect.

A reduced test suite for protocol conformance testing

A reduced test suite for protocol conformance testing