Existing Test Suite Research Articles

Hybrid Data Approach for Selecting Effective test Cases During the Regression Testing

Abstract In the software industry, software testing becomes more important in the entire software development life cycle. Software testing is one of the fundamental components of software quality assurances. Software Testing Life Cycle (STLC)is a process involved in testing the complete software, which includes Regression Testing, Unit Testing, Smoke Testing, Integration Testing, Interface Testing, System Testing & etc. In the STLC of Regression testing, test case selection is one of the most important concerns for effective testing as well as cost of the testing process. During the Regression testing, executing all the test cases from existing test suite is not possible because that takes more time to test the modified software. This paper proposes new Hybrid approach that consists of modified Greedy approach for handling the test case selection and Genetic Algorithm uses effective parameter like Initial Population, Fitness Value, Test Case Combination, Test Case Crossover and Test Case Mutation for optimizing the tied test suite. By doing this, effective test cases are selected and minimized the tied test suite to reduce the cost of the testing process. Finally the result of proposed approach compared with conventional greedy approach and proved that our approach is more effective than other existing approach.

Hierarchical regression test case selection using slicing

In this paper, we propose a novel regression test case selection approach by decomposing an object-oriented (OO) program into packages, classes, methods and statements that are affected by some modification made to the program. This decomposition is based on the proposed hierarchical slicing of an OO program. By mapping these decompositions to the existing test suite, we select a new reduced regression test suite and add some new test cases, if necessary, to retest the modified program. We apply hierarchical slicing on a suitable intermediate graph proposed for representing an OO program. This intermediate graph representation corresponds to all the possible dependences among the different parts of an OO program. We improve the scalability of the intermediate graph to a considerable extent by identifying and removing the redundant edges from the graph and thus detect the affected program parts in less time. The average reduction in time achieved for all the ten programs under experimentation is approximately 28.1%. The test cases that cover these affected parts of the program are then selected for regression testing. The average reduction in the number of test cases selected for regression testing of experimental programs is approximately 56.3%.

Hierarchical regression test case selection using slicing

In this paper, we propose a novel regression test case selection approach by decomposing an object-oriented (OO) program into packages, classes, methods and statements that are affected by some modification made to the program. This decomposition is based on the proposed hierarchical slicing of an OO program. By mapping these decompositions to the existing test suite, we select a new reduced regression test suite and add some new test cases, if necessary, to retest the modified program. We apply hierarchical slicing on a suitable intermediate graph proposed for representing an OO program. This intermediate graph representation corresponds to all the possible dependences among the different parts of an OO program. We improve the scalability of the intermediate graph to a considerable extent by identifying and removing the redundant edges from the graph and thus detect the affected program parts in less time. The average reduction in time achieved for all the ten programs under experimentation is approximately 28.1%. The test cases that cover these affected parts of the program are then selected for regression testing. The average reduction in the number of test cases selected for regression testing of experimental programs is approximately 56.3%.

A combined approach to optimize the test suite size in regression testing

Testing is an important software engineering activity which is widely used by testers to reveal faults and improve the quality of the software by omitting or removing the detected faults. After required subsequent modifications in the software, there is a need to re-validate the software at each release. During this, new test cases are to be added to the existing test suite to check the new functionality implemented in the release. Enhancement of the test suite results in obvious test suite size increment and may also contain redundant test cases. Because of time and resource limits, revalidating the improved software every possible time is so tedious and complex. So, it is advantageous to develop such mechanism that increases the coverage and diversity of test cases in a test suite to optimize the results of test suite minimization and prioritization. The proposed approach can be beneficial for testers to solve the regression testing problems.

State-based models in regression test suite prioritization

Testing software products is very expensive and time consuming, especially for large software systems with extensive regression testing. During regression testing, a modified system is often re-tested using an existing test suite. Since test suites can be very large, testers are interested in detecting faults in the modified system as early as possible. Test prioritization tries to order test cases for execution in a way that increases the chances of the early detection of faults. Most of the existing test prioritization methods are based on the code of the system under test, but model-based test prioritization has been lately proposed. Most of the existing model-based test prioritization methods can be used only when models are modified during system maintenance. In this paper, we present model-based prioritization for a class of modifications for which models are not modified (only the source code is modified). After identifying the elements of the model related to the modified source code, information collected during the execution of the model is used to prioritize tests for execution. Here, we present and compare existing and new model-based test prioritization methods focused on this class of modifications. The major motivation for presenting these methods is to provide system developers with simple and yet effective test prioritization techniques for early fault detection. Statistical analysis of the empirical study, which compares the effectiveness of the presented methods in terms of early fault detection, show that compared to random ordering of test cases, model-based test prioritization significantly improve the effectiveness of test prioritization with respect to early fault detection.

Mutant reduction based on dominance relation for weak mutation testing

Context: As a fault-based testing technique, mutation testing is effective at evaluating the quality of existing test suites. However, a large number of mutants result in the high computational cost in mutation testing. As a result, mutant reduction is of great importance to improve the efficiency of mutation testing.Objective: We aim to reduce mutants for weak mutation testing based on the dominance relation between mutant branches.Method: In our method, a new program is formed by inserting mutant branches into the original program. By analyzing the dominance relation between mutant branches in the new program, the non-dominated one is obtained, and the mutant corresponding to the non-dominated mutant branch is the mutant after reduction.Results: The proposed method is applied to test ten benchmark programs and six classes from open-source projects. The experimental results show that our method reduces over 80% mutants on average, which greatly improves the efficiency of mutation testing.Conclusion: We conclude that dominance relation between mutant branches is very important and useful in reducing mutants for mutation testing.

DEVELOPMENT AND VALIDATION OF AN IMPROVED TEST SELECTION AND PRIORITIZATION ALGORITHM BASED ON ACO

Regression testing is an important and often costly software maintenance activity. Retesting the software using existing test suite whenever modifications are made to the system, in order to regain confidence in correctness of the system, is called as Regression Testing. Regression test suites are often too large to re-execute in the given time and cost constraints. Reordering of the test suite is done according to appropriate criteria like code, branch, condition and fault coverage, etc. This process is known as Test Suite Prioritization. We can also select a subset of the original test suite on the basis of some criteria, often called as Regression Test Selection. The research problem that arises from this is the choice of technique or process to be used for selecting and prioritizing according to one or more of the chosen criteria(s). Ant Colony Optimization (ACO) is one such technique that was used by Singh et al. for solving Time-Constrained Test Suite Selection and Prioritization problem using Fault Exposing Potential (FEP). In this paper, we propose improvements to the existing algorithm along with details of the time complexity of the algorithm. It was very convincing to implement the technique considering the results obtained. Implementation of the proposed algorithm has also been demonstrated. The tool was repeatedly run on sample programs by changing the time constraint criterion. The analysis shows the usefulness and effectiveness of using ACO technique for test suite selection and prioritization.

Understanding the effect of time-constraint bounded novel technique for regression test selection and prioritization

It is the demand of our ever-advancing IT industry that software be updated in order to continue its use. Such a modification should not introduce any unwanted new faults in the system. For this, the existing test suite needs to be rerun, often called as regression testing. The main challenge during the regression testing process is not to exceed the desired time and budge deadlines. As a consequence various techniques such as test case selection, minimization and prioritization are used. This paper proposes and analyzes the effect of time constraint on an ant colony optimization based technique for Regression test selection and prioritization. It has been found that with an increase in the applied time constraint, there are more chances to get an optimum selected and prioritized test suite. Also it was found that the complexity of our algorithm depends on the size of the test suite and the applied time constraint and is independent of the number of faults being mutated or any other input variable.

An Optimal Technique for Reducing the Effort of Regression Test

Regression test selection techniques are proposed often but are many times inaccurate when used with larger systems. The proposed new selection technique will be safer, more precise, and can handle the object-oriented features even in larger systems through its phases. Selecting the subset of the test case from the existing test suite is an important problem in regression testing and is addressed in the regression test selection technique. Safe regression test selection technique selects and identifies the program parts that are affected by the change. The test selection is performed by matching the identified change information with the coverage information. A tool is implemented that reduces the testing effort efficiently and the result shows that it can achieve considerable savings in the regression testing time.

An Optimal Technique for Reducing the Effort of Regression Test

Regression test selection techniques are proposed often but are many times inaccurate when used with larger systems. The proposed new selection technique will be safer, more precise, and can handle the object-oriented features even in larger systems through its phases. Selecting the subset of the test case from the existing test suite is an important problem in regression testing and is addressed in the regression test selection technique. Safe regression test selection technique selects and identifies the program parts that are affected by the change. The test selection is performed by matching the identified change information with the coverage information. A tool is implemented that reduces the testing effort efficiently and the result shows that it can achieve considerable savings in the regression testing time.

Regression Test Reduction for Object-Oriented Software: A Control Call Graph Based Technique and Associated Tool

This paper presents a selective regression testing technique and an associated tool for object-oriented software. The technique is based on the concept of Control Call Graphs, which are a reduced form of traditional Control Flow Graphs. It uses static analysis of the source code of the program. The developed tool (1) identifies the Control Call Paths potentially impacted by changes, (2) selects, from an existing test suite, the appropriate test cases, and (3) generates new JUnit test cases for control call paths that are not covered by existing tests (new ones, or those whose structure has been modified after changes). In this way, the approach supports an incremental update of the test suite. The selected JUnit test cases, including the new ones, are automatically executed. Three concrete case studies are reported to provide evidence of the feasibility of the approach and its benefits in terms of reduction of regression testing effort.

Size-Constrained Regression Test Case Selection Using Multicriteria Optimization

To ensure that a modified software system has not regressed, one approach is to rerun existing test cases. However, this is a potentially costly task. To mitigate the costs, the testing effort can be optimized by executing only a selected subset of the test cases that are believed to have a better chance of revealing faults. This paper proposes a novel approach for selecting and ordering a predetermined number of test cases from an existing test suite. Our approach forms an Integer Linear Programming problem using two different coverage-based criteria, and uses constraint relaxation to find many close-to-optimal solution points. These points are then combined to obtain a final solution using a voting mechanism. The selected subset of test cases is then prioritized using a greedy algorithm that maximizes minimum coverage in an iterative manner. The proposed approach has been empirically evaluated and the results show significant improvements over existing approaches for some cases and comparable results for the rest. Moreover, our approach provides more consistency compared to existing approaches.

Validating pragmatic reuse tasks by leveraging existing test suites

SUMMARYTraditional industrial practice often involves the ad hoc reuse of source code that was not designed for that reuse. Such pragmatic reuse tasks play an important role in disciplined software development. Pragmatic reuse has been seen as problematic due to a lack of systematic support, and an inability to validate that the reused code continues to operate correctly within the target system. Although recent work has successfully systematized support for pragmatic reuse tasks, the issue of validation remains unaddressed. In this paper, we present a novel approach and tool to semi‐automatically reuse and transform relevant portions of the test suite associated with pragmatically reused code, as a means to validate that the relevant constraints from the originating system continue to hold, while minimizing the burden on the developer. We conduct a formal experiment with experienced developers, to compare the application of our approach versus the use of a standard IDE (the ‘manual approach’). We find that, relative to the manual approach, our approach: reduces task completion time; improves instruction coverage by the reused test cases; and improves the correctness of the reused test cases. Copyright © 2012 John Wiley & Sons, Ltd.

Regression Optimizer a Multi Coverage Criteria Test Suite Minimization Technique

Regression test suites are developed and maintained throughout the lifetime of the software product. For testers, it is common practice to add new testcases to the existing regression test suite, with intent to test new features in the software product or to capture any newly discovered fault. Many a times the intention is to check whether the program is sufficiently tested or not. This is done by measuring code coverage. In case if not, then additional tests are added until the test suite has achieved a specified coverage level according to a specific criterion. Due to this continuous addition of testcases, regression test suites tend to grow in size. As a result, multiple testcases may exist which may test the same feature or same set of requirements. Test Suite minimization techniques identify redundant test cases from a test suite based on some criterion. In this paper we propose a novel test suite minimization technique which identifies redundancy in a given test suite based on multiple coverage criteria for example function, function call stack, line and branch coverage of given test cases. Paper also talks about the benefits of our approach over other existing test suite minimization techniques. General Terms Regression test suite minimization technique.

Fault-based test suite prioritization for specification-based testing

Context Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in nature and not theoretically driven; hence, they do not necessarily provide sound estimates. Also, these techniques are not applicable when the source code is not available or when the software is tested for the first time. Objective We propose and develop the novel notion of fault-based prioritization of test cases which directly utilizes the theoretical knowledge of their fault-detecting ability and the relationships among the test cases and the faults in the prescribed fault model, based on which the test cases are generated. Method We demonstrate our approach of fault-based prioritization by applying it to the testing of the implementation of logical expressions against their specifications. We then validate our proposal by an empirical study that evaluates the effectiveness of prioritization techniques using two different metrics. Results A theoretically guided fault-based prioritization technique generally outperforms other techniques under study, as assessed by two different metrics. Our empirical results also show that the technique helps to reveal all target faults by executing only about 72% of the prioritized test suite, thereby reducing the effort required in testing. Conclusions The fault-based prioritization approach is not only applicable to the instance empirically validated in this paper, but should also be adaptable to other fault-based testing strategies. We also envisage new research directions to be opened up by our work.

USING PROGRAM SLICING TO IMPROVE THE EFFICIENCY AND EFFECTIVENESS OF CLUSTER TEST SELECTION

Cluster test selection is a new successful approach to select a subset of the existing test suite in regression testing. In this paper, program slicing is introduced to improve the efficiency and effectiveness of cluster test selection techniques. A static slice is computed on the modified code. The execution profile of each test case is filtered by the program slice to highlight the parts of software affected by modification, called slice filtering. The slice filtering reduces the data dimensions for cluster analysis, such that the cost of cluster test selection is saved dramatically. The experiment results show that the slice filtering techniques could reduce the cost of cluster test selection significantly and could also improve the effectiveness of cluster test selection modestly. Therefore, cluster test selection by filtering has more potential scalability to deal with large software.

Regression test suite prioritization using system models

SUMMARYDuring regression testing, a modified system is often retested using an existing test suite. Since the size of the test suite may be very large, testers are interested in detecting faults in the modified system as early as possible during this retesting process. Test prioritization attempts to order tests for execution so that the chances of early detection of faults during retesting are increased. The existing prioritization methods are based on the source code of the system under test. In this paper, we present and evaluate two model‐based selective methods and a dependence‐based method of test prioritization utilizing the state‐based model of the system under test. These methods assume that the modifications are made both on the system under test and its model. The existing test suite is executed on the system model and information about this execution is used to prioritize tests. Execution of the model is inexpensive as compared with execution of the system under test; therefore, the overhead associated with test prioritization is relatively small. In addition, we present an analytical framework for evaluation of test prioritization methods. This framework may reduce the cost of evaluation as compared with the framework that is based on observation. We have performed an empirical study in which we compared different test prioritization methods. The results of the empirical study suggest that system models may improve the effectiveness of test prioritization with respect to early fault detection. Copyright © 2011 John Wiley & Sons, Ltd.

A survey of combinatorial testing

Combinatorial Testing (CT) can detect failures triggered by interactions of parameters in the Software Under Test (SUT) with a covering array test suite generated by some sampling mechanisms. It has been an active field of research in the last twenty years. This article aims to review previous work on CT, highlights the evolution of CT, and identifies important issues, methods, and applications of CT, with the goal of supporting and directing future practice and research in this area. First, we present the basic concepts and notations of CT. Second, we classify the research on CT into the following categories: modeling for CT, test suite generation, constraints, failure diagnosis, prioritization, metric, evaluation, testing procedure and the application of CT. For each of the categories, we survey the motivation, key issues, solutions, and the current state of research. Then, we review the contribution from different research groups, and present the growing trend of CT research. Finally, we recommend directions for future CT research, including: (1) modeling for CT, (2) improving the existing test suite generation algorithm, (3) improving analysis of testing result, (4) exploring the application of CT to different levels of testing and additional types of systems, (5) conducting more empirical studies to fully understand limitations and strengths of CT, and (6) combining CT with other testing techniques.

Regression Test Selection Technique for Multi-Programming Language

Regression testing is a maintenance activity applied to modified software to provide confidence that the changed parts are correct and that the unchanged parts have not been adversely affected by the modifications. Regression test selection techniques reduce the cost of regression testing, by selecting a subset of an existing test suite to use in retesting modified programs. This paper presents the first general regression-test-selection technique, which based on code and allows selecting test cases for any programs written in any programming language. Then it handles incomplete program. We also describe RTSDiff, a regression-test-selection system that implements the proposed technique. The results of the empirical studied that performed in four programming languages java, C#, C++ and Visual basic show that the efficiency and effective in reducing the size of test suit.

Evaluating the Commissioning and QA of Intensity Modulated Arc Therapy using a Benchmark IMRT Planning Suite

Volumetric Intensity Modulated Arc Therapy (IMAT) is often implemented using a planning and delivery system that has already been commissioned for static beam IMRT. This study is designed to provide benchmark data for commissioning IMAT on these dual-purpose systems. The primary aim of this phase of the study was to assess whether an existing test suite could be used for evaluating IMAT commissioning and QA, with an emphasis on comparisons to traditional IMRT delivery techniques.