Test Case Reduction Research Articles

Evaluation of the fixed‐point iteration of minimizing delta debugging

AbstractThe minimizing Delta Debugging (DDMIN) was among the first algorithms designed to automate the task of reducing test cases. Its popularity is based on the characteristics that it works on any kind of input, without knowledge about the input structure. Several studies proved that smaller outputs can be produced faster with more advanced techniques (e.g., building a tree representation of the input and reducing that data structure); however, if the structure is unknown or changing frequently, maintaining the descriptors might not be resource‐efficient. Therefore, in this paper, we focus on the evaluation of the novel fixed‐point iteration of minimizing Delta Debugging (DDMIN*) on publicly available test suites related to software engineering. Our experiments show that DDMIN* can help reduce inputs further by 48.08% on average compared to DDMIN (using lines as the units of the reduction). Although the effectiveness of the algorithm improved, it comes with the cost of additional testing steps. This study shows how the characteristics of the input affect the results and when it pays off using DDMIN*.

Optimizing Mutation-based Fault Localization through Contribution-based Test Case Reduction

Optimizing Mutation-based Fault Localization through Contribution-based Test Case Reduction

Test case reduction and SWOA optimization for distributed agile software development using regression testing

Test case reduction and SWOA optimization for distributed agile software development using regression testing

Delta4Ms: Improving mutation‐based fault localization by eliminating mutant bias

AbstractFault localization is a complex, costly and time‐consuming task in software debugging. Numerous automated techniques have been developed to expedite this process. Mutation‐based fault localization (MBFL) is one of the most widely studied techniques which uses mutation analysis to generate mutants for revealing potential faults in the program. However, our theoretical analysis exposes an inherent conflict between the fundamental assumption and the essential meaning of existing MBFL suspiciousness. This conflict is caused by mutant bias. Intuitively, the suspiciousness can be corrected by eliminating the mutant bias for more accurately measuring the faulty probability of the corresponding mutant statement. In this paper, we introduce Delta4Ms, a fault localization approach designed to eliminate mutant bias. Delta4Ms integrates the principles of signal theory, modelling the actual suspiciousness and mutant bias as the desired and false signal components, respectively. Based on theoretical derivation, the average suspiciousness of mutants serves as an estimate of mutant bias. Delta4Ms effectively mitigates mutant bias, extracting the desired signal and yielding corrected suspiciousness for fault localization. To precisely estimate mutant bias, higher order mutants (HOMs) are incorporated. We conduct an extensive experimental evaluation of Delta4Ms on 320 real‐fault programs from Codeflaws. The results indicate that our model significantly outperforms existing SBFL and MBFL techniques, showing a considerable improvement in fault localization effectiveness. We further assessed the robustness of Delta4Ms by examining different HOM ratios and HOM generation strategies. Moreover, Delta4Ms achieves a substantial reduction in mutation execution cost and minimal accuracy loss through the implementation of test case reduction. Finally, we perform preliminary experiments on 15 real‐fault programs from the Defects4J benchmark to assess the generalization of the model's fault localization effectiveness.

An empirical study to compare three web test automation approaches: NLP‐based, programmable, and capture&replay

AbstractA new advancement in test automation is the use of natural language processing (NLP) to generate test cases (or test scripts) from natural language text. NLP is innovative in this context and promises of reducing test cases creation time and simplifying understanding for “non‐developer” software testers as well. Recently, many vendors have launched on the market many proposals of NLP‐based tools and testing frameworks but their superiority has never been empirically validated. This paper investigates the adoption of NLP‐based test automation in the web context with a series of case studies conducted to compare the costs of the NLP testing approach—measured in terms of test cases development and test cases evolution—with respect to more consolidated approaches, that is, programmable (or script‐based) testing and capture&replay testing. The results of our study show that NLP‐based test automation appears to be competitive for small‐ to medium‐sized test suites such as those considered in our empirical study. It minimizes the total cumulative cost (development and evolution) and does not require software testers with programming skills.

An Automated Framework for Detecting Change in the Source Code and Test Case Change Recommendation

Improvements and acceleration in software development have contributed towards high-quality services in all domains and all fields of industry, causing increasing demands for high-quality software developments. The industry is adopting human resources with high skills, advanced methodologies, and technologies to match the high-quality software development demands to accelerate the development life cycle. In the software development life cycle, one of the biggest challenges is the change management between the version of the source codes. Various reasons, such as changing the requirements or adapting available updates or technological upgrades, can cause the source code's version. The change management affects the correctness of the software service's release and the number of test cases. It is often observed that the development life cycle is delayed due to a lack of proper version control and due to repetitive testing iterations. Hence the demand for better version control-driven test case reduction methods cannot be ignored. The parallel research attempts propose several version control mechanisms. Nevertheless, most version controls are criticized for not contributing toward the test case generation of reduction. Henceforth, this work proposes a novel probabilistic rule-based test case reduction method to simplify the software development's testing and version control mechanism. Software developers highly adopt the refactoring process for making efficient changes such as code structure and functionality or applying changes in the requirements. This work demonstrates very high accuracy for change detection and management. This results in higher accuracy for test case reductions. The outcome of this work is to reduce the development time for the software to make the software development industry a better and more efficient world.

The k + 1 Symmetric Test Pattern for Smart Contracts

A smart contract is a pivotal notion in blockchain technology. Distributed applications contain smart contracts verifying the fulfillment of the conditions, which determine the execution of transactions between the blockchain network nodes. Those software-controlled logical conditions are called verification rules. As the number of conditions increases, the complexity of smart contract testing rapidly grows. This paper aims to propose a smart contract testing pattern that significantly limits the needed number of test cases. For evaluation expression with four verification rules, the pattern usage reduces the number of test cases by 68.75% in relation to the full coverage of logical value combinations. With the increase in the number of logical conditions, not only the number of test cases but also their percentage decreases. Starting from seven verification rules in the evaluation expression, the percentage reduction of test cases exceeds 90%. As a result, the cost of preparing and maintaining test case suites may be substantially cut. It should be emphasized that test execution time can be reduced even by 3 orders of magnitude (from seconds to milliseconds). Such an approach is highly important for regression testing, especially when used in continuous software integration, delivery, and deployment approaches.

The effect of hoisting on variants of Hierarchical Delta Debugging

AbstractMinimizing failing test cases is an important preprocessing step on the path of debugging. If much of a test case that triggered a bug does not contribute to the actual failure, then the time required to fix the bug can increase considerably. However, test case reduction itself can be a time‐consuming task, especially if done manually. Therefore, automated minimization techniques have been proposed, the minimizing Delta Debugging and the Hierarchical Delta Debugging (HDD) algorithms being the most well known. In this paper, we investigated the input format of HDD, searching for structures that the algorithm cannot reduce. Motivated by the findings, we have created an algorithmic framework that enabled the use of transformations other than pruning. Furthermore, with the Transformation‐based Minimization framework, we propose to extend HDD and its coarse and recursive variants with a reduction method that does not prune subtrees but replaces them with compatible subtrees further down the hierarchy, called hoisting. We have evaluated various combinations of pruning and hoisting on multiple test suites and found that hoisting can help to further reduce the size of test cases by 27% on average and by 80% as best case compared with the baseline algorithm.

Optimum Test Suite Using Fault-Type Coverage-Based Ant Colony Optimization Algorithm

Software Product Lines(SPLs) covers a mixture of features for testing Software Application Program(SPA). Testing cost reduction is a major metric of software testing. In combinatorial testing(CT), maximization of fault type coverage and test suite reduction plays a key role to reduce the testing cost of SPA. Metaheuristic Genetic Algorithm(GA) do not offer best outcome for test suite optimization problem due to mutation operation and required more computational time. So, Fault-Type Coverage Based Ant Colony Optimization(FTCBACO) algorithm is offered for test suite reduction in CT. FTCBACO algorithm starts with test cases in test suite and assign separate ant to each test case. Ants elect best test cases by updating of pheromone trails and selection of higher probability trails. Best test case path of ant with least time are taken as optimal solution for performing CT. Hence, FTCBACO Technique enriches reduction rate of test suite and minimizes computational time of reducing test cases efficiently for CT.

Test Case Prioritization, Selection, and Reduction Using Improved Quantum-Behaved Particle Swarm Optimization.

The emerging areas of IoT and sensor networks bring lots of software applications on a daily basis. To keep up with the ever-changing expectations of clients and the competitive market, the software must be updated. The changes may cause unintended consequences, necessitating retesting, i.e., regression testing, before being released. The efficiency and efficacy of regression testing techniques can be improved with the use of optimization approaches. This paper proposes an improved quantum-behaved particle swarm optimization approach for regression testing. The algorithm is improved by employing a fix-up mechanism to perform perturbation for the combinatorial TCP problem. Second, the dynamic contraction-expansion coefficient is used to accelerate the convergence. It is followed by an adaptive test case selection strategy to choose the modification-revealing test cases. Finally, the superfluous test cases are removed. Furthermore, the algorithm’s robustness is analyzed for fault as well as statement coverage. The empirical results reveal that the proposed algorithm performs better than the Genetic Algorithm, Bat Algorithm, Grey Wolf Optimization, Particle Swarm Optimization and its variants for prioritizing test cases. The findings show that inclusivity, test selection percentage and cost reduction percentages are higher in the case of fault coverage compared to statement coverage but at the cost of high fault detection loss (approx. 7%) at the test case reduction stage.

On test case reduction for testing safety properties of manufacturing systems

This paper presents an approach to reduce the number of test cases, and hence testing time for the safe input-output conformance simulation relation (safe-IOCOS). The safe-IOCOS relation requires the implementation to be trace equivalent with respect to the specification only for traces composed of safety behaviors, which makes safe-IOCOS a suitable relation to test safety properties in practical settings. However, in typical manufacturing systems, multiple safety behaviors are typically associated with each nominal operation in the implementation. Thus, if safe-IOCOS is used industrially then testing for safety related faults becomes time consuming as the traces composed of same safety behaviors gets tested multiple times. This is possible either if the target states reached after the execution of traces have the same past behavior or the same future behavior. To remedy this, two reduction methods are proposed in this paper, subset construction and bisimulation equivalence. Both reduction methods preserve the traces of the system. Using both subset construction and bisimulation, a given specification can be maximally reduced and then used to implement the manufacturing system. The implementation based on a maximally reduced bisimilar specification allows the test engineer to omit test cases if the same safety behavior has already been tested. Furthermore, faults related to missing safety behaviors that are associated with multiple traces can be uncovered more efficiently compared to if the non-reduced specification is used for testing. To summarize, testing is a laborious problem, which can benefit from methods that enable reduction in testing time and makes the testing procedure efficient in terms of uncovering errors.

An Optimized Surrogate Based Technique for Reducing the Mutation Test Cost Using Satin Bower Bird Algorihm

Fault based testing is also known as mutation testing. In this, the faults are voluntarily introduced in the code to check for the feasibility of the program. If the introduced faults does not produce any variations in the program, it results that the program need debugging for perfect operations. It is a costly test due to its intensive operations in finding the bugs. In order to reduce the expense for mutation testing, automated tools and optimization algorithms were introduced. Surrogate based optimization is also a type of reducing mutation testing cost in all possible ways by using three conditions as objective functions. It able to minimize the test case and achieve high mutation score, but only for few code snipped. To bridge this gap, here an enhanced version of surrogate based optimized is proposed. The existing technique is enhanced by using satin bower bird optimization for efficient test case reduction and kill all the mutants in the snippet. The proposed method is tested on few snippets like calculator, hash table and stack and so on using EmuJava. The proposed optimized surrogate based optimization able to outperform the existing technique by having high mutation score of above 80% for all testing Java snippets. It also able to reduce the computational time and cost by reducing the test cases minimum of 950 for stack snippet and only 50 iterations for processing the CGPA snippet. Therefore, the proposed satin bower bird based surrogate optimization able to reduce the computational cost along with high percentage for killing the mutants. Hence, the proposed approach is best for cost minimization in mutation testing.

Test Suite Reduction and Prioritization Framework in Regression Testing

Most of the test cases minimization reduced test cases during regression testing to generate new test suite to cover the same software requirements.The objective of this paper is to present new framework that integrate the idea of minimization and prioritization.Hence, reduction and prioritization able to reduce test cases based on the statements covered by the previous test cases to avoid redundancy.Beginning from the reduction of the test cases, followed by weighted prioritizationaccording to their usefulness.The framework was tested using sample test suite and the results obtained shown increases on the average percentage of faults detection (APFD). Future plan is to test on the larger size of test suite.

Fault-type coverage based ant colony optimization algorithm for attaining smaller test suite

<table width="0" border="1" cellspacing="0" cellpadding="0"><tbody><tr><td valign="top" width="593"><p>In this paper, we proposed Fault-Type Coverage Based Ant Colony Optimization (FTCBACO) technique for test suite optimization. An algorithm starts with initialization of FTCBACO factors using test cases in test suite. Then, assign separate ant to each test case called vertex. Each ant chooses best vertices to attain food source called objective of the problem by means of updating of pheromone trails and higher probability trails. This procedure is repeated up to the ant reaches food source. In FTCBACO algorithm, minimal number of test cases with less execution time chosen by an ant to cover all faults type (objective) are taken as optimal solution. We measured the performance of FTCBACO against Greedy approach and Additional Greedy Approach in terms of fault type coverage, test suite size and execution time. However, the heuristic Greedy approach and Additional Greedy approach required more execution time and maximum test suite size to provide the best resolution for test suite optimization problem. Statistical investigations are performed to finalize the performance significance of FTCBACO with other approaches that concludes FTCBACO technique enriches the reduction rate of test suite and minimizes execution time of reducing test cases efficiently.</p></td></tr></tbody></table>

GUI Test Case Prioritization using Social Network Analysis

Graphical User Interfaces (GUIs) has become popular in today software products due to its user-friendly merit. However, GUI tests are costly and suffer explosive test cases. Moreover, software evolution causes the GUIs inevitably changed resulting in harder testing. Regression test has been widely used for GUIs software testing. It is considered to be the most expensive phase in the software testing process, though. Among several techniques, test case prioritization would help identify the reduced test cases in the regression testing suite, still maintain the same criteria as the original number of test cases in order to solve the limitation of time and cost in this phase. In this paper, we propose using network centrality from social network analysis to prioritize the test cases and compare each parameter to find a suitable parameter for GUI test case prioritization.

A hybrid approach to perform test case prioritisation and reduction for software product line testing

Software Product Line (SPL) is a popular research area in software engineering which deals with many products generated simultaneously. There is a requirement for a tool or framework that can test all the products. SPL implies Test Case Prioritisation (TCP) and Test Case Reduction (TCR) techniques to alleviate the problem of testing. This paper proposes a hybrid approach which combines K-Means and Principal Component Analysis (PCA) approaches to perform SPL testing. The experimental results show that the hybrid approach-based order provides better results than the random, similarity, ICPL algorithm-based order using APFD metric. Our proposed approach reduces the test cases to the minimum amount and is also used to perform efficient SPL testing that not only improves the effectiveness and efficiency of fault detection but also improves the effectiveness of Test Suite Reduction (TSR).

A hybrid approach to perform test case prioritisation and reduction for software product line testing

Software Product Line (SPL) is a popular research area in software engineering which deals with many products generated simultaneously. There is a requirement for a tool or framework that can test all the products. SPL implies Test Case Prioritisation (TCP) and Test Case Reduction (TCR) techniques to alleviate the problem of testing. This paper proposes a hybrid approach which combines K-Means and Principal Component Analysis (PCA) approaches to perform SPL testing. The experimental results show that the hybrid approach-based order provides better results than the random, similarity, ICPL algorithm-based order using APFD metric. Our proposed approach reduces the test cases to the minimum amount and is also used to perform efficient SPL testing that not only improves the effectiveness and efficiency of fault detection but also improves the effectiveness of Test Suite Reduction (TSR).

A Technique for Transitioning Of Plan Driven Software Development Method to Distributed Agile Software Development

In the era of twenty first century where human being is moving forward from semi – automated systems to fully automatic and machine learning based systems, software development (SD) mechanisms have also moved from static and pretended process following mechanisms to fast, ready for advices and adaptability based software development methods. Due to this traditional software development methods following heavy documentation based step by step processes have become obsolete and software development industry is transitioning towards dynamic, incremental, iterative, customer – centered, light – documentation based system development mechanisms referred as Agile Software development (ASD). The transition becomes more attention grabbing in distributed environment referred as distributed Agile software development (DASD). Certain processes like – task allocation, backlog prioritization, test case reduction and prioritization use manual processes rather than semi or fully automated processes. This methodology of transformation needs transitioning changes in existing DASD systems. In this paper we have proposed framework for transitioning of plan driven software development method to distributed Agile software development. The major area of concern is to automate the manual processes followed and convert them to semi – automated and fully – automated processes.

An Automated Framework for Detecting Change in the Source Code and Test Case Change Recommendation

Improvements and acceleration in software development has contributed towards high quality services in all domains and all fields of industry causing increasing demands for high quality software developments. In order to match with the high-quality software development demands, the software development industry is adopting human resources with high skills, advanced methodologies and technologies for accelerating the development life cycle. In the software development life cycle, one of the biggest challenges is the change management between versions of the source codes. The versing of the source code can be caused by various reasons such as change in the requirements or adaptation of functional update or technological upgradations. The change management does not only affect the correctness of the release for the software service, rather also impact the number of test cases. It is often observed that, the development life cycle is delayed due to lack of proper version control and due to the improver version control, the repetitive testing iterations. Hence the demand for better version control driven test case reduction methods cannot be ignored. A number of version control mechanisms are proposed by the parallel research attempts. Nevertheless, most of the version controls are criticized for not contributing towards the test case generation of reduction. Henceforth, this work proposes a novel probabilistic refactoring detection and rule-based test case reduction method in order to simplify the testing and version control mechanism for the software development. The refactoring process is highly adopted by the software developers for making efficient changes such as code structure, functionality or apply change in the requirements. This work demonstrates a very high accuracy for change detection and management. This results into a higher accuracy for test case reductions. The final outcome of this work is to reduce the development time for the software for making the software development industry a better and efficient world.

IETCR: An Information Entropy Based Test Case Reduction Strategy for Mutation-Based Fault Localization

Mutation-based fault localization (MBFL) is a recently proposed technique with the advantage of high fault localization accuracy. However, such a mutation analysis based technique is difficult to be accepted by industry due to its huge computational cost on mutation analysis. There are three ways to improve MBFL's efficiency, which are reducing the number of mutants, optimizing the mutants' execution process, and reducing the number of test cases. The former two ways have been mainly studied and shown promising results, but for the latter way, the related studies are limited since this kind of method will reduce the precision of MBFL. In this paper, we mainly focus on the latter way and propose an information entropy based test case reduction (IETCR) strategy for MBFL. In particular, we first calculate the entropy change of test cases and select a proportion of them according to their value. Then we use a reduced test suite to execute mutants. To show the effectiveness of the IETCR strategy, we choose six real-world programs with 112 faulty versions. In terms of mutation reduction rate, we find MBFL with the IETCR strategy can reduce 56.3%~88.3% cost while keeping almost the same fault localization accuracy when compared to the original MBFL without test case reduction. Moreover, we use Wilcoxon signed-rank test for statistical analysis, which shows that there is no statistically significant difference between MBFL with IETCR strategy and the original MBFL.