Size Of Test Suite Research Articles

Quality of Automated Program Repair on Real-World Defects

Automated program repair is a promising approach to reducing the costs of manual debugging and increasing software quality. However, recent studies have shown that automated program repair techniques can be prone to producing patches of low quality, overfitting to the set of tests provided to the repair technique, and failing to generalize to the intended specification. This paper rigorously explores this phenomenon on real-world Java programs, analyzing the effectiveness of four well-known repair techniques, GenProg, Par, SimFix, and TrpAutoRepair, on defects made by the projects’ developers during their regular development process. We find that: (1) When applied to real-world Java code, automated program repair techniques produce patches for between 10.6 and 19.0 percent of the defects, which is less frequent than when applied to C code. (2) The produced patches often overfit to the provided test suite, with only between 13.8 and 46.1 percent of the patches passing an independent set of tests. (3) Test suite size has an extremely small but significant effect on the quality of the patches, with larger test suites producing higher-quality patches, though, surprisingly, higher-coverage test suites correlate with lower-quality patches. (4) The number of tests that a buggy program fails has a small but statistically significant positive effect on the quality of the produced patches. (5) Test suite provenance, whether the test suite is written by a human or automatically generated, has a significant effect on the quality of the patches, with developer-written tests typically producing higher-quality patches. And (6) the patches exhibit insufficient diversity to improve quality through some method of combining multiple patches. We develop JaRFly, an open-source framework for implementing techniques for automatic search-based improvement of Java programs. Our study uses JaRFly to faithfully reimplement GenProg and TrpAutoRepair to work on Java code, and makes the first public release of an implementation of Par. Unlike prior work, our study carefully controls for confounding factors and produces a methodology, as well as a dataset of automatically-generated test suites, for objectively evaluating the quality of Java repair techniques on real-world defects.

Finding Minimum Locating Arrays Using a CSP Solver

Combinatorial interaction testing is an efficient software testing strategy. If all interactions among test parameters or factors needed to be covered, the size of a required test suite would be prohibitively large. In contrast, this strategy only requires covering $t$-wise interactions where $t$ is typically very small. As a result, it becomes possible to significantly reduce test suite size. Locating arrays aim to enhance the ability of combinatorial interaction testing. In particular, $(\overline{1}, t)$-locating arrays can not only execute all $t$-way interactions but also identify, if any, which of the interactions causes a failure. In spite of this useful property, there is only limited research either on how to generate locating arrays or on their minimum sizes. In this paper, we propose an approach to generating minimum locating arrays. In the approach, the problem of finding a locating array consisting of $N$ tests is represented as a Constraint Satisfaction Problem (CSP) instance, which is in turn solved by a modern CSP solver. The results of using the proposed approach reveal many $(\overline{1}, t)$-locating arrays that are smallest known so far. In addition, some of these arrays are proved to be minimum.

Optimal Test Case Selection Using Ant Colony and Rough Sets

Test case selection helps in improving quality of test suites by removing ambiguous, redundant test cases, thereby reducing the cost of software testing. Various works carried out have chosen test cases based on single parameter and optimized the test cases using single objective employing single strategies. In this article, a parameter selection technique is combined with an optimization technique for optimizing the selection of test cases. A two-step approach has been employed. In first step, the fuzzy entropy-based filtration is used for test case fitness evaluation and selection. In second step, the improvised ant colony optimization is employed to select test cases from the previously reduced test suite. The experimental evaluation using coverage parameters namely, average percentage statement coverage and average percentage decision coverage along with suite size reduction, demonstrate that by using this proposed approach, test suite size can be reduced, reducing further the computational effort incurred.

Ant colony algorithm to generate t-way test suite with constraints

T-way testing is one of the testing techniques offered to generate test suites. It focuses on interactions of input parameters based on strength. Besides that, t-way is used to overcome exhaustive testing problem. Another important aspect in t-way is constraints. It forbids certain interactions of input parameters. Therefore, the final test suite contains only valid interactions. T-way testing is NP-hard problem. No single strategy can always generates the best test suite size at all time for all configurations. Thus, Const-TTSGA strategy has been developed to generate test suites. The strategy supports constraints variable strength. Const-TTSGA is a metaheuristic strategy which applies ant colony algorithm to generate the best test cases. Two types of experiments have been conducted; constraints uniform and variable strength which consists of few other configurations. Results obtained are compared to benchmarked results. Const-TTSGA outperformed other strategy for costraints uniform strength experiments except for one configuration. However, the strategy outperformed other strategies for constraints variable strength experiments.

Diversity-Oriented Test Suite Generation for EFSM Model

In this article, test diversity has been suggested to be a valid way to improve test suite effectiveness. Extended finite state machine (EFSM) is a widely used formal model, but little attention is paid on the test suite generation with more diversity. EFSM test suite generation involves test paths generation and test data generation. Considering the discrepancy between test paths has a more crucial impact on the diversity of test suite, compared with the difference between test data, this article, therefore, mainly concerns the test paths generation with more diversity for EFSM models. Hence, the factors that influence the discrepancy between test paths are investigated. Then based on these factors, an integrated distance metric is designed to evaluate the dissimilarity between test paths, and a diversity measurement for EFSM test suite is presented. Furthermore, a diversity-oriented test suite generation (DOTSG) method is proposed where a dissimilarity-based fitness function and diversity-oriented update strategy are adopted in traditional coverage-oriented EFSM test suite generation (COTSG) by genetic algorithm. The experimental results show that, compared to COTSG, our DOTSG can not only generate more diverse test suite to satisfy a certain coverage criteria, improving the fault detection capability of the test suite, but also decrease the evolution time cost and the size of test suite generated.

Implementation of Sine Cosine Algorithm (SCA) for Combinatorial Testing

Before being released to the market, software should be screened to ensure that the quality assurance measurement goals have been attained. To achieve this, one of the types of testing sorts is combinatorial testing (CT) aimed at discovering the faults that occur by interacting with the software. A minimization strategy for test cases is indeed important for optimizing test cases and reducing time. As NP hard (where NP is a non-deterministic polynomial) is the problem of generating the minimum test suite of combinatorial interaction testing (CIT). this paper discusses the implementation, and validation of an efficient strategy for t-way testing. The main contribution of the sine cosine algorithm SCA is to show that the strategy was sufficiently competitive as compared to other strategies in terms of the generated test suite size. Unlike most paper. The main contribution of SCA is to show the generation of test data for a high coverage strength (t < 12).

A Modified Artificial Bee Colony Based Test Suite Generation Strategy for Uniform T-Way Testing

Today, t-way testing has been widely known with the ability to reduce test suite size compared to exhaustive testing. At the same time, it has been proven by many researchers to provide maximum bug detection capability. Thus, various t-way strategies were developed since the past three decades. The paper proposed a new test generation strategy, named Modified Artificial Bee Colony T-Way Test Suite Generation (MABCTS). It supports uniform strength t-way testing. Experimentation results are compared with present strategies and produced comparable results. Since t-way testing is considered an NP-hard problem, there are no strategies that can be demanded to produce the best results.

Multi-objective test suite optimization for detection and localization of software faults

Testing of software is done with an intention to find faults. If a fault is there then it needs to be detected, located and then resolved. Fault detection and localization are adjoining activities and thus it is difficult to combine them. Fault detection requires test information that helps in detecting the faults early whereas fault localization requires information which helps in locating the faults accurately. But test information is one common thing that is required for both the activities. This pre-condition helps to effectively combine both fault detection and localization. In this research work, a code and mutant coverage based multi-objective approach has been proposed to produce a minimized test suite having the ability of both detecting and locating faults. For optimization of test cases, NSGA-II algorithm has been used. Results on the projects of Defects4j repository depicts that the proposed approach is able to produce minimized test suites having the capability of detecting 95.16% of faults and locating all detected faults with fault localization score almost equivalent to that of the complete test suite. The average percentage of reduction in test suite size is 78% which is a good reduction percentage with the given fault detection and localization scores.

An Effective Regression Test Case Selection Using Hybrid Whale Optimization Algorithm

Test suite optimization is an ever-demanded approach for regression test cost reduction. Regression testing is conducted to identify any adverse effects of maintenance activity on previously working versions of the software. It consumes almost seventy percent of the overall software development lifecycle budget. Regression test cost reduction is therefore of vital importance. Test suite optimization is the most explored approach to reduce the test suite size to re-execute. This article focuses on test suite optimization as a regression test case selection, which is a proven N-P hard combinatorial optimization problem. The authors have proposed an almost safe regression test case selection approach using a Hybrid Whale Optimization Algorithm and empirically evaluated the same on subject programs retrieved from the Software Artifact Infrastructure Repository with Bat Search and ACO-based regression test case selection approaches. The analyses of the obtained results indicate an improvement in the fault detection ability of the proposed approach over the compared ones with significant reduction in test suite size.

Comparing Graph-Based Algorithms to Generate Test Cases from Finite State Machines

Model-Based Testing (MBT) is a well-known technique that employs formal models to represent reactive systems’ behavior and generates test cases. Such systems have been specified and verified using mostly Finite State Machines (FSMs). There is a plethora of test generation algorithms in the literature; most of them are based on graphs once an FSM can be formally defined as a graph. Nevertheless, there is a lack of studies on analyzing cost and efficiency of FSM-based test generation algorithms. This study compares graph-based algorithms adopted to generate test cases from FSM models. In particular, we compare the Chinese Postman Problem (CPP) and H-Switch Cover (HSC) algorithms with the well-known Depth-First Search (DFS) and Breadth-First Search (BFS) algorithms in the context of covering all-transitions and all-transition-pairs criteria in an FSM. First, a systematic literature mapping was conducted to summarize the methods that have been adopted in MBT, considering FSMs. Second, the main methods found were implemented and analyzed on randomly-generated FSMs, as well as real-world models that represent embedded systems of space applications. To make comparisons, we considered analyses in terms of cost (time), efficiency (mutant analysis) and characteristics of the generated test suites (number of test cases, average length of test cases, largest and smallest test cases, standard deviation and distribution of test cases). In general, CPP presented the best results in terms of number of test cases and test suite size. In addition, CPP also presented low distribution of average length compared to other algorithms.

Adequate vs. inadequate test suite reduction approaches

Abstract Context: Regression testing is an important activity that allows ensuring the correct behavior of a system after changes. As the system grows, the time and resources to perform regression testing increase. Test Suite Reduction (TSR) approaches aim to speed up regression testing by removing obsolete or redundant test cases. These approaches can be classified as adequate or inadequate. Adequate TSR approaches reduce test suites and completely preserve test requirements (e.g., covered statements) of the original test suites. Inadequate TSR approaches do not preserve test requirements. The percentage of satisfied test requirements indicates the inadequacy level. Objective: We compare some state-of-the-art adequate and inadequate TSR approaches with respect to the size of reduced test suites and their fault-detection capability. We aim to increase our body of knowledge on TSR approaches by comparing: (i) well-known traditional adequate TSR approaches; (ii) their inadequate variants; and (iii) several variants of a novel Clustering-Based (CB) approach for (adequate and inadequate) TSR. Method: We conducted an experiment to compare adequate and inadequate TSR approaches. This comparison is founded on a public dataset containing information on real faults. Results: The most important findings from our experiment can be summarized as follows: (i) there is not an inadequate TSR approach that outperforms the others; (ii) some inadequate variants of the CB approach, and few traditional inadequate approaches, outperform the adequate ones in terms of reduction in test suite size with a negligible effect on fault-detection capability; and (iii) the CB approach is less sensitive than the other inadequate approaches, that is, variations in the inadequacy level have small effect on reduction in test suite size and on loss in fault-detection capability. Conclusions: These findings imply that inadequate TSR approaches and especially the CB approach might be appealing because they lead to a greater reduction in test suite size (with respect to the adequate ones) at the expense of a small loss in fault-detection capability.

Functional decomposition—A contribution to overcome the parameter space explosion during validation of highly automated driving

Objective: Particular testing by functional decomposition of the automated driving function can potentially contribute to reducing the effort of validating highly automated driving functions. In this study, the required size of test suites for scenario-based testing and the potential to reduce it by functional decomposition are quantified for the first time.Methods: The required size of test suites for scenario-based approval of a so-called Autobahn-Chauffeur (SAE Level 3) is analyzed for an exemplary set of scenarios. Based on studies of data from failure analyses in other domains, the possible range for the required test coverage is narrowed down and suitable discretization steps, as well as ranges for the influence parameters, are assumed. Based on those assumptions, the size of the test suites for testing the complete system is quantified. The effects that lead to a reduction in the parameter space for particular testing of the decomposed driving function are analyzed and the potential to reduce the validation effort is estimated by comparing the resulting test suite sizes for both methods.Results: The combination of all effects leads to a reduction in the test suites’ size by a factor between 20 and 130, depending on the required test coverage. This means that the size of the required test suite can be reduced by 95–99% by particular testing compared to scenario-based testing of the complete system.Conclusions: The reduction potential is a valuable contribution to overcome the parameter space explosion during the validation of highly automated driving. However, this study is based on assumptions and only a small set of exemplary scenarios. Thus, the findings have to be validated in further studies.

Fault coverage-based test suite optimization method for regression testing: learning from mistakes-based approach

This paper presents a novel method referred as fault coverage-based test suite optimization (FCBTSO) for regression test suite optimization. FCBTSO is proposed based on Harrolds–Gupta–Soffa (HGS) test suite reduction method, and it follows the phenomenon: “learning from mistakes”. We conducted computational experiments on 12 versions of benchmarked programs retrieved from software artefact infrastructure repository and dummy fault matrix test. The performance of the proposed FCBTSO is measured against the traditional test suite reduction methods (Greedy method, Additional Greedy, HGS, and Enhanced HGS) by following the performance measures: fault coverage, execution time and reduced optimized test suite size. Rigorous statistical tests are conducted to determine the performance significance, which indicates that FCBTSO outperforms other approaches implemented with respect to the execution time that includes the execution time of the proposed approach to find the optimized test suite and the execution time of test cases in the optimized test suite.

Empirical Analysis of Object-Oriented Software Test Suite Evolution

The software system is evolving over the time, thus, the test suite must be repaired according to the changing code. Updating test cases manually is a time-consuming activity, especially for large test suites, which motivate the recent development of automatically repairing test techniques. To develop an effective automatic repair technique that reduces the effort of development and the cost of evolution, the developer should understand how the test suite evolves in practice. This investigation aims to conduct a comprehensive empirical study on eight Java systems with many versions of these systems and their test suites to find out how the test suite is evolving, and to find the relationship between the change in the program and the corresponding evolution in the test suite. This study showed that the test suite size is mostly increased, where the test suite complexity is stabilized. The increase (or decrease) in the code size will mostly increase (or decrease) the test suite size. However, the increasing or decreasing in the code complexity is offset by stabilizing the test suite complexity. Moreover, the percentage of the code coverage tends to be increased more than decreased, but in the mutation coverage, the opposite is true.

A Systematic Literature Review of Test Case Prioritization Using Genetic Algorithms

Regression testing is the essential process of software maintenance and evolution phase of the software development life cycle for assuring the quality and reliability of updated software. Test case prioritization is the technique of regression testing to reduce the time and effort required for regression testing. Search-based algorithms are used to enhance the efficiency and effectiveness of the method. Among these search-based optimization algorithms, genetic algorithms are becoming more popular among researchers since the last decade. In this paper, we are doing a systematic literature review, i.e., a secondary study of test case prioritization using genetic algorithms. The objective of this review is to examine and classify the current state of use of the genetic algorithm in test case prioritization. In other words, to give a base for the advancement of test case prioritization research using genetic algorithms. With the use of the systematic literature review protocol, we selected the most relevant studies (20 out of 384) from the appropriate repositories by using a set of search keywords, inclusion/exclusion criteria and the quality assessment of studies. The data extraction and synthesis process and the taxonomic classification are used to answer the research questions. We also performed a rigorous analysis of the techniques by comparing them on research methodology, the prioritization method, dataset specification, test suite size, types of genetic algorithms used, performance metrics, and the validation criteria. The whole process took four months for comprehensive analysis and classification of primary studies. We observed that the parameter settings, the type of operators, the probabilistic rate of operators, and fitness function design have a significant impact on the quality of the solutions obtained. This systematic literature review yields that genetic algorithms have great potential in solving test case prioritization problems, and the area is open for further improvements. Future researchers can fill the research gaps by following the suggestions given in the review. From this review, we found that the use of the appropriate approach can make a genetic algorithm based test case prioritization one of the effective methods in regression testing.

Evaluation of Mutation Testing in a Nuclear Industry Case Study

For software quality assurance, many safety-critical industries appeal to the use of dynamic testing and structural coverage criteria. However, there are reasons to doubt the adequacy of such practices. Mutation testing has been suggested as an alternative or complementary approach but its cost has traditionally hindered its adoption by industry, and there are limited studies applying it to real safety-critical code. This paper evaluates the effectiveness of state-of-the-art mutation testing on safety-critical code from within the U.K. nuclear industry, in terms of revealing flaws in test suites that already meet the structural coverage criteria recommended by relevant safety standards. It also assesses the practical feasibility of implementing such mutation testing in a real setting. We applied a conventional selective mutation approach to a C codebase supplied by a nuclear industry partner and measured the mutation score achieved by the existing test suite. We repeated the experiment using trivial compiler equivalence (TCE) to assess the benefit that it might provide. Using a conventional approach, it first appeared that the existing test suite only killed 82% of the mutants, but applying TCE revealed that it killed 92%. The difference was due to equivalent or duplicate mutants that TCE eliminated. We then added new tests to kill all the surviving mutants, increasing the test suite size by 18% in the process. In conclusion, mutation testing can potentially improve fault detection compared to structural-coverage-guided testing, and may be affordable in a nuclear industry context. The industry feedback on our results was positive, although further evidence is needed from application of mutation testing to software with known real faults.

A learning algorithm for optimizing continuous integration development and testing practice

SummaryContinuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large‐scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault‐detection effectiveness of continuous integration tests using historical test records and combining this information with coverage‐based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest‐all approach. The results further demonstrate no significant reduction in fault‐detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.

A Theoretical and Empirical Study of Diversity-Aware Mutation Adequacy Criterion

Diversity has been widely studied in software testing as a guidance towards effective sampling of test inputs in the vast space of possible program behaviors. However, diversity has received relatively little attention in mutation testing. The traditional mutation adequacy criterion is a one-dimensional measure of the total number of killed mutants. We propose a novel, diversity-aware mutation adequacy criterion called distinguishing mutation adequacy criterion, which is fully satisfied when each of the considered mutants can be identified by the set of tests that kill it, thereby encouraging inclusion of more diverse range of tests. This paper presents the formal definition of the distinguishing mutation adequacy and its score. Subsequently, an empirical study investigates the relationship among distinguishing mutation score, fault detection capability, and test suite size. The results show that the distinguishing mutation adequacy criterion detects 1.33 times more unseen faults than the traditional mutation adequacy criterion, at the cost of a 1.56 times increase in test suite size, for adequate test suites that fully satisfies the criteria. The results show a better picture for inadequate test suites; on average, 8.63 times more unseen faults are detected at the cost of a 3.14 times increase in test suite size.

A Kidney Algorithm for Pairwise Test Suite Generation

Pairwise testing can greatly minimize the cost of software testing and also increase the ability of fault detection. Nevertheless, generating the most optimal test suite is an NP-complete problem and still an open area for research. The test case generation is the most active area of the pairwise testing research. Metaheuristic algorithms have been broadly used for solving difficult optimization problems as well as proving their effectiveness to get most optimal solutions. Kidney algorithm (KA) is a recent metaheuristic algorithm. This study introduces a new pairwise strategy by adapting KA; which is the first time to adapt KA in generating the test suite. The proposed strategy is called Pairwise Kidney Strategy (PKS). This study also highlights the PKS design; in addition, compare its performance with other reported strategies in the literature in terms of test suite size. Experiment results show that PKS has very competitive results as compared with other strategies.

Comparative Study Between Flower Pollination Algorithm and Cuckoo Search Algorithm for t -Way Test Data Generation

T-way testing is a sampling approach for test data generation. Recently, adapting meta-heuristic algorithms for t-way testing is very attractive in order to find a minimum subset of test data that can test a system overall. As a consequence, several meta-heuristic algorithms have been used as the basis of t-way strategies. In order to guide software tester (and engineers in general) to select the best algorithm for the problem at hand, there is a need to evaluate and benchmark the performance of each strategy against common case studies. This paper presents a comparative study between two meta-heuristic strategies for t-way test data generation: Flower Pollination Algorithm (FPA) and Cuckoo Search (CS). Our experiments have performed on a real-world case study. Experimental results demonstrate that FPA appears to produce better results in most of the test cases in term of test suite size and convergence rate owing to its ability for controlling local and global search.