Regression Test Prioritization Research Articles

Regression test prioritization leveraging source code similarity with tree kernels

AbstractRegression test prioritization (RTP) is an active research field, aiming at re‐ordering the tests in a test suite to maximize the rate at which faults are detected. A number of RTP strategies have been proposed, leveraging different factors to reorder tests. Some techniques include an analysis of changed source code, to assign higher priority to tests stressing modified parts of the codebase. Still, most of these change‐based solutions focus on simple text‐level comparisons among versions. We believe that measuring source code changes in a more refined way, capable of discriminating between mere textual changes (e.g., renaming of a local variable) and more structural changes (e.g., changes in the control flow), could lead to significant benefits in RTP, under the assumption that major structural changes are also more likely to introduce faults. To this end, we propose two novel RTP techniques that leverage tree kernels (TK), a class of similarity functions largely used in Natural Language Processing on tree‐structured data. In particular, we apply TKs to abstract syntax trees of source code, to more precisely quantify the extent of structural changes in the source code, and prioritize tests accordingly. We assessed the effectiveness of the proposals by conducting an empirical study on five real‐world Java projects, also used in a number of RTP‐related papers. We automatically generated, for each considered pair of software versions (i.e., old version, new version) in the evolution of the involved projects, 100 variations with artificially injected faults, leading to over 5k different software evolution scenarios overall. We compared the proposed prioritization approaches against well‐known prioritization techniques, evaluating both their effectiveness and their execution times. Our findings show that leveraging more refined code change analysis techniques to quantify the extent of changes in source code can lead to relevant improvements in prioritization effectiveness, while typically introducing negligible overheads due to their execution.

On the Effectiveness of Feature Selection Techniques in the Context of ML-Based Regression Test Prioritization

On the Effectiveness of Feature Selection Techniques in the Context of ML-Based Regression Test Prioritization

Multi-Objective Fault-Coverage Based Regression Test Selection and Prioritization Using Enhanced ACO_TCSP

Regression testing of the software during its maintenance phase, requires test case prioritization and selection due to the dearth of the allotted time. The resources and the time in this phase are very limited, thus testers tend to use regression testing methods such as test case prioritization and selection. The current study evaluates the effectiveness of testing with two major goals: (1) Least running time and (2) Maximum fault coverage possible. Ant Colony Optimization (ACO) is a well-known soft computing technique that draws its inspiration from nature and has been widely researched, implemented, analyzed, and validated for regression test prioritization and selection. Many versions of ACO approaches have been prolifically applied to find solutions to many non-polynomial time-solvable problems. Hence, an attempt has been made to enhance the performance of the existing ACO_TCSP algorithm without affecting its time complexity. There have been efforts to enhance the exploration space of various paths in each iteration and with elite exploitation, reducing the total number of iterations required to converge to an optimal path. Counterbalancing enhanced exploration with intelligent exploitation implies that the run time is not adversely affected, the same has also been empirically validated. The enhanced algorithm has been compared with the existing ACO algorithm and with the traditional approaches. The approach has also been validated on four benchmark programs to empirically evaluate the proposed Enhanced ACO_TCSP algorithm. The experiment revealed the increased cost-effectiveness and correctness of the algorithm. The same has also been validated using the statistical test (independent t-test). The results obtained by evaluating the proposed approach against other reference techniques using Average Percentage of Faults Detected (APFD) metrics indicate a near-optimal solution. The multiple objectives of the highest fault coverage and least running time were fruitfully attained using the Enhanced ACO_TCSP approach without compromising the complexity of the algorithm.

Colosseum: Regression Test Prioritization by Delta Displacement in Test Coverage

The problem of test-case prioritization has been pursued for over three decades now and continues to be one of the active topics in software testing research. In this paper, we focus on a code-coverage based regression test-prioritization solution ( <inline-formula><tex-math notation="LaTeX">${{\sf Colosseum}}$</tex-math></inline-formula> ) that takes into account the position of changed (delta) code elements (basic-blocks) along the loop-free straight-line execution path of the regression test-cases. We propose a heuristic that logically associates each of these paths with three parameters: (i) the offset (displacement <i>a</i> ) of the first delta from the starting basic-block, (ii) the offset (displacement <i>c</i> ) of the last delta from the terminating basic block, and (iii) the average scattering (displacement <i>b</i> ) within all the intermediate basic-blocks. We hypothesize that a regression test-case path with a shorter overall displacement has a good chance of propagating the affects of the code-changes to the observable outputs in the program. <inline-formula><tex-math notation="LaTeX">${{\sf Colosseum}}$</tex-math></inline-formula> prioritizes test-cases with smaller overall displacements and executes them early in the regression test-execution cycle. The underlying intuition is that the probability of a test-case revealing a regression fault depends on the probability of the corresponding change propagation. The change in this context can potentially lead to an error. Extending this logic, delta displacement provides an approximation to failed error propagation. Evaluation on 20 open-source C projects from the Software-artifact Infrastructure Repository and GitHub (totaling: 694,512 SLOC, 280 versions, and 69,305 test-cases) against four state-of-the-art prioritizations reveals that: <inline-formula><tex-math notation="LaTeX">${{\sf Colosseum}}$</tex-math></inline-formula> outperforms the competitors with an overall 84.61% success in terms of 13 prioritization effectiveness metrics, majority of which prefer to execute top- <inline-formula><tex-math notation="LaTeX">$k\%$</tex-math></inline-formula> prioritized test-cases.

Hansie: Hybrid and consensus regression test prioritization

Traditionally, given a test-suite and the underlying system-under-test, existing test-case prioritization heuristics report a permutation of the original test-suite that is seemingly best according to their criteria. However, we observe that a single heuristic does not perform optimally in all possible scenarios, given the diverse nature of software and its changes. Hence, multiple individual heuristics exhibit effectiveness differently. Interestingly, together, the heuristics bear the potential of improving the overall regression test selection across scenarios. In this paper, we pose the test-case prioritization as a rank aggregation problem from social choice theory. Our solution approach, named Hansie, is two-flavored: one involving priority-aware hybridization, and the other involving priority-blind computation of a consensus ordering from individual prioritizations. To speed-up test-execution, Hansie executes the aggregated test-case orderings in a parallel multi-processed manner leveraging regular windows in the absence of ties, and irregular windows in the presence of ties. We show the benefit of test-execution after prioritization and introduce a cost-cognizant metric (EPL) for quantifying overall timeline latency due to load-imbalance arising from uniform or non-uniform parallelization windows. We evaluate Hansie on 20 open-source subjects totaling 287,530 lines of source code, 69,305 test-cases, and with parallelization support of up to 40 logical CPUs.

Multi objective test case selection and prioritization using African buffalo optimization

To prevent a software from going obsolete, it is mandatory to update it in order to meet the current ever changing requirements. Each update is accompanied by a regression testing cost. Attempts to reduce this cost and effort have been made for over three decades now, but the search for a better technique continues. This paper offers an novel method to solve the regression test prioritization and selection issue in a nature inspired way using African Buffalo Optimization. A new ABO_TCSP algorithm has been proposed and explained with an example. The example has been tested using APFD metrics. The results prove to be optimal for the considered example and hence encourages the researchers to further validate and test the proposed approach.

Mahtab: Phase-wise acceleration of regression testing for C

Software regression testing consists of offline, online, and execution phases which are executed sequentially. The offline phase involves code instrumentation and test-coverage collection. Subsequently, the online phase performs program differencing, test-suite selection and prioritization. Finally, the selected test-cases are executed against the new version of software for its re-validation. Regression testing is a time-consuming process and is often on the critical path of the project. To improve the turn-around time of software development cycle, our goal is to reduce regression testing time across all phases using multi-core parallelization. This poses several challenges that stem from I/O, dependence on third-party libraries, and inherently sequential components in the overall testing process. We propose parallelization test-windows to effectively partition test-cases across threads. To measure the benefit of prioritization coupled with multi-threaded execution, we propose a new metric, EPSilon, for rewarding failure observation frequency in the timeline of test-execution. To measure the rate of code-change coverage due to regression test prioritization, we introduce ECC, a variant of the widely used APFD metric. We illustrate the effectiveness of our approach using the popular Software-artifact Infrastructure Repository (SIR) and five real-world projects from GitHub.

An Attempt to Design and Implement Contiki and Cooja Regression Test Suites using Combinatorial Testing

Although several studies have been conducted on regression test selection, augmentation, prioritization, and pruning, regression test suite creation is performed in an ad hoc manner for most software systems by selecting a few crucial parameters and their combinations. Regression test suite creation can be rigorously approached by using the combinatorial testing (CT) method. Although this approach is advantageous, it has been used in only a few cases. This paper presents a practical approach for applying CT and augmenting the Contiki and Cooja base regression test suites with the National Institute of Standards and Technology tools.

Industrially Applicable System Regression Test Prioritization in Production Automation

When changes are performed on an automated production system (aPS), new faults can be accidentally introduced in the system, which are called regressions. A common method for finding these faults is regression testing. In most cases, this regression testing process is performed under high time pressure and on-site in a very uncomfortable environment. Until now, there is no automated support for finding and prioritizing system test cases regarding the fully integrated aPS that are suitable for finding regressions. Thus, the testing technician has to rely on personal intuition and experience, possibly choosing an inappropriate order of test cases, finding regressions at a very late stage of the test run. Using a suitable prioritization, this iterative process of finding and fixing regressions can be streamlined and a lot of time can be saved by executing test cases likely to identify new regressions earlier. Thus, an approach is presented in this paper that uses previously acquired runtime data from past test executions and performs a change identification and impact analysis to prioritize test cases that have a high probability to unveil regressions caused by side effects of a system change. The approach was developed in cooperation with reputable industrial partners active in the field of aPS engineering, ensuring a development in line with industrial requirements. An industrial case study and an expert evaluation were performed, showing promising results.

Evolutionary Cost Cognizant Regression Test Prioritization for Object-Oriented Programs Based on Fault Dependency

Regression testing performed to provide confidence on the newly or updated software system is a resource consuming process. To ease this process, various techniques are developed. One such technique, test case prioritization, orders test cases with respect to the goals such that the most important test case in achieving those goals is scheduled earlier during the testing session. Among such performance goals, the rate of faults detections, measure how faults are detected quickly throughout the regression testing process. Improved dependency detection among faults provides faster feedback to the developers which gives chance to debug leading faults earlier in time. One other goal, the rate of fault severity detection, measure how fast severe fault can be detected in the testing process. Although, previous works address these issues but assumed that the costs of executing test cases and severities of detected faults are the same. However, costs of test and severities of faults varied. Furthermore, they did not consider incorporating evolution process such as applying genetic algorithms to their technique. In this work, we proposed an evolutionary cost-cognizant regression testing approach that prioritizes test case according to the rate of severity detection of test cases connected with dependent faults using genetic algorithms. The aim is to reveal more severe leading faults earlier using least cost in executing the test suite and to measure the efficacy of the technique using APFDc.

Prioritizing test cases for regression techniques using metaheuristic techniques

Regression testing attempts to find new errors after some alterations have been done in the software. Previously run test cases are re-executed to check whether historical faults have re-emerged or to validate the functionality of the software. In the past, several methods have been analyzed for reducing the cost of regression testing, which includes regression test selection, regression test minimization and regression test prioritization. Test Suite Prioritization techniques orders the test cases so that all the test cases are executed on a priority basis. Artificial bee colony (ABC) algorithm is a very powerful and efficient algorithm influenced by the intelligent nature of bees. In this paper, we have applied the ABC algorithm for test suite prioritization and the comparison has been made with Genetic Algorithm and Cuscuta Search. From our experimental results, we can infer that results of Cuscuta Search are low as compared to ABC and GA for small test suites.

The network of faults: a complex network approach to prioritize test cases for regression testing

Regression testing is performed to provide confidence that changes in a part of software do not affect other parts of the software. An execution of all existing test cases is the best way to re-establish this confidence. However, regression testing is an expensive process--there might be insufficient resources (e.g., time, workforce) to allow for the re-execution of all test cases. Regression test prioritization techniques attempt to re-order a regression test suite based on some criteria so that highest priority test cases are executed earlier. In this study, we prioritize test cases for regression testing based on the dependency network of faults. In software testing, it is common that some faults are the consequences of other faults (leading faults). Dependent faults can be removed if and only if the leading faults have been removed. Our goal is to prioritize test cases so that test cases that have exposed the leading faults in the system testing phase, are executed first in regression testing. The leading faults are modeled as the most central faults in the fault dependency network. We present ComReg, a test-case prioritization technique based on the dependency network of faults. We model a fault dependency network as a directed graph and identify leading faults to prioritize test cases for regression testing. We use a centrality aggregation technique which considers six network representative centrality metrics to identify leading faults in the fault dependency network. We also discuss the use of fault communities to select an arbitrary percentage of the test cases from a prioritized regression test suite. We conduct a case study that evaluates the effectiveness and applicability of the proposed method. We obtain a fault dependency network from the development of a vocabulary learning software. We found that the fault network is a small-world graph with distinguishable community structure. The leading faults are common in all centralities and a re-ordering of test cases is feasible for regression testing based on those leading faults. Our method outperforms traditional regression testing prioritization techniques in detecting fault dependencies. Our modeling of the network of faults provides insights into the requirement of recognizing fault dependencies while re-ordering regression test suites for both research and practice. The dependency model needs further evaluation and improvement considering relevant resources (e.g., man-hours).

Regression Testing Prioritization, Selection and Reduction using Hybrid Criteria

Regression testing is a software testing technique. Testing and validating the part of code are the activity performed within different phases. Tasks of regression testing are: Test Case Prioritization, Test Suite Selection, Test case reduction which give the guarantee that no intended fault is produced while modifying the code. This paper hybrid all the criteria’s in different prospective with existing techniques. Selecting and choosing minimum number of test cases according to the result is our major goal. It will give solution to certain unnecessary results found after testing that further seem to be diminished in execution time. In our work we are formulizing the swarm algorithm for hybrid criteria. Hybrid criteria use Rank, Merge and Choice for building the test cases from test suite for minimizing the redundancy. Branch technique is used if one of test cases fails or does not show any result then next option can be used. Swarm algorithms give additional functions for having effective result with less time and effort. Initial seed value for hybrid criteria’s is taken randomly. This research will lead to give better efficiency in regression testing using hybrid criteria. Path Coverage deals with the test case selection as it gives all the details of test cases. General Terms Event Coverage, Statement Coverage, Execution Time, Fault Detected, Priority, ACO

Impact analysis in the presence of dependence clusters using Static Execute After in WebKit

SUMMARYImpact analysis based on code dependence can provide opportunities to identify parts of the software affected by a change. Because changes usually have far reaching effects in programs, effective and efficient impact analysis is vital. Static Execute After (SEA) is a relation on procedures that is efficiently computable and accurate enough to be a candidate for the use in impact analysis in practice. To assess the applicability of SEA in terms of capturing real defects, we present results on integrating it into the build system of WebKit, a large, open source software system, and on related experiments. We show that a large number of real defects can be captured by impact sets computed by SEA, albeit many of them are large. We demonstrate that this is not an issue in applying it to regression test prioritization, but generally it can be an obstacle in the path to efficient use of impact analysis. We believe that the main reason for large impact sets is the formation of dependence clusters in code. As apparently dependence clusters cannot be easily avoided in the majority of cases, we focus on determining the effects these clusters have on impact analysis and regression test prioritization. Copyright © 2013 John Wiley &amp; Sons, Ltd.

Analyzing test case selection &amp; prioritization using ACO

Regression testing is primarily a maintenance activity that is performed frequently to ensure the validity of the modified software. In such cases, due to time and cost constraints, the entire test suite cannot be run. Thus, it becomes essential to select or prioritize the tests in order to cover maximum faults in minimum time. Recently, Ant Colony Optimization (ACO), which is a new way to solve time constraint prioritization problem, has been utilized. This paper presents the analysis of the regression test prioritization technique to reorder test suites in time constraint environment along with the sample runs on various programs. Our analysis concluded that the ACO finds better orderings at higher values of the time constraint (TC). The correctness of the technique has also been recorded to be near optimal at an average.

Test case prioritization using ant colony optimization

Regression testing is primarily a maintenance activity that is performed frequently to ensure the validity of the modified software. In such cases, due to time and cost constraints, the entire test suite cannot be run. Thus, it becomes essential to prioritize the tests in order to cover maximum faults in minimum time. In this paper, ant colony optimization is used, which is a new way to solve time constraint prioritization problem. This paper presents the regression test prioritization technique to reorder test suites in time constraint environment along with an algorithm that implements the technique.