Test Case Prioritization Research Articles

Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software

Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable software

A Learning Classifier System for Automated Test Case Prioritization and Selection

A Learning Classifier System for Automated Test Case Prioritization and Selection

Fault-Based Test Case Prioritization of Regression Testing Using Genetic Algorithm

Prioritization of test case technique schedules the test suite in an order which will increase the efficiency and effectiveness of the system. The test case executed in an order which enhances the probability of finding the fault and also it finds the severe fault at the early stage. The prioritization of test case is proved to be useful for the regression testing. While much research is going on code coverage based prioritization technique and requirement based test case prioritization technique is not used by many. In this paper, test case prioritization technique based on fault of regression technique is proposed from the requirement specification, to improve the software quality and to increase the fault detection rate. In this paper, test case prioritization is done based the six main factors (i) requirement based on customer priority (ii) based on changes made in requirement specification (iii) complexity in Implementation of code (iv) re-usability (v) flow of application (vi) impact of the fault.

A Systematic Literature Review on prioritizing software test cases using Markov chains

Software Testing is a costly activity since the size of the test case set tends to increase as the construction of the software evolves. Test Case Prioritization (TCP) can reduce the effort and cost of software testing. TCP is an activity where a subset of the existing test cases is selected in order to maximize the possibility of finding defects. On the other hand, Markov Chains representing a reactive system, when solved, can present the occupation time of each of their states. The idea is to use such information and associate priority to those test cases that consist of states with the highest probabilities. The objective of this paper is to conduct a survey to identify and understand key initiatives for using Markov Chains in TCP. Aspects such as approaches, developed techniques, programming languages , analytical and simulation results, and validation tests are investigated. A Systematic Literature Review (SLR) was conducted considering studies published up to July 2021 from five different databases to answer the three research questions. From SLR, we identified 480 studies addressing Markov Chains in TCP that have been reviewed in order to extract relevant information on a set of research questions. The final 12 studies analyzed use Markov Chains at some stage of test case prioritization in a distinct way, that is, we found that there is no strong relationship between any of the studies, not only on how the technique was used but also in the context of the application. Concerning the fields of application of this subject, 6 forms of approach were found: Controlled Markov Chain, Usage Model, Model-Based Test, Regression Test , Statistical Test, and Random Test. This demonstrates the versatility and robustness of the tool. A large part of the studies developed some prioritization tool, being its validation done in some cases analytically and in others numerically, such as: Measure of the software specification, Optimal Test Transition Probabilities, Adaptive Software Testing, Automatic Prioritization, Ant Colony Optimization , Model Driven approach, and Monte Carlo Random Testing. • Software Testing is a costly activity since the size of the test case set tends to increase as the construction of the software evolves. • Test Case Prioritization can reduce the effort and cost of software testing. • Markov Chains can be efficiently used in Test Case Prioritization.

Test case prioritization using partial attention

Test case prioritization (TCP) aims to reorder the regression test suite with a goal of increasing the fault detection rate. Various TCP techniques have been proposed based on different prioritization strategies. Among them, the greedy-based techniques are the most widely-used TCP techniques. However, existing greedy-based techniques usually reorder all candidate test cases in prioritization iterations, resulting in both efficiency and effectiveness problems. In this paper, we propose a generic partial attention mechanism, which adopts the previous priority values (i.e., the number of additionally-covered code units) to avoid considering all candidate test cases. Incorporating the mechanism with the additional-greedy strategy, we implement a novel coverage-based TCP technique based on partition ordering (OCP). OCP first groups the candidate test cases into different partitions and updates the partitions on the descending order. We conduct a comprehensive experiment on 19 versions of Java programs and 30 versions of C programs to compare the effectiveness and efficiency of OCP with six state-of-the-art TCP techniques: total-greedy, additional-greedy, lexicographical-greedy, unify-greedy, art-based, and search-based. The experimental results show that OCP achieves a better fault detection rate than the state-of-the-arts. Moreover, the time costs of OCP are found to achieve 85%–99% improvement than most state-of-the-arts.

Assigning the Test Case Priorities Using Butterfly Optimization Algorithm for Software Test

Software maintenance is the longest process of SDLC. It continues with the distribution of applications till the software is not in operation. Software modifications are an unavoidable aspect of the life cycle of software growth. The optimization of software testing is still an important task, as the average percentage of detected failures (APFD), the average percentage of Branch Coverage detection (APBCD), and output of time are unsatisfactory in priority test cases. We also suggested in this document to give priority to test cases using Butterfly Algorithms optimally. We use Butterfly Algorithm with a fitness function specified with a similitude-distance model to optimise the ordering of test cases. Three testing suites selected from the software testing case repository experimented with within 3 benchmarking programmes. Our Test Case Prioritization technique (TCP) was seen better than current works with the Butterfly APFD Algorithm as the output matrix. Overall APFD results show Butterfly Algorithm being a successful competitor in TCP applications.

Analytic hierarchy process-based regression test case prioritization technique enhancing the fault detection rate

Analytic hierarchy process-based regression test case prioritization technique enhancing the fault detection rate

Test Case Prioritization, Selection, and Reduction Using Improved Quantum-Behaved Particle Swarm Optimization.

The emerging areas of IoT and sensor networks bring lots of software applications on a daily basis. To keep up with the ever-changing expectations of clients and the competitive market, the software must be updated. The changes may cause unintended consequences, necessitating retesting, i.e., regression testing, before being released. The efficiency and efficacy of regression testing techniques can be improved with the use of optimization approaches. This paper proposes an improved quantum-behaved particle swarm optimization approach for regression testing. The algorithm is improved by employing a fix-up mechanism to perform perturbation for the combinatorial TCP problem. Second, the dynamic contraction-expansion coefficient is used to accelerate the convergence. It is followed by an adaptive test case selection strategy to choose the modification-revealing test cases. Finally, the superfluous test cases are removed. Furthermore, the algorithm’s robustness is analyzed for fault as well as statement coverage. The empirical results reveal that the proposed algorithm performs better than the Genetic Algorithm, Bat Algorithm, Grey Wolf Optimization, Particle Swarm Optimization and its variants for prioritizing test cases. The findings show that inclusivity, test selection percentage and cost reduction percentages are higher in the case of fault coverage compared to statement coverage but at the cost of high fault detection loss (approx. 7%) at the test case reduction stage.

On the use of commit-relevant mutants

Applying mutation testing to test subtle program changes, such as program patches or other small-scale code modifications, requires using mutants that capture the delta of the altered behaviours. To address this issue, we introduce the concept of commit-relevant mutants, which are the mutants that interact with the behaviours of the system affected by a particular commit. Therefore, commit-aware mutation testing, is a test assessment metric tailored to a specific commit. By analysing 83 commits from 25 projects involving 2,253,610 mutants in both C and Java, we identify the commit-relevant mutants and explore their relationship with other categories of mutants. Our results show that commit-relevant mutants represent a small subset of all mutants, which differs from the other classes of mutants (subsuming and hard-to-kill), and that the commit-relevant mutation score is weakly correlated with the traditional mutation score (Kendall/Pearson 0.15-0.4). Moreover, commit-aware mutation analysis provides insights about the testing of a commit, which can be more efficient than the classical mutation analysis; in our experiments, by analysing the same number of mutants, commit-aware mutants have better fault-revelation potential (30% higher chances of revealing commit-introducing faults) than traditional mutants. We also illustrate a possible application of commit-aware mutation testing as a metric to evaluate test case prioritisation.

A Quantitative Function for Estimating the Comparative Values of Software Test Cases

Software testing is becoming more critical to ensure that software functions properly. As the time, effort, and funds invested in software testing activities have been increased significantly, these resources still cannot meet the increasing demand of software testing. Managers must allocate testing resources to the test cases effectively in uncovering important defects. This study builds a value function that can quantify the relative value of a test case and thus play a significant role in prioritizing test cases, addressing the resource constraint issues in software testing and serving as a foundation of AI for software testing. The authors conducted a Monte Carlo simulation to exhibit application of the final value function.

Value-based cost-cognizant test case prioritization for regression testing.

Software Test Case Prioritization (TCP) is an effective approach for regression testing to tackle time and budget constraints. The major benefit of TCP is to save time through the prioritization of important test cases first. Existing TCP techniques can be categorized as value-neutral and value-based approaches. In a value-based fashion, the cost of test cases and severity of faults are considered whereas, in a value-neutral fashion these are not considered. The value-neutral fashion is dominant over value-based fashion, and it assumes that all test cases have equal cost and all software faults have equal severity. But this assumption rarely holds in practice. Therefore, value-neutral TCP techniques are prone to produce unsatisfactory results. To overcome this research gap, a paradigm shift is required from value-neutral to value-based TCP techniques. Currently, very limited work is done in a value-based fashion and to the best of the authors’ knowledge, no comprehensive review of value-based cost-cognizant TCP techniques is available in the literature. To address this problem, a systematic literature review (SLR) of value-based cost-cognizant TCP techniques is presented in this paper. The core objective of this study is to combine the overall knowledge related to value-based cost-cognizant TCP techniques and to highlight some open research problems of this domain. Initially, 165 papers were reviewed from the prominent research repositories. Among these 165 papers, 21 papers were selected by using defined inclusion/exclusion criteria and quality assessment procedures. The established questions are answered through a thorough analysis of the selected papers by comparing their research contributions in terms of the algorithm used, the performance evaluation metric, and the results validation method used. Total 12 papers used an algorithm for their technique but 9 papers didn’t use any algorithm. Particle Swarm Optimization (PSO) Algorithm is dominantly used. For results validation, 4 methods are used including, Empirical study, Experiment, Case study, and Industrial case study. The experiment method is dominantly used. Total 6 performance evaluation metrics are used and the APFDc metric is dominantly used. This SLR yields that value-orientation and cost cognition are vital in the TCP process to achieve its intended goals and there is great research potential in this research domain.

Improved novel bat algorithm for test case prioritization and minimization

Improved novel bat algorithm for test case prioritization and minimization

Assessing expert system-assisted literature reviews with a case study

Given the large numbers of publications in software engineering, frequent literature reviews are required to keep current on work in specific areas. One tedious work in literature reviews is to find relevant studies amongst thousands of non-relevant search results. In theory, expert systems can assist in finding relevant work but those systems have primarily been tested in simulations rather than in application to actual literature reviews. Hence, few researchers have faith in such expert systems. Accordingly, using a realistic case study, this paper assesses how well our state-of-the-art expert system can help with literature reviews.The assessed literature review aimed at identifying test case prioritization techniques for automated UI testing, specifically from 8,349 papers on IEEE Xplore. This corpus was studied with an expert system that incorporates an incrementally updated human-in-the-loop active learning tool. Using that expert system, in three hours, we found 242 relevant papers from which we identified 12 techniques representing the state-of-the-art in test case prioritization when source code information is not available. These results were then validated by six other graduate students manually exploring the same corpus. Without the expert system, this task would have required 53 h and would have found 27 additional papers. That is, our expert system achieved 90% recall with 6% of the human effort cost when compared to a conventional manual method. Significantly, the same 12 state-of-the-art test case prioritization techniques were identified by both the expert system and the manual method. That is, the 27 papers missed by the expert system would not have changed the conclusion of the literature review.Hence, if this result generalizes, it endorses the use of our expert system to assist in literature reviews.

Test case prioritization using modified genetic algorithm and ant colony optimization for regression testing

Test case prioritization using modified genetic algorithm and ant colony optimization for regression testing

Test Cases Prioritization Using Ant Colony Optimization and Firefly Algorithm

Test Cases Prioritization Using Ant Colony Optimization and Firefly Algorithm

Multi-Objective Crow Search and Fruit Fly Optimization for Combinatorial Test Case Prioritization

This paper proposes a novel test case prioritization technique, namely multi-objective crow search and fruitfly optimization (MOCSFO) for test case prioritization. The proposed MOCSFO is designed by integrating crow search algorithm (CSA) and chaotic fruitfly optimization algorithm (CFOA). The optimal test cases are selected based on newly modelled fitness function, which consist of two parameters, namely average percentage of combinatorial coverage (APCC) and normalized average of the percentage of faults detected (NAPFD). The test case to be selected is decided using a searching criterion or fitness based on sequential weighed coverage size. Accordingly, the effective searching criterion is formulated to determine the optimal test cases based on the constraints. The experimentation of the proposed MOCSFO method is performed by considering the performance metrics, like NAPFD and APCC. The proposed MOCSFO outperformed the existing methods with enhanced NAPFD of 0.7, and APCC of 0.837.

Test case prioritization with neuron valuation based pattern

• Filtering error-triggering data in testing DNNs is cost-effective. • The set of test cases may contain various types of data. • Patterns for DNN-based classifiers can help in prioritizing the abnormal inputs. • Our pattern consists of neurons and the computed thresholds for pattern matching. • Evaluation results show our advantage in prioritizing error-triggering samples. Deep neural networks (DNNs) have been increasingly adopted in various applications. Systematic verification and validation are essential techniques to guarantee the quality of such systems. Testing is one of the feasible solutions for system validation. However, testing DNNs usually requires a large number of test cases and leads to the high labeling cost. Meanwhile, the test cases may contain various noisy data, which also increases the burden of testing. In this paper, we put forwards a test case prioritization method for DNN-based classifiers, which assigns high priorities to those cases that can lead to problematic classifications. The test cases are prioritized w.r.t. the pattern extracted from the neuron outputs. The difference between the pattern acquired from the training set and the neurons matched from a given input determines the priority of the input. For a trained model, the method consists of two steps. First, we collect neuron output values over the training set and construct a neuron-based pattern for every class of training samples. Second, the metrics are computed according to the comparison between neuron output values of the DNN model from an input and the selected pattern. We carry out the experimentation over three popular datasets with various neural network structures. The extensive experimental results demonstrate that the efficiency and the generalizability of the prioritization method outperform most of the existing techniques.

Dissimilarity‐based test case prioritization through data fusion

AbstractTest case prioritization (TCP) aims at scheduling test case execution so that more important test cases are executed as early as possible. Many TCP techniques have been proposed, according to different concepts and principles, with dissimilarity‐based TCP (DTCP) prioritizing tests based on the concept of test case dissimilarity: DTCP chooses the next test case from a set of candidates such that the chosen test case is farther away from previously selected test cases than the other candidates. DTCP techniques typically only use one aspect/granularity of the information or features from test cases to support the prioritization process. In this article, we adopt the concept of data fusion to propose a new family of DTCP techniques, data‐fusion‐driven DTCP (DDTCP), which attempts to use different information granularities for prioritizing test cases by dissimilarity. We performed an empirical study involving 30 versions of five subject programs, investigating the testing effectiveness and efficiency by comparing DDTCP against DTCP techniques that use a dissimilarity granularity. The experimental results show that not only does DDTCP have better fault‐detection rates than single‐granularity DTCP techniques, but it also appears to only incur similar prioritization costs. The results also show that DDTCP remains robust over multiple system releases.

Validating Switch Variable of Flower Pollination Algorithm in Regression Test Case Prioritization

This study aims at analysing and validating the impact of switch variable of Flower Pollination Algorithm (FPA) in regression test case prioritization. The value of switch variable decides whether local or global optimization will take place. The present study makes an attempt to analyse the impact of switch variable with values ranging from 0.8 to 0.1 on test case prioritization in regression testing. The present study is an extension of the prior work done to implement FPA for test case prioritization in regression testing. To validate the study two case studies written in Java programming were used. The results were validated using the APFD metrics. From the empirical results it was deduced that values ranging between 0.8 and 0.5 of switch variable gave promising results for test case prioritization.

Test Case Prioritization Using Partial Attention

Test case prioritization (TCP) aims to reorder the regression test suite with a goal of increasing the fault detection rate. Various TCP techniques have been proposed based on different prioritization strategies. Among them, the greedy-based techniques are the most widely-used TCP techniques. However, existing greedy-based techniques usually reorder all candidate test cases in prioritization iterations, resulting in both efficiency and effectiveness problems. In this paper, we propose a generic partial attention mechanism, which adopts the previous priority values (i.e., the number of additionally-covered code units) to avoid considering all candidate test cases. Incorporating the mechanism with the additional-greedy strategy, we implement a novel coverage-based TCP technique based on partition ordering (OCP). OCP first groups the candidate test cases into different partitions and updates the partitions on the descending order. We conduct a comprehensive experiment on 19 versions of Java programs and 30 versions of C programs to compare the effectiveness and efficiency of OCP with six state-of-the-art TCP techniques: total-greedy, additional-greedy, lexicographical-greedy, unify-greedy, art-based, and search-based. The experimental results show that OCP achieves a better fault detection rate than the state-of-the-arts. Moreover, the time costs of OCP are found to achieve 85%-99% improvement than most state-of-the-arts.