Test Case Prioritization Method Research Articles

Exploiting DBSCAN and Combination Strategy to Prioritize the Test Suite in Regression Testing

Test case prioritization techniques improve the fault detection rate by adjusting the execution sequence of test cases. For static black-box test case prioritization techniques, existing methods generally improve the fault detection rate by increasing the early diversity of execution sequences based on string distance differences. However, such methods have a high time overhead and are less stable. This paper proposes a novel test case prioritization method (DC-TCP) based on density-based spatial clustering of applications with noise (DBSCAN) and combination policies. By introducing a combination strategy to model the inputs to generate a mapping model, the test inputs are mapped to consistent types to improve generality. The DBSCAN method is then used to refine the classification of test cases further, and finally, the Firefly search strategy is introduced to improve the effectiveness of sequence merging. Extensive experimental results demonstrate that the proposed DC-TCP method outperforms other methods in terms of the average percentage of faults detected and exhibits advantages in terms of time efficiency when compared to several existing static black-box sorting methods.

A Hybrid Meta-Heuristic Approach for Test Case Prioritization and Optimization

The application of the test case prioritization method is a key part of system testing intended to think it through and sort out the issues early in the development stage. Traditional prioritization techniques frequently fail to take into account the complexities of big-scale test suites, growing systems and time constraints, therefore cannot fully fix this problem. The proposed study here will deal with a meta-heuristic hybrid method that focuses on addressing the challenges of the modern time. The strategy utilizes genetic algorithms alongside a black hole as a means to create a smooth tradeoff between exploring numerous possibilities and exploiting the best one. The proposed hybrid algorithm of genetic black hole (HGBH) uses the capabilities of considering the imperatives such as code coverage, fault finding rate and execution time from search algorithms in our hybrid approach to refine test cases considerations repetitively. The strategy accomplished this by putting experiments on a large-scale project of industrial software developed. The hybrid meta-heuristic technique ends up being better than the routine techniques. It helps in higher code coverage, which, in turn, enables to detect crucial defects at an early stage and also to allocate the testing resources in a better way. In particular, the best APFD value was 0.9321, which was achieved in 6 generations with 4.879 seconds the value to which the computer was run. Besides these, , the approach resulted in the mean value of APFD as 0.9247 and 0.9302 seconds which took from 10.509 seconds to 30.372 seconds. The carried out experiment proves the feasibility of this approach in implementing complex systems and consistently detecting the changes, enabling it to adapt to rapidly changing systems. In the end, this research provides us with a new hybrid meta-heuristic way of test case prioritization and optimization, which, in turn, helps to tackle the obstacles caused by large-scale test cases and constantly changing systems.

The MADAG Strategy for Fault Location Techniques

Spectrum-based fault localization (SBFL), which utilizes spectrum information of test cases to calculate the suspiciousness of each statement in a program, can reduce developers’ effort. However, applying redundant test cases from a test suite to fault localization incurs a heavy burden, especially in a restricted resource environment, and it is expensive and infeasible to inspect the results of each test input. Prioritizing/selecting appropriate test cases is important to enable the practical application of the SBFL technique. In addition, we must ensure that applying the selected tests to SBFL can achieve approximately the effectiveness of fault localization with whole tests. This paper presents a test case prioritization/selection strategy, namely the Minimal Aggregate of the Diversity of All Groups (MADAG). The MADAG strategy prioritizes/selects test cases using information on the diversity of the execution trace of each test case. We implemented and applied the MADAG strategy to 233 faulty versions of the Siemens and UNIX programs from the Software-artifact Infrastructure Repository. The experiments show that (1) the MADAG strategy uses only 8.99 and 14.27 test cases, with an average of 18, from the Siemens and UNIX test suites, respectively, and the SBFL technique has approximate effectiveness for fault localization on all test cases and outperforms the previous best test case prioritization method; (2) we verify that applying whole tests from the test suite may not achieve the better effectiveness in fault localization compared with the tests selected by MADAG strategy.

Test case prioritization using test case diversification and fault-proneness estimations

Regression testing activities greatly reduce the risk of faulty software release. However, the size of the test suites grows throughout the development process, resulting in time-consuming execution of the test suite and delayed feedback to the software development team. This has urged the need for approaches such as test case prioritization (TCP) and test-suite reduction to reach better results in case of limited resources. In this regard, proposing approaches that use auxiliary sources of data such as bug history can be interesting. We aim to propose an approach for TCP that takes into account test case coverage data, bug history, and test case diversification. To evaluate this approach we study its performance on real-world open-source projects. The bug history is used to estimate the fault-proneness of source code areas. The diversification of test cases is preserved by incorporating fault-proneness on a clustering-based approach scheme. The proposed methods are evaluated on datasets collected from the development history of five real-world projects including 357 versions in total. The experiments show that the proposed methods are superior to coverage-based TCP methods. The proposed approach shows that improvement of coverage-based and fault-proneness-based methods is possible by using a combination of diversification and fault-proneness incorporation.

Test case prioritization with neuron valuation based pattern

• Filtering error-triggering data in testing DNNs is cost-effective. • The set of test cases may contain various types of data. • Patterns for DNN-based classifiers can help in prioritizing the abnormal inputs. • Our pattern consists of neurons and the computed thresholds for pattern matching. • Evaluation results show our advantage in prioritizing error-triggering samples. Deep neural networks (DNNs) have been increasingly adopted in various applications. Systematic verification and validation are essential techniques to guarantee the quality of such systems. Testing is one of the feasible solutions for system validation. However, testing DNNs usually requires a large number of test cases and leads to the high labeling cost. Meanwhile, the test cases may contain various noisy data, which also increases the burden of testing. In this paper, we put forwards a test case prioritization method for DNN-based classifiers, which assigns high priorities to those cases that can lead to problematic classifications. The test cases are prioritized w.r.t. the pattern extracted from the neuron outputs. The difference between the pattern acquired from the training set and the neurons matched from a given input determines the priority of the input. For a trained model, the method consists of two steps. First, we collect neuron output values over the training set and construct a neuron-based pattern for every class of training samples. Second, the metrics are computed according to the comparison between neuron output values of the DNN model from an input and the selected pattern. We carry out the experimentation over three popular datasets with various neural network structures. The extensive experimental results demonstrate that the efficiency and the generalizability of the prioritization method outperform most of the existing techniques.

A Comparative Evaluation of Test Coverage Techniques Effectiveness

Software systems have become complex and challenging to develop and maintain because of the large size of test cases with increased scalability issues. Test case prioritization methods have been successfully utilized in test case management. However, the prohibitively exorbitant cost of large test cases is now the mainstream in the software industry. The growth of agile test-driven development has increased the expectations for software quality. Yet, our knowledge of when to use various path testing criteria for cost-effectiveness is inadequate due to the inherent complexity in software testing. Existing researches attempted to address the issue without effectively tackling the scalability of large test suites to reduce time in regression testing. In order to provide a more accurate way of fault detection in software projects, we introduced novel coverage criteria, called Incremental Cluster-based test case Prioritization (ICP), and investigated its potentials by making a comparative evaluation with three un-clustered traditional coverage-based criteria: Prime-Path Coverage (PPC), Edge-Pair Coverage (EPC) and Edge Coverage (EC) based on mutation analysis. By clustering test suites, based on their dynamic run-time behavior, the number of pair-wise comparisons is reduced significantly. To compare, we analyzed 20 functions from 25 C programs, instrumented faults into the programs, and used the Mull mutation tool to generate mutants and perform a statistical analysis of the results. The experimental results show that ICP can lead to cost-effective improvements in fault detection.

Test Case Prioritization—ANT Algorithm With Faults Severity

Abstract Regression testing is applied whenever a code changes, ensuring that the modifications fixed the fault and no other faults are introduced. Due to a large number of test cases to be run, test case prioritization is one of the strategies that allows to run the test cases with the highest fault rate first. The aim of the paper is to present an optimized test case prioritization method inspired by ant colony optimization, test case prioritization–ANT. The criteria used by the optimization algorithm are the number of faults not covered yet by the selected test cases and the sum of severity of the faults. The cost, i.e. time execution, for test cases is considered in the computation of the pheromone deposited on the graph’s edges. The average percentage of fault detected metric, as best selection criterion, is used to uncover maximum faults with the highest severity, and reducing the regression testing time. Several experiments are considered, detailed and discussed, comparing various algorithm parameter’s alternatives. A benchmark project is also used to validate the proposed approach. The obtained results are encouraging, being a cornerstone for new perspectives to be considered.

Model-Based Test Case Prioritization Using an Alternating Variable Method for Regression Testing of a UML-Based Model

Many test case prioritization (TCP) studies based on regression testing using a code-based development approach have appeared. However, few studies on model-based mutation testing have explored what kind of fault seeding is appropriate or how much the code-based results differ. In this paper, as automatic seeding for the mutation generation, several mutation operators were employed for the UML statechart. Here, we suggest mutation testing employing the model-based development approach and a new TCP method based on an alternating variable method (AVM). We statistically compare the average percentage of the fault detection (APFD) results of the new method to other TCP methods such as a greedy algorithm for code coverage or fault exposure possibility. Finally, in empirical studies, the model-based TCP results for a power window switch module, a body control module, and a passive entry and start system are derived; these are real industrial challenges in the automotive industry.

Nowatorska metoda priorytetyzacji przypadków testowych oparta na prognozowaniu błędów instrukcji kodu oprogramowania numerycznego

Test case prioritization (TCP) has been considerably utilized to arrange the implementation order of test cases, which contributes to improve the efficiency and resource allocation of software regression testing. Traditional coverage-based TCP techniques, such as statement-level, method/function-level and class-level, only leverages program code coverage to prioritize test cases without considering the probable distribution of defects. However, software defect data tends to be imbalanced following Pareto principle. Instinctively, the more vulnerable the code covered by the test case is, the higher the priority it is. Besides, statement-level coverage is a more fine-grained method than function-level coverage or class-level coverage, which can more accurately formulate test strategies. Therefore, we present a test case prioritization approach based on statement software defect prediction to tame the limitations of current coverage-based techniques in this paper. Statement metrics in the source code are extracted and data pre-processing is implemented to train the defect predictor. And then the defect detection rate of test cases is calculated by combining the prioritization strategy and prediction results. Finally, the prioritization performance is evaluated in terms of average percentage faults detected in four open source datasets. We comprehensively compare the performance of the proposed method under different prioritization strategies and predictors. The experimental results show it is a promising technique to improve the prevailing coverage-based TCP methods by incorporating statement-level defect-proneness. Moreover, it is also concluded that the performance of the additional strategy is better than that of max and total, and the choice of the defect predictor affects the efficiency of the strategy.

A Model-Based Test Case Prioritization Approach Based on Fault Urgency and Severity

With the aggrandizement scale of software system, the number of test cases has grown explosively. Test case prioritization (TCP) has been widely used in software testing to effectively improve testing efficiency. However, traditional TCP methods are mostly based on software code and they are difficult to apply to model-based testing. Moreover, existing model-based TCP techniques often do not take the likely distribution of faults into consideration, yet software faults are not often equally distributed in the system, and test cases that cover more fault prone modules are more likely to reveal faults so that they should be run with a higher priority. Therefore, in this paper, we provide a TCP approach based on Hidden Markov Model (HMM), to detect faults as earlier as possible and reduce the cost of modification. This approach consists of the following main parts: (1) transforming the Unified Modeling Language (UML) sequence diagram to HMM; (2) estimating the fault urgency according to fault priority and probability; (3) estimating the fault severity by analyzing the weight of the state in the HMM; (4) generating test case priority from fault urgency and fault severity, then prioritizing test case. The proposed approach is implemented on unmanned aerial vehicles (UAV) flight control system to perform TCP. The experimental results show that our proposed TCP approach can effectively enhance the probability of earlier fault detection and improve the efficiency and stability as compared to other prioritization techniques, such as original prioritization, random prioritization, additional prioritization and EPS-UML.

Incorporating fault-proneness estimations into coverage-based test case prioritization methods

Context: During the development process of a software program, regression testing is used to ensure that the correct behavior of the software is retained after updates to the source code. This regression testing becomes costly over time as the number of test cases increases and it makes sense to prioritize test cases in order to execute fault-detecting test cases as soon as possible. There are many coverage-based test case prioritization (TCP) methods that only use the code coverage data to prioritize test cases. By incorporating the fault-proneness estimations of code units into the coverage-based TCP methods, we can improve such techniques. Objective: In this paper, we aim to propose an approach which improves coverage-based TCP methods by considering the fault-proneness distribution over code units. Further, we present the results of an empirical study that shows using our proposed approach significantly improves the additional strategy, which is a widely used coverage-based TCP method. Method: The approach presented in this study uses the bug history of the software in order to introduce a defect prediction method to learn a neural network model. This model is then used to estimate fault-proneness of each area of the source code and then the estimations are incorporated into coverage-based TCP methods. Our proposed approach is a general idea that can be applied to many coverage-based methods, such as the additional and total TCP methods. Results: The proposed methods are evaluated on datasets collected from the development history of five real-world projects including 357 versions in total. The experiments show that using an appropriate bug history can improve coverage-based TCP methods.

Regression Test Cases Prioritization Using Clustering and Code Change Relevance

Regression testing is important for maintaining software quality. However, the cost of regression testing is relatively high. Test case prioritization is one way to reduce this cost. Test case prioritization techniques sort test cases for regression testing based on their importance. In this paper, we design and implement a test case prioritization method based on the location of a change. The method consists of three steps: (1) clustering test cases, (2) prioritizing the clusters with respect to the relevance of the clusters to a code change, and (3) test case prioritization within each cluster based on metrics. We propose a metric for measuring test case importance based on Requirement Complexity, Code Complexity, and Code Coverage. To evaluate our method, we apply it on a launch interceptor problem program, and measure the inclusiveness and precision for clusters of test cases with respect to code change in specific test cases. Our results show that our proposed change-based prioritization method increases the likelihood of executing more relevant test cases earlier.

Role of Test Case Prioritization based on Regression Testing using Clustering

Software testing is the most important part of software development life cycle. There are various types in software testing which have their own different functionalities. Among them regression testing is most useful functional type of testing which is done in the software maintence phase. This testing is used to check the errors when any change is made in the existing system. To make system efficient and effective, techniques of test case prioritization are use. The reduction in the cost of testing and fault detection capabilities of testing should be done by test case prioritization. This technique is also applied on different algorithms to improve their efficiency. Many clustering algorithms may also use test case prioritization method to increase the efficiency in code coverage. Prioritization techniques that incorporate a clustering approach and utilize code coverage, code complexity to increase the effectiveness of the prioritization.

Static test case prioritization using topic models

Software development teams use test suites to test changes to their source code. In many situations, the test suites are so large that executing every test for every source code change is infeasible, due to time and resource constraints. Development teams need to prioritize their test suite so that as many distinct faults as possible are detected early in the execution of the test suite. We consider the problem of static black-box test case prioritization (TCP), where test suites are prioritized without the availability of the source code of the system under test (SUT). We propose a new static black-box TCP technique that represents test cases using a previously unused data source in the test suite: the linguistic data of the test cases, i.e., their identifier names, comments, and string literals. Our technique applies a text analysis algorithm called topic modeling to the linguistic data to approximate the functionality of each test case, allowing our technique to give high priority to test cases that test different functionalities of the SUT. We compare our proposed technique with existing static black-box TCP techniques in a case study of multiple real-world open source systems: several versions of Apache Ant and Apache Derby. We find that our static black-box TCP technique outperforms existing static black-box TCP techniques, and has comparable or better performance than two existing execution-based TCP techniques. Static black-box TCP methods are widely applicable because the only input they require is the source code of the test cases themselves. This contrasts with other TCP techniques which require access to the SUT runtime behavior, to the SUT specification models, or to the SUT source code.

A Test Case Prioritization Method with Practical Weight Factors

A Test Case Prioritization Method with Practical Weight Factors

회귀 테스팅의 테스트 케이스 우선 순위화 기법의 실험적 연구

테스트 케이스 우선 순위화는 회귀 테스팅이 시간 제약 하에서 주어진 모든 테스트 케이tm를 수행할 수 없을 때 테스트 케이스의 실행 순서를 스캐쥴링하는 것이다. 본 논문에서는 장기적인 회귀 테스팅 환경에서 과거의 테스트 실행 및 오류 검출 정보를 활용한 HED우선 순위화 방법을 제안하고, 이를 기존의 Random 및 LRU 방법과 비교하였다. 본 실험을 통해 몇 가지 중요한 통찰을 얻을 수 있었다. 첫째, 우선 순위화 방법들이 프로그램의 특성에 따라 성능 면에서 상호 보완적이라는 점이다. 오류를 찾는 테스트 케이스들을 많이 갖고 있는 프로그램의 경우에는 Random이 효과적이고, 상대적으로 오류를 찾는 테스트 케이스의 비율이 작은 경우에는 제안된 HED방법이 좋은 성능을 보였으며, 중간 정도인 경우에는 LRU 방법이 효과적이었다. 둘째, 전체적인 성능이 테스트 스위트의 크기에 영향을 많이 받는다는 점이다. 테스트 스위트의 크기를 달리하여 실험한 결과 오류의 수명 값과 그 성능 순위에 차이를 보였다. 마지막으로 전체 테스트 케이스의 <TEX>$20\%$</TEX>만을 실행하여도 전체 테스트 케이스 모두를 실행하는 것과 성능 면에서 유사한 결과를 얻을 수 있다는 점 등이다. Test case prioritization methods schedule test cases for execution when we can not practically run all test cases for regression testing. We proposed a new prioritization method that is based on historical execution and mr detection data. And we conducted an experiment to compare the proposed method with existing Random and LRU methods using the fault age under the long run environment as criterion. The experiment shows several interesting results. First, our results show that they are complementary. Random method shows good performance for programs that have many error-detectable test cases and HED is more effective for the programs that can be detected by very small amount of test cases. But LRU is more effective for the programs that have relatively medium amount of error detectable test cases. Next, the performance of prioritization method is affected by the size of test suites. Two experiments that have different size of test suites show considerably different fault ages and performance order. And lastly, the <TEX>$20\%$</TEX> of test cases shows considerably good performance compared to the execution result of the full test suite.