Test Case Prioritization Research Articles

A Systematic Literature Review of Test Case Prioritization Technique on Software Product Line Testing

A Systematic Literature Review of Test Case Prioritization Technique on Software Product Line Testing

Nonlinear Reinforcement Learning-Based Dynamic Test Case Prioritization with Anomaly Detection for Continuous Integration

This research article introduces DynamicR-TCP, a nonlinear approach to dynamic test case prioritization (TCP) leveraging reinforcement learning and anomaly detection. Designed to enhance the efficiency of software testing in continuous integration (CI) environments, the proposed model dynamically adapts to changing conditions by learning complex patterns from historical test data. A reinforcement learning agent, employing policy and value networks, guides nonlinear prioritization by optimizing the sequence of test case executions. The model integrates a sliding window strategy for adaptive focus on smaller test subsets and uses an experience replay buffer for iterative learning. An anomaly detection layer monitors test outcomes for deviations, triggering retraining when needed to maintain reliability. Experimental evaluation using the Defects4J dataset demonstrates the model’s superior fault detection rates, faster time to first fault, and higher computational efficiency compared to contemporary methods. An ablation study highlights the nonlinear synergy among the key components—reinforcement learning, sliding windows, and anomaly detection—showing significant performance gains when combined. This comprehensive framework advances software testing by providing an adaptive, efficient, and scalable solution, ensuring robust performance in dynamic CI environments.

Accelerating Time-to-Market in Life Insurance: The Power of AI-Driven Test Automation Frameworks

The Life Insurance sector grapples with distinct challenges, balancing multifaceted operational workflows, rigorous compliance standards, and the surging demand for swift software deployment. Conventional Quality Assurance (QA) paradigms often fall short in delivering the responsiveness required by today's digital transformation efforts. This paper examines how Artificial Intelligence (AI)-enhanced test automation frameworks can substantially truncate time-to-market while ensuring software excellence. By leveraging advanced predictive analytics, strategic test case prioritization, and integrated Continuous Integration/Continuous Delivery (CI/CD) processes, Life Insurance enterprises can accelerate software launch cycles with improved dependability, thereby meeting the ever-evolving expectations of the marketplace and regulatory entities.

Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios

Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a ( \(2D\) ) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.

A Comprehensive Assessment Model for Requirement-based Test Case Prioritization: Integrating Internal Factors and Dependency Analysis through Expert Evaluation

A Comprehensive Assessment Model for Requirement-based Test Case Prioritization: Integrating Internal Factors and Dependency Analysis through Expert Evaluation

An Enhancement of Multi-Factor Weighted Approach Technique in Prioritizing Test Cases by Comparing Similarity Distance

An Enhancement of Multi-Factor Weighted Approach Technique in Prioritizing Test Cases by Comparing Similarity Distance

Prioritizing test cases for deep learning-based video classifiers

The widespread adoption of video-based applications across various fields highlights their importance in modern software systems. However, in comparison to images or text, labelling video test cases for the purpose of assessing system accuracy can lead to increased expenses due to their temporal structure and larger volume. Test prioritization has emerged as a promising approach to mitigate the labeling cost, which prioritizes potentially misclassified test inputs so that such inputs can be identified earlier with limited time and manual labeling efforts. However, applying existing prioritization techniques to video test cases faces certain limitations: they do not account for the unique temporal information present in video data. Unlike static image datasets that only contain spatial information, video inputs consist of multiple frames that capture the dynamic changes of objects over time. In this paper, we propose VRank, the first test prioritization approach designed specifically for video test inputs. The fundamental idea behind VRank is that video-type tests with a higher probability of being misclassified by the evaluated DNN classifier are considered more likely to reveal faults and will be prioritized higher. To this end, we train a ranking model with the aim of predicting the probability of a given test input being misclassified by a DNN classifier. This prediction relies on four types of generated features: temporal features (TF), video embedding features (EF), prediction features (PF), and uncertainty features (UF). We rank all test inputs in the target test set based on their misclassification probabilities. Videos with a higher likelihood of being misclassified will be prioritized higher. We conducted an empirical evaluation to assess the performance of VRank, involving 120 subjects with both natural and noisy datasets. The experimental results reveal VRank outperforms all compared test prioritization methods, with an average improvement of 5.76%∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}46.51% on natural datasets and 4.26%∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}53.56% on noisy datasets.

Model-based diversity-driven learn-to-rank test case prioritization

Model-based Test Case Prioritization utilizing similarity metrics has proved effective in software testing. However, the utility of similarity metrics in it varies with test scenarios, hindering its universal effectiveness and performance optimization. To tackle this problem, we propose a Diversity-driven Learn-to-rank model-based TCP approach, named DLTCP, for optimizing early fault detection performance. Our method first employs the whale optimization algorithm to search for a suitable set of similarity metrics from a pool of existing candidates. This search process determines which metrics should be used. According to each selected metric, test cases are then prioritized. The resulting test case rankings are used as the training data for DLTCP. Finally, the proposed method incorporates random forest to train a ranking model for test case prioritization. As such, it can fuse multiple similarity metrics to improve the TCP performance. We conduct extensive experiments to evaluate our method’s performance using the average percentage fault detected (APFD) as metric. The experimental results show that DLTCP achieve an average APFD value of 0.953 for seven classic benchmark models, which is 11.37% higher than that of the state-of-the-art algorithms. It can well select a set of similarity metrics for effective fusion, demonstrating competitive performance in early fault detection.

Retraction Note: Test case prioritization using a Hybrid Chaotic Flower-fruit fly optimization algorithm with multiple objectives

Retraction Note: Test case prioritization using a Hybrid Chaotic Flower-fruit fly optimization algorithm with multiple objectives

Object Constraint Language based test case optimization with modified Average Percentage of Fault Detection metric

AbstractTesting is one of the most time‐consuming and unpredictable processes within the software development life cycle. As a result, many test case optimization (TCO) techniques have been proposed to make this process more scalable. Object Constraint Language (OCL) was initially introduced as a constraint language to provide additional details to Unified Modeling Language models. However, as OCL continues to evolve, an increasing number of systems are being expressed by this language. Despite this growth, a noticeable research gap exists for the testing of systems whose specifications are expressed in OCL. In our previous work, we verified the effectiveness and efficiency of performing the test case prioritization (TCP) process for these systems. In this study, we extend our previous work by integrating the test case minimization (TCM) process to determine whether TCM can also benefit the testing process under the context of OCL. The evaluation of TCO approaches often relies on well‐established metrics such as the average percentage of fault detection (APFD). However, the suitability of APFD for model‐based testing (MBT) is not ideal. This paper addresses this limitation by proposing a modification to the APFD metric to enhance its viability for MBT scenarios. We conducted four case studies to evaluate the feasibility of integrating the TCM and TCP processes into our proposed approach. In these studies, we applied the multi‐objective optimization algorithm NSGA‐II and the genetic algorithm independently to the TCM and TCP processes. The objective was to assess the effectiveness and efficiency of combining TCM and TCP in enhancing the testing phase. Through experimental analysis, the results highlight the benefits of integrating TCM and TCP in the context of OCL‐based testing, providing valuable insights for practitioners and researchers aiming to optimize their testing efforts. Specifically, the main contributions of this work include the following: (1) we introduce the integration of the TCM process into the TCO process for systems expressed by OCL. This integration benefits the testing process further by reducing redundant test cases while ensuring sufficient coverage. (2) We comprehensively analyze the limitations associated with the commonly used metric, APFD, and then, a modified version of the APFD metric has been proposed to overcome these weaknesses. (3). We systematically evaluate the effectiveness and efficiency of OCL‐based TCO processes on four real‐world case studies with different complexities.

Security Development Lifecycle-Based Adaptive Reward Mechanism for Reinforcement Learning in Continuous Integration Testing Optimization

Continuous automated testing throughout each cycle can ensure the security of the continuous integration (CI) development lifecycle. Test case prioritization (TCP) is a critical factor in optimizing automated testing, which prioritizes potentially failed test cases and improves the efficiency of automated testing. In CI automated testing, the TCP is a continuous decision-making process that can be solved with reinforcement learning (RL). RL-based CITCP can continuously generate a TCP strategy for each CI development lifecycle, with the reward mechanism as the core. The reward mechanism consists of the reward function and the reward strategy. However, there are new challenges to RL-based CITCP in real-industry CI testing. With high-frequency iteration, the reward function is often calculated with a fixed length of historical information, ignoring the spatial characteristics of the current cycle. Therefore, the dynamic time window (DTW)-based reward function is proposed to perform the reward calculation, which adaptively adjusts the recent historical information range based on the integration cycle. Moreover, with low-failure testing, the reward strategy usually only rewards failure test cases, which creates a sparse reward problem in RL. To address this issue, the similarity-based reward strategy is proposed, which increases the reward objects of some passed test cases, similar to the failure test cases. The DTW-based reward function and the similarity-based reward strategy together constitute the proposed adaptive reward mechanism in RL-based CITCP. To validate the effectiveness of the adaptive reward mechanism, experimental verification is carried out on 13 industrial data sets. The experimental results show that the adaptive reward mechanism can improve the TCP effect, where the average NAPFD is maximally improved by 7.29%, the average Recall is maximally improved by 6.04% and the average TTF is improved by 6.81 positions with a maximum of 63.77.

Test case prioritization based on fault sensitivity analysis using ranked NSGA-2

This paper discusses regression testing in software maintenance, focusing on test case prioritization to verify modifications to software functionality efficiently. The primary goal is to rank test cases, prioritizing those covering more code or faults with minimal execution time. The challenge lies in prioritizing numerous test cases generated during development and maintenance. Various algorithms, including greedy approaches and meta-heuristic techniques, address this challenge. The paper introduces a ranking-based non-dominated sorting genetic algorithm (NSGA-2) for test case prioritization, emphasizing cases sensitive to faults caused by modifications or new functionality. Historical data is prioritized, with key objectives including the sensitive index, execution cost, and average percentage of fault detection (APFD). The proposed model is tested on handcrafted and benchmark Java-based applications, comparing its performance to state-of-the-art algorithms in test case prioritization.

Optimizing test case prioritization through ranked NSGA-2 for enhanced fault sensitivity analysis

Optimizing test case prioritization through ranked NSGA-2 for enhanced fault sensitivity analysis

Exploiting DBSCAN and Combination Strategy to Prioritize the Test Suite in Regression Testing

Test case prioritization techniques improve the fault detection rate by adjusting the execution sequence of test cases. For static black-box test case prioritization techniques, existing methods generally improve the fault detection rate by increasing the early diversity of execution sequences based on string distance differences. However, such methods have a high time overhead and are less stable. This paper proposes a novel test case prioritization method (DC-TCP) based on density-based spatial clustering of applications with noise (DBSCAN) and combination policies. By introducing a combination strategy to model the inputs to generate a mapping model, the test inputs are mapped to consistent types to improve generality. The DBSCAN method is then used to refine the classification of test cases further, and finally, the Firefly search strategy is introduced to improve the effectiveness of sequence merging. Extensive experimental results demonstrate that the proposed DC-TCP method outperforms other methods in terms of the average percentage of faults detected and exhibits advantages in terms of time efficiency when compared to several existing static black-box sorting methods.

Enhancing Automated Test Case Generation Through KNN algorithm in Software Testing

In the ever-evolving world of software development, assuring the stability and resilience of software systems remains a vital task. This study aims to tackle this problem by suggesting a novel method for generating automated test cases. It involves incorporating the robust KNN algorithm into the domain of data mining methods. The research intends to exploit previous testing data to detect patterns and trends, increasing the production of effective and targeted test scenarios.
 The literature study highlights the increasing overlap between data mining and software testing, underscoring the need for sophisticated methods to address the requirements of contemporary, ever-changing applications. In this regard, our study presents the KNN algorithm as a new addition to the collection of data mining tools for software testing. The process entails gathering extensive historical testing data, including information on test cases, errors, and the attributes of tested software applications. By using rigorous preprocessing and feature selection techniques, the KNN method is used to examine the correlations between various variables and the probability of flaws. The KNN model is trained and optimized using the dataset, and the resultant predictions are then used in an automated system for generating test cases. The experimental results confirm the efficacy of the suggested method in finding crucial test scenarios and prioritizing test cases according to past fault trends. The practical usefulness and advantages of incorporating the KNN algorithm into the automated test case-generating process are further confirmed by real-world case studies. In conclusion, this study expands the discourse on data mining in software testing by proposing the KNN algorithm as a vital component in automated test case development. By using KNN, testing teams can adjust to the ever-changing characteristics of contemporary applications, leading to enhanced efficiency and effectiveness in testing procedures. The results of this research enhance the continuous development of software quality assurance methods and provide opportunities for additional investigation at the convergence of machine learning and software testing

On the use of contextual information for machine learning based test case prioritization in continuous integration development

Context:In most software organizations, Continuous Integration (CI) is a common practice usually subject to some budgets. Consequently, prioritizing test cases to be executed in the CI cycle is fundamental. The idea is first to execute test cases with higher failure-proneness to provide rapid feedback and decrease costs. To perform this task approaches in the literature adopt failure history and Machine Learning (ML). However, in addition to the failure history, it is also important to consider information from the CI context of the organizations and the application domain. Objective:For this end, we introduce a contextual information approach for ML algorithms. Such an approach considers information from the testing activity that can be easily collected, such as test case execution time, size, and complexity. We implement the approach by introducing two contextual versions of the algorithms: Multi-Armed Bandit (MAB) and Random Forest (RF). Method:Six systems are used to compare both contextual algorithms and to evaluate their performance regarding their corresponding non-contextual versions, considering three different budgets. Results:Contextual algorithms perform better when indicators related to test time reduction are considered, as the contextual information they use is related to execution time. Regarding NAPFD and APFDc, the non-contextual algorithms have better general performance, but both contextual versions obtain competitive results. Conclusions:The contextual versions implemented can capture the desired context information in the prioritization without negatively impacting their performance regarding fault-detection.

Fault sensitivity index-based multi-objective testcase prioritization

Abstract Test case prioritization (TCP) is a regression technique that sequences test cases by assigning priority based on specific criteria defined by software testers. Various parameters, such as code coverage, statement coverage, and method coverage, are utilized in Test Case Prioritization (TCP), wherein metaheuristic techniques are widely employed to determine the optimal order of test cases based on these specified parameters. However, simply applying these techniques does not ensure the satisfaction of all the needs of software testers. This paper introduces an empirical study that employs the multi-objective test case prioritization (MOTCP) technique to prioritize the test cases based on target points defined by software testers. The study calculates a Software Complexity Index (SCI) at the code level, identifying fault-prone areas. Furthermore, a Test-case Complexity Index (TCI) is also used for prioritization. The proposed technique incorporates various target points defined by the software tester to calculate SCI and TCI, which serve as our main objectives for TCP. A detailed analysis is also performed to examine the impact of these target points on the generated optimal order of test cases. Finally, the proposed model is compared with other state-of-the-art techniques across various evaluation parameters.

Multi-factor based Regression Test Case Prioritization using Fuzzy Logic

The maintenance level activity generally done after the modification in the software to check whether it is functioning right or not is termed as regression testing. Test case prioritization, a key practice, involves strategically ordering test cases based on specific criteria to enhance the efficiency of fault detection within a condensed time frame. The fuzzy rule base serves as an alternative to the conventional crisp value set, offering a nuanced approach beyond binary outcomes (Yes or No). The primary objective of this research is to address critical factors often overlooked in existing literature on prioritization. Notably, prevalent approaches focus on singular factors during test case prioritization, highlighting the need for a comprehensive technique. To enhance the prioritization of test cases, there is a demand for a method that considers multi-factors or combinations thereof, ultimately increasing effectiveness. This paper introduces an innovative approach a multi-factors regression test-case prioritization technique utilizing fuzzy rules. The methodology aims to optimize the prioritization of test cases, striking a balance between effectiveness and time efficiency. Fuzzy rules are formulated to assess the effectiveness of a prioritized set of test cases in developing the proposed approach. A user-friendly tool has been developed to facilitate the application of this technique, allowing users to input relevant factors and subsequently prioritize test cases accordingly. Through extensive experiments using the developed tool, the effectiveness of the proposed approach has been validated. The results demonstrate that the priority lists of test cases generated for different projects, considering multi-factors, show greater promise compared to techniques relying solely on a single factor for prioritization.

An empirically based object-oriented testing using Machine learning

INTRODUCTION: The rapid growth of machine learning has the potential to revolutionize various industries and applications by automating complex tasks and enhancing efficiency. Effective software testing is crucial for ensuring software quality and minimizing resource expenses in software engineering. Machine learning techniques play a vital role in software testing by aiding in test case prioritization, predicting software defects, and analyzing test results. OBJECTIVES: The primary objective of this study is to explore the use of machine learning algorithms for software defect prediction. METHODS: Machine Learning models including Random Forest Classifier, Logistic Regression, K Nearest Neighbors, Gradient Boosting Classifiers, Catboost Classifier, and Convolutional Neural Networks have been employed for the study. The dataset includes a wide range of features relevant to software defect prediction and evaluates the performance of different prediction models. The study also focussed on developing hybrid models using stacking classifiers, which combine multiple individual models to improve accuracy. RESULTS: The experimental results show that the hybrid models combining CatBoost and Convolutional Neural Network have outperformed individual models, achieving the highest accuracy of 89.5%, highlighting the effectiveness of combining machine learning algorithms for software defect prediction. CONCLUSION: In conclusion, this study sheds light on the pivotal role of machine learning in enhancing software defect prediction.

An Improved Deep Learning Based Test Case Prioritization Using Deep Reinforcement Learning

An Improved Deep Learning Based Test Case Prioritization Using Deep Reinforcement Learning