Test Case Prioritization Research Articles

Accelerating Time-to-Market in Life Insurance: The Power of AI-Driven Test Automation Frameworks

The Life Insurance sector grapples with distinct challenges, balancing multifaceted operational workflows, rigorous compliance standards, and the surging demand for swift software deployment. Conventional Quality Assurance (QA) paradigms often fall short in delivering the responsiveness required by today's digital transformation efforts. This paper examines how Artificial Intelligence (AI)-enhanced test automation frameworks can substantially truncate time-to-market while ensuring software excellence. By leveraging advanced predictive analytics, strategic test case prioritization, and integrated Continuous Integration/Continuous Delivery (CI/CD) processes, Life Insurance enterprises can accelerate software launch cycles with improved dependability, thereby meeting the ever-evolving expectations of the marketplace and regulatory entities.

Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios

Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a ( \(2D\) ) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.

A Comprehensive Assessment Model for Requirement-based Test Case Prioritization: Integrating Internal Factors and Dependency Analysis through Expert Evaluation

A Comprehensive Assessment Model for Requirement-based Test Case Prioritization: Integrating Internal Factors and Dependency Analysis through Expert Evaluation

An Enhancement of Multi-Factor Weighted Approach Technique in Prioritizing Test Cases by Comparing Similarity Distance

An Enhancement of Multi-Factor Weighted Approach Technique in Prioritizing Test Cases by Comparing Similarity Distance

Prioritizing test cases for deep learning-based video classifiers

The widespread adoption of video-based applications across various fields highlights their importance in modern software systems. However, in comparison to images or text, labelling video test cases for the purpose of assessing system accuracy can lead to increased expenses due to their temporal structure and larger volume. Test prioritization has emerged as a promising approach to mitigate the labeling cost, which prioritizes potentially misclassified test inputs so that such inputs can be identified earlier with limited time and manual labeling efforts. However, applying existing prioritization techniques to video test cases faces certain limitations: they do not account for the unique temporal information present in video data. Unlike static image datasets that only contain spatial information, video inputs consist of multiple frames that capture the dynamic changes of objects over time. In this paper, we propose VRank, the first test prioritization approach designed specifically for video test inputs. The fundamental idea behind VRank is that video-type tests with a higher probability of being misclassified by the evaluated DNN classifier are considered more likely to reveal faults and will be prioritized higher. To this end, we train a ranking model with the aim of predicting the probability of a given test input being misclassified by a DNN classifier. This prediction relies on four types of generated features: temporal features (TF), video embedding features (EF), prediction features (PF), and uncertainty features (UF). We rank all test inputs in the target test set based on their misclassification probabilities. Videos with a higher likelihood of being misclassified will be prioritized higher. We conducted an empirical evaluation to assess the performance of VRank, involving 120 subjects with both natural and noisy datasets. The experimental results reveal VRank outperforms all compared test prioritization methods, with an average improvement of 5.76%∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}46.51% on natural datasets and 4.26%∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim $$\\end{document}53.56% on noisy datasets.

Model-based diversity-driven learn-to-rank test case prioritization

Model-based Test Case Prioritization utilizing similarity metrics has proved effective in software testing. However, the utility of similarity metrics in it varies with test scenarios, hindering its universal effectiveness and performance optimization. To tackle this problem, we propose a Diversity-driven Learn-to-rank model-based TCP approach, named DLTCP, for optimizing early fault detection performance. Our method first employs the whale optimization algorithm to search for a suitable set of similarity metrics from a pool of existing candidates. This search process determines which metrics should be used. According to each selected metric, test cases are then prioritized. The resulting test case rankings are used as the training data for DLTCP. Finally, the proposed method incorporates random forest to train a ranking model for test case prioritization. As such, it can fuse multiple similarity metrics to improve the TCP performance. We conduct extensive experiments to evaluate our method’s performance using the average percentage fault detected (APFD) as metric. The experimental results show that DLTCP achieve an average APFD value of 0.953 for seven classic benchmark models, which is 11.37% higher than that of the state-of-the-art algorithms. It can well select a set of similarity metrics for effective fusion, demonstrating competitive performance in early fault detection.

Retraction Note: Test case prioritization using a Hybrid Chaotic Flower-fruit fly optimization algorithm with multiple objectives

Retraction Note: Test case prioritization using a Hybrid Chaotic Flower-fruit fly optimization algorithm with multiple objectives

Security Development Lifecycle-Based Adaptive Reward Mechanism for Reinforcement Learning in Continuous Integration Testing Optimization

Continuous automated testing throughout each cycle can ensure the security of the continuous integration (CI) development lifecycle. Test case prioritization (TCP) is a critical factor in optimizing automated testing, which prioritizes potentially failed test cases and improves the efficiency of automated testing. In CI automated testing, the TCP is a continuous decision-making process that can be solved with reinforcement learning (RL). RL-based CITCP can continuously generate a TCP strategy for each CI development lifecycle, with the reward mechanism as the core. The reward mechanism consists of the reward function and the reward strategy. However, there are new challenges to RL-based CITCP in real-industry CI testing. With high-frequency iteration, the reward function is often calculated with a fixed length of historical information, ignoring the spatial characteristics of the current cycle. Therefore, the dynamic time window (DTW)-based reward function is proposed to perform the reward calculation, which adaptively adjusts the recent historical information range based on the integration cycle. Moreover, with low-failure testing, the reward strategy usually only rewards failure test cases, which creates a sparse reward problem in RL. To address this issue, the similarity-based reward strategy is proposed, which increases the reward objects of some passed test cases, similar to the failure test cases. The DTW-based reward function and the similarity-based reward strategy together constitute the proposed adaptive reward mechanism in RL-based CITCP. To validate the effectiveness of the adaptive reward mechanism, experimental verification is carried out on 13 industrial data sets. The experimental results show that the adaptive reward mechanism can improve the TCP effect, where the average NAPFD is maximally improved by 7.29%, the average Recall is maximally improved by 6.04% and the average TTF is improved by 6.81 positions with a maximum of 63.77.

Test case prioritization based on fault sensitivity analysis using ranked NSGA-2

This paper discusses regression testing in software maintenance, focusing on test case prioritization to verify modifications to software functionality efficiently. The primary goal is to rank test cases, prioritizing those covering more code or faults with minimal execution time. The challenge lies in prioritizing numerous test cases generated during development and maintenance. Various algorithms, including greedy approaches and meta-heuristic techniques, address this challenge. The paper introduces a ranking-based non-dominated sorting genetic algorithm (NSGA-2) for test case prioritization, emphasizing cases sensitive to faults caused by modifications or new functionality. Historical data is prioritized, with key objectives including the sensitive index, execution cost, and average percentage of fault detection (APFD). The proposed model is tested on handcrafted and benchmark Java-based applications, comparing its performance to state-of-the-art algorithms in test case prioritization.

Optimizing test case prioritization through ranked NSGA-2 for enhanced fault sensitivity analysis

Optimizing test case prioritization through ranked NSGA-2 for enhanced fault sensitivity analysis

Exploiting DBSCAN and Combination Strategy to Prioritize the Test Suite in Regression Testing

Test case prioritization techniques improve the fault detection rate by adjusting the execution sequence of test cases. For static black-box test case prioritization techniques, existing methods generally improve the fault detection rate by increasing the early diversity of execution sequences based on string distance differences. However, such methods have a high time overhead and are less stable. This paper proposes a novel test case prioritization method (DC-TCP) based on density-based spatial clustering of applications with noise (DBSCAN) and combination policies. By introducing a combination strategy to model the inputs to generate a mapping model, the test inputs are mapped to consistent types to improve generality. The DBSCAN method is then used to refine the classification of test cases further, and finally, the Firefly search strategy is introduced to improve the effectiveness of sequence merging. Extensive experimental results demonstrate that the proposed DC-TCP method outperforms other methods in terms of the average percentage of faults detected and exhibits advantages in terms of time efficiency when compared to several existing static black-box sorting methods.

Enhancing Automated Test Case Generation Through KNN algorithm in Software Testing

In the ever-evolving world of software development, assuring the stability and resilience of software systems remains a vital task. This study aims to tackle this problem by suggesting a novel method for generating automated test cases. It involves incorporating the robust KNN algorithm into the domain of data mining methods. The research intends to exploit previous testing data to detect patterns and trends, increasing the production of effective and targeted test scenarios.
 The literature study highlights the increasing overlap between data mining and software testing, underscoring the need for sophisticated methods to address the requirements of contemporary, ever-changing applications. In this regard, our study presents the KNN algorithm as a new addition to the collection of data mining tools for software testing. The process entails gathering extensive historical testing data, including information on test cases, errors, and the attributes of tested software applications. By using rigorous preprocessing and feature selection techniques, the KNN method is used to examine the correlations between various variables and the probability of flaws. The KNN model is trained and optimized using the dataset, and the resultant predictions are then used in an automated system for generating test cases. The experimental results confirm the efficacy of the suggested method in finding crucial test scenarios and prioritizing test cases according to past fault trends. The practical usefulness and advantages of incorporating the KNN algorithm into the automated test case-generating process are further confirmed by real-world case studies. In conclusion, this study expands the discourse on data mining in software testing by proposing the KNN algorithm as a vital component in automated test case development. By using KNN, testing teams can adjust to the ever-changing characteristics of contemporary applications, leading to enhanced efficiency and effectiveness in testing procedures. The results of this research enhance the continuous development of software quality assurance methods and provide opportunities for additional investigation at the convergence of machine learning and software testing

On the use of contextual information for machine learning based test case prioritization in continuous integration development

Context:In most software organizations, Continuous Integration (CI) is a common practice usually subject to some budgets. Consequently, prioritizing test cases to be executed in the CI cycle is fundamental. The idea is first to execute test cases with higher failure-proneness to provide rapid feedback and decrease costs. To perform this task approaches in the literature adopt failure history and Machine Learning (ML). However, in addition to the failure history, it is also important to consider information from the CI context of the organizations and the application domain. Objective:For this end, we introduce a contextual information approach for ML algorithms. Such an approach considers information from the testing activity that can be easily collected, such as test case execution time, size, and complexity. We implement the approach by introducing two contextual versions of the algorithms: Multi-Armed Bandit (MAB) and Random Forest (RF). Method:Six systems are used to compare both contextual algorithms and to evaluate their performance regarding their corresponding non-contextual versions, considering three different budgets. Results:Contextual algorithms perform better when indicators related to test time reduction are considered, as the contextual information they use is related to execution time. Regarding NAPFD and APFDc, the non-contextual algorithms have better general performance, but both contextual versions obtain competitive results. Conclusions:The contextual versions implemented can capture the desired context information in the prioritization without negatively impacting their performance regarding fault-detection.

Fault sensitivity index-based multi-objective testcase prioritization

Abstract Test case prioritization (TCP) is a regression technique that sequences test cases by assigning priority based on specific criteria defined by software testers. Various parameters, such as code coverage, statement coverage, and method coverage, are utilized in Test Case Prioritization (TCP), wherein metaheuristic techniques are widely employed to determine the optimal order of test cases based on these specified parameters. However, simply applying these techniques does not ensure the satisfaction of all the needs of software testers. This paper introduces an empirical study that employs the multi-objective test case prioritization (MOTCP) technique to prioritize the test cases based on target points defined by software testers. The study calculates a Software Complexity Index (SCI) at the code level, identifying fault-prone areas. Furthermore, a Test-case Complexity Index (TCI) is also used for prioritization. The proposed technique incorporates various target points defined by the software tester to calculate SCI and TCI, which serve as our main objectives for TCP. A detailed analysis is also performed to examine the impact of these target points on the generated optimal order of test cases. Finally, the proposed model is compared with other state-of-the-art techniques across various evaluation parameters.

Multi-factor based Regression Test Case Prioritization using Fuzzy Logic

The maintenance level activity generally done after the modification in the software to check whether it is functioning right or not is termed as regression testing. Test case prioritization, a key practice, involves strategically ordering test cases based on specific criteria to enhance the efficiency of fault detection within a condensed time frame. The fuzzy rule base serves as an alternative to the conventional crisp value set, offering a nuanced approach beyond binary outcomes (Yes or No). The primary objective of this research is to address critical factors often overlooked in existing literature on prioritization. Notably, prevalent approaches focus on singular factors during test case prioritization, highlighting the need for a comprehensive technique. To enhance the prioritization of test cases, there is a demand for a method that considers multi-factors or combinations thereof, ultimately increasing effectiveness. This paper introduces an innovative approach a multi-factors regression test-case prioritization technique utilizing fuzzy rules. The methodology aims to optimize the prioritization of test cases, striking a balance between effectiveness and time efficiency. Fuzzy rules are formulated to assess the effectiveness of a prioritized set of test cases in developing the proposed approach. A user-friendly tool has been developed to facilitate the application of this technique, allowing users to input relevant factors and subsequently prioritize test cases accordingly. Through extensive experiments using the developed tool, the effectiveness of the proposed approach has been validated. The results demonstrate that the priority lists of test cases generated for different projects, considering multi-factors, show greater promise compared to techniques relying solely on a single factor for prioritization.

An empirically based object-oriented testing using Machine learning

INTRODUCTION: The rapid growth of machine learning has the potential to revolutionize various industries and applications by automating complex tasks and enhancing efficiency. Effective software testing is crucial for ensuring software quality and minimizing resource expenses in software engineering. Machine learning techniques play a vital role in software testing by aiding in test case prioritization, predicting software defects, and analyzing test results. OBJECTIVES: The primary objective of this study is to explore the use of machine learning algorithms for software defect prediction. METHODS: Machine Learning models including Random Forest Classifier, Logistic Regression, K Nearest Neighbors, Gradient Boosting Classifiers, Catboost Classifier, and Convolutional Neural Networks have been employed for the study. The dataset includes a wide range of features relevant to software defect prediction and evaluates the performance of different prediction models. The study also focussed on developing hybrid models using stacking classifiers, which combine multiple individual models to improve accuracy. RESULTS: The experimental results show that the hybrid models combining CatBoost and Convolutional Neural Network have outperformed individual models, achieving the highest accuracy of 89.5%, highlighting the effectiveness of combining machine learning algorithms for software defect prediction. CONCLUSION: In conclusion, this study sheds light on the pivotal role of machine learning in enhancing software defect prediction.

An Improved Deep Learning Based Test Case Prioritization Using Deep Reinforcement Learning

An Improved Deep Learning Based Test Case Prioritization Using Deep Reinforcement Learning

An exploratory study of history-based test case prioritization techniques on different datasets

In regression testing, Test case prioritization (TCP) is a technique to arrange all the available test cases. TCP techniques can improve fault detection performance which is measured by the average percentage of fault detection (APFD). History-based TCP is one of the TCP techniques that consider the history of past data to prioritize test cases. The issue of equal priority allocation to test cases is a common problem for most TCP techniques. However, this problem has not been explored in history-based TCP techniques. To solve this problem in regression testing, most of the researchers resort to random sorting of test cases. This study aims to investigate equal priority in history-based TCP techniques. The first objective is to implement different history-based TCP techniques. The second objective is to explore the problem of equal priority in history-based TCP techniques. The third objective is to explore random sorting as a solution to the problem of equal priority in history-based TCP techniques. Datasets of historical records of test cases from conventional and modern sources were collected. History-based TCP techniques were applied to different datasets. The History-based TCP techniques were checked for the problem of equal priority. Then random sorting was used as a solution to the problem of equal priority. Finally, the results were elaborated in terms of APFD and execution time. The results indicate that history-based techniques also suffer from the problem of equal priority like other types of TCP techniques. Secondly, random sorting does not produce optimal results while trying to solve the problem of equal priority in history-based TCP. Furthermore, random sorting deteriorates the results of history-based TCP techniques when employed to solve the problem of equal priority. One should resort to random sorting if no other solution exists. The decision to choose the best solution requires a cost-benefit analysis keeping in view the context and solution under consideration.

A Hybrid Meta-Heuristic Approach for Test Case Prioritization and Optimization

The application of the test case prioritization method is a key part of system testing intended to think it through and sort out the issues early in the development stage. Traditional prioritization techniques frequently fail to take into account the complexities of big-scale test suites, growing systems and time constraints, therefore cannot fully fix this problem. The proposed study here will deal with a meta-heuristic hybrid method that focuses on addressing the challenges of the modern time. The strategy utilizes genetic algorithms alongside a black hole as a means to create a smooth tradeoff between exploring numerous possibilities and exploiting the best one. The proposed hybrid algorithm of genetic black hole (HGBH) uses the capabilities of considering the imperatives such as code coverage, fault finding rate and execution time from search algorithms in our hybrid approach to refine test cases considerations repetitively. The strategy accomplished this by putting experiments on a large-scale project of industrial software developed. The hybrid meta-heuristic technique ends up being better than the routine techniques. It helps in higher code coverage, which, in turn, enables to detect crucial defects at an early stage and also to allocate the testing resources in a better way. In particular, the best APFD value was 0.9321, which was achieved in 6 generations with 4.879 seconds the value to which the computer was run. Besides these, , the approach resulted in the mean value of APFD as 0.9247 and 0.9302 seconds which took from 10.509 seconds to 30.372 seconds. The carried out experiment proves the feasibility of this approach in implementing complex systems and consistently detecting the changes, enabling it to adapt to rapidly changing systems. In the end, this research provides us with a new hybrid meta-heuristic way of test case prioritization and optimization, which, in turn, helps to tackle the obstacles caused by large-scale test cases and constantly changing systems.

Deep learning-powered test case prioritisation in continuous integration: a comparative study and efficiency analysis

Deep learning-powered test case prioritisation in continuous integration: a comparative study and efficiency analysis