Test Case Prioritization Approach Research Articles

A Hybrid Meta-Heuristic Approach for Test Case Prioritization and Optimization

The application of the test case prioritization method is a key part of system testing intended to think it through and sort out the issues early in the development stage. Traditional prioritization techniques frequently fail to take into account the complexities of big-scale test suites, growing systems and time constraints, therefore cannot fully fix this problem. The proposed study here will deal with a meta-heuristic hybrid method that focuses on addressing the challenges of the modern time. The strategy utilizes genetic algorithms alongside a black hole as a means to create a smooth tradeoff between exploring numerous possibilities and exploiting the best one. The proposed hybrid algorithm of genetic black hole (HGBH) uses the capabilities of considering the imperatives such as code coverage, fault finding rate and execution time from search algorithms in our hybrid approach to refine test cases considerations repetitively. The strategy accomplished this by putting experiments on a large-scale project of industrial software developed. The hybrid meta-heuristic technique ends up being better than the routine techniques. It helps in higher code coverage, which, in turn, enables to detect crucial defects at an early stage and also to allocate the testing resources in a better way. In particular, the best APFD value was 0.9321, which was achieved in 6 generations with 4.879 seconds the value to which the computer was run. Besides these, , the approach resulted in the mean value of APFD as 0.9247 and 0.9302 seconds which took from 10.509 seconds to 30.372 seconds. The carried out experiment proves the feasibility of this approach in implementing complex systems and consistently detecting the changes, enabling it to adapt to rapidly changing systems. In the end, this research provides us with a new hybrid meta-heuristic way of test case prioritization and optimization, which, in turn, helps to tackle the obstacles caused by large-scale test cases and constantly changing systems.

Requirement based Test Case Prioritization for System Testing

System Testing encompasses a large number of test cases, which may not be able to get executed due to constrained time, budget and limitation of the resources. Therefore, the test cases must be prioritized in some order such that the critical and most required functionality can be tested early. In this paper, a hierarchical approach for system test case prioritization based on requirements has been proposed that maps requirements on the system test cases. This approach analyzes and assigns value to each requirement based on a comprehensive set of twelve factors thereby prioritizing the requirements. Further, the prioritized requirement is mapped on the highly relevant module and then prioritized set of test cases. To analyze the effectiveness of this approach, a case study of income tax calculator software [1] has been taken. The existing as well as the proposed approach were applied and analyzed on this software. The results show the efficacy of the proposed approach in terms of fault detection and severity early.

Code Similarity and Location-Awareness Automatic Program Repair

Automatic program repair has drawn more and more attention since software quality is facing increasing challenges. In existing approaches, the unlimited search space is considered to be the main limitation in finding the correct patch. So how to reduce the search space to improve the efficiency of automatic program repair remains a problem to be solved. In this work, we represent a similarity-based and location-awareness-based automatic program repair (SLARepair). SLARepair takes the similarity between codes as important search information. The search space is further subdivided by the location-awareness strategy to improve search efficiency. In addition, to better guide the search process, a new fitness function is designed for genetic programming, which brings notable improvements. Moreover, the patch verification time is further reduced by utilizing the test case prioritization approach combined with test case filtering. Extensive experiments demonstrate that our SLARepair outperforms the state-of-the-art approaches on the Defects4J benchmark and achieves competitive performances.

A novel test case prioritization approach for black‐box testing based on K‐medoids clustering

AbstractRegression testing is an essential and expensive process in software testing. However, there may be insufficient resources for the execution of all test cases during regression testing. Test case prioritization (TCP) techniques improve the efficiency of regression testing by adjusting the test case execution sequence. Traditional TCP techniques usually rely on the historical execution information of the software under test for more efficient results. String distance‐based TCP (SD‐TCP) avoids these limitations; it uses only the textual difference information of the test cases themselves for prioritization. However, the time overhead on the sorting process of this method is not ideal, and the extreme test case inputs have an impact on the stability of the method. To address these problems, we propose a novel test case prioritization strategy, it first classifies the test cases more finely using the K‐medoids algorithm and then transforms the set into subsequences and improves the early diversity by greedy sorting within clusters. Finally, the test cases are selected through a polling strategy to compose the execution sequence. Extensive experimental results demonstrate that the proposed approach outperforms SD‐TCP in better time efficiency on test case prioritization; it also has a higher average percentage of fault detected (APFD) value than random prioritization (RP) and SD‐TCP.

An Evaluation of Ranking-to-Learn Approaches for Test Case Prioritization in Continuous Integration

Continuous Integration (CI) environments is a practice adopted by most organizations that allows frequent integration of software changes, making software evolution more rapid and cost-effective. Such environments require dynamic Test Case Prioritization (TCP) approaches that adapt better to the test budgets and frequent addition/removal of test cases. In this sense, Ranking-to-Learn approaches have been proposed and are more suitable for CI constraints. By observing past prioritizations and guided by reward functions, they learn the best prioritization for a given commit. In order to contribute for improvements and direct future research, this work evaluates how far the solutions produced by these approaches are from optimal solutions produced by a deterministic approach (ground truth). To this end, we consider two learning-based approaches i) RETECS, which is based on Reinforcement Learning; and ii) COLEMAN, an approach based on Multi-Armed Bandit. The evaluation was conducted with twelve systems, three test budgets, two reward functions, and six measures concerning fault detection effectiveness, early fault detection, test time reduction in the CI cycles, prioritization time, and accuracy. Our findings have some implications for the approaches application and reward function choice. The approaches are applicable in real scenarios and produce solutions very close to the optimal ones, respectively, in 92% and 75% of the cases. Both approaches have some limitations to learn with few historical test data (a small number of CI Cycles) and deal with a large test case set, in which many failures are distributed over many test cases.

Test case prioritization using test case diversification and fault-proneness estimations

Regression testing activities greatly reduce the risk of faulty software release. However, the size of the test suites grows throughout the development process, resulting in time-consuming execution of the test suite and delayed feedback to the software development team. This has urged the need for approaches such as test case prioritization (TCP) and test-suite reduction to reach better results in case of limited resources. In this regard, proposing approaches that use auxiliary sources of data such as bug history can be interesting. We aim to propose an approach for TCP that takes into account test case coverage data, bug history, and test case diversification. To evaluate this approach we study its performance on real-world open-source projects. The bug history is used to estimate the fault-proneness of source code areas. The diversification of test cases is preserved by incorporating fault-proneness on a clustering-based approach scheme. The proposed methods are evaluated on datasets collected from the development history of five real-world projects including 357 versions in total. The experiments show that the proposed methods are superior to coverage-based TCP methods. The proposed approach shows that improvement of coverage-based and fault-proneness-based methods is possible by using a combination of diversification and fault-proneness incorporation.

Multiobjective Test Case Prioritization Using Test Case Effectiveness: Multicriteria Scoring Method

Modified source code validation is done by regression testing. In regression testing, the time and resources are limited, in which we have to select the minimal test cases from test suites to reduce execution time. The test case minimization process deals with the optimization of the regression testing by removing redundant test cases or prioritizing the test cases. This study proposed a test case prioritization approach based on multiobjective particle swarm optimization (MOPSO) by considering minimum execution time, maximum fault detection ability, and maximum code coverage. The MOPSO algorithm is used for the prioritization of test cases with parameters including execution time, fault detection ability, and code coverage. Three datasets are selected to evaluate the proposed MOPSO technique including TreeDataStructure, JodaTime, and Triangle. The proposed MOPSO is compared with the no ordering, reverse ordering, and random ordering technique for evaluating the effectiveness. The higher values of results represent the more effectiveness and the efficiency of the proposed MOPSO as compared to other approaches for TreeDataStructure, JodaTime, and Triangle datasets. The result is presented to 100-index mode relevant from low to high values; after that, test cases are prioritized. The experiment is conducted on three open-source java applications and evaluated using metrics inclusiveness, precision, and size reduction of a matrix of the test suite. The results revealed that all scenarios performed well in acceptable mode, and the technique is 17% to 86% more effective in terms of inclusiveness, 33% to 85% more effective in terms of precision, and 17% minimum to 86% maximum in size reduction of metrics.

A Dissimilarity with Dice-Jaro-Winkler Test Case Prioritization Approach for Model-Based Testing in Software Product Line

A Dissimilarity with Dice-Jaro-Winkler Test Case Prioritization Approach for Model-Based Testing in Software Product Line

Test Case Prioritization Approach for Sequence of Events Using Complexity Factor

Test case prioritization (TCP) is a method to prioritize and schedule test cases. Some approaches have been introduced to minimize the time, cost, and effort for testing the software based on the test cases that are higher priority. Since the more complex the software program, the more intensive the test should be carried out. Thus, complexity is one of the factors that affect the effectiveness of the test case prioritization. However, the existing approaches for measuring complexity have some limitations. This is due to inaccuracy in finding the weightage value for complexity as the value is useful to determine the test case prioritization. Consequently, a complexity metric measurement is needed to determine the weightage value. Hence, this paper presents work on TCP using complexity factors to enhance the accuracy in prioritizing the test cases for event sequences. This work uses Branch Coverage Expectation (BCE) for complexity measurement, in which BCE has been proven its usefulness empirically in the previous research. The event-weightage value based on the complexity is then assigned and used to prioritize the test cases while the Average Percentage of Fault Detected (APFD) metric is used to evaluate the proposed approach. A tool has been developed to ease the process as well as to facilitate the evaluation purposes. The results show the need to combine the complexity factor with other factors to improve the proposed TCP's effectiveness.

Test Case Prioritization Approach for Sequence of Events Using Complexity Factor

Test case prioritization (TCP) is a method to prioritize and schedule test cases. Some approaches have been introduced to minimize the time, cost, and effort for testing the software based on the test cases that are higher priority. Since the more complex the software program, the more intensive the test should be carried out. Thus, complexity is one of the factors that affect the effectiveness of the test case prioritization. However, the existing approaches for measuring complexity have some limitations. This is due to inaccuracy in finding the weightage value for complexity as the value is useful to determine the test case prioritization. Consequently, a complexity metric measurement is needed to determine the weightage value. Hence, this paper presents work on TCP using complexity factors to enhance the accuracy in prioritizing the test cases for event sequences. This work uses Branch Coverage Expectation (BCE) for complexity measurement, in which BCE has been proven its usefulness empirically in the previous research. The event-weightage value based on the complexity is then assigned and used to prioritize the test cases while the Average Percentage of Fault Detected (APFD) metric is used to evaluate the proposed approach. A tool has been developed to ease the process as well as to facilitate the evaluation purposes. The results show the need to combine the complexity factor with other factors to improve the proposed TCP's effectiveness.

An Ontology Based Test Case Prioritization Approach in Regression Testing

Regression testing is a widely studied research area, with the aim of meeting the quality challenges of software systems. To achieve a software system of good quality, we face high consumption of resources during testing. To overcome this challenge, test case prioritization (TCP) as a sub-type of regression testing is continuously investigated to achieve the testing objectives. This study provides an insight into proposing the ontology-based TCP (OTCP) approach, aimed at reducing the consumption of resources for the quality improvement and maintenance of software systems. The proposed approach uses software metrics to examine the behavior of classes of software systems. It uses Binary Logistic Regression (BLR) and AdaBoostM1 classifiers to verify correct predictions of the faulty and non-faulty classes of software systems. Reference ontology is used to match the code metrics and class attributes. We investigated five Java programs for the evaluation of the proposed approach, which was used to achieve code metrics. This study has resulted in an average percentage of fault detected (APFD) value of 94.80%, which is higher when compared to other TCP approaches. In future works, large sized programs in different languages can be used to evaluate the scalability of the proposed OTCP approach.

Trend Application of Machine Learning in Test Case Prioritization: A Review on Techniques

Software quality can be assured by passing the process of software testing. However, software testing process involve many phases which lead to more resources and time consumption. To reduce these downsides, one of the approaches is to adopt test case prioritization (TCP) where numerous works has indicated that TCP do improve the overall software testing performance. TCP does have several kinds of techniques which have their own strengths and weaknesses. As for this review paper, the main objective of this paper is to examine deeper on machine learning (ML) techniques based on research questions created. The research method for this paper was designed in parallel with the research questions. Consequently, 110 primary studies were selected where, 58 were journal articles, 50 were conference papers and 2 considered as others articles. For overall result, it can be said that ML techniques in TCP has trending in recent years yet some improvements are certainly welcomed. There are multiple ML techniques available, in which each technique has specified potential values, advantages, and limitation. It is notable that ML techniques has been considerably discussed in TCP approach for software testing.

Nowatorska metoda priorytetyzacji przypadków testowych oparta na prognozowaniu błędów instrukcji kodu oprogramowania numerycznego

Test case prioritization (TCP) has been considerably utilized to arrange the implementation order of test cases, which contributes to improve the efficiency and resource allocation of software regression testing. Traditional coverage-based TCP techniques, such as statement-level, method/function-level and class-level, only leverages program code coverage to prioritize test cases without considering the probable distribution of defects. However, software defect data tends to be imbalanced following Pareto principle. Instinctively, the more vulnerable the code covered by the test case is, the higher the priority it is. Besides, statement-level coverage is a more fine-grained method than function-level coverage or class-level coverage, which can more accurately formulate test strategies. Therefore, we present a test case prioritization approach based on statement software defect prediction to tame the limitations of current coverage-based techniques in this paper. Statement metrics in the source code are extracted and data pre-processing is implemented to train the defect predictor. And then the defect detection rate of test cases is calculated by combining the prioritization strategy and prediction results. Finally, the prioritization performance is evaluated in terms of average percentage faults detected in four open source datasets. We comprehensively compare the performance of the proposed method under different prioritization strategies and predictors. The experimental results show it is a promising technique to improve the prevailing coverage-based TCP methods by incorporating statement-level defect-proneness. Moreover, it is also concluded that the performance of the additional strategy is better than that of max and total, and the choice of the defect predictor affects the efficiency of the strategy.

LSTM-based deep learning for spatial–temporal software testing

Continuous integration (CI) software development practice has become more and more popular. Regression testing occurs very frequently in CI. Test case suites constantly change since new test cases are inserted and obsolete test case are removed in each cycle. The software developer hunts for quick-feedback of faults because of time constraint. An embedded software usually includes the spatial–temporal data in CI. The efficiency of regression testing for the embedded software is related to the space–time. To achieve ideal regression testing goals for the embedded software in CI, this paper proposes a novel test case prioritization approach using LSTM-Based (Long short-term memory) deep learning. LSTM is a time series prediction model. It can predict the probability of each test case detection fault in the next cycle according to the testing history information of all the previous CI cycles. The priority of test case can be obtained dynamically under the guidance of the probability. The experiments are conducted on two industrial data sets. The results verify that compared with some exiting test case prioritization approaches, our approach has better performance for embedded software as follows: (1) improve the prioritization effectiveness, (2) increase the fault detection rate in CI environment, and (3) decrease the testing execution time through automatic reduction the obsolete test cases.

A Multi-Armed Bandit Approach for Test Case Prioritization in Continuous Integration Environments

Continuous Integration (CI) environments have been increasingly adopted in the industry to allow frequent integration of software changes, making software evolution faster and cost-effective. In such environments, Test Case Prioritization (TCP) techniques play an important role to reduce regression testing costs, establishing a test case execution order that usually maximizes early fault detection. Existing works on TCP in CI environments (TCPCI) present some limitations. Few pieces of work consider CI particularities, such as the test case volatility, that is, they do not consider the dynamic environment of the software life-cycle in which new test cases can be added or removed (discontinued), characteristic related to the Exploration versus Exploitation (EvE) dilemma. To solve such a dilemma an approach needs to balance: i) the diversity of test suite; and ii) the quantity of new test cases and test cases that are error-prone or that comprise high fault-detection capabilities. To deal with this, most approaches use, besides the failure-history, other measures that rely on code instrumentation or require additional information, such as testing coverage. However, to maintain the information updated can be difficult and time-consuming, not scalable due to the test budget of CI environments. In this context, and to properly deal with the TCPCI problem, this work presents an approach based on Multi-Armed Bandit (MAB) called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">COLEMAN</monospace> ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>C</b>ombinatorial V<b>O</b>lati<b>LE</b> <b>M</b>ulti-Armed B<b>AN</b>dit</i> ). The TCPCI problem falls into the category of volatile and combinatorial MAB, because multiple arms (test cases) need to be selected, and they are added or removed over the cycles. We conducted an evaluation considering three time budgets and eleven systems. The results show the applicability of our approach and that <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">COLEMAN</monospace> outperforms the most similar approach from literature in terms of early fault detection and performance.

Risk-Based Test Case Prioritization by Correlating System Methods and Their Associated Risks

Regression testing aims to ensure the quality of a software after modification. However, re-executing the entire test suite in regression testing is a time-consuming, costly, and tedious process that often requires additional budget and time. Thus, limited resources always result in early termination and poor quality software. Test case prioritization aims to improve regression testing by re-scheduling the test cases in a manner that could increase the fault detection rate. Risk-based testing has gained popularity in the area of software testing. However, most of the existing methods compute risk values manually, which makes these methods tiresome, laborious, and slow. In this paper, we propose a semi-automatic risk-based test case prioritization approach based on software modification information and methods (functions) invocation relationship. The objective of this research is to make risk-based testing more systematic and flexible by automating the risk assessment process, and find high-risk faults early. The proposed approach utilizes the requirements modification information, complexity, and size of the methods as the risk indicating factors. We applied an automated procedure to extract these risk factors and compute the risk values of the system methods. The proposed approach is empirically evaluated with two software applications with multiple versions in terms of its fault detection rate, both overall and in risky modules. The experimental results suggest that our proposed approach improves test efficiency by spotting defects early overall and even earlier in the high-risk modules than existing state-of-the-art approaches.

Regression Test Case Prioritization Based on Fixed Size Candidate Set ART Algorithm

Regression testing is a very time-consuming and expensive testing activity. Many test case prioritization techniques have been proposed to speed up regression testing. Previous studies show that no one technique is always best. Random strategy, as the simplest strategy, is not always so bad. Particularly, when a test suite has higher fault detection capability, the strategy can generate a better result. Nevertheless, due to the randomness, the strategy is not always as satisfactory as expected. In this context, we present a test case prioritization approach using fixed size candidate set adaptive random testing algorithm to reduce the effect of randomness and improve fault detection effectiveness. The distance between pair-wise test cases is assessed by exclusive OR. We designed and conducted empirical studies on eight C programs to validate the effectiveness of the proposed approach. The experimental results, confirmed by a statistical analysis, indicate that the approach we proposed is more effective than random and the total greedy prioritization techniques in terms of fault detection effectiveness. Although the presented approach has comparable fault detection effectiveness to ART-based and the additional greedy techniques, the time cost is much lower. Consequently, the proposed approach is much more cost-effective.

A Model-Based Test Case Prioritization Approach Based on Fault Urgency and Severity

With the aggrandizement scale of software system, the number of test cases has grown explosively. Test case prioritization (TCP) has been widely used in software testing to effectively improve testing efficiency. However, traditional TCP methods are mostly based on software code and they are difficult to apply to model-based testing. Moreover, existing model-based TCP techniques often do not take the likely distribution of faults into consideration, yet software faults are not often equally distributed in the system, and test cases that cover more fault prone modules are more likely to reveal faults so that they should be run with a higher priority. Therefore, in this paper, we provide a TCP approach based on Hidden Markov Model (HMM), to detect faults as earlier as possible and reduce the cost of modification. This approach consists of the following main parts: (1) transforming the Unified Modeling Language (UML) sequence diagram to HMM; (2) estimating the fault urgency according to fault priority and probability; (3) estimating the fault severity by analyzing the weight of the state in the HMM; (4) generating test case priority from fault urgency and fault severity, then prioritizing test case. The proposed approach is implemented on unmanned aerial vehicles (UAV) flight control system to perform TCP. The experimental results show that our proposed TCP approach can effectively enhance the probability of earlier fault detection and improve the efficiency and stability as compared to other prioritization techniques, such as original prioritization, random prioritization, additional prioritization and EPS-UML.

Evolutionary Cost-cognizant Test Case Selection and Prioritization for Object-Oriented Programs

Test case prioritization (TCP) is a software testing technique that finds an ideal ordering of test cases for regression testing, so that testers can obtain the maximum benefit of their test suite, even if the testing process is stop at some arbitrary point. The recent trend of software development uses OO paradigm. This paper proposed a cost-cognizant TCP approach for object-oriented software that uses path-based integration testing. Path-based integration testing will identify the possible execution path and extract these paths from the Java System Dependence Graph (JSDG) model of the source code using forward slicing technique. Afterward evolutionary algorithm (EA) was employed to prioritize test cases based on the severity detection per unit cost for each of the dependent faults. The proposed technique was known as Evolutionary Cost-Cognizant Regression Test Case Prioritization (ECRTP) and being implemented as regression testing approach for experiment.

A review of object-oriented approach for test case prioritization

&lt;p&gt;The most essential phase in regression testing is Test Case Prioritization (TCP), with its primary objective to increase the fault detection rate at different stages during testing. Prior to achieving the objective, existing evidence of techniques in TCP must be synthesized and analyzed. At present, fault detection for TCP based on object-oriented features only consider statement, module, and class level. The important features of object-oriented (OO) programming like inheritance and polymorphism have not been fully explored for fault detection in TCP. Such OO concepts are important for test case selection and in turn for ranking the test cases (prioritization). This paper reviews various test case prioritization techniques specific to OO systems. This review is hoped to highlight the importance and usage of TCP in relation to object-oriented software development lifecycle.&lt;/p&gt;