Average Percentage Of Faults Detected Research Articles

Test case prioritization based on fault sensitivity analysis using ranked NSGA-2

This paper discusses regression testing in software maintenance, focusing on test case prioritization to verify modifications to software functionality efficiently. The primary goal is to rank test cases, prioritizing those covering more code or faults with minimal execution time. The challenge lies in prioritizing numerous test cases generated during development and maintenance. Various algorithms, including greedy approaches and meta-heuristic techniques, address this challenge. The paper introduces a ranking-based non-dominated sorting genetic algorithm (NSGA-2) for test case prioritization, emphasizing cases sensitive to faults caused by modifications or new functionality. Historical data is prioritized, with key objectives including the sensitive index, execution cost, and average percentage of fault detection (APFD). The proposed model is tested on handcrafted and benchmark Java-based applications, comparing its performance to state-of-the-art algorithms in test case prioritization.

An exploratory study of history-based test case prioritization techniques on different datasets

In regression testing, Test case prioritization (TCP) is a technique to arrange all the available test cases. TCP techniques can improve fault detection performance which is measured by the average percentage of fault detection (APFD). History-based TCP is one of the TCP techniques that consider the history of past data to prioritize test cases. The issue of equal priority allocation to test cases is a common problem for most TCP techniques. However, this problem has not been explored in history-based TCP techniques. To solve this problem in regression testing, most of the researchers resort to random sorting of test cases. This study aims to investigate equal priority in history-based TCP techniques. The first objective is to implement different history-based TCP techniques. The second objective is to explore the problem of equal priority in history-based TCP techniques. The third objective is to explore random sorting as a solution to the problem of equal priority in history-based TCP techniques. Datasets of historical records of test cases from conventional and modern sources were collected. History-based TCP techniques were applied to different datasets. The History-based TCP techniques were checked for the problem of equal priority. Then random sorting was used as a solution to the problem of equal priority. Finally, the results were elaborated in terms of APFD and execution time. The results indicate that history-based techniques also suffer from the problem of equal priority like other types of TCP techniques. Secondly, random sorting does not produce optimal results while trying to solve the problem of equal priority in history-based TCP. Furthermore, random sorting deteriorates the results of history-based TCP techniques when employed to solve the problem of equal priority. One should resort to random sorting if no other solution exists. The decision to choose the best solution requires a cost-benefit analysis keeping in view the context and solution under consideration.

Optimizing Prioritization of Crowdsourced Test Reports of Web Applications through Image-to-Text Conversion

Collating vast test reports is a time-consuming and laborious task in crowdsourced testing. Crowdsourced test reports are usually presented in two ways, one as text and the other as images, which have symmetrical content. Researchers have proposed many text- and image-based methods for prioritizing crowdsourced test reports of mobile applications. However, crowdsourced test reports of web applications typically have clearer textual descriptions of errors but noisier screenshots of errors. This gap motivates us to propose a method for prioritizing crowdsourced test reports of web applications to detect all errors earlier. In this paper, we integrate text and image information from test reports to enhance the analysis process. First, we use the natural language processing (NLP) technique to extract textual features of error descriptions and then symmetrically extract image features of error screenshots, i.e., we use the optical character recognition (OCR) technique to obtain textual information in the screenshots and then also use the NLP technique to extract features. To validate our approach, we conduct experiments on 717 test reports. The experimental results show that our method has a higher APFD (average percentage fault detection) and shorter runtime than state-of-the-art prioritization methods.

Multi-Objective Fault-Coverage Based Regression Test Selection and Prioritization Using Enhanced ACO_TCSP

Regression testing of the software during its maintenance phase, requires test case prioritization and selection due to the dearth of the allotted time. The resources and the time in this phase are very limited, thus testers tend to use regression testing methods such as test case prioritization and selection. The current study evaluates the effectiveness of testing with two major goals: (1) Least running time and (2) Maximum fault coverage possible. Ant Colony Optimization (ACO) is a well-known soft computing technique that draws its inspiration from nature and has been widely researched, implemented, analyzed, and validated for regression test prioritization and selection. Many versions of ACO approaches have been prolifically applied to find solutions to many non-polynomial time-solvable problems. Hence, an attempt has been made to enhance the performance of the existing ACO_TCSP algorithm without affecting its time complexity. There have been efforts to enhance the exploration space of various paths in each iteration and with elite exploitation, reducing the total number of iterations required to converge to an optimal path. Counterbalancing enhanced exploration with intelligent exploitation implies that the run time is not adversely affected, the same has also been empirically validated. The enhanced algorithm has been compared with the existing ACO algorithm and with the traditional approaches. The approach has also been validated on four benchmark programs to empirically evaluate the proposed Enhanced ACO_TCSP algorithm. The experiment revealed the increased cost-effectiveness and correctness of the algorithm. The same has also been validated using the statistical test (independent t-test). The results obtained by evaluating the proposed approach against other reference techniques using Average Percentage of Faults Detected (APFD) metrics indicate a near-optimal solution. The multiple objectives of the highest fault coverage and least running time were fruitfully attained using the Enhanced ACO_TCSP approach without compromising the complexity of the algorithm.

Mutation-Based Minimal Test Suite Generation for Boolean Expressions

Boolean expressions are highly involved in control flows of programs and software specifications. Coverage criteria for Boolean expressions aim at producing minimal test suites to detect software faults. There exist various testing criteria, efficiency of which is usually evaluated through mutation analysis. This paper proposes an integer programming-based minimal test suite generation technique relying on mutation analysis. The proposed technique also takes into account the cost of fault detection. The technique is optimal such that the resulting test suite guarantees to detect all the mutants under given fault assumptions, while maximizing the average percentage of fault detection of a test suite. Therefore, the approach presented can also be considered as a reference method to check the efficiency of any common technique. The method is evaluated using four well-known real benchmark sets of Boolean expressions and is also exemplary compared with MCDC criterion. The results show that the test suites generated by the proposed method provide better fault coverage values and faster fault detection.

A review paper: optimal test cases for regression testing using artificial intelligent techniques

<span lang="EN-US">The goal of the testing process is to find errors and defects in the software being developed so that they can be fixed and corrected before they are delivered to the customer. Regression testing is an essential quality testing technique during the maintenance phase of the program as it is performed to ensure the integrity of the program after modifications have been made. With the development of the software, the test suite becomes too large to be fully implemented within the given test cost in terms of budget and time. Therefore, the cost of regression testing using different techniques should be reduced, here we dealt many methods such as retest all technique, regression test selection technique (RTS) and test case prioritization technique (TCP). The efficiency of these techniques is evaluated through the use of many metrics such as average percentage of fault detected (APFD), average percentage block coverage (APBC) and average percentage decision coverage (APDC). In this paper we dealt with these different techniques used in test case selection and test case prioritization and the metrics used to evaluate their efficiency by using different techniques of artificial intelligent and describe the best of all.</span>

A novel test case prioritization approach for black‐box testing based on K‐medoids clustering

AbstractRegression testing is an essential and expensive process in software testing. However, there may be insufficient resources for the execution of all test cases during regression testing. Test case prioritization (TCP) techniques improve the efficiency of regression testing by adjusting the test case execution sequence. Traditional TCP techniques usually rely on the historical execution information of the software under test for more efficient results. String distance‐based TCP (SD‐TCP) avoids these limitations; it uses only the textual difference information of the test cases themselves for prioritization. However, the time overhead on the sorting process of this method is not ideal, and the extreme test case inputs have an impact on the stability of the method. To address these problems, we propose a novel test case prioritization strategy, it first classifies the test cases more finely using the K‐medoids algorithm and then transforms the set into subsequences and improves the early diversity by greedy sorting within clusters. Finally, the test cases are selected through a polling strategy to compose the execution sequence. Extensive experimental results demonstrate that the proposed approach outperforms SD‐TCP in better time efficiency on test case prioritization; it also has a higher average percentage of fault detected (APFD) value than random prioritization (RP) and SD‐TCP.

A Comparative Analysis of Bat and Genetic Algorithms for Test Case Prioritization in Regression Testing

Regression testing is carried out to ensure that software modifications do not introduce new potential bugs to the existing software. Existing test cases are applied in the testing, such test cases can run into thousands, and there is not much time to execute all of them. Test Case Prioritization (TCP) is a technique to order test cases so that the test cases potentially revealing more faults are performed first. With TCP being deemed an optimization problem, several metaheuristic nature-inspired algorithms such as Bat, Genetic, Ant colony, and Firefly algorithms have been proposed for TCP. These algorithms have been compared theoretically or based on a single metric. This study employed an experimental design to offer an in-depth comparison of bat and genetic algorithms for TCP. Unprioritized test cases and a brute-force approach were used for comparison. Average Percentage Fault Detection (APFD)- a popular metric, execution time and memory usage were used to evaluate the algorithms’ performance. The study underscored the importance of test case prioritization and established the superiority of the Genetic algorithm over the bat algorithm for TCP in APFD. No stark differences were recorded regarding memory usage and execution time for the two algorithms. Both algorithms seemed to scale well with the growth of test cases.

Analisis Efektifitas Algoritma FAST Menggunakan Metrik Average Percentage Fault Detection dan Waktu Eksekusi Pada Test Case Prioritization

It uses regression testing to validate the modification results so as not to cause new errors in features that are already functioning correctly. The implementation of regression testing is generally done by re-executing the test suite used in the previous test. One of the crucial issues in regression testing is determining a strategic approach to reuse existing test suites. Testing with retesting all test cases will result in a long and expensive test. Thus, it is necessary to use a test case selection approach. The research study focuses on analyzing the effectiveness of the FAST algorithm, namely the FAST-pw and FAST-all algorithms, for executing test cases based on priority. The research method uses experiments through running algorithms on test cases that have been implanted with faults in the software with a test case size of 10,000 to 20,000 lines of the program. The effectiveness parameter uses the test case execution time and the percentage of fault detection using the average percentage fault detection (APFD) metric. The results showed that the FAST algorithm, namely the FAST-pw and FAST-all algorithms, had good effectiveness values ​​when applied to TCP testing with small to medium-sized SUTs, namely test cases that tested 1,000 – 20,000 program lines.

Value-Based Test Case Prioritization for Regression Testing Using Genetic Algorithms

Test Case Prioritization (TCP) techniques perform better than other regression test optimization techniques including Test Suite Reduction (TSR) and Test Case Selection (TCS). Many TCP techniques are available, and their performance is usually measured through a metric Average Percentage of Fault Detection (APFD). This metric is value-neutral because it only works well when all test cases have the same cost, and all faults have the same severity. Using APFD for performance evaluation of test case orders where test cases cost or faults severity varies is prone to produce false results. Therefore, using the right metric for performance evaluation of TCP techniques is very important to get reliable and correct results. In this paper, two value-based TCP techniques have been introduced using Genetic Algorithm (GA) including Value-Cognizant Fault Detection-Based TCP (VCFDB-TCP) and Value-Cognizant Requirements Coverage-Based TCP (VCRCB-TCP). Two novel value-based performance evaluation metrics are also introduced for value-based TCP including Average Percentage of Fault Detection per value (APFDv) and Average Percentage of Requirements Coverage per value (APRCv). Two case studies are performed to validate proposed techniques and performance evaluation metrics. The proposed GA-based techniques outperformed the existing state-of-the-art TCP techniques including Original Order (OO), Reverse Order (REV-O), Random Order (RO), and Greedy algorithm.

Application of the Law of Minimum and Dissimilarity Analysis to Regression Test Case Prioritization

Regression testing is one of the most expensive processes in testing. Prioritizing test cases in regression testing is critical for the goal of detecting the faults sooner within a large set of test cases. We propose a test case prioritization (TCP) technique for regression testing called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LoM-Score</i> inspired by the Law of Minimum (LoM) from biology. This technique calculates the impact probabilities of methods calculated by change impact analysis with forward slicing and orders test cases according to LoM. However, this ordering doesn’t consider the possibility that consecutive test cases may be covering the same methods repeatedly. Thereby, such ordering can delay the time of revealing faults that exist in other methods. To solve this problem, we enhance the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LoM-Score</i> TCP technique with an adaptive approach, namely with a dissimilarity-based coordinate analysis approach. The dissimilarity-based coordinate analysis uses Jaccard Similarity for calculating the similarity coefficients between test cases in terms of covered methods and the enhanced technique called Dissimilarity-LoM-Score ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dis-LoM-Score</i> ) applies a penalty with respective on the ordered test cases. We performed our case study on 10 open-source Java projects from Defects4J, which is a dataset of real bugs and an infrastructure for controlled experiments provided for software engineering researchers. Then, we hand-seeded multiple mutants generated by Major, which is a mutation testing tool. Then we compared our TCP techniques <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">LoM-Score</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Dis-LoM-Score</i> with the four traditional TCP techniques based on their Average Percentage of Faults Detected (APFD) results.

Test Case Prioritization in Unit and Integration Testing: A Shuffled-Frog-Leaping Approach

Both unit and integration testing are incredibly crucial for almost any software application because each of them operates a distinct process to examine the product. Due to resource constraints, when software is subjected to modifications, the drastic increase in the count of test cases forces the testers to opt for a test optimization strategy. One such strategy is test case prioritization (TCP). Existing works have propounded various methodologies that re-order the system-level test cases intending to boost either the fault detection capabilities or the coverage efficacy at the earliest. Nonetheless, singularity in objective functions and the lack of dissimilitude among the re-ordered test sequences have degraded the cogency of their approaches. Considering such gaps and scenarios when the meteoric and continuous updations in the software make the intensive unit and integration testing process more fragile, this study has introduced a memetics-inspired methodology for TCP. The proposed structure is first embedded with diverse parameters, and then traditional steps of the shuffled-frog-leaping approach (SFLA) are followed to prioritize the test cases at unit and integration levels. On 5 standard test functions, a comparative analysis is conducted between the established algorithms and the proposed approach, where the latter enhances the coverage rate and fault detection of re-ordered test sets. Investigation results related to the mean average percentage of fault detection (APFD) confirmed that the proposed approach exceeds the memetic, basic multi-walk, PSO, and optimized multi-walk by 21.7%, 13.99%, 12.24%, and 11.51%, respectively.

Modified Harris Hawks Optimization Based Test Case Prioritization for Software Testing

Generally, software testing is considered as a proficient technique to achieve improvement in quality and reliability of the software. But, the quality of test cases has a considerable influence on fault revealing capability of software testing activity. Test Case Prioritization (TCP) remains a challenging issue since prioritizing test cases is unsatisfactory in terms of Average Percentage of Faults Detected (APFD) and time spent upon execution results. TCP is mainly intended to design a collection of test cases that can accomplish early optimization using preferred characteristics. The studies conducted earlier focused on prioritizing the available test cases in accelerating fault detection rate during software testing. In this aspect, the current study designs a Modified Harris Hawks Optimization based TCP (MHHO-TCP) technique for software testing. The aim of the proposed MHHO-TCP technique is to maximize APFD and minimize the overall execution time. In addition, MHHO algorithm is designed to boost the exploration and exploitation abilities of conventional HHO algorithm. In order to validate the enhanced efficiency of MHHO-TCP technique, a wide range of simulations was conducted on different benchmark programs and the results were examined under several aspects. The experimental outcomes highlight the improved efficiency of MHHO-TCP technique over recent approaches under different measures.

Crowdsourced Test Report Prioritization Based on Text Classification

In crowdsourced testing, crowd workers from different places help developers conduct testing and submit test reports for the observed abnormal behaviors. Developers manually inspect each test report and make an initial decision for the potential bug. However, due to the poor quality, test reports are handled extremely slowly. Meanwhile, due to the limitation of resources, some test reports are not handled at all. Therefore, some researchers attempt to resolve the problem of test report prioritization and have proposed many methods. However, these methods do not consider the impact of duplicate test reports. In this paper, we focus on the problem of test report prioritization and present a new method named DivClass by combining a diversity strategy and a classification strategy. First, we leverage Natural Language Processing (NLP) techniques to preprocess crowdsourced test reports. Then, we build a similarity matrix by introducing an asymmetric similarity computation strategy. Finally, we combine the diversity strategy and the classification strategy to determine the inspection order of test reports. To validate the effectiveness of DivClass, experiments are conducted on five crowdsourced test report datasets. Experimental results show that DivClass achieves 0.8887 in terms of APFD (Average Percentage of Fault Detected) and improves the state-of-the-art technique DivRisk by 14.12% on average. The asymmetric similarity computation strategy can improve DivClass by 4.82% in terms of APFD on average. In addition, empirical results show that DivClass can greatly reduce the number of inspected test reports.

Applying test case prioritization to software microbenchmarks

Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case prioritization (TCP), a regression testing technique, to software microbenchmarks may help capturing large performance regressions sooner upon new versions. This may especially be beneficial for microbenchmark suites, because they take considerably longer to execute than unit test suites. However, it is unclear whether traditional unit testing TCP techniques work equally well for software microbenchmarks. In this paper, we empirically study coverage-based TCP techniques, employing total and additional greedy strategies, applied to software microbenchmarks along multiple parameterization dimensions, leading to 54 unique technique instantiations. We find that TCP techniques have a mean APFD-P (average percentage of fault-detection on performance) effectiveness between 0.54 and 0.71 and are able to capture the three largest performance changes after executing 29% to 66% of the whole microbenchmark suite. Our efficiency analysis reveals that the runtime overhead of TCP varies considerably depending on the exact parameterization. The most effective technique has an overhead of 11% of the total microbenchmark suite execution time, making TCP a viable option for performance regression testing. The results demonstrate that the total strategy is superior to the additional strategy. Finally, dynamic-coverage techniques should be favored over static-coverage techniques due to their acceptable analysis overhead; however, in settings where the time for prioritzation is limited, static-coverage techniques provide an attractive alternative.

Test Case Prioritization Using Bat Algorithm

Background: Regression testing is very important stage of software maintenance and evolution phase. The software keeps on updating and to preserve the software quality, it needs to be retested every time it is updated. Due to limited resources, complete testing of the software becomes tedious task. The probable solution to this problem is to execute those test cases first that are more important prior to less important test cases. Objective: Optimization methods need to be acquired for efficient test case prioritization in minimum time, while maintaining the quality of the software. Various nature-inspired algorithms like genetic algorithm, particle swarm optimization and ant colony optimization, etc. have been applied for prioritizing test cases. In this paper, we have applied a relatively new nature-inspired optimization method, namely, Bat algorithm that utilizes the echolocation, loudness and pulse emission rate of bats in prioritizing the test cases. Methods: The proposed algorithm is experimented on sample case study of timetable management system and evaluated with the help of popular evaluation metric Average Percentage of Fault Detection. Results: Results have been compared with the baseline approaches i.e. untreated, random and reverse prioritization and well-established optimization method i.e. genetic algorithm and we have found a considerable increase in the value of evaluation metric. Conclusion: This preliminary study shows that the bat algorithm have great potential to solve test case prioritization problems.

Test Case Prioritization Based on Early Fault Detection Technique

Background: In Regression testing when any changes made to already tested program it should not affect to other part of program. When some part of code is modified then it is necessary to validate the modified code. In the life cycle of software development, regression testing is an important and expensive activity. Objective: Our objective is to save regression testing time by reducing the cost and time of retesting of the modified code as well as affected components of the program. Objective is to prioritize the test case for object oriented program based on early fault detection. Methods: A new model is proposed using fuzzy inference system for regression testing. This paper presents a fuzzy inference system based on the use of inputs like fault detection rate, execution time and requirement coverage. The proposed system allows tester to prioritize test case based on linguistic rules. Results: In this paper performance is measured by using Average Percentage Fault Detection metric. We analysed for both prioritized and non-prioritized test suite using average percentage fault detection metric. Conclusion: We developed fuzzy expert model which helps to takes better decision than other expert system for regression testing. We conclude that, prioritization of test case will decrease the regression testing time.

Test scenario prioritization from user requirements for web-based software

User requirements are the building blocks for development of software applications. User requirements decide the width and breadth of any software. Nowadays, test scenarios are prepared from the user requirements, which give the test engineers ample scope to review the test plan thoroughly before doing software testing. Regression testing is carried out to know the effect of requirement changes on the functionalities and performance of the software. Test scenario prioritization, which is one of the techniques to perform regression testing, maximizes the ease of debugging for the system under test. The code based test scenario prioritization and model based test scenario prioritization have their own limitations. So, to achieve ease of debugging and to get more time for reviewing the test plans, researchers are now working on test scenario prioritization using requirements collected from the end users. In this paper, we propose an approach named Requirement based test scenario prioritization to prioritize test scenarios using requirements collected from end users for developing software applications. The user’s functional requirements are collected and is assigned with some weight depending upon different factors like complexity of implementing the requirements, type of release of the requirements, requirement volatility, coupling between requirements etc. Test scenarios are generated from requirements collected from the end users. Then, the final priority weight of each test scenario is found out by considering the weight of each requirement covered by the corresponding test scenario and the percentage of requirements coverage made by each test scenario. The test scenarios are prioritized based on the final priority weight. The proposed approach is evaluated using average percentage of fault detection metric and is found to be very efficient in early test scenario prioritization and detection of faults.

Test Case Prioritization Approach for Sequence of Events Using Complexity Factor

Test case prioritization (TCP) is a method to prioritize and schedule test cases. Some approaches have been introduced to minimize the time, cost, and effort for testing the software based on the test cases that are higher priority. Since the more complex the software program, the more intensive the test should be carried out. Thus, complexity is one of the factors that affect the effectiveness of the test case prioritization. However, the existing approaches for measuring complexity have some limitations. This is due to inaccuracy in finding the weightage value for complexity as the value is useful to determine the test case prioritization. Consequently, a complexity metric measurement is needed to determine the weightage value. Hence, this paper presents work on TCP using complexity factors to enhance the accuracy in prioritizing the test cases for event sequences. This work uses Branch Coverage Expectation (BCE) for complexity measurement, in which BCE has been proven its usefulness empirically in the previous research. The event-weightage value based on the complexity is then assigned and used to prioritize the test cases while the Average Percentage of Fault Detected (APFD) metric is used to evaluate the proposed approach. A tool has been developed to ease the process as well as to facilitate the evaluation purposes. The results show the need to combine the complexity factor with other factors to improve the proposed TCP's effectiveness.

An Ontology Based Test Case Prioritization Approach in Regression Testing

Regression testing is a widely studied research area, with the aim of meeting the quality challenges of software systems. To achieve a software system of good quality, we face high consumption of resources during testing. To overcome this challenge, test case prioritization (TCP) as a sub-type of regression testing is continuously investigated to achieve the testing objectives. This study provides an insight into proposing the ontology-based TCP (OTCP) approach, aimed at reducing the consumption of resources for the quality improvement and maintenance of software systems. The proposed approach uses software metrics to examine the behavior of classes of software systems. It uses Binary Logistic Regression (BLR) and AdaBoostM1 classifiers to verify correct predictions of the faulty and non-faulty classes of software systems. Reference ontology is used to match the code metrics and class attributes. We investigated five Java programs for the evaluation of the proposed approach, which was used to achieve code metrics. This study has resulted in an average percentage of fault detected (APFD) value of 94.80%, which is higher when compared to other TCP approaches. In future works, large sized programs in different languages can be used to evaluate the scalability of the proposed OTCP approach.