Test Case Selection Techniques Research Articles

Test case selection and prioritization using machine learning: a systematic literature review

Regression testing is an essential activity to assure that software code changes do not adversely affect existing functionalities. With the wide adoption of Continuous Integration (CI) in software projects, which increases the frequency of running software builds, running all tests can be time-consuming and resource-intensive. To alleviate that problem, Test case Selection and Prioritization (TSP) techniques have been proposed to improve regression testing by selecting and prioritizing test cases in order to provide early feedback to developers. In recent years, researchers have relied on Machine Learning (ML) techniques to achieve effective TSP (ML-based TSP). Such techniques help combine information about test cases, from partial and imperfect sources, into accurate prediction models. This work conducts a systematic literature review focused on ML-based TSP techniques, aiming to perform an in-depth analysis of the state of the art, thus gaining insights regarding future avenues of research. To that end, we analyze 29 primary studies published from 2006 to 2020, which have been identified through a systematic and documented process. This paper addresses five research questions addressing variations in ML-based TSP techniques and feature sets for training and testing ML models, alternative metrics used for evaluating the techniques, the performance of techniques, and the reproducibility of the published studies. We summarize the results related to our research questions in a high-level summary that can be used as a taxonomy for classifying future TSP studies.

A Systematic Analysis of Regression Test Case Selection: A Multi-Criteria-Based Approach

In applied software engineering, the algorithms for selecting the appropriate test cases are used to perform regression testing. The key objective of this activity is to make sure that modification in the system under test (SUT) has no impact on the overall functioning of the updated software. It is concluded from the literature that the efficacy of the test case selection solely depends on the following metrics, namely, the execution cost of the test case, the lines of the code covered in unit time also known as the code coverage, the ability to capture the potential faults, and the code modifications. Furthermore, it is also observed that the approaches for the regression testing developed so far generated results by focusing on one or two parameters. In this paper, our key objectives are twofold: one is to explore the importance of the role of each metric in detail. The secondary objective is to study the combined effect of these metrics in test case selection task that is capable of achieving more than one objective. In this paper, a detailed and comprehensive review of the work related to regression testing is provided in a very distinct and principled way. This survey will be useful for the researchers contributing to the field of regression testing. It is noteworthy that our systematic literature review (SLR) included the noteworthy work published from 2007 to 2020. Our study observed that about 52 relevant studies focused on all of the four metrics to perform their respective tasks. The results also revealed that about 30% of the different categories of regression test case reported the results using metaheuristic regression test selection (RTS). Similarly, about 31% of the literature reported results using the generic regression test case selection techniques. Most of the researchers focus on the datasets, namely, Software-Artefact Infrastructure Repository (SIR), JodaTime, TreeDataStructure, and Apache Software Foundation. For validation purpose, following parameters were focused, namely, the inclusiveness, precision, recall, and retest-all.

K-Step Crossover Method based on Genetic Algorithm for Test Suite Prioritization in Regression Testing

Software is an integration of numerous programming modules  (e.g., functions, procedures, legacy system, reusable components, etc.) tested and combined to build the entire module. However, some undesired faults may occur due to a change in modules while performing validation and verification. Retesting of entire software is a costly affair in terms of money and time. Therefore, to avoid retesting of entire software, regression testing is performed. In regression testing, an earlier created test suite is used to retest the software system's modified module. Regression Testing works in three manners; minimizing test cases, selecting test cases, and prioritizing test cases. In this paper, a two-phase algorithm has been proposed that considers test case selection and test case prioritization technique for performing regression testing on several modules ranging from a smaller line of codes to huge line codes of procedural language. A textual based differencing algorithm has been implemented for test case selection. Program statements modified between two modules are used for textual differencing and utilized to identify test cases that affect modified program statements. In the next step, test case prioritization is implemented by applying the Genetic Algorithm for code/condition coverage. Genetic operators: Crossover and Mutation have been applied over the initial population (i.e. test cases), taking code/condition coverage as fitness criterion to provide a prioritized test suite. Prioritization algorithm can be applied over both original and reduced test suite depending upon the test suite's size or the need for accuracy. In the obtained results, the efficiency of the prioritization algorithms has been analyzed by the Average Percentage of Code Coverage (APCC) and Average Percentage of Code Coverage with cost (APCCc). A comparison of the proposed approach is also done with the previously proposed methods and it is observed that APCC & APCCc values achieve higher percentage values faster in the case of the prioritized test suite in contrast to the non-prioritized test suite.

Test Case Prioritization & Selection for an Object Oriented Software using Genetic Algorithm.

In this paper our aim is to propose a Test Case Selection and Prioritization technique for OOP for ordering the test cases as per in accordance with their priority for finding the faults in the OOS. We have used the heuristic Genetic Algorithm, in order to generating the order of these prioritized test cases for a given OOS. The motive is to put a test case first into the ordered sequence that may have the highest prospective of finding an error in the given OOS & then soon.

The Application Of Machine Learning In Test Case Prioritization - A Review

Regression testing is very important but also a very costly and time-consuming activity that ensures the developers that changes in the application will not bring new errors. Retest all, selection of test cases and prioritization of test cases (TCP) approaches are used to enhance the efficiency and effectiveness in regression testing. While test case selection techniques decrease testing time and cost, it can exclude some critical test cases that can detect the faults. On the other hand, test case prioritization considers all test cases and execute them until resources are exhausted or all test cases are executed, while always focusing on the most important ones. Over the years, machine learning has found wide usage in solving different problems in software engineering. Software development and maintenance problems can be defined as learning problems and machine learning techniques have shown to be very effective in solving these problems. In the range of application of machine learning, machine learning techniques have also found usage in solving the test case prioritization problem. In this paper, we investigate the application of machine learning techniques in test case prioritization. We survey some of the most recent studies made in this field and provide information like techniques of machine learning used in TCP process, metrics used to measure the effectiveness of the proposed methods, data used to define the priority of test cases and some advantages or limitations of application of machine learning in TCP.

Test case prioritization and selection technique in continuous integration development environments: a case study

Regression testing is a very important activity in continuous integration development environments. Software engineers frequently integrate new or changed code that involves in a new regression testing. Furthermore, regression testing in continuous integration development environments is together with tight time constraints. It is also impossible to re-run all the test cases in regression testing. Test case prioritization and selection technique are often used to render continuous integration processes more cost-effective. According to multi objective optimization, we present a test case prioritization and selection technique, TCPSCI, to satisfy time constraints and achieve testing goals in continuous integration development environments. Based on historical failure data, testing coverage code size and testing execution time, we order and select test cases. The test cases of the maximize code coverage, the shorter execution time and revealing the latest faults have the higher priority in the same change request. The case study results show that using TCPSCI has a higher cost-effectiveness comparing to the manually prioritization.

Effective Regression Test Case Selection

Regression test case selection techniques attempt to increase the testing effectiveness based on the measurement capabilities, such as cost, coverage, and fault detection. This systematic literature review presents state-of-the-art research in effective regression test case selection techniques. We examined 47 empirical studies published between 2007 and 2015. The selected studies are categorized according to the selection procedure, empirical study design, and adequacy criteria with respect to their effectiveness measurement capability and methods used to measure the validity of these results. The results showed that mining and learning-based regression test case selection was reported in 39% of the studies, unit level testing was reported in 18% of the studies, and object-oriented environment (Java) was used in 26% of the studies. Structural faults, the most common target, was used in 55% of the studies. Overall, only 39% of the studies conducted followed experimental guidelines and are reproducible. There are 7 different cost measures, 13 different coverage types, and 5 fault-detection metrics reported in these studies. It is also observed that 70% of the studies being analyzed used cost as the effectiveness measure compared to 31% that used fault-detection capability and 16% that used coverage.

Full modification coverage through automatic similarity-based test case selection

Context: This paper presents the similarity approach for regression testing (SART), where a similarity-based test case selection technique (STCS) is used in a model-based testing process to provide selection of test cases exercising modified parts of a specification model. Unlike other model-based regression testing techniques, SART relies on similarity analysis among test cases to identify modifications, instead of comparing models, hence reducing the dependency on specific types of model.Objective: To present convincing evidence of the usage of similarity measures for modification-traversing test case selection.Method: We investigate SART in a case study and an experiment. The case study uses artefacts from industry and should be seen as a sanity check of SART, while the experiment focuses on gaining statistical power through the generation of synthetical models in order to provide convincing evidence of SART’s effectiveness. Through posthoc analysis we obtain p-values and effect sizes to observe statistically significant differences between treatments with respect to transition and modification coverage.Results: The case study with industrial artefacts revealed that SART is able to uncover the same number of defects as known similarity-based test case selection techniques. In turn, the experiment shows that SART, unlike the other investigated techniques, presents 100% modification coverage. In addition, all techniques covered a similar percentage of model transitions.Conclusions: In summary, not only does SART provide transition and defect coverage equal to known STCS techniques, but it exceeds greatly in covering modified parts of the specification model, being a suitable candidate for model-based regression testing.

Aware Time in Bug Fix—A Novel Automatic Test Case Selection for Prioritization of Version Control

The supreme goal of the Automatic Test case selection techniques is to guarantee systematic coverage, to recognize the usual error forms and to lessen the test of redundancy. It is unfeasible to carry out all the test cases consistently. For this reason, the test cases are picked and prioritize it. The major goal of test case prioritization is to prioritize the test case sequence and finds faults as early as possible to improve the efficiency. Regression testing is used to ensure the validity and the enhancement part of the changed software. In this paper, we propose a new path compression technique (PCUA) for both old version and new version of BPEL dataset. In order to analyze the enhancement part of an application and to find an error in an enhancement part of an application, center of the tree has been calculated. Moreover in the comparative analysis, our proposed PCUA- COT technique is compared with the existing XPFG technique in terms of time consuming and error detection in the path of an enhancement part of BPEL dataset. The experimental results have been shown that our proposed work is better than the existing technique in terms of time consuming and error detection.

A Review: Study of Test Case Generation Techniques

Software testing is very labor intensive task for developing software and improving its quality. According to some researchers and software professionals,50% of the time, cost and effort are spent on software testing.To test software, generating test cases is the most important task.Testing can be done either manually or automatically by using various testing tools. In today’s scenario software are testedautomatically with the help of tools as it is a fast and accurate process of testing software. Although various testing tools are available in market and are used by testers to test the software and to generate test cases and test data automatically.There are various techniques available for generating test cases like fuzzy logic, finite state machine, neural networks, genetic algorithms, soft computing, genetic programming, evolutionary computation and many others.This paper presents various test case generation methods, test case minimization, selection, and prioritization and evaluation techniques.This paper also focuses on various test case prioritization and selection techniques that help the test engineers to schedule and rank the test cases to reduce the total effort, time and the cost.

Ants Optimization for Minimal Test Case Selection and Prioritization as to Reduce the Cost of Regression Testing

testing is the major process in software development life cycle. Regression testing is very costly and inevitable activity that is to be performed in a restricted environment to ensure the validity of modified software. It is inefficient to re- run every test case from test suite when some kind of modification is done in the software. Test case selection and prioritization techniques select and organize the test cases in a test suite based on some criteria such that the faults are covered quickly with minimum execution time. This task can be done on basis of the Ant Colony Optimization technique (ACO) of Swarm Intelligence as it is not deeply studied yet. The main objective of this thesis is to solve the path problem: Means to find the shortest path and Resolve the time problem: Means to minimize the time of finding shortest path. Because of time and cost constraint, it is not possible to perform extensive regression testing. Techniques such as test case selection and prioritization are used to solve the problem of time and cost constraints. In this paper we are modifying the previous technique to get better results in case of execution time and then the Effectiveness of techniques is checked with the help of APFD metric.

Application of Search Algorithms for Model Based Regression Testing

UML models have gained their significance as reported in the literature. The use of a model to describe the behavior of a system is a proven and major advantage to test. With the help of Model Based Testing (MBT), it is possible to automatically generate test cases. When MBT is applied on large industrial systems, there is problem to sampling the test cases from the suit of entire test because it is difficult to execute the huge number of test cases being generated. The motivation of this study is to design a multi objective genetic algorithm based test case selection technique which can select the most appropriate subset of test cases. NSGA (Non-dominated Sorting Genetic Algorithm) is used as an optimization algorithm and its fitness function is improved for selecting test cases from the dataset. It is concluded that there is a room to improve the performance of NSGA algorithm by means of tailoring its respective fitness function.

Achieving scalable model-based testing through test case diversity

The increase in size and complexity of modern software systems requires scalable, systematic, and automated testing approaches. Model-based testing (MBT), as a systematic and automated test case generation technique, is being successfully applied to verify industrial-scale systems and is supported by commercial tools. However, scalability is still an open issue for large systems, as in practice there are limits to the amount of testing that can be performed in industrial contexts. Even with standard coverage criteria, the resulting test suites generated by MBT techniques can be very large and expensive to execute, especially for system level testing on real deployment platforms and network facilities. Therefore, a scalable MBT technique should be flexible regarding the size of the generated test suites and should be easily accommodated to fit resource and time constraints. Our approach is to select a subset of the generated test suite in such a way that it can be realistically executed and analyzed within the time and resource constraints, while preserving the fault revealing power of the original test suite to a maximum extent. In this article, to address this problem, we introduce a family of similarity-based test case selection techniques for test suites generated from state machines. We evaluate 320 different similarity-based selection techniques and then compare the effectiveness of the best similarity-based selection technique with other common selection techniques in the literature. The results based on two industrial case studies, in the domain of embedded systems, show significant benefits and a large improvement in performance when using a similarity-based approach. We complement these analyses with further studies on the scalability of the technique and the effects of failure rate on its effectiveness. We also propose a method to identify optimal tradeoffs between the number of test cases to run and fault detection.

Tool Support For Test Case Selection In Regression Testing

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible. As the software evolves the size of test suites also grows. Due to limited resources, basic problem in regression testing is to select the important test cases from the available test suit. In this test case selection techniques unimportant test cases are discarded to save the time and resources. A number of techniques have been proposed for test case selection. Most of them are code based. So, they have not proved viable in all the situations. In this article we present an approach based on specifications. It uses classification tree method using classification tree editor tool. The classification-tree method provides a systematic way for software testers to derive test cases by considering important relevant aspects that are identified from the specification. The method has been used in many real-life applications and shown to be effective. The proposed approach is very effective to reduce manual effort in the generation and selection of test cases. We are using CTE XL tool for supporting this enhanced process which shows how to integrate weighting factors into classification trees which leads to automatic generation of selected test suites. CTE XL supports test case selection by occurrence probability, error probability, or risk because it is based on user requirements.

다양한 웹 기반 소프트웨어의 테스트를 위한 효율적인 테스트 케이스의 생성

인터넷을 통한 정보의 교류와 비즈니스가 활발해지면서 웹 기반 소프트웨어도 확대되고 있고 그만큼 품질 측면의 노력이 중요하다. 웹 컨텐츠의 링크나 웹 기반 프로그램을 검증하는 연구가 시도되었으나 다양한 웹 기반 구성 방법들을 커버하는 방법은 찾기 어려웠고 또한 적용 실험이 부족하다. 이 논문에서는 다양한 동적인 웹 기반 소프트웨어들을 타입별로 나누어 보고 동일한 방법으로 모델링 한 후 테스트 케이스를 생성하는 방법을 제안한다. 웹 기반 소프트웨어를 구성하는 개체들을 파악한 후 이를 개체 제어흐름 그래프(Object Control of Flow Graph)와 개체 관계 다이어그램(Object Relationship Diagram)으로 모델링 한다. 모델에서 테스트 경로를 파악하고 테스트 키 포인트를 구동하는 테스트 케이스를 찾는다. 제안한 방법으로 다섯 가지 타입의 웹 기반 소프트웨어를 적용하여 실용성을 보였으며 이제까지 제안된 방법과 어떤 차이를 보이는지 비교하였다. As information and business communication via Internet are growing up, web-based software is wide spread and more important on the viewpoint of software qualify than stand-alone. Research on verification of web content links and web-based Program was tried, but has short on covering various types of web based software and making experiments to be applied in real testing practice. This paper suggests a modeling technique to be applied to dynamic and various types of web-based software. First, it identifies each elements consisting of web-based software and then construct a model of Object Control Flow Graph and Object Relationship Diagram. We can generate test cases covering all test paths of ORD or invoking key points test route. Suggested modeling method and test case selection technique are verified by applying five types of web-based software and compared with other web-based test techniques.

FAITH, a general purpose protocol test system for ISDN

This paper describes the key features of FAITH, a protocol independent test system for ISDN protocols. It is based on TTCN as test notation and a test life cycle that is compliant with ISO9646. TTCN specifications are automatically compiled into executable code. FAITH supports the test specification, automatic test implementation, test execution, automatic test report generation of the test life cycle and additionally multi party testing methods and extensive test case selection techniques.