Regression Test Suite Research Articles

Hybrid Framework To Exclude Similar and Faulty Test Cases In Regression Testing

Regression testing is a crucial phase in the software development lifecycle that makes sure that new changes/updates in the software system don’t introduce defects or don’t affect adversely the existing functionalities. However, as the software systems grow in complexity, the number of test cases in regression suite can become large which results into more testing time and resource consumption. In addition, the presence of redundant and faulty test cases may affect the efficiency of the regression testing process. Therefore, this paper presents a new Hybrid Framework to Exclude Similar & Faulty Test Cases in Regression Testing (ETCPM) that utilizes automated code analysis techniques and historical test execution data to identity and exclude redundant, similar and faulty test cases from the given regression suite. Our experimental results clearly show the benefits of the ETCPM framework in terms of reduction in the testing time, optimization of the resource allocation, and improvement in the overall quality of regression test suite. ETCPM enables software development teams to achieve faster and reliable regression testing by intelligent exclusion of similar and fault test cases, which yields in reduction in the software delivery cycles and better end user satisfaction.

Quantification of Regression Test Suite Execution Time in Parallel Execution Setup with Weighted Test Suite Split Algorithm

Regression test suite execution time study focus is essentially on two aspects. They are execution time reduction and making effective use of available hardware resources and manpower. This paper investigates how the regression test suite can be split into subsets to make use of parallel execution across several machines with identical execution speeds and asymmetrical execution speeds. In the symmetrical execution speed setup, long test execution time test cases are evenly distributed across all the hardware machines. However, in asymmetrical execution speed machines, more test cases are distributed to speed machines to make efficient use of hardware resources. In all the situations where there is automation tool to execute the individual test cases of test suite this approach can be employed to make effective use of hardware resources and to keep the execution time within bounds. The Algorithms can also be used in situations where there are queues involved and serial, fixed time service takes place for each of the entity being served.

Quantification of Regression Test Suite Execution Time in Parallel Execution Setup with Weighted Test Suite Split Algorithm

Regression test suite execution time study focus is essentially on two aspects. They are execution time reduction and making effective use of available hardware resources and manpower. This paper investigates how the regression test suite can be split into subsets to make use of parallel execution across several machines with identical execution speeds and asymmetrical execution speeds. In the symmetrical execution speed setup, long test execution time test cases are evenly distributed across all the hardware machines. However, in asymmetrical execution speed machines, more test cases are distributed to speed machines to make efficient use of hardware resources. In all the situations where there is automation tool to execute the individual test cases of test suite this approach can be employed to make effective use of hardware resources and to keep the execution time within bounds. The Algorithms can also be used in situations where there are queues involved and serial, fixed time service takes place for each of the entity being served.

A model-free autonomous performance testing for human recognition using different types of software-intensive autonomous systems

The model-based methods were utilized in order to produce the test cases for the behavioral model of a software system. Run test cases habitually or physically facilitates premature identification of requirement errors. Regression test suite design is thought-provoking as well as significant task in this automated test design. General techniques of regression testing comprise rerunning formerly accomplished tests as well as inspecting whether program behavior has modified as well as formerly fixed faults have recurred. Regression testing is carried out with the intension of assessing a system skillfully by means of logically picking the right least set of tests essential to suitably cover a particular modification. Then again, the relapse testing occasions of experiment prioritization, test suite decrease, and relapse test choice are commonly focused on conditions, which recognize the experiments to pick or the experiment to run thusly in independent framework. As indicated by experiment prioritization, experiments are very much arranged ward upon some condition just as experiments with greatest need are run first to achieve a presentation objective. If there should be an occurrence of test suite decrease/minimization, experiment, which end up being ended over the long haul are dismissed from the test suite with the intension of making a minor arrangement of experiments. In the event of relapse test determination, from a prevalent unique suite, a subset of experiments is picked.

Comparative Analysis of Test Case Prioritization Using Ant Colony Optimization Algorithm and Genetic Algorithm

After it is published, every software system will get an upgrade, requiring it to adapt to meet the ever-changing client needs thus regression testing becomes one of the most important operations in any software system. As it is too expensive to repeat the execution of all the test cases available from a previous version of the software system, numerous ways to optimize the regression test suite have evolved, one of which is test case prioritizing. This study was carried out to test and compare the effectiveness of evolutionary algorithms and swarm intelligence algorithms, represented by the Genetic Algorithm (GA) and Ant Colony Optimization (ACO) algorithms. They will be implemented to find the Average Percentage Fault Detected (APFD), execution time, and Big O notation, as these are critical aspects in software testing to ensure high-quality products are produced on time. This study employs data from two separate investigations on comparable issues, denoted as Case Study One and Case Study Two. TCP has been extensively used in recent years, but not much research has been conducted to analyze and evaluate the performance of genetic algorithms (GA) and ant colony optimization (ACO) in a test case prioritization context. The algorithms were compared using APFD, execution time. The APFD and execution time values of 50, 100, 150, and 200 iterations for ACO and GA for both datasets are conducted. Both algorithms were determined to work on O() notation, which indicates they should scale up their execution process similarly on different input scales. Both algorithms performed well in their respective roles. ACO has shown to be more valuable than GA in terms of APFD and GA has shown to be more valuable than ACO in terms of execution time.

A model-based mutation framework for IEC61131-3 manufacturing systems

Abstract This article presents a model-based mutation framework for creating mutant systems implemented in IEC61131-3 languages. The framework is based on a meta-model of the IEC61131-3 standard. It supports 21 mutation operations, such as the addition or removal of POUs or the mutation of Structured Text and Sequential Functional Charts. Moreover, operations can be combined to create higher-order changes, such as Type III code clones. The mutation framework can be used to evaluate and improve development tools for supporting the evolution of manufacturing systems, such as regression test suite generators or clone detectors.

Patching Locking Bugs Statically with Crayons

The Linux Kernel is a world-class operating system controlling most of our computing infrastructure: mobile devices, Internet routers and services, and most of the supercomputers. Linux is also an example of low-level software with no comprehensive regression test suite (for good reasons). The kernel’s tremendous societal importance imposes strict stability and correctness requirements. These properties make Linux a challenging and relevant target for static automated program repair (APR). Over the past decade, a significant progress has been made in dynamic APR. However, dynamic APR techniques do not translate naturally to systems without tests. We present a static APR technique addressing sequential locking API misuse bugs in the Linux Kernel. We attack the key challenge of static APR, namely, the lack of detailed program specification, by combining static analysis with machine learning to complement the information presented by the static analyzer. In experiments on historical real-world bugs in the kernel, we were able to automatically re-produce or propose equivalent patches in 85% of the human-made patches, and automatically rank them among the top three candidates for 64% of the cases and among the top five for 74%.

Test case prioritization using partial attention

Test case prioritization (TCP) aims to reorder the regression test suite with a goal of increasing the fault detection rate. Various TCP techniques have been proposed based on different prioritization strategies. Among them, the greedy-based techniques are the most widely-used TCP techniques. However, existing greedy-based techniques usually reorder all candidate test cases in prioritization iterations, resulting in both efficiency and effectiveness problems. In this paper, we propose a generic partial attention mechanism, which adopts the previous priority values (i.e., the number of additionally-covered code units) to avoid considering all candidate test cases. Incorporating the mechanism with the additional-greedy strategy, we implement a novel coverage-based TCP technique based on partition ordering (OCP). OCP first groups the candidate test cases into different partitions and updates the partitions on the descending order. We conduct a comprehensive experiment on 19 versions of Java programs and 30 versions of C programs to compare the effectiveness and efficiency of OCP with six state-of-the-art TCP techniques: total-greedy, additional-greedy, lexicographical-greedy, unify-greedy, art-based, and search-based. The experimental results show that OCP achieves a better fault detection rate than the state-of-the-arts. Moreover, the time costs of OCP are found to achieve 85%–99% improvement than most state-of-the-arts.

Towards a Shift in Regression Test Suite Development Approach in Agile

Objective: From the literature review, it is evident that the concept of “regression testing” inherited in agile software testing originates from software maintenance practices. Therefore, the existing algorithms for regression testing revolve around the software maintenance principles rather than agile methodology. The objective of this paper is to evaluate the degree of fitness of the existing regression test-suite development algorithms for performing the regression testing in agile. Methods: This paper performs a systematic literature review for research work published from 2006 to 2018, which includes survey of the existing regression testing algorithms to identify and overcome the challenges associated with them while performing regression testing in agile. This research paper considers the four research questions into scope for analyzing the fitness of existing regression test-suite development algorithm for performing regression testing under agile methodology. Further, this paper attempts to propose approach for the development of the regression test-suite suitable for regression testing under agile methodology. Results: The current regression test-suite development algorithm were found unsuitable for performing the regression testing under agile methodology due to the newly identified four key challenges associated with them. Conclusion: The current regression test-suite development algorithms aligned with software maintenance principles rather than agile methodology. In addition, the newly proposed approach for regression test-suite development found to be easily adaptable by agile teams as it aligns with agile methodology principles. Finally, this paper recommends the adoption of agile principle through the newly proposed approach for developing regression test-suite for performing regression testing under agile methodology.

ExVivoMicroTest: ExVivo Testing of Microservices

AbstractMicroservice‐based applications consist of multiple services that can evolve independently. When a service must be updated, it is first tested with in‐house regression test suites. However, the test suites that are executed are usually designed without the exact knowledge about how the services will be accessed and used in the field; therefore, they may easily miss relevant test scenarios, failing to prevent the deployment of faulty services. To address this problem, we introduce ExVivoMicroTest, an approach that analyzes the execution of deployed services at run‐time in the field, in order to generate test cases for future versions of the same services. ExVivoMicroTest implements lightweight monitoring and tracing capabilities, to inexpensively record executions that can be later turned into regression test cases that capture how services are used in the field. To prevent accumulating an excessive number of test cases, ExVivoMicroTest uses a test coverage model that can discriminate the recorded executions between the ones that are worth to be turned into test cases and the ones that should be discarded. The resulting test cases use a mocked environment that fully isolates the service under test from the rest of the system to faithfully reply interactions. We assessed ExVivoMicroTest with the PiggyMetrics and Train Ticket open source microservice applications and studied how different configurations of the monitoring and tracing logic impact on the capability to generate test cases.

Test Case Prioritization Using Partial Attention

Test case prioritization (TCP) aims to reorder the regression test suite with a goal of increasing the fault detection rate. Various TCP techniques have been proposed based on different prioritization strategies. Among them, the greedy-based techniques are the most widely-used TCP techniques. However, existing greedy-based techniques usually reorder all candidate test cases in prioritization iterations, resulting in both efficiency and effectiveness problems. In this paper, we propose a generic partial attention mechanism, which adopts the previous priority values (i.e., the number of additionally-covered code units) to avoid considering all candidate test cases. Incorporating the mechanism with the additional-greedy strategy, we implement a novel coverage-based TCP technique based on partition ordering (OCP). OCP first groups the candidate test cases into different partitions and updates the partitions on the descending order. We conduct a comprehensive experiment on 19 versions of Java programs and 30 versions of C programs to compare the effectiveness and efficiency of OCP with six state-of-the-art TCP techniques: total-greedy, additional-greedy, lexicographical-greedy, unify-greedy, art-based, and search-based. The experimental results show that OCP achieves a better fault detection rate than the state-of-the-arts. Moreover, the time costs of OCP are found to achieve 85%-99% improvement than most state-of-the-arts.

Comparison of Multi-Objective Evolutionary Algorithms to Prioritize Regression Test Cases

Regression testing is one of the most critical testing activities among software product verification activities. Nevertheless, resources and time constraints could inhibit the execution of a full regression test suite, hence leaving us in confusion on what test cases to run to preserve the high quality of software products. Different techniques can be applied to prioritize test cases in resource-constrained environments, such as manual selection, automated selection, or hybrid approaches. Different Multi-Objective Evolutionary Algorithms (MOEAs) have been used in this domain to find an optimal solution to minimize the cost of executing a regression test suite while obtaining maximum fault detection coverage as if the entire test suite was executed. MOEAs achieve this by selecting set of test cases and determining the order of their execution. In this paper, three Multi Objective Evolutionary Algorithms, namely, NSGA-II, IBEA and MoCell are used to solve test case prioritization problems using the fault detection rate and branch coverage of each test case. The paper intends to find out what’s the most effective algorithm to be used in test cases prioritization problems, and which algorithm is the most efficient one, and finally we examined if changing the fitness function would impose a change in results. Our experiment revealed that NSGA-II is the most effective and efficient MOEA; moreover, we found that changing the fitness function caused a significant reduction in evolution time, although it did not affect the coverage metric.

An Efficient Regression Test Suite Optimization Approach Using Hybrid Spider Monkey Optimization Algorithm

Regression testing validates the modified software and safeguards against the introduction of new errors during modification. A number of test suite optimization techniques relying on meta-heuristic techniques have been proposed to find the minimal set of test cases to execute for regression purposes. This paper proposes a hybrid spider monkey optimization based regression test suite optimization approach and empirically compares its performance with three other approaches based on bat search, ant colony, and cuckoo search. The authors conducted an empirical study with various subjects retrieved from software artifact infrastructure repository. Fault coverage and execution time of algorithm are used as fitness measures to meet the optimization criteria. Extensive experiments are conducted to evaluate the performance of the proposed approach with other search-based approaches under study using various statistical tests like m-way ANOVA and post hoc tests including odds ratio. Results indicate the superiority of the proposed approach in most of the cases and comparable in others.

SIGSOFT Outstanding Doctoral Dissertation Award

As software becomes more important and ubiquitous, high quality software also becomes crucial. We depend on software developers who write the software to also maintain and improve its quality. When developers make changes to software, they rely on continuous integration [6] and regression testing [15] to check that changes do not break existing functionality. Continuous integration (CI) automates the process of building and testing software after every change. The process of running tests on the code after every change is known as regression testing. The goal of regression testing is to allow developers to detect and fix faults early on, ideally the moment the faults are introduced. Regression testing is widely used in both industry and open source, but regression testing suffers from two main challenges: (1) regression testing is costly, and (2) regression test suites often contain flaky tests.

On introducing automatic test case generation in practice: A success story and lessons learned

The level and quality of automation dramatically affects software testing activities, determines costs and effectiveness of the testing process, and largely impacts on the quality of the final product. While costs and benefits of automating many testing activities in industrial practice (including managing the quality process, executing large test suites, and managing regression test suites) are well understood and documented, the benefits and obstacles of automatically generating system test suites in industrial practice are not well reported yet, despite the recent progresses of automated test case generation tools. Proprietary tools for automatically generating test cases are becoming common practice in large software organizations, and commercial tools are becoming available for some application domains and testing levels. However, generating system test cases in small and medium-size software companies is still largely a manual, inefficient and ad-hoc activity.This paper reports our experience in introducing techniques for automatically generating system test suites in a medium-size company. We describe the technical and organizational obstacles that we faced when introducing automatic test case generation in the development process of the company, and present the solutions that we successfully experienced in that context. In particular, the paper discusses the problems of automating the generation of test cases by referring to a customized ERP application that the medium-size company developed for a third party multinational company, and presents ABT2.0, the test case generator that we developed by tailoring ABT, a research state-of-the-art GUI test generator, to their industrial environment. This paper presents the new features of ABT2.0, and discusses how these new features address the issues that we faced.

Exposing bugs in JavaScript engines through test transplantation and differential testing

JavaScript is a popular programming language today with several implementations competing for market dominance. Although a specification document and a conformance test suite exist to guide engine development, bugs occur and have important practical consequences. Implementing correct engines is challenging because the spec is intentionally incomplete and evolves frequently. This paper investigates the use of test transplantation and differential testing for revealing functional bugs in JavaScript engines. The former technique runs the regression test suite of a given engine on another engine. The latter technique fuzzes existing inputs and then compares the output produced by different engines with a differential oracle. We conducted experiments with engines from five major players—Apple, Facebook, Google, Microsoft, and Mozilla—to assess the effectiveness of test transplantation and differential testing. Our results indicate that both techniques revealed several bugs, many of which are confirmed by developers. We reported 35 bugs with test transplantation (23 of these bugs confirmed and 19 fixed) and reported 24 bugs with differential testing (17 of these confirmed and 10 fixed). Results indicate that most of these bugs affected two engines—Apple’s JSC and Microsoft’s ChakraCore (24 and 26 bugs, respectively). To summarize, our results show that test transplantation and differential testing are easy to apply and very effective in finding bugs in complex software, such as JavaScript engines.

Multi-objective regression test suite optimization using three variants of adaptive neuro fuzzy inference system.

In the process of software development, regression testing is one of the major activities that is done after making modifications in the current system or whenever a software system evolves. But, the test suite size increases with the addition of new test cases and it becomes in-efficient because of the occurrence of redundant, broken, and obsolete test cases. For that reason, it results in additional time and budget to run all these test cases. Many researchers have proposed computational intelligence and conventional approaches for dealing with this problem and they have achieved an optimized test suite by selecting, minimizing or reducing, and prioritizing test cases. Currently, most of these optimization approaches are single objective and static in nature. But, it is mandatory to use multi-objective dynamic approaches for optimization due to the advancements in information technology and associated market challenges. Therefore, we have proposed three variants of self-tunable Adaptive Neuro-fuzzy Inference System i.e. TLBO-ANFIS, FA-ANFIS, and HS-ANFIS, for multi-objective regression test suites optimization. Two benchmark test suites are used for evaluating the proposed ANFIS variants. The performance of proposed ANFIS variants is measured using Standard Deviation and Root Mean Square Error. A comparison of experimental results is also done with six existing methods i.e. GA-ANFIS, PSO-ANFIS, MOGA, NSGA-II, MOPSO, and TOPSIS and it is concluded that the proposed method effectively reduces the size of regression test suite without a reduction in the fault detection rate.

A water resource simulator in Python

A new generalised water resource network modelling Python library, Pywr, is presented. Given hydrological inflows, Pywr simulates customisable water allocation and operation rules throughout complex multi-purpose managed water systems at each user-defined time-step. The model uses a low-level interface to existing linear programming solvers for fast priority-based optimisation-driven simulation. The library uses an object based system for users to provide input data and record simulation outputs. A novel multi-scenario simulation method provides an almost 4-fold improvement in model run-times and supports calculating robustness metrics across scenarios. A flexible interface to specify multi-objective optimisation formulations as part of a model’s input file is included. These features enable analysts to apply advanced water planning approaches, such as robust decision making and robust optimisation, to real systems. The library is available under the GPLv3 open source licence, includes several examples and a regression test suite.

A Fault-Based Testing Approach in Safety Critical Medical Systems

The advent of technology has opened unprecedented opportunities in health care delivery system as the demand for intelligent and knowledge-based systems has increased as modern medical practices become more knowledge-intensive. As a result of this, there is greater need to investigate the pervasiveness of software faults in Safety critical medical systems for proper diagnosis. The sheer volume of code in these systems creates significant concerns about the quality of the software. The rate of untimely deaths nowadays is alarming partly due to the medical device used to carry out the diagnosis process. A safety-critical medical (SCM) system is a complex system in which the malfunctioning of software could result in death, injury of the patient or damage to the environment. The malfunctioning of the software could be as a result of the inadequacy in software testing due to test suit problem or oracle problem. Testing a SCM system poses great challenges to software testers. One of these challenges is the need to generate a limited number of test cases of a given regression test suite in a manner that does not compromise its defect detection ability. This paper presents a novel five-stage fault-based testing procedure for SCM, a model-based approach to generate test cases for differential diagnosis of Tuberculosis. We used Prime Path Coverage and Edge-Pair Coverage as coverage criteria to ensure maximum coverage to identify feasible paths. We analyzed the proposed testing procedure with the help of three metrics consisting of Fault Detection Density, Fault Detection Effectiveness and Mutation Adequacy Score. We evaluated the effectiveness of our testing procedure by running the suggested test cases on a sample historical data of tuberculosis patients. The experimental results show that our developed testing procedure has some advantages such as creating mutant graphs and Fuzzy Cognitive Map Engine while resolving the problem of eliminating infeasible test cases for effective decision making.

Compiler fuzzing: how much does it matter?

Despite much recent interest in randomised testing (fuzzing) of compilers, the practical impact of fuzzer-found compiler bugs on real-world applications has barely been assessed. We present the first quantitative and qualitative study of the tangible impact of miscompilation bugs in a mature compiler. We follow a rigorous methodology where the bug impact over the compiled application is evaluated based on (1) whether the bug appears to trigger during compilation; (2) the extent to which generated assembly code changes syntactically due to triggering of the bug; and (3) whether such changes cause regression test suite failures, or whether we can manually find application inputs that trigger execution divergence due to such changes. The study is conducted with respect to the compilation of more than 10 million lines of C/C++ code from 309 Debian packages, using 12% of the historical and now fixed miscompilation bugs found by four state-of-the-art fuzzers in the Clang/LLVM compiler, as well as 18 bugs found by human users compiling real code or as a by-product of formal verification efforts. The results show that almost half of the fuzzer-found bugs propagate to the generated binaries for at least one package, in which case only a very small part of the binary is typically affected, yet causing two failures when running the test suites of all the impacted packages. User-reported and formal verification bugs do not exhibit a higher impact, with a lower rate of triggered bugs and one test failure. The manual analysis of a selection of the syntactic changes caused by some of our bugs (fuzzer-found and non fuzzer-found) in package assembly code, shows that either these changes have no semantic impact or that they would require very specific runtime circumstances to trigger execution divergence.