Existing Test Suite Research Articles

DeLoSo: Detecting Logic Synthesis Optimization Faults Based on Configuration Diversity

Logic synthesis tools are the core components of digital circuit design, which convert programs written in hardware description languages into gate-level netlists, and optimize the netlists. However, the netlist optimization is complex, with numerous optimization parameters to be configured. Any minor optimization faults in logic synthesis tools may cause circuit diagrams to significantly deviate from the original design, posing risks in target systems. We propose DeLoSo, the De tector of Lo gic S ynthesis o ptimization faults, the first method specifically designed to identify potential faults in the optimization processes of logic synthesis tools. DeLoSo relies on netlist differences and parameter variations to guide the generation of diverse Logic Synthesis Optimization Configuration (LSOC) combinations to thoroughly test the optimization process. DeLoSo consists of three components: LSOC generator, which generates diverse LSOC combinations through configuration recombination and mutation; LSOC diversity evaluator, which assesses the diversity of optimization configurations; and LSOC validator, which validates the generated LSOC combinations to discover optimization faults. DeLoSo identified 19 faults in two established logic synthesis tools (i.e., Vivado and Yosys); 15 of them have been fixed by vendors. Particularly, the community maintainers of Yosys have considered incorporating DeLoSo into Yosys’ existing test suite.

A test amplification bot for Pharo/Smalltalk

Test amplification exploits the knowledge embedded in an existing test suite to strengthen it. A typical test amplification technique transforms the initial tests into additional test methods that increase the mutation coverage. Although past research demonstrated the benefits, additional steps need to be taken to incorporate test amplifiers in the everyday workflow of developers. This paper describes a proof-of-concept bot integrating Small-Amp with GitHub-Actions. The bot decides for itself which tests to amplify and does so within a limited time budget. To integrate the bot into the GitHub-Actions workflow, we incorporate three special-purpose features: (i) prioritization (to fit the process within a given time budget), (ii) sharding (to split lengthy tests into smaller chunks), and (iii) sandboxing (to make the amplifier crash-resilient). We evaluate our approach by installing the proof-of-concept extension of Small-Amp on five open-source projects deployed on GitHub. Our results show that a test amplification bot is feasible at a project level by integrating it into the build system. Moreover, we quantify the impact of prioritization, sharding, and sandboxing so that other test amplifiers may benefit from these special-purpose features. Our proof-of-concept demonstrates that the entry barrier for adopting test amplification can be significantly lowered.

DeltaDroid : Dynamic Delivery Testing in Android

Android is a highly fragmented platform with a diverse set of devices and users. To support the deployment of apps in such a heterogeneous setting, Android has introduced dynamic delivery —a new model of software deployment in which optional, device- or user-specific functionalities of an app, called Dynamic Feature Modules (DFMs) , can be installed, as needed, after the app’s initial installation. This model of app deployment, however, has exacerbated the challenges of properly testing Android apps. In this article, we first describe the results of an extensive study in which we formalized a defect model representing the various conditions under which DFM installations may fail. We then present DeltaDroid —a tool aimed at assisting the developers with validating dynamic delivery behavior in their apps by augmenting their existing test suite. Our experimental evaluation using real-world apps corroborates DeltaDroid ’s ability to detect many crashes and unexpected behaviors that the existing automated testing tools cannot reveal.

Multimodal multi-objective optimization: Comparative study of the state-of-the-art

Multimodal multi-objective problems (MMOPs) commonly arise in real-world problems where distant solutions in decision space correspond to very similar objective values. To obtain all solutions for MMOPs, many multimodal multi-objective evolutionary algorithms (MMEAs) have been proposed. For now, few studies have encompassed most of the recently proposed representative MMEAs and made a comparative comparison. In this study, we first review the related works during the last two decades. Then, we choose 12 state-of-the-art algorithms that utilize different diversity-maintaining techniques and compared their performance on existing test suites. Experimental results indicate the strengths and weaknesses of different techniques on different types of MMOPs, thus providing guidance on how to select/design MMEAs in specific scenarios.

Analisis Efektifitas Algoritma FAST Menggunakan Metrik Average Percentage Fault Detection dan Waktu Eksekusi Pada Test Case Prioritization

It uses regression testing to validate the modification results so as not to cause new errors in features that are already functioning correctly. The implementation of regression testing is generally done by re-executing the test suite used in the previous test. One of the crucial issues in regression testing is determining a strategic approach to reuse existing test suites. Testing with retesting all test cases will result in a long and expensive test. Thus, it is necessary to use a test case selection approach. The research study focuses on analyzing the effectiveness of the FAST algorithm, namely the FAST-pw and FAST-all algorithms, for executing test cases based on priority. The research method uses experiments through running algorithms on test cases that have been implanted with faults in the software with a test case size of 10,000 to 20,000 lines of the program. The effectiveness parameter uses the test case execution time and the percentage of fault detection using the average percentage fault detection (APFD) metric. The results showed that the FAST algorithm, namely the FAST-pw and FAST-all algorithms, had good effectiveness values ​​when applied to TCP testing with small to medium-sized SUTs, namely test cases that tested 1,000 – 20,000 program lines.

Production Monitoring to Improve Test Suites

In this paper, we propose to use production executions to improve the quality of testing for certain methods of interest for developers. These methods can be methods that are not covered by the existing test suite, or methods that are poorly tested. We devise an approach called PANKTI which monitors applications as they execute in production, and then automatically generates differential unit tests, as well as derived oracles, from the collected data. PANKTI's monitoring and generation focuses on one single programming language, Java. We evaluate it on three real-world, open-source projects: a videoconferencing system, a PDF manipulation library, and an e-commerce application. We show that PANKTI is able to generate differential unit tests by monitoring target methods in production, and that the generated tests improve the quality of the test suite of the application under consideration.

Mutation testing of PL/SQL programs

Mutation testing is a prominent technique for evaluating the effectiveness of a test suite. Existing tools developed for supporting this technique are applicable for mainstream programming languages like C and Java. Mutation testing tools and mutation operators used by these tools are inherently language-specific. Moreover, there is a lack of industrial case studies for evaluating mutation testing tools and techniques in practice. In this article, we introduce muPLSQL, a tool for applying mutation testing on PL/SQL programs, facilitating automation for both mutant generation and test execution. We utilized existing mutation operators that are applicable for PL/SQL. In addition, we introduced some operators specifically for this language. We conducted an industrial case study for evaluating the applicability and usefulness of our tool and mutation testing in general. We applied mutation testing on a business support software system. muPLSQL generated a total of 5,939 mutants. The number of live mutants was 680. Manual inspection of live mutants led to improvements of the existing test suite. In addition, we found 8 faults in source code during the inspection process. Test execution against the mutants required around 40 h. The overall effort was almost one person month.

A Framework for the Regression Testing of Model-to-Model Transformations

Background: Model transformations play a key role in Model-Driven Engineering (MDE). Testing model transformation is an important activity to ensure the quality and correctness of the generated models. However, during the evolution and maintenance of these model transformation programs, frequently testing them by running a large number of test cases can be costly. Regression test selection is a form of testing, which selects tests from an existing test suite to test a modified program. Aim: The aim of the paper is to present a test selection approach for the regression testing of model transformations. The selected test case suite should be smaller in size than the full test suite, thereby reducing the testing overhead, while at the same time the fault detection capability of the full test suite should not be compromised. Method: approach is based on the use of a traceability mapping of test cases with their corresponding rules to select the affected test items. The approach is complemented with a tool that automates the proposed process. Results: Our experiments show that the proposed approach succeeds in reducing the size of the selected test case suite, and hence its execution time, while not compromising the fault detection capability of the full test suite. Conclusion: The experimental results confirm that our regression test selection approach is cost-effective compared to a retest strategy.

Realistic Constrained Multiobjective Optimization Benchmark Problems From Design

Multiobjective optimization is increasingly used in engineering to design new systems and to identify design tradeoffs. Yet, design problems often have objective functions and constraints that are expensive and highly nonlinear. Combinations of these features lead to poor convergence and diversity loss with common algorithms that have not been specifically designed for constrained optimization. Constrained benchmark problems exist, but they do not necessarily represent the challenges of engineering problems. In this article, a framework to design electro-mechanical actuators, called multiobjective design of actuators (MODAct), is presented and 20 constrained multiobjective optimization test problems are derived from the framework with a specific focus on constraints. The full source code is made available to ease its use. The effects of the constraints are analyzed through their impact on the Pareto front as well as on the convergence performance. A constraint landscape analysis approach is followed and extended with three new metrics to characterize the search and objective spaces. The features of MODAct are compared to existing test suites to highlight the differences. In addition, a convergence analysis using NSGA-II, NSGA-III, and C-TAEA on MODAct and existing test suites suggests that the design problems are indeed difficult due to the constraints. In particular, the number of simultaneously violated constraints in newly generated solutions seems key in understanding the convergence challenges. Thus, MODAct offers an efficient framework to analyze and handle constraints in future optimization algorithm design.

A Fuzzy Rule Based Approach for Test Case Selection Probability Estimation in Regression Testing

Regression testing is a very essential activity during maintenance of software. Due to constraints of time and cost, it is not possible to re-execute every test case with respect to every change occurred. Thus, a technique is required that selects and prioritises the test cases efficiently. This paper proposes a novel fuzzy rule-based approach for selecting and ordering a number of test cases from an existing test suite to predict the selection probability of test cases using multiple factors. The test cases, which have ability to find high fault detection rate with maximum coverage and minimum execution time to test are selected. The results specify the effectiveness of the proposed model for predicting the selection probability of individual test cases.

Scripted GUI testing of Android open-source apps: evolution of test code and fragility causes

Evidence from empirical studies suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular, GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI testing fragility, i.e., GUI test classes failing or needing interventions because of modifications in the AUT or in its GUI arrangement and definition. The objective of our study is to examine the diffusion of test classes created with a set of popular GUI Automation Frameworks for Android apps, the amount of changes required to keep test classes up to date, and the amount of code churn in existing test suites, along with the underlying modifications in the AUT that caused such modifications. We defined 12 metrics to characterize the evolution of test classes and test methods, and a taxonomy of 28 possible causes for changes to test code. To perform our experiments, we selected six widely used open-source GUI Automation Frameworks for Android apps. We evaluated the diffusion of the tools by mining the GitHub repositories featuring them, and computed our set of metrics on the projects. Applying the Grounded Theory technique, we then manually analyzed diff files of test classes written with the selected tools, to build from the ground up a taxonomy of causes for modifications of test code. We found that none of the considered GUI automation frameworks achieved a major diffusion among open-source Android projects available on GitHub. For projects featuring tests created with the selected frameworks, we found that test suites had to be modified often – specifically, about 8% of developers’ modified LOCs belonged to test code and that a relevant portion (around 50% on average) of those modifications were induced by modifications in GUI definition and arrangement. Test code written with GUI automation fromeworks proved to need significant interventions during the lifespan of a typical Android open-source project. This can be seen as an obstacle for developers to adopt this kind of test automation. The evaluations and measurements of the maintainance needed by test code wrtitten with GUI automation frameworks, and the taxonomy of modification causes, can serve as a benchmark for developers, and the basis for the formulation of actionable guidelines and the development of automated tools to help mitigating the issue.

A Scalable Test Suite for Continuous Dynamic Multiobjective Optimization.

Dynamic multiobjective optimization (DMO) has gained increasing attention in recent years. Test problems are of great importance in order to facilitate the development of advanced algorithms that can handle dynamic environments well. However, many of the existing dynamic multiobjective test problems have not been rigorously constructed and analyzed, which may induce some unexpected bias when they are used for algorithmic analysis. In this paper, some of these biases are identified after a review of widely used test problems. These include poor scalability of objectives and, more important, problematic overemphasis of static properties rather than dynamics making it difficult to draw accurate conclusion about the strengths and weaknesses of the algorithms studied. A diverse set of dynamics and features is then highlighted that a good test suite should have. We further develop a scalable continuous test suite, which includes a number of dynamics or features that have been rarely considered in literature but frequently occur in real life. It is demonstrated with empirical studies that the proposed test suite is more challenging to the DMO algorithms found in the literature. The test suite can also test algorithms in ways that existing test suites cannot.

A fuzzy rule-based approach for test case selection probability estimation in regression testing

Regression testing is a very essential activity during maintenance of software. Due to constraints of time and cost, it is not possible to re-execute every test case with respect to every change occurred. Thus, a technique is required that selects and prioritises the test cases efficiently. This paper proposes a novel fuzzy rule-based approach for selecting and ordering a number of test cases from an existing test suite to predict the selection probability of test cases using multiple factors. The test cases, which have ability to find high fault detection rate with maximum coverage and minimum execution time to test are selected. The results specify the effectiveness of the proposed model for predicting the selection probability of individual test cases.

Evaluation of Mutation Testing in a Nuclear Industry Case Study

For software quality assurance, many safety-critical industries appeal to the use of dynamic testing and structural coverage criteria. However, there are reasons to doubt the adequacy of such practices. Mutation testing has been suggested as an alternative or complementary approach but its cost has traditionally hindered its adoption by industry, and there are limited studies applying it to real safety-critical code. This paper evaluates the effectiveness of state-of-the-art mutation testing on safety-critical code from within the U.K. nuclear industry, in terms of revealing flaws in test suites that already meet the structural coverage criteria recommended by relevant safety standards. It also assesses the practical feasibility of implementing such mutation testing in a real setting. We applied a conventional selective mutation approach to a C codebase supplied by a nuclear industry partner and measured the mutation score achieved by the existing test suite. We repeated the experiment using trivial compiler equivalence (TCE) to assess the benefit that it might provide. Using a conventional approach, it first appeared that the existing test suite only killed 82% of the mutants, but applying TCE revealed that it killed 92%. The difference was due to equivalent or duplicate mutants that TCE eliminated. We then added new tests to kill all the surviving mutants, increasing the test suite size by 18% in the process. In conclusion, mutation testing can potentially improve fault detection compared to structural-coverage-guided testing, and may be affordable in a nuclear industry context. The industry feedback on our results was positive, although further evidence is needed from application of mutation testing to software with known real faults.

Test Suite Minimization in Regression Testing Using Hybrid Approach of ACO and GA

This article describes about the application of search-based techniques in regression testing and compares the performance of various search-based techniques for software testing. Test cases tend to increase exponentially as the software is modified. It is essential to remove redundant test cases from the existing test suite. Regression testing is very costly and must be performed in restricted ways to ensure the validity of the existing software. There exist different methods to improve the quality of test cases in terms of the number of faults covered, opposed to the number of statements covered in a minimum time. Different methods exist for this purpose, such as minimization, test case selection, and test case prioritization. In this article, search-based methods are applied to improve the quality of the test suite in order to select a minimum set of test cases which covers all the statements in a minimum time. The whole approach is named search based regression testing. In this paper, the performance of different metaheuristics for test suite minimization problem is also compared with a hybrid approach of ant colony optimization algorithm and genetic algorithm.

Test suite generation with the Many Independent Objective (MIO) algorithm

Context: Automatically generating test suites is intrinsically a multi-objective problem, as any of the testing targets (e.g., statements to execute or mutants to kill) is an objective on its own. Test suite generation has peculiarities that are quite different from other more regular optimization problems. For example, given an existing test suite, one can add more tests to cover the remaining objectives. One would like the smallest number of small tests to cover as many objectives as possible, but that is a secondary goal compared to covering those targets in the first place. Furthermore, the amount of objectives in software testing can quickly become unmanageable, in the order of (tens/hundreds of) thousands, especially for system testing of industrial size systems.Objective: To overcome these issues, different techniques have been proposed, like for example the Whole Test Suite (WTS) approach and the Many-Objective Sorting Algorithm (MOSA). However, those techniques might not scale well to very large numbers of objectives and limited search budgets (a typical case in system testing). In this paper, we propose a novel algorithm, called Many Independent Objective (MIO) algorithm. This algorithm is designed and tailored based on the specific properties of test suite generation.Method: An empirical study was carried out for test suite generation on a series of artificial examples and seven RESTful API web services. The EvoMaster system test generation tool was used, where MIO, MOSA, WTS and random search were compared.Results: The presented MIO algorithm resulted having the best overall performance, but was not the best on all problems.Conclusion: The novel presented MIO algorithm is a step forward in the automation of test suite generation for system testing. However, there are still properties of system testing that can be exploited to achieve even better results.

Agent-Based Regression Test Case Generation using Class Diagram, Use cases and Activity Diagram

In Regression testing, test case generation is a process of generating test cases from the existing test suite to ensure that modifications made in the system have not affected its existing functionality. The use of modeling based UML class diagram identifies changes at syntax level, whereas use cases and activity diagram identify changes at semantic level. In this research, we have used the combination of UML class diagram, use cases and activity diagram to identify changes at both syntax and semantics level. We compared UML class diagrams, use cases and activity diagrams of old and modified code to identify these changes. It is found that the use of UML class diagram, use cases and activity diagram results in better identification of changes and hence leads to efficient test case generation. Additionally, agents developed in Java Agent Development Environment are used to collect these changes from different stake holders in the distributed environment. The distribution of testing tasks among mobile agents reduces the average time required for generation of test cases in regression testing.

Avoiding useless mutants

Mutation testing is a program-transformation technique that injects artificial bugs to check whether the existing test suite can detect them. However, the costs of using mutation testing are usually high, hindering its use in industry. Useless mutants (equivalent and duplicated) contribute to increase costs. Previous research has focused mainly on detecting useless mutants only after they are generated and compiled. In this paper, we introduce a strategy to help developers with deriving rules to avoid the generation of useless mutants. To use our strategy, we pass as input a set of programs. For each program, we also need a passing test suite and a set of mutants. As output, our strategy yields a set of useless mutants candidates. After manually confirming that the mutants classified by our strategy as "useless" are indeed useless, we derive rules that can avoid their generation and thus decrease costs. To the best of our knowledge, we introduce 37 new rules that can avoid useless mutants right before their generation. We then implement a subset of these rules in the MUJAVA mutation testing tool. Since our rules have been derived based on artificial and small Java programs, we take our MUJAVA version embedded with our rules and execute it in industrial-scale projects. Our rules reduced the number of mutants by almost 13% on average. Our results are promising because (i) we avoid useless mutants generation; (ii) our strategy can help with identifying more rules in case we set it to use more complex Java programs; and (iii) our MUJAVA version has only a subset of the rules we derived.

AN EFFICIENT APPROACH OF REGRESSION TESTING USING HIERARCHICAL DECOMPOSITION SLICING

The purpose of regression testing is to ensure that bug fixes and new functionality introduced in a new version of a software do not adversely affect the correct functionality inherited from the previous version. I have proposed a new slicing method to decompose a Java program into affected packages, classes, methods and statements identified with respect to the modification made in the program.Because java program has hierarchical characteristics(Package/Class/Method), this decomposition is based on this characteristic. I also have proposed an intermediate representation for Java programs that shows Package Level, Class Level, and Method Level Dependency. I have named this intermediate representation as Program Dependency Tree(PDT). This PDT is used to identify the program parts that are possibly affected by the change made to the program. With the help of PDT, we can obtain program slices. The packages, classes, methods, and statements thus affected are identified by traversing the Program Dependency Tree. The proposed approach maps the decomposed slice (comprising of the affected program parts) with the coverage information of the existing test suite to select the appropriate test cases for regression testing.

Combinatorial First Order Polynomial Coverage Based Test Suites Prioritization for Improving Software System Quality

Combinatorial Testing (CT) is performed to ensure the development of a software system quality. In CT, many research works has been designed for test suite minimization. But, existing test suite minimization techniques does not covers the more number of test cases for detecting the maximum faults in software programs. Therefore, there is a requirement for new test suite minimization technique for improving software system quality with higher number of test cases. To optimize the order of interactions being tested and to reduce the number of test suites generated best in terms of coverage, Combinatorial First Order Polynomial Coverage Based Prioritization (CFOP-CP) technique is proposed. For monitoring the order of interactions at time interval ‘ t ’ and time interval ‘ t-1 ’, First Order Polynomial (FOP) function is used. Therefore, FOP function predicts the faults interactions in software program efficiently which resulting in higher fault interaction prediction accuracy. After that, Coverage-based Test Suites Prioritization is performed to prioritize test suites best in terms of coverage and therefore provides global coverage for finding faults in software programs. Finally, Similarity-based Test Suite Selection is performed to reduce the number of test suites for detecting maximum number of faults in software programs which in turn helps for improving the software system quality. The CFOP-CP technique conducts the experimental works on parameters such as fault interaction prediction accuracy, testing cost and coverage rate. The experimental result shows that the CFOP-CP technique is able to improve the coverage rate for software fault detection and also reduces the testing cost for improving the software system quality when compared to state-of-the-art-works. Keywords: Combinatorial Testing, Test Suite, Fault Interaction, Coverage, Test Cases, Prioritization