ARUS: A Tool for Automatically Removing Unnecessary Stubbings from Test Suites

Software Testing is an important aspect of the real time software development process. Software testing always assures the quality of software product. As associated with software testing, there are few very important issues where there is a need to pay attention on it in the process of software development test. These issues are generation of effective test case and test suite as well as optimization of test case and suite while doing testing of software product. The important issue is that testing time of the test case and test suite. It is very much important that after development of software product effective testing should be performed. So to overcome these issues of optimization, we have proposed new approach for test suite optimization using genetic algorithm (GA). Genetic algorithm is evolutionary in nature so it is often used for optimization of problem by researcher. In this paper, our aim is to study various selections methods like tournament selection, rank selection and roulette wheel selection and then we apply this genetic algorithm (GA) on various programs which will generate optimized test suite with parameters like fitness value of test case, test suite and take minimum amount of time for execution after certain preset generation. In this paper our main objectives as per the experimental investigation, we show that tournament selection works very fine as compared to other methods with respect fitness selection of test case and test suites, testing time of test case and test suites as well as number of requirements.

Empirically evaluating Greedy-based test suite reduction methods at different levels of test suite complexity

Analysis on Test Case Assortment and Priortization Strategy

Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The reliability of software correlates with the quality of the test suite used to test it. Therefore, the quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system.This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Further, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes.The method compares test suites in a platform independent manner. The test suites, which are actually compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. Further, it allows us to use any arbitrary criterion for devising equivalence classes which makes this method viable in various situations.We exercise our method over a couple of case studies to demonstrate its viability and effectiveness. The effectiveness of the method is validated using mutation and coverage analyses.

Database Management System (DBMS) developers have implemented extensive test suites to test their DBMSs. For example, the SQLite test suites contain over 92 million lines of code. Despite these extensive efforts, test suites are not systematically reused across DBMSs, leading to wasted effort. Integration is challenging, as test suites use various test case formats and rely on unstandardized test runner features. We present a unified test suite, SQuaLity, in which we integrated test cases from three widely-used DBMSs, SQLite, PostgreSQL, and DuckDB. In addition, we present an empirical study to determine the potential of reusing these systems' test suites. Our results indicate that reusing test suites is challenging: First, test formats and test runner commands vary widely; for example, SQLite has 4 test runner commands, while MySQL has 112 commands with additional features, to, for example, execute file operations or interact with a shell. Second, while some test suites contain mostly standard-compliant statements (e.g., 99% in SQLite), other test suites mostly test non-standardized functionality (e.g., 31% of statements in the PostgreSQL test suite are nonstandardized). Third, test reuse is complicated by various explicit and implicit dependencies, such as the need to set variables and configurations, certain test cases requiring extensions not present by default, and query results depending on specific clients. Despite the above findings, we have identified 3 crashes, 3 hangs, and multiple compatibility issues across four different DBMSs by executing test suites across DBMSs, indicating the benefits of reuse. Overall, this work represents the first step towards test-case reuse in the context of DBMSs, and we hope that it will inspire follow-up work on this important topic.

The OpenWorm Project is an international open-source collaboration to create a multi-scale model of the organism Caenorhabditis elegans At each scale, including subcellular, cellular, network and behaviour, this project employs one or more computational models that aim to recapitulate the corresponding biological system at that scale. This requires that the simulated behaviour of each model be compared with experimental data both as the model is continuously refined and as new experimental data become available. Here we report the use of SciUnit, a software framework for model validation, to attempt to achieve these goals. During project development, each model is continuously subjected to data-driven 'unit tests' that quantitatively summarize model-data agreement, identifying modelling progress and highlighting particular aspects of each model that fail to adequately reproduce known features of the biological organism and its components. This workflow is publicly visible via both GitHub and a web application and accepts community contributions to ensure that modelling goals are transparent and well-informed.This article is part of a discussion meeting issue 'Connectome to behaviour: modelling C. elegans at cellular resolution'.

Continuous integration advocates to run the test suite of a project frequently, e.g., for every code change committed to a shared repository. This process imposes a high computational cost and sometimes also a high human cost, e.g., when developers must wait for the test suite to pass before a change appears in the main branch of the shared repository. However, only 4% of all test suite invocations turn a previously passing test suite into a failing test suite. The question arises whether running the test suite for each code change is really necessary. This paper presents continuous test suite failure prediction, which reduces the cost of continuous integration by predicting whether a particular code change should trigger the test suite at all. The core of the approach is a machine learning model based on features of the code change, the test suite, and the development history. We also present a theoretical cost model that describes when continuous test suite failure prediction is worthwhile. Evaluating the idea with 15k test suite runs from 242 open-source projects shows that the approach is effective at predicting whether running the test suite is likely to reveal a test failure. Moreover, we find that our approach improves the AUC over baselines that use features proposed for just-in-time defect prediction and test case failure prediction by 13.9% and 2.9%, respectively. Overall, continuous test suite failure prediction can significantly reduce the cost of continuous integration.

Test suites play an important role in developing reliable software applications. Generally, the behaviour of software applications is verified by executing test suites to find defects. The quality of a test suite needs to be evaluated and enriched (if needed) especially for testing critical systems, such as plane-navigation system. This paper presents a novel method for comparing concrete and executable test suites using equivalence classes. This comparison identifies gaps in test suites with respect to each other. These gaps indicate potential weaknesses in the test suites. Furthermore, this method provides a mechanism to enrich the test suites using these gaps. In this method, we devise equivalence classes, and associate each test case to an equivalence class. We, then, simulate the comparison of test suites by comparing sets of equivalence classes. The method compares test suites in a platform independent manner. The test suites, which are compared, are smaller than the original test suites because the redundant test cases are removed from the test suites, which makes it efficient. We exercise our method over three case studies to demonstrate its viability and effectiveness. The first case study illustrates the application of the method and evaluates its effectiveness using a mutation analysis. The second case study evaluates its effectiveness using mutation and coverage analyses. The final case study evaluates it on a real case study, which is Lucene search engine.

Many researchers have been focusing on building combinatorial test generators having the best possible performances, in terms of smaller test suites and shorter generation times. The majority of tools generates test suites from scratch. This means that when the test suite must be regenerated, the old tests are discarded and a new test suite is built. However, there are many cases in which old test cases, possibly written by hand, need to be (or could be) included in the final test suite, and the test suite completed with new tests in order to reach the desired level of combinatorial coverage. These existing tests that are reused are generally called seed tests. Seed tests could be important for testing domain-specific critical parts of the system, or they could represent old test suites that must be enriched to reach the desired (possibly higher) strength of coverage. In this paper, we propose a new architecture for incremental test generation that starts from existing test seeds. This new architecture is supported by the pMEDICI+ tool which extends our previous effort done for pMEDICI. We evaluate the proposed approach on the benchmarks given in the context of the second edition of the CT-Competition and w.r.t. two application scenarios. For each scenario, we automatically generate seed tests and then we apply pMEDICI+ to obtain the desired test suite. The experiments highlight that using incremental test generation can contribute significantly in the reduction of test generation time and, in many cases, in the reduction of the test suite size.

The availability of effective test suites is critical for the development and maintenance of reliable software systems. To increase test effectiveness, software developers tend to employ larger and larger test suites. The recent availability of software tools for automatic test generation makes building large test suites affordable, therefore contributing to accelerating this trend. However, large test suites, though more effective, are resources and time consuming and therefore cannot be executed frequently. Reducing them without decreasing code coverage is a needed compromise between efficiency and effectiveness of the test, hence enabling a more regular check of the software under development. We propose a novel approach, namely REDUNET, to reduce a test suite while keeping the same code coverage. We integrate this approach in a complete framework for the automatic generation of efficient and effective test suites, which includes test suite generation, code coverage analysis, and test suite reduction. Our approach formulates the test suite reduction as a set cover problem and applies integer linear programming and a network-based optimisation, which takes advantage of the properties of the control flow graph. We find the optimal set of test cases that keeps the same code coverage in fractions of seconds on real software projects and test suites generated automatically by Randoop. The results on ten real software systems show that the proposed approach finds the optimal minimisation and achieves up to 90% reduction and more than 50% reduction on all systems under analysis. On the largest project our reduction algorithm performs more than three times faster than both integer linear programming alone and the state-of-the-art heuristic Harrold Gupta Soffa.

Objectives: Software testing is the significant part of software development and is essential to confirm the quality of the software. The test suites developed for this purpose can be used again and updated repeatedly as the software advances. Subsequently, novel test cases will be added to the test suite and because of that, the size of the test suite will become bigger. Moreover, the test suite becomes redundant. Thus executing/re-executing the large test suite consumes more time and also increases the cost of testing. Therefore, in order to minimize the cost and the time of testing, it is essential to minimize the test suite. Methods/Statistical Analysis: Thus, the focus of this paper is to minimize the test suite by discovering a group of test cases that gives the same or better coverage as the original test suite based on some condition. Finding: In this study, the minimization is achieved by using a Gravitational Bee Search (GBS) algorithm, this algorithm is derived by combining artificial bee colony and gravitational search algorithms. Then, the Fuzzy operation is applied for prioritization to achieve efficient test suite. The algorithm searches for the optimum solution by calculating fitness values using coverage information. The search process is repetitive until a reduced test suite is identified. Application/ Improvement: The proposed algorithm is applied on an online ticket booking system and the results shows that the proposed system is approximately 15% to 20% more efficient than the existing system respective to the number of test cases and execution time.Keywords: Gravitational Bee Search, Software Testing, Test Suite Minimization, Test Coverage

Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be modified condition/decision coverage (MC/DC) adequate. Despite its cost, there is evidence that MC/DC is an effective verification technique and can help to uncover safety faults. As the software is modified and new test cases are added to the test suite, the test suite grows and the cost of regression testing increases. To address the test-suite size problem, researchers have investigated the use of test-suite reduction algorithms, which identify a reduced test suite that provides the same coverage of the software according to some criterion as the original test suite, and test-suite prioritization algorithms, which identify an ordering of the test cases in the test suite according to some criteria or goals. Existing test-suite reduction and prioritization techniques, however, may not be effective in reducing or prioritizing MC/DC-adequate test suites because they do not consider the complexity of the criterion. This paper presents new algorithms for test-suite reduction and prioritization that can be tailored effectively for use with MC/DC. The paper also presents the results of empirical studies of these algorithms.

Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be modified condition/decision coverage (MC/DC) adequate. Despite its cost, there is evidence that MC/DC is an effective verification technique, and can help to uncover safety faults. As the software is modified and new test cases are added to the test suite, the test suite grows, and the cost of regression testing increases. To address the test-suite size problem, researchers have investigated the use of test-suite reduction algorithms, which identify a reduced test suite that provides the same coverage of the software, according to some criterion, as the original test suite, and test-suite prioritization algorithms, which identify an ordering of the test cases in the test suite according to some criteria or goals. Existing test-suite reduction and prioritization techniques, however, may not be effective in reducing or prioritizing MC/DC-adequate test suites because they do not consider the complexity of the criterion. The paper presents new algorithms for test-suite reduction and prioritization that can be tailored effectively for use with MC/DC. The paper also presents the results of a case study of the test-suite reduction algorithm.

The software system is evolving over the time, thus, the test suite must be repaired according to the changing code. Updating test cases manually is a time-consuming activity, especially for large test suites, which motivate the recent development of automatically repairing test techniques. To develop an effective automatic repair technique that reduces the effort of development and the cost of evolution, the developer should understand how the test suite evolves in practice. This investigation aims to conduct a comprehensive empirical study on eight Java systems with many versions of these systems and their test suites to find out how the test suite is evolving, and to find the relationship between the change in the program and the corresponding evolution in the test suite. This study showed that the test suite size is mostly increased, where the test suite complexity is stabilized. The increase (or decrease) in the code size will mostly increase (or decrease) the test suite size. However, the increasing or decreasing in the code complexity is offset by stabilizing the test suite complexity. Moreover, the percentage of the code coverage tends to be increased more than decreased, but in the mutation coverage, the opposite is true.

Combinatorial Interaction Testing is a widely used method for testing intricate systems. In most cases, the test suites are generated from scratch. However, there are cases when testers may want to reuse existing tests, in order to include them in a new test suite, both for enhancing the performance of the generation process or because those tests are valuable for checking the functioning of the system under test in critical conditions. In this paper, we propose a general framework for dealing with existing test suites using combinatorial test generators. We also discuss the definition of partial tests and test suites, and the scenarios in which partial tests should or could be reused. Finally, we compare the most common tools for completing test suites, namely ACTS, PICT, and pMEDICI+, using different incompleteness levels in the seeds. ACTS with seeds generally performed the best in terms of test suite size and generation time. The other two tools, namely PICT and pMEDICI+, were slower and produced larger test suites on average. We have found that using seeds could sometimes come with a cost, especially in the scenario where test cases are partial and completing them is not always cost-effective in terms of generation time. The choice of re-using or throwing away existing tests must be based on use case-specific requirements. We do not recommend using seeds when they are composed of partial test cases, providing that they are not required for some other reason. On the contrary, we envision the use of partial test suites when a test suite with higher strength is needed.