Test Case Generation Algorithm Research Articles

Knowledge transfer based many-objective approach for finding bugs in multi-path loops

AbstractGenerating test cases is essential for discovering software bugs. However, finding bugs in multi-path loops is challenging, especially when bugs can only be exposed after a specific interleaving execution order. Furthermore, the number of paths increases exponentially when the iteration count increases. It results in running out of resources (e.g., allocated search budget) in existing algorithms for test case generation before finding program bugs. To overcome these challenges and improve the effectiveness of existing algorithms in finding bugs among multi-path loops, we propose a path-oriented test case generation framework to explore loop paths of bounded depth. To fulfill this task, we first present a method that groups loop paths based on their latent similarities. Then, we formulate the problem of generating test cases to cover loop paths as a many-objective optimization one. The many-objective optimization problem is designed to optimize one group of loop paths at a time. Finally, a many-objective test case generation framework is used to generate test cases for covering loop paths. In this framework, a knowledge transfer scheme is incorporated to reuse suitable archived test cases at the start of the search process for a similar group of paths. We have implemented the proposed knowledge transfer scheme with generalized differential evolution 3 (KT-GDE3) and evaluated it with benchmarks from the competition on software verification (SV-COMP16 and SV-COMP21). The experimental results show that KT-GDE3 outperforms alternative search-based test case generation algorithms in bug finding capability for multi-path loops. To evaluate the effectiveness of the proposed knowledge transfer scheme, our approach is compared with the variant without this scheme (called KT-GDE3-unarchived) in the benchmarks. The corresponding experimental results verify our assumption that the knowledge transfer scheme can significantly enhance the optimization process of multi-objective algorithms such as generalized differential evolution when finding bugs in multi-path loops.

Test Case Generation Evaluator for the Implementation of Test Case Generation Algorithms Based on Learning to Rank

Test Case Generation Evaluator for the Implementation of Test Case Generation Algorithms Based on Learning to Rank

Cost-effective test case generation with the hyper-heuristic for software product line testing

Implementation of optimization algorithm for test case generation in Model-Based Testing (MBT) for Software Product Line (SPL) has been increasing, due to the demand for optimal test case results with a balanced trade-off between cost and effectiveness measure. This paper proposed a hyper-heuristic test cases generation approach in MBT for SPL called Improvement Selection Rules-Modified Choice Function (ISR-MCF). ISR-MCF is implemented with three search operators which are Non-Dominated Sorting Genetic Algorithm II with low-level heuristic (NSGA-II-LLH), Strength Pareto Evolutionary with Low-Level Heuristic (SPEA 2-LLH) and Particle Swarm Optimization with Low-Level Heuristic (PSO-LLH). The approach was evaluated with a test model and the result shows that the proposed ISR-MCF with NSGA-II-LLH outperforms other existing rules for minimization measure (size of a test suite and execution time and maximization measure (coverage criteria).

Coverage-Guided Testing for Recurrent Neural Networks

Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">testRNN</small> , which is then evaluated on a set of LSTM benchmarks. Experiments confirm that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">testRNN</small> has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">testRNN</small> enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing.

Performance of Enhanced Multiple-Searching Genetic Algorithm for Test Case Generation in Software Testing

Test case generation is an important process in software testing. However, manual generation of test cases is a time-consuming process. Automation can considerably reduce the time required to create adequate test cases for software testing. Genetic algorithms (GAs) are considered to be effective in this regard. The multiple-searching genetic algorithm (MSGA) uses a modified version of the GA to solve the multicast routing problem in network systems. MSGA can be improved to make it suitable for generating test cases. In this paper, a new algorithm called the enhanced multiple-searching genetic algorithm (EMSGA), which involves a few additional processes for selecting the best chromosomes in the GA process, is proposed. The performance of EMSGA was evaluated through comparison with seven different search-based techniques, including random search. All algorithms were implemented in EvoSuite, which is a tool for automatic generation of test cases. The experimental results showed that EMSGA increased the efficiency of testing when compared with conventional algorithms and could detect more faults. Because of its superior performance compared with that of existing algorithms, EMSGA can enable seamless automation of software testing, thereby facilitating the development of different software packages.

Proposal of Iterative Genetic Algorithm for Test Suite Generation

Huge and reputed software industries are expected to deliver quality products. However, industry suffers from a loss of approximately $500 billion due to shoddy software quality. The quality of the product in terms of its accuracy, efficiency, and reliability can be revamped through testing by focusing attention on testing the product through effective test case generation and prioritization. The authors have proposed a test-case generation technique based on iterative listener genetic algorithm that generates test cases automatically. The proposed technique uses its adaptive nature and solves the issues like redundant test cases, inefficient test coverage percentage, high execution time, and increased computation complexity by maintaining the diversity of the population which will decrease the redundancy in test cases. The performance of the technique is compared with four existing test-case generation algorithms in terms of computational complexity, execution time, coverage, and it is observed that the proposed technique outperformed.

Automatic Test Case and Test Oracle Generation Based on Functional Scenarios in Formal Specifications for Conformance Testing

Testing a program to confirm whether it consistently implements its requirements specification is a necessary but time-consuming activity in software development. Automatic testing based on specifications can significantly alleviate the workload and cost, but faces a challenge of how to ensure that both the user’s concerns in the specification and possible execution paths in the program are all covered. In this paper, we describe a new method, called “Vibration-Method” or simply “V-Method”, for automatic generation of test cases and test oracle from model-based formal specifications, aiming to address this challenge. The proposed method is suitable for testing information systems in which rich data types are used. Supporting the principle of “divide and conquer”, the method provides a specific technique for generating test cases based on functional scenarios defined in the specification, test case generation criteria, automatic test case generation algorithms, and a well-defined mechanism for deriving test oracle. We elaborate on the method by discussing how initial test cases can be automatically generated, how additional necessary test cases are produced using the “vibration” technique, and how a test oracle can be automatically derived for a group of test cases. We also describe a controlled experiment to evaluate the effectiveness of the method and discuss the important issues in relation to the performance and applicability of the method.

Non-Deterministic Delay Behavior Testing of Chinese Train Control System Using UPPAAL-TRON

The Chinese Train Control System level 3 (CTCS-3) is an open and real-time safety-critical system. Due to the non-deterministic delay behavior in the environment, there is a large number of stimuli to the train control system with different patterns of arrival times. Control strategies must be well considered, as either the logic of the function fails or the real-time constraints are dissatisfied, may directly lead to significant human injury or financial loss. In this paper, we introduce the notion of test specification and the relativized timed input/output conformance based on timed automata theory. A new test case generation and execution algorithm has been proposed, by which the tester can reset Implementation Under Test (IUT) whenever they want, make it more sense in specific domain fields. We apply UPPAAL-TRON based online conformance testing framework to test nondeterministic delay behavior of Radio Block Center (RBC) handover scenario of CTCS-3 against critical safety, real-time, and liveness properties (defined in the system requirement specification (SRS) documents). Our experiments show that assurance of collision avoidance and train non-derailment can be guaranteed, but emergency brake intervention may happen with low probability in this scenario under the requirements specified in the SRS document.

A Systematic Study on Factors Impacting GUI Traversal-Based Test Case Generation Techniques for Android Applications

Many test case generation algorithms have been proposed to test Android apps through their graphical user interfaces. However, no systematic study on the impact of the core design elements in these algorithms on effectiveness and efficiency has been reported. This paper presents the first controlled experiment to examine three key design factors, each of which is popularly used in GUI traversal-based test case generation techniques. These three major factors are definition of GUI state equivalence, state search strategy, and waiting time strategy in between input events. The empirical results on 33 Android apps with real faults revealed interesting results. First, different choices of GUI state equivalence led to significant difference on failure detection rate and extent of code coverage. Second, searching the GUI state hierarchy randomly is as effective as searching it systematically. Last but not the least, the choices on when to fire the next input event to the app under test is immaterial so long as the length of the test session is practically long enough such as 1 h. We also found two new GUI state equivalence definitions that are statistically as effective as the existing best strategy for GUI state equivalence.

Prioritized Process Test: An Alternative to Current Process Testing Strategies

Testing processes and workflows in information and Internet of Things systems is a major part of the typical software testing effort. Consistent and efficient path-based test cases are desired to support these tests. Because certain parts of software system workflows have a higher business priority than others, this fact has to be involved in the generation of test cases. In this paper, we propose a Prioritized Process Test (PPT), which is a model-based test case generation algorithm that represents an alternative to currently established algorithms that use directed graphs and test requirements to model the system under test. The PPT accepts a directed multigraph as a model to express priorities, and edge weights are used instead of test requirements. To determine the test-coverage level of test cases, a test-depth-level concept is used. We compared the presented PPT with five alternatives (i.e. the Process Cycle Test (PCT), a naive reduction of test set created by the PCT, Brute Force algorithm, Set-covering-Based Solution and Matching-based Prefix Graph Solution) for edge coverage and edge-pair coverage. To assess the optimality of the path-based test cases produced by these strategies, we used 14 metrics based on the properties of these test cases and 59 models that were created for three real-world systems. For all edge coverage, the PPT produced more optimal test cases than the alternatives in terms of the majority of the metrics. For edge-pair coverage, the PPT strategy yielded similar results to those of the alternatives. Thus, the PPT strategy is an applicable alternative as it reflects both the required test coverage level and the business priority in parallel.

Hyper-Heuristic Based Strategy for Pairwise Test Case Generation

Pairwise testing significantly reduces testing efforts of contemporary software systems by efficiently sampling their exorbitant number of parameter configurations. Meta-heuristic based pairwise test generation strategies appeared effective in the recent literature for pairwise testing. However, meta-heuristics require substantial information of the problem domain before producing optimal results. As alternative to meta-heuristics, hyper-heuristics have been introduced. Hyper-heuristics promotes generality by using a high-level heuristic as chief selector from a set of low-level heuristics. The suitability of hyper-heuristics for optimization problems motivated us to adopt the Exponential Monte Carlo hyper-heuristic as a basis for our proposed pairwise test case generation strategy called Pairwise_HHH. Based on the published benchmarking results, Pairwise_HHH gives competitive results in many of the parameter configurations considered. Pairwise_HHH serves as our research vehicle to investigate the effective use of hyper-heuristic based algorithm for pairwise test case generation.

A hybrid approach to testing for nonfunctional faults in embedded systems using genetic algorithms

SummaryEmbedded systems are challenging to program correctly, because they use an interrupt‐driven programming paradigm and run in resource‐constrained environments. This leads to various classes of nonfunctional faults that can be detected only by customized verification techniques. These nonfunctional faults are specifically related to usage of resources such as time and memory. For example, the presence of interrupts can induce delays in interrupt servicing and in system execution time. Such delays can occur when multiple interrupt service routines and interrupts of different priorities compete for resources on a given CPU. As another example, stack overflows are caused when the combination of active methods and interrupt invocations on the stack grows too large, and these can lead to data loss and other significant device failures. To detect these types of nonfunctional faults, developers need to estimate worst‐case resource usage. Most existing approaches for calculating such estimates are based on static analysis; however, these have a tendency to overapproximate the resources needed. Dynamic techniques such as random testing, in contrast, often underapproximate resource usage. In this article, we present SimEspresso, a framework that uses a combination of static analysis and a test case generation algorithm to estimate worst‐case resource usage. There are three different worst‐case resource usage scenarios that we consider: (1) worst‐case execution times, (2) worst‐case interrupt latencies, and (3) worst‐case stack usage. SimEspresso first uses static analysis to identify program paths and interrupt interleavings that potentially lead to worst‐case scenarios. It then uses a genetic algorithm to generate test cases that guide program execution down these paths, using these particular interrupt interleavings. We performed an empirical study to evaluate the effectiveness of SimEspresso; our results show that SimEspresso is more effective than static analysis approaches and improves significantly over the state of the art dynamic technique, random test case generation. We also find that when we use only the genetic algorithm, omitting the static analysis, SimEspresso performs almost as effectively, but takes significantly longer to complete its task.

Controlling Occurrence Frequencies of Parameter Values in Pair-Wise Testing

Pair-wise testing is a widely used strategy of software testing. It requires testing every pair of parameter values at least once. This paper focuses weighting of parameter values for this testing strategy. Weighting is an added feature which allows the tester to prioritize different parameter values by specifying their desired frequency of occurrence in a test suite. This feature is desirable as it allows the tester to have more control over the resulting test suite. However, there has been not much research on weighting: to our knowledge, all existing weighting methods treat weights as a second class requirement and cannot generate a test suite that sufficiently respects the given weights. Aiming to overcome this problem, this paper proposes a weighting method which can be used in combination of any one-test-at-a-time greedy test case generation algorithm. By comparing the parameter value distribution in the current test suite and the ideal one specified by the given weights, the method generates each test case so that the resulting test suite can reflect the weights as accurately as possible. The usefulness of the method is demonstrated through empirical results.

Test Scenario Design for Intelligent Driving System Ensuring Coverage and Effectiveness

Intelligent vehicle greatly benefits traffic safety, efficiency and driving comfortable. With the development of intelligent driving technology and its application, it is becoming increasingly important to do effective and comprehensive tests before putting on the market. Comprehensively considering the cost of test, an automatic generation method of test scenarios is proposed to ensure both coverage and effectiveness. Based on the analyzed key infuence factors of an intelligent driving system, the analytic hierarchy process (AHP) is used to determine their importance and accordingly an complex index is defined, based on which an improved test case generation algorithm based on the pairwise independent combinatorial testing tool (PICT) is proposed to ensuring both combinational coverage and complexity of test cases. Finally, the test scenario is generated by clustering these discrete test cases considering similarity and complexity. The high complex test cases are preferred to be combined together and conducted preferentially to increase the test efficiency. The effectiveness of this method is validated by applying it on a lane departure warning system (LDW).

Проведение итеративного динамического анализа приложений, предоставляющих графический интерфейс пользователя

This paper is dedicated to practical research in the field of automated testing and analysis of software that features a graphical user interface. Current tendencies in user interface development favour semi-automatic approaches that employ human experts to create and prepare test suites. The ever-increasing complexity of software leads to decreased effectiveness of these approaches, especially when one considers large amounts of computational resources available during development. We present a fully automatic approach to dynamic analysis of program graphical interfaces. Our approach is based on the open-source GUITAR tool, which we have identified among other industrial and academic tools as the one closest to full automation. While highly efficient, GUITAR nevertheless has certain drawbacks and limitations which might cause insufficient accuracy in modelling the graphical interface structure and its individual elements. In turn, these limitations lead to fragments of graphical interface not getting processed during analysis or cause incorrect (i.e. not reproducible in practice) test cases to be generated. Our contributions include a set of modifications: incremental graphical interface model generation, improved identification of graphical interface element attributes and side effects, and finally a test case generation algorithm that focuses on reaching unprocessed graphical interface elements to check their functionality and improve the model. We have tested our modifications on a set of open source projects originally checked by GUITAR developers and achieved positive results: increased precision of GUI structure model and theoretically can decrease number of inapplicable test cases. Finally, we discuss several potential improvements for future work, including, in particular, the use of dynamic symbolic execution methods.

CenUMs – concurrency enhanced usage models for statistical testing of complex systems with concurrent streams of use

Model-based statistical testing with Markov chain usage model (MCUMs) represents a highly automated test approach. It also boosts test planning by providing relevant parameters that aid decision making. However, MCUMs are not suitable for testing modern systems that allow concurrent streams of use. Concurrent aspects result in a state space explosion in the model. They complicate the model specification and increase the calculation time for parameters. Therefore, we introduce concurrency enhanced usage model (CenUMs). They represent a natural extension to MCUMs by composite states with regions, which are used to formalize concurrent streams of use. This allows a simplified and reasonable specification as it hides the state space explosion. We also provide configurable algorithms for test case generation that take composite states into account. Additionally, a new analysis method is outlined that leads to a significant improvement of the calculation time for parameters. An analysis of its arithmetic complexity and results from experiments are provided.

Genetic Algorithm Based Test Data Generation for Structured Software Testing

Conventional software testing is time consuming and expensive task. The techniques for automated test case generation of software is very important as it can reduce the time and cost of this process. Search based test data generation received lot of attention in last two decades for solving this problem. The well known metaheuristic search technique genetic algorithm is used to obtain a specific coverage in software testing. This technique automatically generates test data in order to obtain branch coverage in software testing. The authors first use a technique to identify the unique paths from a control flow graph and then apply genetic algorithm for test case generation. This paper defines the basic research issues of genetic algorithm like chromosome, fitness function, crossing over, mutation in this problem domain and uses a case study to illustrate it.

Empirical Validation of Test Case Generation based on All-edge Coverage Criteria

Software testing assesses the functionality and correctness of the software through analysis and execution. It is done by exercising appropriate number of test cases so that no part of the program is left untested. Presence of multiple loops in a program makes it unlikely or impossible to test all paths. Therefore researchers try to find the subsets of the test cases, which when tested give confidence of complete testing. However, the subsets of paths are based on some testing criteria. In this research paper GA approach has been used to find out the subset of paths of the test program that fulfills all edge coverage criteria. The Genetic Algorithm for Test Case Generation (GATCG) proposed in this work generates reduced number of paths for a test program. These paths are termed as prime paths. The proposed GATCG technique makes use of the concept of prime paths to reduce the cost of testing. The efficiency of proposed algorithm is established from the results, in terms of number of iterations and time consumed in generating the prime paths for test programs.

An approach of security testing for third‐party component based on state mutation

ABSTRACTIt is essential to study an effective approach of security testing for third‐party component. In this paper, to effectively trigger implicit vulnerabilities of third‐party components, an approach of security testing for third‐party component is proposed based on state mutation. To start with, executable method sequences of components are transformed into extended finite state machine. Then, according to characteristics of condition conflict and behavior conflict, two test case generation algorithms are addressed, that is, Operations Conflict Sequences Generation Algorithm and Conditions Conflict Sequences Generation Algorithm, which are designed to generate inaccessible sequences of behavior and condition conflicts. These conflict sequences are run. Furthermore, the security detecting algorithms are addressed to detect implicit vulnerabilities of third‐party components, and then, testing report of component security is obtained. In the end, some experiments are conducted on the basis of the proposed approach, and the experimental results show that the proposed approach can effectively detect security exceptions of third‐party components. Copyright © 2015 John Wiley &amp; Sons, Ltd.

A Novel Testing Model for SOA based Services

SOA (Service-Oriented Architecture) filled the gap between software and commercial enterprise. SOA integrates multiple web services. We bear to secure the caliber of web services that gives guarantee about what network services work and their output results. There is close to work has to be performed for an automatic test case generation for SOA based services. But, full coverage of XML elements is missing. To the best of our knowledge this all works do not attempt to cover all possible elements of the XML schema presents in the WSDL file. There is also a need to apply different assertions on each service operation for generating the test cases. To overcome this problem we proposed a novel testing model for SOA based application. This new testing model helps us to get the automatic test cases of SOA based application. We build up our new test model with the aid of our proposed test case generation algorithm and test case selection algorithm. In the end, we generate the test suite execution results and find the coverage of XML schema elements present in the WSDL file.