Test Case Generation Problem Research Articles

Finding Cross-Rule Optimization Bugs in Datalog Engines

Datalog is a popular and widely-used declarative logic programming language. Datalog engines apply many cross-rule optimizations; bugs in them can cause incorrect results. To detect such optimization bugs, we propose an automated testing approach called Incremental Rule Evaluation (IRE), which synergistically tackles the test oracle and test case generation problem. The core idea behind the test oracle is to compare the results of an optimized program and a program without cross-rule optimization; any difference indicates a bug in the Datalog engine. Our core insight is that, for an optimized, incrementally-generated Datalog program, we can evaluate all rules individually by constructing a reference program to disable the optimizations that are performed among multiple rules. Incrementally generating test cases not only allows us to apply the test oracle for every new rule generated—we also can ensure that every newly added rule generates a non-empty result with a given probability and eschew recomputing already-known facts. We implemented IRE as a tool named Deopt, and evaluated Deopt on four mature Datalog engines, namely Soufflé, CozoDB, μZ, and DDlog, and discovered a total of 30 bugs. Of these, 13 were logic bugs, while the remaining were crash and error bugs. Deopt can detect all bugs found by queryFuzz, a state-of-the-art approach. Out of the bugs identified by Deopt, queryFuzz might be unable to detect 5. Our incremental test case generation approach is efficient; for example, for test cases containing 60 rules, our incremental approach can produce 1.17× (for DDlog) to 31.02× (for Soufflé) as many valid test cases with non-empty results as the naive random method. We believe that the simplicity and the generality of the approach will lead to its wide adoption in practice.

An Automated Path-Focused Test Case Generation with Dynamic Parameterization Using Adaptive Genetic Algorithm (AGA) for Structural Program Testing

Various software engineering paradigms and real-time projects have proved that software testing is the most critical and highly important phase in the SDLC. In general, software testing takes approximately 40–60% of the total effort and time involved in project development. Generating test cases is the most important process in software testing. There are many techniques involved in the automatic generation of these test cases which aim to find a smaller group of cases that could allow for an adequacy level to be achieved which will hence reduce the effort and cost involved in software testing. In the structural testing of a product, the auto-generation of test cases that are path focused in an efficient manner is a challenging process. These are often considered optimization problems and hence search-based methods such as genetic algorithm (GA) and swarm optimizations have been proposed to handle this issue. The significance of the study is to address the optimization problem of automatic test case generation in search-based software engineering. The proposed methodology aims to close the gap of genetic algorithms acquiring local minimum due to poor diversity. Here, dynamic adjustment of cross-over and mutation rate is achieved by calculating the individual measure of similarity and fitness and searching for the more global optimum. The proposed method is applied and experimented on a benchmark of five industrial projects. The results of the experiments have confirmed the efficiency of generating test cases that have optimum path coverage.

MTGAN: Extending Test Case set for Deep Learning Image Classifier

During the recent several years, deep learning has achieved excellent results in image recognition, voice processing, and other research areas, which has set off a new upsurge of research and application. Internal defects and external malicious attacks may threaten the safe and reliable operation of a deep learning system and even cause unbearable consequences. The technology of testing deep learning systems is still in its infancy. Traditional software testing technology is not applicable to test deep learning systems. In addition, the characteristics of deep learning such as complex application scenarios, the high dimensionality of input data, and poor interpretability of operation logic bring new challenges to the testing work. This paper focuses on the problem of test case generation and points out that adversarial examples can be used as test cases. Then the paper proposes MTGAN which is a framework to generate test cases for deep learning image classifiers based on Generative Adversarial Network. Finally, this paper evaluates the effectiveness of MTGAN.

A Critical Review on Automated Test Case Generation for Conducting Combinatorial Testing Using Particle Swarm Optimization

In software development life cycle, testing plays the significant role to verify requirement specification, analysis, design, coding and to estimate the reliability of software system. A test manager can write a set of test cases manually for the smaller software systems. However, for the extensive software system, normally the size of test suite is large, and the test suite is prone to an error committed like omissions of important test cases, duplication of some test cases and contradicting test cases etc. When test cases are generated automatically by a tool in an intelligent way, test errors can be eliminated. In addition, it is even possible to reduce the size of test suite and thereby to decrease the cost & time of software testing.It is a challenging job to reduce test suite size. When there are interacting inputs of Software under Test (SUT), combinatorial testing is highly essential to ensure higher reliability from 72 % to 91 % or even more than that. A meta-heuristic algorithm like Particle Swarm Optimization (PSO) solves optimization problem of automated combinatorial test case generation. Many authors have contributed in the field of combinatorial test case generation using PSO algorithms.We have reviewed some important research papers on automated test case generation for combinatorial testing using PSO. This paper provides a critical review of use of PSO and its variants for solving the classical optimization problem of automatic test case generation for conducting combinatorial testing.

Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets

The test case generation is intrinsically a multi-objective problem, since the goal is covering multiple test targets (e.g., branches). Existing search-based approaches either consider one target at a time or aggregate all targets into a single fitness function (whole-suite approach). Multi and many-objective optimisation algorithms (MOAs) have never been applied to this problem, because existing algorithms do not scale to the number of coverage objectives that are typically found in real-world software. In addition, the final goal for MOAs is to find alternative trade-off solutions in the objective space, while in test generation the interesting solutions are only those test cases covering one or more uncovered targets. In this paper, we present Dynamic Many-Objective Sorting Algorithm (DynaMOSA), a novel many-objective solver specifically designed to address the test case generation problem in the context of coverage testing. DynaMOSA extends our previous many-objective technique Many-Objective Sorting Algorithm (MOSA) with dynamic selection of the coverage targets based on the control dependency hierarchy. Such extension makes the approach more effective and efficient in case of limited search budget. We carried out an empirical study on 346 Java classes using three coverage criteria (i.e., statement, branch, and strong mutation coverage) to assess the performance of DynaMOSA with respect to the whole-suite approach (WS), its archive-based variant (WSA) and MOSA. The results show that DynaMOSA outperforms WSA in 28 percent of the classes for branch coverage (+8 percent more coverage on average) and in 27 percent of the classes for mutation coverage (+11 percent more killed mutants on average). It outperforms WS in 51 percent of the classes for statement coverage, leading to +11 percent more coverage on average. Moreover, DynaMOSA outperforms its predecessor MOSA for all the three coverage criteria in 19 percent of the classes with +8 percent more code coverage on average.

Theoretical analysis of dynamic selection of switching auxiliary objectives on XdivK problem

Subject of Research. The paper deals with analysis of the EA+RL method inefficiency reasons on XdivK optimization problem with switching auxiliary objectives. It is proposed to modify the EA+RL method. The EA+RL method increases efficiency of an evolutionary algorithm by introducing auxiliary objectives. The XdivK problem is characterized by a large number of local optima. Switching objectives help to escape from local optima on some stages of optimization while being obstructive on the other stages. Method. To perform theoretical analysis of the EA+RL method and its proposed modification, the corresponding optimization process was modeled by Markov chains. The number of fitness function evaluations needed to reach the optimum was estimated based on the analysis of transition probabilities. Main Results. The EA+RL method and its proposed modification were theoretically analyzed on the XdivK problem with switching auxiliary objectives. It was proved that the proposed modification ignores obstructive objectives contrary to the EA+RL method. The lower and upper bounds on the running time of the proposed modification were obtained. Practical Relevance. The proposed modification increases the efficiency of EA+RL method, successfully used to solve NP-hard optimization problems, such as the test case generation problem.

Simulink model testing method based on program mutation

In order to solve the current problems( expensive testing cost and low efficiency of test case generation) in mutation test for Simulink models,a mutation testing process and an optimized set of mutation operators were proposed for Simulink models based on program mutation according to the current mutation operators for the Simulink models. Experiments show that this set of mutation operators can effectively reduce the generation number of mutation models without prejudice to the mutation score of testing case set,thus it will effectively save the testing cost. Then a search-based test case generation method for Simulink models mutation testing was described. The test case generation problem was transformed into the objective function minimization problem,and the test cases which can kill the mutation models were ultimately obtained through the optimization of objective function by algorithm of simulated annealing. Finally,the application of a typical case for the method verified the correctness and effectiveness.

Using genetic algorithms to generate test sequences for complex timed systems

The generation of test data for state-based specifications is a computationally expensive process. This problem is magnified if we consider that time constraints have to be taken into account to govern the transitions of the studied system. The main goal of this paper is to introduce a complete methodology, supported by tools, that addresses this issue by representing the test data generation problem as an optimization problem. We use heuristics to generate test cases. In order to assess the suitability of our approach we consider two different case studies: a communication protocol and the scientific application BIPS3D. We give details concerning how the test case generation problem can be presented as a search problem and automated. Genetic algorithms (GAs) and random search are used to generate test data and evaluate the approach. GAs outperform random search and seem to scale well as the problem size increases. It is worth to mention that we use a very simple fitness function that can be easily adapted to be used with other evolutionary search techniques.

Efficient coverage of parallel and hierarchical stateflow models for test case generation

SUMMARYThis paper is concerned with test case generation from Simulink/Stateflow (SL/SF) models with a focus on coverage of SF model elements. Coverage of the SF component in a model is a difficult task because of two primary reasons: (i) the SF component itself may lie deep in the SL/SF model in which case, inputs have to pass through a complex chain of SL blocks to reach the SF block and (ii) nonlinear constraints in the model are difficult to solve using constraint solvers. Hierarchy and parallelism in the SF model add further complexity to the problem. The existing approaches flatten such SF elements, and generate test cases from the flattened finite state machines. Handling of issues (i) and (ii) has already been discussed in earlier research. In this paper, we present a method of covering SF components, which does not require to flatten any hierarchy or parallelism in the components. This not only makes the test case generation problem efficient but also addresses the problem of scalability. We have implemented this method and performed a number of medium‐sized case studies. The results show improved performance over the results obtained by some commercial tools. Copyright © 2011 John Wiley & Sons, Ltd.

Combinatorial Testing: Principles and Methods

PDF HTML阅读 XML下载 导出引用 引用提醒 组合测试:原理与方法 DOI: 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: Supported by the National Natural Science Foundation of China under Grant Nos.60673044, 60633010 (国家自然科学基金) Combinatorial Testing: Principles and Methods Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:组合测试能够在保证错误检出率的前提下采用较少的测试用例测试系统.但是,组合测试用例集的构造问题的复杂度是NP完全的.组合测试方法的有效性和复杂性吸引了组合数学领域和软件工程领域的学者们对其进行深入的研究.总结了近年来在组合测试方面的研究进展,主要内容包括:组合测试准则的研究、组合测试生成问题与其他NP完全问题的联系、组合测试用例的数学构造方法、采用计算机搜索的组合测试生成方法以及基于组合测试的错误定位技术. Abstract:Combinatorial testing can use a small number of test cases to test systems while preserving fault detection ability. However, the complexity of test case generation problem for combinatorial testing is NP-complete. The efficiency and complexity of this testing method have attracted many researchers from the area of combinatorics and software engineering. This paper summarizes the research works on this topic in recent years. They include: various combinatorial test criteria, the relations between the test generation problem and other NP-complete problems, the mathematical methods for constructing test cases, the computer search techniques for test generation and fault localization techniques based on combinatorial testing. 参考文献 相似文献 引证文献

Enhancing Test Coverage by Back-tracing Model-checker Counterexamples

The automatic detection of unreachable coverage goals and generation of tests for "corner-case" scenarios is crucial to make testing and simulation based verification more effective. In this paper we address the problem of coverability analysis and test case generation in modular and component based systems. We propose a technique that, given an uncovered branch in a component, either establishes that the branch cannot be covered or produces a test case at the system level which covers the branch. The technique is based on the use of counterexamples returned by model checkers, and exploits compositionality to cope with large state spaces typical of real applications.

Automated technology mapping

Logic “mapping,” or “transformation,” refers to the process of converting a logic design from one form of specification to another. The output is usually a specific technology implementation and the input could range from a previous technology implementation to a high-level design language. Motivated initially by the problem of test case generation for new technologies, a logic transformation system, known as the Technology Mapping System (TMS), was developed. This system has focused on the problem of technology-to-technology mapping involving gate array or standard cell logic families. TMS makes use of an intermediate notation, called GLN, and uses several forms of “rules” to control the mapping process. This paper discusses the history and general operation of TMS, and makes a comparison of transformations from different types of sources.