Automated Test Case Generation Tools Research Articles

On introducing automatic test case generation in practice: A success story and lessons learned

The level and quality of automation dramatically affects software testing activities, determines costs and effectiveness of the testing process, and largely impacts on the quality of the final product. While costs and benefits of automating many testing activities in industrial practice (including managing the quality process, executing large test suites, and managing regression test suites) are well understood and documented, the benefits and obstacles of automatically generating system test suites in industrial practice are not well reported yet, despite the recent progresses of automated test case generation tools. Proprietary tools for automatically generating test cases are becoming common practice in large software organizations, and commercial tools are becoming available for some application domains and testing levels. However, generating system test cases in small and medium-size software companies is still largely a manual, inefficient and ad-hoc activity.This paper reports our experience in introducing techniques for automatically generating system test suites in a medium-size company. We describe the technical and organizational obstacles that we faced when introducing automatic test case generation in the development process of the company, and present the solutions that we successfully experienced in that context. In particular, the paper discusses the problems of automating the generation of test cases by referring to a customized ERP application that the medium-size company developed for a third party multinational company, and presents ABT2.0, the test case generator that we developed by tailoring ABT, a research state-of-the-art GUI test generator, to their industrial environment. This paper presents the new features of ABT2.0, and discusses how these new features address the issues that we faced.

Scented since the beginning: On the diffuseness of test smells in automatically generated test code

Software testing represents a key software engineering practice to ensure source code quality and reliability. To support developers in this activity and reduce testing effort, several automated unit test generation tools have been proposed. Most of these approaches have the main goal of covering as more branches as possible. While these approaches have good performance, little is still known on the maintainability of the test code they produce, i.e.,whether the generated tests have a good code quality and if they do not possibly introduce issues threatening their effectiveness. To bridge this gap, in this paper we study to what extent existing automated test case generation tools produce potentially problematic test code. We consider seven test smells, i.e.,suboptimal design choices applied by programmers during the development of test cases, as measure of code quality of the generated tests, and evaluate their diffuseness in the unit test classes automatically generated by three state-of-the-art tools such as Randoop, JTExpert, and Evosuite. Moreover, we investigate whether there are characteristics of test and production code influencing the generation of smelly tests. Our study shows that all the considered tools tend to generate a high quantity of two specific test smell types, i.e.,Assertion Roulette and Eager Test, which are those that previous studies showed to negatively impact the reliability of production code. We also discover that test size is correlated with the generation of smelly tests. Based on our findings, we argue that more effective automated generation algorithms that explicitly take into account test code quality should be further investigated and devised.

Automated test case generation tool based on Pycparser

Automated test case generation tool based on Pycparser

An optimization approach for automated unit test generation tools using multi-objective evolutionary algorithms

High code coverage is measured by the process of software testing typically using automatic test case generation tools. This standard approach is usually used for unit testing to improve software reliability. Most automated test case generation tools focused just on code coverage without considering its cost and redundancy between generated test cases. To obtain optimized high code coverage and to ensure minimum cost and redundancy a Multi-Objectives Evolutionary Algorithm approach (MOEA) is set in motion. An efficient approach is proposed and applied to different algorithms from MOEA Frame from the separate library with three fitness functions for Coverage, Cost, and Redundancy. Four MEOA algorithms have been proven reliable to reach above the 90 percent code coverage: NSGAII, Random, SMSEMOA,v and ε-MOEA. These four algorithms are the key factors behind the MOEA approach.

Achieving scalable mutation-based generation of whole test suites

Without complete formal specification, automatically generated software tests need to be manually checked in order to detect faults. This makes it desirable to produce the strongest possible test set while keeping the number of tests as small as possible. As commonly applied coverage criteria like branch coverage are potentially weak, mutation testing has been proposed as a stronger criterion. However, mutation based test generation is hampered because usually there are simply too many mutants, and too many of these are either trivially killed or equivalent. On such mutants, any effort spent on test generation would per definition be wasted. To overcome this problem, our search-based EvoSuite test generation tool integrates two novel optimizations: First, we avoid redundant test executions on mutants by monitoring state infection conditions, and second we use whole test suite generation to optimize test suites towards killing the highest number of mutants, rather than selecting individual mutants. These optimizations allowed us to apply EvoSuite to a random sample of 100 open source projects, consisting of a total of 8,963 classes and more than two million lines of code, leading to a total of 1,380,302 mutants. The experiment demonstrates that our approach scales well, making mutation testing a viable test criterion for automated test case generation tools, and allowing us to analyze the relationship of branch coverage and mutation testing in detail.

Automated requirements-based test case generation

Black-box testing is a technique in which test cases are derived from requirements without regard to the internal structure of the implementation. In current practice, the black-box test cases are derived manually from requirements. Manually deriving test cases from requirements is a costly and time consuming process. In this paper, we present the notion of autogenerating black-box test cases from requirements, that can result in dramatic time and cost savings.To accomplish this, we use requirements formalized as temporal logic properties. We define coverage metrics directly on the structure of the formalized requirements, and use an automated test case generation tool, like the model checker, to generate test cases from formal requirements that satisfy the desired criteria. To evaluate the effectiveness of black-box test suites generated in this manner, we measure the implementation coverage achieved by the test suites, and their fault-finding effectiveness. In [11], we conducted a preliminary investigation using a close to production model of a Flight Guidance System developed at Rockwell Collins Inc. We autogenerated requirements-based test suites for three different requirements coverage criteria and evaluated them by measuring the implementation coverage achieved.

BEAT

BEAT is a Web-based system that generates fault-based test cases from Boolean expressions. It is based on the integration of our several fault-based test case selection strategies. The generated test cases are considered to be fault-based, because they are aiming at the detection of particular faults. For example, when the Boolean expression is in irredundant disjunctive normal form, test cases generated by BEAT can guarantee the detection of seven common types of operator and operand faults, which may occur during the implementation process. Apart from being an automated test case generation tool developed for software testing practitioners, it also can be used as a self-learning and training tool for students as well as software testing practitioners.