Automatic Test Case Generation Research Articles

Multi‐objective optimization‐based and fault localization‐oriented test case generation for novice programs

SummaryOnline judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum‐based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high‐quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program‐oriented Multi‐Objective Optimization‐Based Fault Localization‐Oriented Test Case Generation (MFTCG) for automatically generating test inputs. Specifically, we use multi‐objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well‐known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.

Automated Test Case Generation for Path Coverage by Using Multi-Objective Particle Swarm Optimization Algorithm with Reinforcement Learning and Relationship Matrix Strategies

Software testing is an important phase in software life cycle, and test case generation occupies the main part of the testing workload. Path coverage is the most stringent and difficult coverage criteria, so automatic test case generation for path coverage (ATCG-PC) is an important task in software testing. ATCG-PC is usually abstracted as an optimization search problem, and the commonly used solution algorithm is swarm intelligence algorithm. This paper proposes a path coverage test case generation algorithm based on improved multi-objective particle swarm optimization (IMOPSO) algorithm. The algorithm uses the multi-objective strategy to generate test cases for a group of similar paths at a time, and is supplemented by the particle action selection strategy based on reinforcement learning and the secondary search algorithm based on relationship matrix, which makes use of the similarity information between paths to speed up the early convergence and help to quickly reach the optimal solution around the local optimal position. By grouping the paths according to the similarity, the algorithm reduces the test case consumption, and improves the efficiency of test case generation. Experimental results show that, compared with the existing methods, the proposed algorithm can achieve higher path coverage with less test case consumption and is also suitable for large-scale programs.

Toward granular search-based automatic unit test case generation

Unit testing verifies the presence of faults in individual software components. Previous research has been targeting the automatic generation of unit tests through the adoption of random or search-based algorithms. Despite their effectiveness, these approaches aim at creating tests by solely optimizing metrics like code coverage, without ensuring that the resulting tests have granularities that would allow them to verify both the behavior of individual production methods and the interaction between methods of the class under test. To address this limitation, we propose a two-step systematic approach to the generation of unit tests: we first force search-based algorithms to create tests that cover individual methods of the production code, hence implementing the so-called intra-method tests; then, we relax the constraints to enable the creation of intra-class tests that target the interactions among production code methods. The assessment of our approach is conducted through a mixed-method research design that combines statistical analyses with a user study. The key results report that our approach is able to keep the same level of code and mutation coverage while providing test suites that are more structured, more understandable and aligned to the design principles of unit testing.

Multi-task collaborative method based on manifold optimization for automated test case generation based on path coverage

Automated test case generation based on path coverage is not only a large-scale black-box optimization problem, but also a key scientific problem in software automatic testing technology. Evolutionary algorithms and other search-based algorithms are representative methods for this problem. However, existing research mainly focuses on the multi-function case, where generating test cases for multiple functions is difficult due to the combinatorial explosion of the path number. In this paper, we propose a multi-task collaborative method based on manifold optimization by considering the topological manifold relationship between the test case space (decision space) and the program path space (target space). This method achieves the goal of collaborative optimization for different function coverage tasks by coordinating the allocation of computing resources and knowledge transfer mechanisms among them. To verify the effectiveness of the proposed method, we compare it with general solution methods for single-function automated test case generation based on path coverage, such as the manifold-inspired search-based algorithm. The experimental results show that the proposed method outperforms the compared single-function optimization algorithms especially on programs with strong coding similarities. This study verifies the feasibility of collaborative optimization methods for solving large-scale black-box optimization problems, such as the automated test case generation based on path coverage, and expands their application scenarios.

MR-Scout : Automated Synthesis of Metamorphic Relations from Existing Test Cases

Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs), that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this paper, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs ), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout . Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.

Test case information extraction from requirements specifications using NLP-based unified boilerplate approach

Automated testing which extracts essential information from software requirements written in natural language offers a cost-effective and efficient solution to error-free software that meets stakeholders’ requirements in the software industry. However, natural language can cause ambiguity in requirements and increase the challenges of automated testing such as test case generation. Negative requirements also cause inconsistency and are often neglected. This research aims to extract test case information (actors, conditions, steps, system response) from positive and negative requirements written in natural language (i.e. English) using natural language processing (NLP). We present a unified boilerplate that combines Rupp's and EARS boilerplates, and serves as the grammar guideline for requirements analysis. Extracted information is populated in a test case template, becoming the building blocks for automated test case generation. An experiment was conducted with three public requirements specifications from PURE datasets to investigate the correctness of information extracted using this proposed approach. The results presented correctness of 50 % (Mdot), 61.7 % (Pointis) and 10 % (Npac) on information extracted. The lower correctness on negative over positive requirements was observed. The correctness by specific categories is also analysed, revealing insights into actors, steps, conditions, and system response extracted from positive and negative requirements.

Multi-task modeling and multifactorial optimization for path coverage problem of automated test case generation

Recent research in automated test case generation (ATCG) focuses on multi-objective optimization using functions based on path structure (F-PS) to solve the path coverage (PC) problem. Despite the similarity among F-PSs, the existing multi-objective optimization models fail to consider using the similarity to effectively promote optimization among multiple objectives. Inspired by the similarity and multitask optimization, this paper first establishes a multitasking path coverage (MtPC) model with two different F-PSs as its tasks. A multifactorial optimization framework for solving MtPC model (MfO-PC) is then proposed to optimize the tasks by assortative mating and to cooperatively generate desired test cases by automatic assignment strategy. Three multifactorial optimization algorithms based on the framework are then designed and tested on twelve benchmark programs. Experimental results show that the effectiveness of the proposed model and the designed algorithms based on MfO-PC framework achieve the highest path coverage with fewer test cases and less running time than some compared state-of-the-art algorithms.

Generating automated test case from sequence diagram using Pre-order Traversal

Automated test case generation is important in software testing to evaluate developed software. The testing process requires test cases that can be based on Software Requirements Specification (SRS) and Software Design Document (SDD). One of UML diagrams included in the previous documentations is sequence diagram. This research proposed a modified pre-order algorithm that can generate test cases automatically from sequence diagrams. Our manual cyclomatic complexity evaluation showed that our method produced accurate test cases based on the sequence of messages in the sequence diagram. This result justifies that the generated test cases match the relevant sequence diagram.

Improving Coverage and Vulnerability Detection in Smart Contract Testing Using Self-Adaptive Learning GA

In the domain of software testing, the generation of test cases is a critical process for detecting system errors and bugs. However, automated test case generation for smart contracts often encounters challenges related to automation, vulnerability diversity, and coverage. This paper presents a novel method, the self-adaptive learning Genetic Algorithm (self-adaptive learning GA), designed to address these issues. Our research methodology incorporates several construction models, namely the Control Dependence Graph (CDG), Control Flow Graph (CFG), and Application Binary Interface (ABI). Initially, the ABI model provides essential information for generating and executing test cases. The CFG model subsequently visualizes potential execution paths through the functions of smart contracts. Ultimately, the CDG model identifies potential vulnerabilities in smart contracts. Using these models, our method enhances automatic test case generation in smart contracts by improving coverage and reducing execution time. We selected a variety of smart contracts from the Decentralized Finance (DeFi) ecosystem for data collection and comparative analysis. The experimental results show superior performance rates, with an average code coverage rate of 98.1%, a total of 3500 vulnerabilities detected, a vulnerability detection rate of 98.7%, a false positive rate of 1.3%, a recall of 98.2%, precision of 98.8%, a path uniqueness rate of 96.4%, false negative rate of 3.5%, an execution time of 25 s, and test case generation time of 16 s. In conclusion, our proposed approach demonstrates a significant improvement over existing methods for test case generation by providing a promising solution for the robustness of smart contracts and security enhancement in the DeFi ecosystem.

Software Quality Assurace

Abstract: Software Quality Assurance (SQA) plays a pivotal role in the software development lifecycle, ensuring the delivery of high-quality software products. This abstract provides a comprehensive overview of the latest advancements in Software Quality Assurance methodologies, practices, and tools. The evolving landscape of software development, characterized by agile methodologies, DevOps practices, and continuous integration, has necessitated a reevaluation and enhancement of SQA processes. The abstract begins by elucidating the fundamental principles of SQA, emphasizing its critical role in detecting and preventing defects throughout the software development process. It explores the shift-left approach, where quality assurance is integrated early in the development cycle, and the benefits of this approach in terms of cost reduction and accelerated time-tomarket. Furthermore, the abstract delves into the integration of artificial intelligence (AI) and machine learning (ML) techniques in SQA processes. The use of AI-driven testing tools for automated test case generation, anomaly detection, and predictive analysis is explored as a means to enhance the efficiency and effectiveness of quality assurance activities. The abstract also discusses the challenges and ethical considerations associated with the use of AI in SQA. Additionally, the abstract highlights the importance of comprehensive test automation strategies in modern SQA. It outlines the advantages of continuous testing, test-driven development (TDD), and behavior-driven development (BDD) methodologies in ensuring robust software quality. The abstract also addresses the challenges of maintaining test automation suites and advocates for a balanced approach that combines automated and manual testing. The abstract concludes by emphasizing the need for a holistic and adaptive SQA framework that aligns with the dynamic nature of software development. It calls for ongoing research and collaboration in the SQA domain to address emerging challenges and capitalize on new opportunities for improving software quality

ESSENT: an arithmetic optimization algorithm with enhanced scatter search strategy for automated test case generation

As one of the main research tasks in software testing, automated test case generation based on path coverage (ATCG-PC) aims to achieve maximum path coverage with a minimized set of test cases. In ATCG-PC, the correlation among the dimensions of test cases is widely utilized in academia to minimize the search efforts of the search-based algorithm. Nevertheless, the information related to target path selection is not utilized, which leads to blind decision-making by the search-based algorithm during the target path selection. Therefore, this paper proposes an enhanced scatter search strategy by using opposition-based learning. An arithmetic optimization algorithm is also proposed to solve ATCG-PC based on the enhanced scatter search strategy, namely, ESSENT. The ESSENT algorithm selects the path with the lowest path entropy among the uncovered paths as the target path, and generates new test cases that cover the target path by modifying the dimensions of the existing test cases. The performance of the ESSENT algorithm is evaluated on six iFogSim subprograms and six Stanford coreNLP subprograms. Experiment results show that the ESSENT algorithm achieves a higher convergence rate than other state-of-the-art algorithms. Furthermore, it enables maximum path coverage with fewer test cases.

White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study

Remote Procedure Call (RPC) is a communication protocol to support client-server interactions among services over a network. RPC is widely applied in industry for building large-scale distributed systems, such as Microservices. Modern RPC frameworks include, for example, Thrift, gRPC, SOFARPC, and Dubbo. Testing such systems using RPC communications is very challenging, due to the complexity of distributed systems and various RPC frameworks the system could employ. To the best of our knowledge, there does not exist any tool or solution that could enable automated testing of modern RPC-based services. To fill this gap, in this article we propose the first approach in the literature, together with an open source tool, for fuzzing modern RPC-based APIs. The approach is in the context of white-box testing with search-based techniques. To tackle schema extraction of various RPC frameworks, we formulate a RPC schema specification along with a parser that allows the extraction from source code of any JVM RPC-based APIs. Then, with the extracted schema we employ a search to produce tests by maximizing white-box heuristics and newly defined heuristics specific to the RPC domain. We built our approach as an extension to an open source fuzzer (i.e., EvoMaster ), and the approach has been integrated into a real industrial pipeline that could be applied to a real industrial development process for fuzzing RPC-based APIs. To assess our novel approach, we conducted an empirical study with two artificial and four industrial web services selected by our industrial partner. In addition, to further demonstrate its effectiveness and application in industrial settings, we report results of employing our tool for fuzzing another 50 industrial APIs autonomously conducted by our industrial partner in their testing processes. Results show that our novel approach is capable of enabling automated test case generation for industrial RPC-based APIs (i.e., 2 artificial and 54 industrial). We also compared with a simple gray-box technique and existing manually written tests. Our white-box solution achieves significant improvements on code coverage. Regarding fault detection, by conducting a careful review with our industrial partner of the tests generated by our novel approach in the selected four industrial APIs, a total of 41 real faults were identified, which have now been fixed. Another 8,377 detected faults are currently under investigation.

Backtracking search optimization algorithm with dual scatter search strategy for automated test case generation

It is a challenge to design an effective algorithm utilizing problem features in automated test case generation for path coverage (ATCG-PC). A feature of ATCG-PC “similar paths are usually executed by similar test cases” was touched by a few scholars and can be further exploited to design more effective algorithms. Inspired by this feature, this paper proposes a two-stage local search strategy, denoted dual scatter search (DS) strategy, which concatenates two improved scatter search strategies with different search behaviors. The first stage aims to fully exploit the discovered test cases to search for desired test cases, and the latter stage aims to mine the unexploited areas of the first stage via using less computational overhead. Then, a backtracking search optimization algorithm with dual scatter search strategy (BSA-DS) is proposed, which incorporates DS strategy into the backtracking search optimization algorithm (BSA) with strong exploration capability. BSA is first introduced into the field of ATCG-PC. The performance of BSA-DS and some state-of-the-art algorithms is tested on twelve popular benchmark programs. Experimental studies demonstrate that BSA-DS achieves the highest path coverage with the fewest test cases and running time on at least eight out of the twelve programs.

InterEvo-TR: Interactive Evolutionary Test Generation With Readability Assessment

Automated test case generation has proven to be useful to reduce the usually high expenses of software testing. However, several studies have also noted the skepticism of testers regarding the comprehension of generated test suites when compared to manually designed ones. This fact suggests that involving testers in the test generation process could be helpful to increase their acceptance of automatically-produced test suites. In this paper, we propose incorporating interactive readability assessments made by a tester into EvoSuite, a widely-known evolutionary test generation tool. Our approach, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">InterEvo-TR</small> , interacts with the tester at different moments during the search and shows different test cases covering the same coverage target for their subjective evaluation. The design of such an interactive approach involves a schedule of interaction, a method to diversify the selected targets, a plan to save and handle the readability values, and some mechanisms to customize the level of engagement in the revision, among other aspects. To analyze the potential and practicability of our proposal, we conduct a controlled experiment in which 39 participants, including academics, professional developers, and student collaborators, interact with <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">InterEvo-TR</small> . Our results show that the strategy to select and present intermediate results is effective for the purpose of readability assessment. Furthermore, the participants' actions and responses to a questionnaire allowed us to analyze the aspects influencing test code readability and the benefits and limitations of an interactive approach in the context of test case generation, paving the way for future developments based on interactivity.

An Automated Path-Focused Test Case Generation with Dynamic Parameterization Using Adaptive Genetic Algorithm (AGA) for Structural Program Testing

Various software engineering paradigms and real-time projects have proved that software testing is the most critical and highly important phase in the SDLC. In general, software testing takes approximately 40–60% of the total effort and time involved in project development. Generating test cases is the most important process in software testing. There are many techniques involved in the automatic generation of these test cases which aim to find a smaller group of cases that could allow for an adequacy level to be achieved which will hence reduce the effort and cost involved in software testing. In the structural testing of a product, the auto-generation of test cases that are path focused in an efficient manner is a challenging process. These are often considered optimization problems and hence search-based methods such as genetic algorithm (GA) and swarm optimizations have been proposed to handle this issue. The significance of the study is to address the optimization problem of automatic test case generation in search-based software engineering. The proposed methodology aims to close the gap of genetic algorithms acquiring local minimum due to poor diversity. Here, dynamic adjustment of cross-over and mutation rate is achieved by calculating the individual measure of similarity and fitness and searching for the more global optimum. The proposed method is applied and experimented on a benchmark of five industrial projects. The results of the experiments have confirmed the efficiency of generating test cases that have optimum path coverage.

Implementation of Automated Test Case Generation in REST API on Android-Based Koperasi Application

This study is focused on developing a data collection system to enhance the performance of Koperasi, an organization with complex data collection. An Android application was created to automate the processing of member and transaction data, significantly improving data processing efficiency. However, building a quality system takes time and requires error-free data processing. To achieve this, Automated Test Case Generation with EvoMaster was used to test the REST API and identify errors. The testing process went through several iterations until almost no errors were found. EvoMaster generated over 19.5 million scenarios and found 78 errors in the REST API in 58 hours, which were promptly fixed between iterations. The use of EvoMaster not only reduced development time but also helped maintain code quality.

ARTE: Automated Generation of Realistic Test Inputs for Web APIs

Automated test case generation for web APIs is a thriving research topic, where test cases are frequently derived from the API specification. However, this process is only partially automated since testers are usually obliged to manually set meaningful valid test inputs for each input parameter. In this article, we present ARTE, an approach for the automated extraction of realistic test data for web APIs from knowledge bases like DBpedia. Specifically, ARTE leverages the specification of the API parameters to automatically search for realistic test inputs using natural language processing, search-based, and knowledge extraction techniques. ARTE has been integrated into RESTest, an open-source testing framework for RESTful APIs, fully automating the test case generation process. Evaluation results on 140 operations from 48 real-world web APIs show that ARTE can efficiently generate realistic test inputs for 64.9% of the target parameters, outperforming the state-of-the-art approach SAIGEN (31.8%). More importantly, ARTE supported the generation of over twice as many valid API calls (57.3%) as random generation (20%) and SAIGEN (26%), leading to a higher failure detection capability and uncovering several real-world bugs. These results show the potential of ARTE for enhancing existing web API testing tools, achieving an unprecedented level of automation.

Using deep learning for selenium web UI functional tests: A case-study with e-commerce applications

The application of test cases for detecting the faults within the software is called software testing. Manual testing is laborious and time-consuming hence automation tools to test software were introduced. Despite the use of automation tools at the User Interface (UI) level of the test pyramid, the limitations of current automation tools like automated test case generation and automated repairing of fragile tests still force us to carry out a large amount of manual testing. In this paper, we propose a novel method using AI to address the given challenges. With our proposed method test cases are automatically generated from the structure of the UI using a pipelined architecture of object detection, text detection and NLP models. We show that the test cases generated by the proposed framework can be translated into executable test scripts using a simple parser. The proposed method generates an average of 98.8% correct executable test cases for the applications under study. We also show the capability of the proposed method in generating new tests automatically when the application is modified. The proposed method generates an average of 98.605% correct executable test cases when the UI is modified for the applications under study. We also empirically prove that a GPU implementation of the proposed framework results in just an additional average runtime of 0.92 seconds per test case which is significantly low given the benefits of automated generation of test scripts and automated repairing of fragile tests.

Test case recommendation based on balanced distance of test targets

Unit testing has been widely regarded as an effective technique to ensure software quality. Writing unit test cases is time-consuming and requires developers to have abundant knowledge and experience. Automated test case generation, a promising technology for liberating developers and improving test efficiency, currently performs not satisfactory in real-world projects. As a complement, test case recommendation (TCR) has been receiving the attention of researchers. TCR can improve the efficiency of test case writing by recommending test case code to developers for their reference and reuse. The overarching idea of TCR techniques is that two similar test targets can reuse each other’s test cases.

Test scenario generation method for autonomous vehicles based on combinatorial testing and Bayesian network

A test scenario generation method based on combinatorial testing (CT) and Bayesian Network for autonomous vehicles is proposed in this paper. Firstly, some parameters are selected to describe the test scenarios which are classified according to road types and driving tasks. Then, the constraint sets for the scenarios with forbidden tuples are established to avoid the generated cases do not conform to the reality, in which the construct constraint set (CCS) algorithm is utilized to compute implied constraints. Furthermore, the Bayesian networks is used as the probabilistic models of the scenarios, where the traffic participants are represented as object nodes and the relative relationships between the participants are converted into the network structures. Finally, an improved automatic efficient test case generator (AETG) is developed to generate test cases. By considering both probability and frequency, the select function is designed for determining the values of scenario parameters. And the generation mode can be changed by modifying the weight and target parameters. The effectiveness of the proposed method is evaluated by generating six typical test scenarios. Compared with other algorithms, the numbers of test cases in the sets generated by this method are less and the probability deviations are smaller.