Test Case Generation Research Articles

Leveraging Gen AI for Automated Test Script and Test Case Generation in SAP ERP Projects

Leveraging Gen AI for Automated Test Script and Test Case Generation in SAP ERP Projects

Evaluating Large Language Models for Software Testing

Large language models (LLMs) have demonstrated significant prowess in code analysis and natural language processing, making them highly valuable for software testing. This paper conducts a comprehensive evaluation of LLMs applied to software testing, with a particular emphasis on test case generation, error tracing, and bug localization across twelve open-source projects. The advantages and limitations, as well as recommendations associated with utilizing LLMs for these tasks, are delineated. Furthermore, we delve into the phenomenon of hallucination in LLMs, examining its impact on software testing processes and presenting solutions to mitigate its effects. The findings of this work contribute to a deeper understanding of integrating LLMs into software testing, providing insights that pave the way for enhanced effectiveness in the field.

Integrating AI in testing automation: Enhancing test coverage and predictive analysis for improved software quality

This research explores the integration of Artificial Intelligence (AI) in testing automation, focusing on its ability to enhance test coverage and enable predictive analysis for improved software quality. As software systems become increasingly complex, traditional testing methods often struggle to meet quality demands. This study evaluates various AI techniques, including machine learning and natural language processing, and their applications in generating test cases, optimizing testing processes, and predicting defects. Through empirical case studies from diverse organizations, we demonstrate significant improvements in test coverage and defect detection rates following AI implementation. The findings highlight the efficiency gains and quality enhancements achieved through AI-driven testing, while also addressing challenges such as data dependency, complexity of implementation, and the need for skilled personnel. This research contributes to the understanding of AI's role in software testing and encourages organizations to adopt these technologies for better quality assurance and faster development cycles.

An Extended Algorithm using a Divide-and-Rule Approach for Boundary Value Analysis and Equivalence Class Partitioning

An extended algorithm using a divide-and-rule approach is proposed in this paper. Instead of generating test cases by only using Boundary Value Analysis and an experience-based technique for combination of input variables with functional dependency like the original algorithm, the extended algorithm applies both Boundary Value Analysis and Equivalence Class Partitioning techniques as well as various techniques for combination of input values. In addition, a new tool provides advantages when compared with the original tool. It provides users with more choices in generating test cases.

Computing Precise Control Interface Specifications

Verifying network programs is challenging because of how they divide labor: the control plane computes high level routes through the network and compiles them to device configurations, while the data plane uses these configurations to realize the desired forwarding behavior. In practice, the correctness of the data plane often assumes that the configurations generated by the control plane will satisfy complex specifications. Consequently, validation tools such as program verifiers, runtime monitors, fuzzers, and test-case generators must be aware of these control interface specifications (ci-specs) to avoid raising false alarms. In this paper, we propose the first algorithm for computing precise ci-specs for network data planes. Our specifications are designed to be efficiently monitorable —concretely, checking that a fixed configuration satisfies a ci-spec can be done in polynomial time. Our algorithm, based on modular program instrumentation, quantifier elimination, and a path-based analysis, is more expressive than prior work, and is applicable to practical network programs. We describe an implementation and show that ci-specs computed by our tool are useful for finding real bugs in real-world data plane programs.

SMT2Test: From SMT Formulas to Effective Test Cases

One of the primary challenges in software testing is generating high-quality test inputs and obtaining corresponding test oracles. This paper introduces a novel methodology to mitigate this challenge in testing program verifiers by employing SMT (Satisfiability Modulo Theories) formulas as a universal test case generator. The key idea is to transform SMT formulas into programs and link the satisfiability of the formulas with the safety property of the programs, allowing the satisfiability of the formulas to act as a test oracle for program verifiers. This method was implemented as a framework named SMT2Test, which enables the transformation of SMT formulas into Dafny and C programs. An intermediate representation was designed to augment the flexibility of this framework, streamlining the transformation for other programming languages and fostering modular transformation strategies. We evaluated the effectiveness of SMT2Test by finding defects in two program verifiers: the Dafny verifier and CPAchecker. Utilizing the SMT2Test framework with the SMT formulas from the SMT competition and SMT solver fuzzers, we discovered and reported a total of 14 previously unknown defects in these program verifiers that were not found by previous methods. After reporting, all of them have been confirmed, and 6 defects have been fixed. These findings show the effectiveness of our method and imply its potential application in testing other programming language infrastructures.

Автоматизований метод перевірки та налагодження тестових сценаріїв на основі формальної моделі

Introduction. Model-based test cases generation is a popular strategy for test automation. It helps to reduce time spent on the development of a test suite and can improve level of coverage. However, many reports show shortage of such test cases in poor quality and doubtable efficiency. Purpose. The main goal of the proposed method is cost-effective validation, assessment, debugging and concretization of generated test cases. The method helps improve quality and efficiency of the test cases, make their scenario meaningful and goal-oriented. The method also develops debugging facilities and simplifies data dependency analysis and test scenario editing. Methods. We propose an automated post-processing method which allows to evaluate path that is examined by the test case, and to make safe changes to the path which will eliminate the shortcomings while leaving the coverage targets of the test case unharmed. The method is based on visualization of the path along the control flow graph of the model with additional information about factual evaluation history of all variables and possible alternative variants of behavior. For consistent substitution of certain values in the signal parameters, which would determine the artifacts of the test environment (such as, for example, files, databases, etc.) and check boundary cases (in predicates of conditions, indexing of arrays, etc.), a method of interactive specification of symbolic traces has been developed. Results. The role of the user in deciding whether to add a test case to the project test suite and make changes to it remains crucial, but to reduce labor intensity, the following processes are automated: evaluation of test scenarios according to certain objective characteristics (level of coverage, ability to detect defects, data cohesion, etc.); highlighting of possible alternatives for making corrections; consistent updating of computations for the corresponding corrections. A prototype was developed based on the proposed methods. The empirical results demonstrated a positive impact on the overall efficiency (ability to detect defects and reduce resource consumption) and quality (meaningfulness, readability, maintenance, usefulness for debugging, etc) of the generated test suites. The method allows to make automatically generated test cases trustable and usable. Conclusion. The proposed toolkit significantly reduces the time spent on researching the results of test generation and validation of the obtained tests and their editing. Unlike existing simulation methods, the proposed method not only informs about the values of variables, but also explores the history of their computations and additionally provides information about admissible alternatives. Further we plan to improve the process of localizing the causes of test failure at execution phase to speed up the search for defects.

SMUP: A technique to improve MC/DC using specified patterns

According to recent study, a number of poorly explored aspects, like program structure and not producing high-quality random inputs for testing, can have a significant impact on the overall efficacy of the testing process. Existing concolic testers found to be more cost-effective than random testers for code coverage. Modified condition/decision coverage (MC/DC) has gained its popularity as the strongest criterion for safety-critical systems proposed by RTCA/DO-178B(C), after multiple condition coverage (MCC). This paper proposes a new source code transformation technique that produces additional test cases to achieve higher MC/DC. The technical contribution of this work is threefold. First, it uses a pattern-based code transformation technique that produces effective MC/DC test cases. This pattern-based approach guides the concolic tester to generate test cases as per the MC/DC requirements. Second, the generated patterns are further simplified to minimize the execution time. Third, we have developed a tool named “Java MC/DC Analyzer” to measure MC/DC score for the input programs. The proposed approach is experimented on thirty Java programs and achieved an average increase of 32.86% MC/DC score which validates our work. Also, we have compared our approach with other code transformation techniques and reported a significant improvement.

DynER: Optimized Test Case Generation for Representational State Transfer (REST)ful Application Programming Interface (API) Fuzzers Guided by Dynamic Error Responses

Modern web services widely provide RESTful APIs for clients to access their functionality programmatically. Fuzzing is an emerging technique for ensuring the reliability of RESTful APIs. However, the existing RESTful API fuzzers repeatedly generate invalid requests due to unawareness of errors in the invalid tested requests and lack of effective strategy to generate legal value for the incorrect parameters. Such limitations severely hinder the fuzzing performance. In this paper, we propose DynER, a new test case generation method guided by dynamic error responses during fuzzing. DynER designs two strategies of parameter value generation for purposefully revising the incorrect parameters of invalid tested requests to generate new test requests. The strategies are, respectively, based on prompting Large Language Model (LLM) to understand the semantics information in error responses and actively accessing API-related resources. We apply DynER to the state-of-the-art fuzzer RESTler and implement DynER-RESTler. DynER-RESTler outperforms foREST on two real-world RESTful services, WordPress and GitLab with a 41.21% and 26.33% higher average pass rate for test requests and a 12.50% and 22.80% higher average number of unique request types successfully tested, respectively. The experimental results demonstrate that DynER significantly improves the effectiveness of test cases and fuzzing performance. Additionally, DynER-RESTler finds three new bugs.

A3Test: Assertion-Augmented Automated Test case generation

Context:Test case generation is a critical yet challenging task in software development. Recently, AthenaTest – a Deep Learning (DL) approach for generating unit test cases has been proposed. However, our revisiting study reveals that AthenaTest can generate less than one-fifth of the test cases correctly, due to a lack of assertion knowledge and test signature verification. Objective:This paper introduces A3Test, a novel DL-based approach to the generation of test cases, enhanced with assertion knowledge and a mechanism to verify consistency of the name and signatures of the tests. A3Test aims to adapt domain knowledge from assertion generation to test case generation. Method:A3Test employs domain adaptation principles and introduces a verification approach to name consistency and test signatures. We evaluate its effectiveness using 5,278 focal methods from the Defects4j dataset. Results:Our findings indicate that A3Test outperforms AthenaTest and ChatUniTest. A3Test generates 2.16% to 395.43% more correct test cases, achieves 2.17% to 34.29% higher method coverage, and 25.64% higher line coverage. A3Test achieves 2.13% to 12.20% higher branch coverage, 2.22% to 12.20% higher mutation scores, and 2.44% to 55.56% more correct assertions compared to both ChatUniTest and AthenaTest respectively for one iteration. When generating multiple test cases per method A3Test still shows improvements and comparable efficacy to ChatUnitTest. A survey of developers reveals that the majority of the participants 70.51% agree that test cases generated by A3Test are more readable than those generated by EvoSuite. Conclusions:A3Test significantly enhances test case generation through its incorporation of assertion knowledge and test signature verification, contributing to the generation of correct test cases.

Optimized automated testing: test case generation and maintenance using latent semantic analysis-based TextRank and particle swarm optimization algorithms

Software development would have to include automated testing to ensure the finished product and performs as intended. However, the process of Test Case Generation and Maintenance can be time-consuming and error-prone, especially when manual methods are used. This research proposes a new approach to improve the efficiency and accuracy of automated testing using latent semantic analysis (LSA)-based TextRank (TR) and particle swarm optimization (PSO) algorithms. The study aims to evaluate the effectiveness of these algorithms in generating and optimizing test cases based on requirements analysis. To retrieve key information from the criteria, methods including text classification (TC), named entity recognition (NER), and sentiment analysis (SA) are used to evaluate the text. Test cases are then generated using LSA-based TR for text summarization and PSO for optimization. The aim of this work is to identify any limitations that need to be addressed and to evaluate the overall efficiency and accuracy of automated testing (AT) using proposed algorithms. The results of this research are expected to have important implications for the software industry, helps to improve the overall efficiency and accuracy of AT. The findings could guide future research that led to the creation of more advanced and effective tools for AT.

Context-Aware Fuzzing for Robustness Enhancement of Deep Learning Models

In the testing-retraining pipeline for enhancing the robustness property of deep learning (DL) models, many state-of-the-art robustness-oriented fuzzing techniques are metric-oriented. The pipeline generates adversarial examples as test cases via such a DL testing technique and retrains the DL model under test with test suites that contain these test cases. On the one hand, the strategies of these fuzzing techniques tightly integrate the key characteristics of their testing metrics. On the other hand, they are often unaware of whether their generated test cases are different from the samples surrounding these test cases and whether there are relevant test cases of other seeds when generating the current one. We propose a novel testing metric called Contextual Confidence (CC). CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Based on this metric, we further propose a novel fuzzing technique Clover as a DL testing technique for the pipeline. In each fuzzing round, Clover first finds a set of seeds whose labels are the same as the label of the seed under fuzzing. At the same time, it locates the corresponding test case that achieves the highest CC values among the existing test cases of each seed in this set of seeds and shares the same prediction label as the existing test case of the seed under fuzzing that achieves the highest CC value. Clover computes the piece of difference between each such pair of a seed and a test case. It incrementally applies these pieces of differences to perturb the current test case of the seed under fuzzing that achieves the highest CC value and to perturb the resulting samples along the gradient to generate new test cases for the seed under fuzzing. Clover finally selects test cases among the generated test cases of all seeds as even as possible and with a preference to select test cases with higher CC values for improving model robustness. The experiments show that Clover outperforms the state-of-the-art coverage-based technique Adapt and loss-based fuzzing technique RobOT by 67%–129% and 48%–100% in terms of robustness improvement ratio, respectively, delivered through the same testing-retraining pipeline. For test case generation, in terms of numbers of unique adversarial labels and unique categories for the constructed test suites, Clover outperforms Adapt by \(2.0\times\) and \(3.5{\times}\) and RobOT by \(1.6\times\) and \(1.7\times\) on fuzzing clean models, and also outperforms Adapt by \(3.4\times\) and \(4.5\times\) and RobOT by \(9.8\times\) and \(11.0\times\) on fuzzing adversarially trained models, respectively.

Test-based patch clustering for automatically-generated patches assessment

Previous studies have shown that Automated Program Repair (apr) techniques suffer from the overfitting problem. Overfitting happens when a patch is run and the test suite does not reveal any error, but the patch actually does not fix the underlying bug or it introduces a new defect that is not covered by the test suite. Therefore, the patches generated by apr tools need to be validated by human programmers, which can be very costly, and prevents apr tool adoption in practice. Our work aims to minimize the number of plausible patches that programmers have to review, thereby reducing the time required to find a correct patch. We introduce a novel light-weight test-based patch clustering approach called xTestCluster, which clusters patches based on their dynamic behavior. xTestCluster is applied after the patch generation phase in order to analyze the generated patches from one or more repair tools and to provide more information about those patches for facilitating patch assessment. The novelty of xTestCluster lies in using information from execution of newly generated test cases to cluster patches generated by multiple APR approaches. A cluster is formed of patches that fail on the same generated test cases. The output from xTestCluster gives developers a) a way of reducing the number of patches to analyze, as they can focus on analyzing a sample of patches from each cluster, b) additional information (new test cases and their results) attached to each patch. After analyzing 902 plausible patches from 21 Java apr tools, our results show that xTestCluster is able to reduce the number of patches to review and analyze with a median of 50%. xTestCluster can save a significant amount of time for developers that have to review the multitude of patches generated by apr tools, and provides them with new test cases that expose the differences in behavior between generated patches. Moreover, xTestCluster can complement other patch assessment techniques that help detect patch misclassifications.

How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?

Software metrics such as coverage or mutation scores have been investigated for the automated quality assessment of test suites. While traditional tools rely on software metrics, the field of self-driving cars (SDCs) has primarily focused on simulation-based test case generation using quality metrics such as the out-of-bound (OOB) parameter to determine if a test case fails or passes. However, it remains unclear to what extent this quality metric aligns with the human perception of the safety and realism of SDCs. To address this (reality) gap, we conducted an empirical study involving 50 participants to investigate the factors that determine how humans perceive SDC test cases as safe, unsafe, realistic, or unrealistic. To this aim, we developed a framework leveraging virtual reality (VR) technologies, called SDC-Alabaster, to immerse the study participants into the virtual environment of SDC simulators. Our findings indicate that the human assessment of safety and realism of failing/passing test cases can vary based on different factors, such as the test's complexity and the possibility of interacting with the SDC. Especially for the assessment of realism, the participants' age leads to a different perception. This study highlights the need for more research on simulation testing quality metrics and the importance of human perception in evaluating SDCs.

Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing

Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria.

MET-MAPF: A Metamorphic Testing Approach for Multi-Agent Path Finding Algorithms

The Multi-Agent Path Finding (MAPF) problem, i.e., the scheduling of multiple agents to reach their destinations, has been widely investigated. Testing MAPF systems is challenging, due to the complexity and variety of scenarios and the agents’ distribution and interaction. Moreover, MAPF testing suffers from the oracle problem, i.e., it is not always clear whether a test shows a failure or not. Indeed, only considering whether the agents reach their destinations without collision is not sufficient. Other properties related to the “quality” of the generated paths should be assessed, e.g., an agent should not follow an unnecessarily long path. To tackle this issue, this paper proposes MET-MAPF, a Metamorphic Testing approach for MAPF systems. We identified ten Metamorphic Relations (MRs) that a MAPF system should guarantee, designed over the environment in which agents operate, the behaviour of the single agents, and the interactions among agents. Starting from the different MRs, MET-MAPF automatically generates test cases addressing them, so possibly exposing different types of failures. Experimental results show that MET-MAPF can indeed find MR violations not exposed by approaches that only consider the completion of the mission as test oracle. Moreover, experiments show that different MRs expose different types of violations.

Fuzzing JavaScript engines with a syntax-aware neural program model

Neural network language modeling has become a remarkable approach in the generation of test cases for fuzzing JavaScript engines. Fuzzers built upon neural language models offer several advantages. They obviate the need for manually developing code generation rules, enable the extraction of patterns from high-quality seed sets, and exhibit commendable portability. Nevertheless, existing works confront challenges in three key aspects: diminished language modeling performance attributable to extensive vocabularies, potential semantic errors within generated test cases, and the limitation of black-box fuzzing, which fails to leverage the internal feedback from the target engine.This paper proposes an innovative neural model-based grey-box fuzzing approach for JavaScript engines. We incorporate the context-free grammar of JavaScript into the neural language model to mitigate the challenges associated with extensive vocabularies, thereby enhancing the model’s performance. Furthermore, to enhance the semantic validity of the generated test cases, we introduce semantic constraints into the mutation process. Notably, this work pioneers the integration of grey-box testing into a fuzzer built upon a neural language model, thereby enhancing the exploration of deep paths. Our prototype, PMFuzz, surpasses NNLM-based counterparts in both language modeling performance and test case generation capabilities. PMFuzz demonstrates a high level of competitiveness in exploring the software state space when compared to traditional coverage-guided grey-box fuzzers. In our evaluation, PMFuzz successfully identified 20 new defects within mainstream JS engines. Eight of them have been confirmed and fixed. Moreover, upon applying our method to C compilers, PMFuzz has revealed 11 new defects.

Basis path coverage testing of MPI programs based on multi-task evolutionary optimization

A Message-Passing Interface (MPI) program usually consists of several processes, and a target path of this program is composed of a target sub-path selected in each process. However, there may be coverage conflict between different target sub-paths in the target path, and there is currently no effective approach to avoid this problem. Therefore, this paper proposes an approach of basis path coverage testing of MPI programs based on multi-task evolutionary optimization, which is used to generate a test case set that meets the criteria of branch and sentence coverage. Firstly, the proposed approach samples a certain number of representative test inputs, and calculates the coverage level of each test input for each target sub-path in the target path, which is used to group the target sub-paths with coverage conflict. Then, we construct an optimization model for each group of the target sub-paths, which can guide test case generation with high effectiveness. Finally, each optimization model is solved by an evolutionary algorithm, and the efficiency of solving these optimization models is improved based on a multi-task framework. The proposed approach is applied to basis path coverage testing of seven benchmark MPI programs under test, and compared with several state-of-the-art approaches. Experimental results indicate that the proposed approach has significant advantages in test case generation, as well as performs good scalability.

Pairwise test case generation with harmony search, one-parameter-at-at-time, seeding, and constraint mechanism integration

Pairwise testing is a method for identifying defects through combinatorial analysis. It involves testing all possible combinations of input parameters in pairs within a system, ensuring that each pair is tested at least once. The field of test case generation is highly active in the realm of combinatorial interaction testing. Research in this area is particularly encouraged, as it falls under the category of non-deterministic polynomial-time hardness. A big challenge in this field is the combinatorial explosion problem. It is about finding the best test suite that covers all possible combinations of interaction strength. In this paper, we present the task of discovering a pairwise test set as a search problem and introduce an innovative testing tool referred to as pairwise test case generation in harmony search algorithm with seeding and constraint mechanism (PHOSC). Experimental results show that PHOSC performs better compared to some existing pairwise strategies in terms of test suite size. Additionally, PHOSC provides a comprehensive framework and serves as a research platform for the generation of pairwise test sets employing the harmony search algorithm. It adopts an approach that focuses on one parameter at a time (OPAT) and incorporates seeding and constraint mechanisms at the same time, thereby enhancing the efficiency and effectiveness of the testing process.

Multi‐objective optimization‐based and fault localization‐oriented test case generation for novice programs

SummaryOnline judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum‐based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high‐quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program‐oriented Multi‐Objective Optimization‐Based Fault Localization‐Oriented Test Case Generation (MFTCG) for automatically generating test inputs. Specifically, we use multi‐objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well‐known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.