Test Case Generation Method Research Articles

DynER: Optimized Test Case Generation for Representational State Transfer (REST)ful Application Programming Interface (API) Fuzzers Guided by Dynamic Error Responses

Modern web services widely provide RESTful APIs for clients to access their functionality programmatically. Fuzzing is an emerging technique for ensuring the reliability of RESTful APIs. However, the existing RESTful API fuzzers repeatedly generate invalid requests due to unawareness of errors in the invalid tested requests and lack of effective strategy to generate legal value for the incorrect parameters. Such limitations severely hinder the fuzzing performance. In this paper, we propose DynER, a new test case generation method guided by dynamic error responses during fuzzing. DynER designs two strategies of parameter value generation for purposefully revising the incorrect parameters of invalid tested requests to generate new test requests. The strategies are, respectively, based on prompting Large Language Model (LLM) to understand the semantics information in error responses and actively accessing API-related resources. We apply DynER to the state-of-the-art fuzzer RESTler and implement DynER-RESTler. DynER-RESTler outperforms foREST on two real-world RESTful services, WordPress and GitLab with a 41.21% and 26.33% higher average pass rate for test requests and a 12.50% and 22.80% higher average number of unique request types successfully tested, respectively. The experimental results demonstrate that DynER significantly improves the effectiveness of test cases and fuzzing performance. Additionally, DynER-RESTler finds three new bugs.

A framework to identify failure scenarios in the control mode transition process for autonomous ships with dynamic autonomy

The dynamic autonomy resulting from control mode transitions in the Maritime Autonomous Surface Ships (MASS) poses a significant risk to its navigation safety. While Systems-Theoretic Process Analysis (STPA) has proven effective in the hazard identification of autonomous ships, the safety challenges inherent in the dynamic autonomy of MASS with seafarers onboard controlled by three possible controllers have not been addressed much. This study proposes a three-phase framework that combines STPA, state machines and Sequentially Timed Events Plotting (STEP) diagram aiming at identifying failure scenarios in the transition process. Findings from a case study highlight the need for such a framework to understand the triggering events for mode transition, the transition process, how transition failure may occur, and refine safety constraints to ensure smooth and safe transitions. The results lay the groundwork for future research into test case generation methods that focus on these identified transition failure scenarios, intending to ensure safe and secure navigation for the progressive introduction of MASS.

Hierarchical Distribution-aware Testing of Deep Learning

With its growing use in safety/security-critical applications, Deep Learning (DL) has raised increasing concerns regarding its dependability. In particular, DL has a notorious problem of lacking robustness. Input added with adversarial perturbations, i.e., Adversarial Examples (AEs) , are easily mispredicted by the DL model. Despite recent efforts made in detecting AEs via state-of-the-art attack and testing methods, they are normally input distribution–agnostic and/or disregard the perceptual quality of adversarial perturbations. Consequently, the detected AEs are irrelevant inputs in the application context or noticeably unrealistic to humans. This may lead to a limited effect on improving the DL model’s dependability, as the testing budget is likely to be wasted on detecting AEs that are encountered very rarely in its real-life operations. In this article, we propose a new robustness testing approach for detecting AEs that considers both the feature-level distribution and the pixel-level distribution, capturing the perceptual quality of adversarial perturbations. The two considerations are encoded by a novel hierarchical mechanism. First, we select test seeds based on the density of feature-level distribution and the vulnerability of adversarial robustness. The vulnerability of test seeds is indicated by the auxiliary information, which are highly correlated with local robustness. Given a test seed, we then develop a novel genetic algorithm–based local test case generation method, in which two fitness functions work alternatively to control the perceptual quality of detected AEs. Finally, extensive experiments confirm that our holistic approach considering hierarchical distributions is superior to the state-of-the-arts that either disregard any input distribution or only consider a single (non-hierarchical) distribution, in terms of not only detecting imperceptible AEs but also improving the overall robustness of the DL model under testing.

Improving Coverage and Vulnerability Detection in Smart Contract Testing Using Self-Adaptive Learning GA

In the domain of software testing, the generation of test cases is a critical process for detecting system errors and bugs. However, automated test case generation for smart contracts often encounters challenges related to automation, vulnerability diversity, and coverage. This paper presents a novel method, the self-adaptive learning Genetic Algorithm (self-adaptive learning GA), designed to address these issues. Our research methodology incorporates several construction models, namely the Control Dependence Graph (CDG), Control Flow Graph (CFG), and Application Binary Interface (ABI). Initially, the ABI model provides essential information for generating and executing test cases. The CFG model subsequently visualizes potential execution paths through the functions of smart contracts. Ultimately, the CDG model identifies potential vulnerabilities in smart contracts. Using these models, our method enhances automatic test case generation in smart contracts by improving coverage and reducing execution time. We selected a variety of smart contracts from the Decentralized Finance (DeFi) ecosystem for data collection and comparative analysis. The experimental results show superior performance rates, with an average code coverage rate of 98.1%, a total of 3500 vulnerabilities detected, a vulnerability detection rate of 98.7%, a false positive rate of 1.3%, a recall of 98.2%, precision of 98.8%, a path uniqueness rate of 96.4%, false negative rate of 3.5%, an execution time of 25 s, and test case generation time of 16 s. In conclusion, our proposed approach demonstrates a significant improvement over existing methods for test case generation by providing a promising solution for the robustness of smart contracts and security enhancement in the DeFi ecosystem.

CanaryExp: A Canary-Sensitive Automatic Exploitability Evaluation Solution for Vulnerabilities in Binary Programs

We propose CanaryExp, an exploitability evaluation solution for vulnerabilities among binary programs protected by StackGuard. CanaryExp devises three novel techniques, namely canary leakage proof of concept generation, canary leaking analysis time exploitation, and dynamic canary-relocation-based exploitability evaluation. The canary leakage proof of concept input generation mechanism first traces the target program’s execution, transforming the execution state into some canary leaking state, from which some canary leaking input is derived. This input can be deemed as proof that some vulnerability that can lead to canary leakage exists. The canary leaking analysis time exploit generation then performs incremental analysis based on the canary leaking input, crafting analysis time exploit that can complete vulnerability exploitation in the analysis time environment. Based on the analysis time exploit, the dynamic canary-relocation-based exploitability evaluation component collects the necessary metadata, on which an exploitation session is automatically constructed that can not only leak the runtime canary and relocate it in the input stream but also evaluate the exploitability of the desired vulnerability. Using a benchmark containing six test programs, eight challenges from some network challenging events and four real-world applications, we demonstrate that CanaryExp can generate canary leaking samples more effectively than existing test case generation methods and automatically evaluate the exploitability for vulnerabilities among programs where the StackGuard protection mechanism is deployed.

DTester: Diversity-Driven Test Case Generation for Web Applications

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is [Formula: see text], [Formula: see text], [Formula: see text] and [Formula: see text] in efficiency, test suite size, diversity and robustness.

Automatic Generation of Test Cases Based on Genetic Algorithm and RBF Neural Network

Software testing plays an important role in improving the quality of software, but the design of test cases requires a lot of manpower, material resources, and time, and designers tend to be subjective when designing test cases. To solve this problem and make the test cases have objectivity and greater coverage, a branch coverage test case automatic generation method based on genetic algorithm and RBF neural network algorithm (GAR) is proposed. In terms of test case generation, based on the genetic algorithm optimized in this paper, a certain number of test case samples are randomly selected to train the RBF neural network to simulate the fitness function and to calculate the individual fitness value. The experiment uses 7 C language codes to automatically generate test cases and compares the experimental data generated by the branch coverage test case generation method based on adaptive genetic algorithm (PDGA), traditional genetic algorithm (SGA), and random test generation method (random) to evaluate the proposed algorithm. The experimental results show that the method is feasible and effective, the branch coverage is increased in the generation of test cases, and the number of iterations of the population is less.

MT-ART: A Test Case Generation Method Based on Adaptive Random Testing and Metamorphic Relation

Most of metamorphic testing (MT) research works focused on the generation and application of metamorphic relations (MRs). There is no clear conclusion about the relationship between test case generation methods and performance of MT. In this article, we introduce a novel method based on adaptive random testing (ART) and MR for MT test case generation. It proposes a family of algorithms for MT test cases generation, named as MT based ART (MT-ART). Three distances are measured to generate the next MT test case. In order to verify the performance of this method, series of experiments on four programs with different numbers of inputs are introduced. The results show that MT-ART performs better than other ART algorithms not only in test effectiveness, but also in test efficiency and test coverage. Based on this article, the following conclusions can be drawn: first, considering the effectiveness of MRs and test cases in MT may lead to better results. In this way, most of the existing research can be improved by this method. This is the most important contribute of our research. Second, not only the source test cases, but also the follow-up test cases can improve the performance of MT. Therefore, they should be considered together during the process of the next test case generation. Third, the average distance performs better than the max distance and the minimum distance in metamorphic test case selection.

A Review of Software Reliability Testing Techniques

In the era of intelligent systems, the safety and reliability of software have received more attention. Software reliability testing is a significant method to ensure reliability, safety and quality of software. The intelligent software technology has not only offered new opportunities but also posed challenges to software reliability technology. The focus of this paper is to explore the software reliability testing technology under the impact of intelligent software technology. In this study, the basic theories of traditional software and intelligent software reliability testing were investigated via related previous works, and a general software reliability testing framework was established. Then, the technologies of software reliability testing were analyzed, including reliability modeling, test case generation, reliability evaluation, testing criteria and testing methods. Finally, the challenges and opportunities of software reliability testing technology were discussed at the end of this paper.

Research on automatic test case generation method of flight control system based on UML state diagram

In order to realize the automatic test of flight control system, the automatic generation method of flight control system test case is studied by using UML state diagram. Using UML extension mechanism, the real-time extension scheme of UML state diagram of flight control system is given. Aiming at the real-time performance of flight control system, the migration equivalence class is constructed by using the time domain equivalence partition method. After the test path is generated according to the corresponding test standard, the test case is generated automatically. Taking a simplified flight control system as an example, the feasibility and practicability of the method are verified by comparing the time of generating test cases with that of generating test cases manually.

Mutation testing with hyperproperties

We present a new method for model-based mutation-driven test case generation. Mutants are generated by making small syntactical modifications to the model or source code of the system under test. A test case kills a mutant if the behavior of the mutant deviates from the original system when running the test. In this work, we use hyperproperties—which allow to express relations between multiple executions—to formalize different notions of killing for both deterministic as well as non-deterministic models. The resulting hyperproperties are universal in the sense that they apply to arbitrary reactive models and mutants. Moreover, an off-the-shelf model checking tool for hyperproperties can be used to generate test cases. Furthermore, we propose solutions to overcome the limitations of current model checking tools via a model transformation and a bounded SMT encoding. We evaluate our approach on a number of models expressed in two different modeling languages by generating tests using a state-of-the-art mutation testing tool.

Test Case Generation Method based on Generative Adversarial Network

Traditional fuzzing tools generate low diversity of test cases and low vulnerability detection efficiency. This paper uses a test case generation model based on a generative confrontation network. The model uses LSTM as the generation network to generate data. The model uses fully connected network to construct a discriminant network for classification, and the test cases with the same protocol format are automatically generated after training. Finally the model uses the generated test cases to attack the protocol system to detect vulnerabilities. This method is compared with the traditional tools AFL, Peach and Sulley on the Modbus protocol, and the validity of the method is verified.

Software test data generation technology based on polymorphic particle swarm evolutionary algorithm

Combinatorial testing is a statute-based software testing method that aims to select a small number of valid test cases from a large combinatorial space of software under test to generate a set of test cases with high coverage and strong error debunking ability. However, combinatorial test case generation is an NP-hard problem that requires solving the combinatorial problem in polynomial time, so a meta-heuristic search algorithm is needed to solve the problem. Compared with other meta-heuristic search algorithms, the particle swarm algorithm is more competitive in terms of coverage table generation scale and execution time. In this paper, we systematically review and summarize the existing research results on generating combinatorial test case sets using particle swarm algorithm, and propose a combinatorial test case generation method that can handle arbitrary coverage strengths by combining the improved one-test-at-a-time strategy and the adaptive particle swarm algorithm for the variable strength combinatorial test problem and the parameter selection problem of the particle swarm algorithm. To address the parameter configuration problem of the particle swarm algorithm, the four parameters of inertia weight, learning factor, population size and iteration number are reasonably set, which makes the particle swarm algorithm more suitable for the generation of coverage tables. For the inertia weights.

Test Case Generation Method Based on User Operation

After the software is published, its users are mainly operators. Although they can find problems with the software, they cannot identify and reproduce software failures. When the software fails, there must be a lot of information related to the failure in the running data of the software to be tested. Therefore, on the basis of obtaining the running data of the software under test, a test case generation method based on user operation is proposed. Based on the generation of software failure, the operation related to the failure is filtered out from the user operation. By taking these operations as the values of the system parameters to be tested, test cases that will inevitably cause software failure are obtained. Finally, the validity of the method is verified by an example, and the test cases obtained are specific to the faults of the software.

Automated Test Case Generation for Effective Spectrum-based Fault Localization

Software testing is the key to ensuring software quality. Test case generation and software fault localization are two important research objects of testing. Spectrum-based fault localization (SBFL) method is a mainstream dynamic fault localization method, which inputs test cases into the program for execution and collects statement coverage information and execution results. By analysing this information, a fault report can be generated. The performance of SBFL is affected by test case quality. Therefore, this paper proposes a test case generation method based on improved genetic algorithm to assist SBFL. This method uses a small number of initial test cases to execute the program, then uses the XGBoost model to pre-process the spectrum information and generates a list of potential fault statements to guide the iterative process of the genetic algorithm. The experimental results show that the test cases generated using the improved genetic algorithm have certain advantages over the test cases generated by general genetic algorithm and the baseline test cases.

Test Case Generation Method for Increasing Software Reliability in Safety-Critical Embedded Systems

Finite-state machines (FSMs) and the W method have been widely used in software testing. However, the W method fails to detect post-processing errors in the implementation under test (IUT) because it ends testing when it encounters a previously visited state. To alleviate this issue, we propose an enhanced fault-detection W method. The proposed method does not stop the test, even if it has reached a previously visited state; it continues to test and check the points that the W method misses. Through various case studies, we demonstrated software testing using the W method and the proposed method. From the results, it can be inferred that the proposed method can more explicitly determine the consistency between design and implementation, and it is a better option for testing larger software. Unfortunately, the testing time of the proposed method is approximately 1.4 times longer than that of the W method because of the added paths. However, our method is more appropriate than the W method for software testing in safety-critical systems, even if this method is time consuming. This is because the error-free characteristics of a safety-critical system are more important than anything else. As a result, our method can be used to increase software reliability in safety-critical embedded systems.

Web service composition on IoT reliability test based on cross entropy

AbstractWeb service has developed the managed IoT application to let connected devices easily and securely interact with cloud applications and other devices. As an important factor for web service, the reliability of web services refers to the probability of web service running success. For modeling web service composition, we should abstract the process of web service composition. Due to the diversity and complexity of web service composition, it is unlikely to do exhaustive testing. In order to improve the quality of web service composition test cases and find out which path leads to the greatest probability of service combination failure, heuristic test case generation method is adopted to obtain the optimal test path. First, the web service composition test is abstracted into the MDP model. The QoS of the web service composition is taken as the software test optimization goal, and the cross‐entropy strategy is used to optimize the test case. The experimental results show that the test profile given by the cross‐strategy is better than the random test strategy. Detect and exclude the same number of software defects. Cross‐entropy strategy can significantly reduce the number of test cases, reduce test costs, and improve defect detection efficiency.

Towards A Test Paths Generation Method for CTCS Level Transition

Test case is an important basis for correctness and safety verification of Chinese Train Control System (CTCS). Focusing on the test cases generation method of UPPAAL which is widely used in CTCS testing activity, the problems are analyzed, and an improved test cases generation method for CTCS is proposed. First, the process and characteristics of UPPAAL test cases generation method are analyzed; then the test requirements of CTCS are studied, and a test cases generation method based on UPPAAL query file is proposed. Finally, taking the level transition function of CTCS as an example, test cases are generated by the proposed method, which shows that this method can meet the test requirements of CTCS.

Model-Based Test Suite Generation Using Mutation Analysis for Fault Localization

Fault localization techniques reduce the effort required when debugging software, as revealed by previous test cases. However, many test cases are required to reduce the number of candidate fault locations. To overcome this disadvantage, various methods were proposed to reduce fault-localization costs by prioritizing test cases. However, because a sufficient number of test cases is required for prioritization, the test-case generation cost remains high. This paper proposes a test-case generation method using a state chart to reduce the number of test suites required for fault localization, minimizing the test-case generation and execution times. The test-suite generation process features two phases: fault-detection test-case generation and fault localization in the test cases. Each phase uses mutation analysis to evaluate test cases; the results are employed to improve the test cases according to the objectives of each phase, using genetic algorithms. We provide useful guidelines for application of a search-based mutational method to a state chart; we show that the proposed method improves fault-localization performance in the test-suite generation phase.

An Automatic Test Case Generation Method based on SysML Activity Diagram

Model-Based Systems Engineering (MBSE) is a new method to engineer the modern large, complex systems. MBSE has a good guarantee for system reliability and it can improve the efficiency of system development. So, MBSE has received extensive attention from academia and industry. SysML is an auxiliary language for MBSE. SysML activity diagram can model the behavior of the system. In this paper, we use SysML activity diagram to generate test case. There are two main things in our work. One is that we have designed an automated SysML activity diagram test case generation method. Second, we develop a test case generation tool to support related applications in the industry.