Test Case Generation Method Research Articles

TAEFuzz : Automatic Fuzzing for Image-based Deep Learning Systems via Transferable Adversarial Examples

Deep learning (DL) components have been broadly applied in diverse applications. Similar to traditional software engineering, effective test case generation methods are needed by industry to enhance the quality and robustness of these deep learning components. To this end, we propose a novel automatic software testing technique, TAEFuzz (Automatic Fuzz -Testing via T ransferable A dversarial E xamples), which aims to automatically assess and enhance the robustness of image-based deep learning (DL) systems based on test cases generated by transferable adversarial examples. TAEFuzz alleviates the over-fitting problem during optimized test case generation and prevents test cases from prematurely falling into local optima. In addition, TAEFuzz enhances the visual quality of test cases through constraining perturbations inserted into sensitive areas of the images. For a system with low robustness, TAEFuzz trains a low-cost denoising module to reduce the impact of perturbations in transferable adversarial examples on the system. Experimental results demonstrate that the test cases generated by TAEFuzz can discover up to 46.1% more errors in the targeted systems, and ensure the visual quality of test cases. Compared to existing techniques, TAEFuzz also enhances the robustness of the target systems against transferable adversarial examples with the perturbation denoising module.

Relevant Information in TDD Experiment Reporting

Experiments are a commonly used method of research in software engineering (SE). Researchers report their experiments following detailed guidelines. However, researchers do not, in the field of test-driven development (TDD) at least, specify how they operationalized the response variables and, particularly, the measurement process. This article has three aims: (i) identify the response variable operationalization components in TDD experiments that study external quality; (ii) study their influence on the experimental results; (iii) determine if the experiment reports describe the measurement process components that have an impact on the results. We used two-part sequential mixed methods research. The first part of the research adopts a quantitative approach applying a statistical analysis of the impact of the operationalization components on the experimental results. The second part follows with a qualitative approach applying a systematic mapping study (SMS). The test suites, intervention types and measurers have an influence on the measurements and results of the statistical analysis of TDD experiments in SE. The test suites have a major impact on both the measurements and the results of the experiments. The intervention type has less impact on the results than on the measurements. While the measurers have an impact on the measurements, this is not transferred to the experimental results. On the other hand, the results of our SMS confirm that TDD experiments do not usually report either the test suites, the test case generation method, or the details of how external quality was measured. A measurement protocol should be used to ensure that the measurements made by different measurers are similar. It is necessary to report the test cases, the experimental task and the intervention type in order to be able to reproduce the measurements and statistical analyses, as well as to replicate experiments and build dependable families of experiments.

DynER: Optimized Test Case Generation for Representational State Transfer (REST)ful Application Programming Interface (API) Fuzzers Guided by Dynamic Error Responses

Modern web services widely provide RESTful APIs for clients to access their functionality programmatically. Fuzzing is an emerging technique for ensuring the reliability of RESTful APIs. However, the existing RESTful API fuzzers repeatedly generate invalid requests due to unawareness of errors in the invalid tested requests and lack of effective strategy to generate legal value for the incorrect parameters. Such limitations severely hinder the fuzzing performance. In this paper, we propose DynER, a new test case generation method guided by dynamic error responses during fuzzing. DynER designs two strategies of parameter value generation for purposefully revising the incorrect parameters of invalid tested requests to generate new test requests. The strategies are, respectively, based on prompting Large Language Model (LLM) to understand the semantics information in error responses and actively accessing API-related resources. We apply DynER to the state-of-the-art fuzzer RESTler and implement DynER-RESTler. DynER-RESTler outperforms foREST on two real-world RESTful services, WordPress and GitLab with a 41.21% and 26.33% higher average pass rate for test requests and a 12.50% and 22.80% higher average number of unique request types successfully tested, respectively. The experimental results demonstrate that DynER significantly improves the effectiveness of test cases and fuzzing performance. Additionally, DynER-RESTler finds three new bugs.

A framework to identify failure scenarios in the control mode transition process for autonomous ships with dynamic autonomy

The dynamic autonomy resulting from control mode transitions in the Maritime Autonomous Surface Ships (MASS) poses a significant risk to its navigation safety. While Systems-Theoretic Process Analysis (STPA) has proven effective in the hazard identification of autonomous ships, the safety challenges inherent in the dynamic autonomy of MASS with seafarers onboard controlled by three possible controllers have not been addressed much. This study proposes a three-phase framework that combines STPA, state machines and Sequentially Timed Events Plotting (STEP) diagram aiming at identifying failure scenarios in the transition process. Findings from a case study highlight the need for such a framework to understand the triggering events for mode transition, the transition process, how transition failure may occur, and refine safety constraints to ensure smooth and safe transitions. The results lay the groundwork for future research into test case generation methods that focus on these identified transition failure scenarios, intending to ensure safe and secure navigation for the progressive introduction of MASS.

Hierarchical Distribution-aware Testing of Deep Learning

With its growing use in safety/security-critical applications, Deep Learning (DL) has raised increasing concerns regarding its dependability. In particular, DL has a notorious problem of lacking robustness. Input added with adversarial perturbations, i.e., Adversarial Examples (AEs) , are easily mispredicted by the DL model. Despite recent efforts made in detecting AEs via state-of-the-art attack and testing methods, they are normally input distribution–agnostic and/or disregard the perceptual quality of adversarial perturbations. Consequently, the detected AEs are irrelevant inputs in the application context or noticeably unrealistic to humans. This may lead to a limited effect on improving the DL model’s dependability, as the testing budget is likely to be wasted on detecting AEs that are encountered very rarely in its real-life operations. In this article, we propose a new robustness testing approach for detecting AEs that considers both the feature-level distribution and the pixel-level distribution, capturing the perceptual quality of adversarial perturbations. The two considerations are encoded by a novel hierarchical mechanism. First, we select test seeds based on the density of feature-level distribution and the vulnerability of adversarial robustness. The vulnerability of test seeds is indicated by the auxiliary information, which are highly correlated with local robustness. Given a test seed, we then develop a novel genetic algorithm–based local test case generation method, in which two fitness functions work alternatively to control the perceptual quality of detected AEs. Finally, extensive experiments confirm that our holistic approach considering hierarchical distributions is superior to the state-of-the-arts that either disregard any input distribution or only consider a single (non-hierarchical) distribution, in terms of not only detecting imperceptible AEs but also improving the overall robustness of the DL model under testing.

Improving Coverage and Vulnerability Detection in Smart Contract Testing Using Self-Adaptive Learning GA

In the domain of software testing, the generation of test cases is a critical process for detecting system errors and bugs. However, automated test case generation for smart contracts often encounters challenges related to automation, vulnerability diversity, and coverage. This paper presents a novel method, the self-adaptive learning Genetic Algorithm (self-adaptive learning GA), designed to address these issues. Our research methodology incorporates several construction models, namely the Control Dependence Graph (CDG), Control Flow Graph (CFG), and Application Binary Interface (ABI). Initially, the ABI model provides essential information for generating and executing test cases. The CFG model subsequently visualizes potential execution paths through the functions of smart contracts. Ultimately, the CDG model identifies potential vulnerabilities in smart contracts. Using these models, our method enhances automatic test case generation in smart contracts by improving coverage and reducing execution time. We selected a variety of smart contracts from the Decentralized Finance (DeFi) ecosystem for data collection and comparative analysis. The experimental results show superior performance rates, with an average code coverage rate of 98.1%, a total of 3500 vulnerabilities detected, a vulnerability detection rate of 98.7%, a false positive rate of 1.3%, a recall of 98.2%, precision of 98.8%, a path uniqueness rate of 96.4%, false negative rate of 3.5%, an execution time of 25 s, and test case generation time of 16 s. In conclusion, our proposed approach demonstrates a significant improvement over existing methods for test case generation by providing a promising solution for the robustness of smart contracts and security enhancement in the DeFi ecosystem.

CanaryExp: A Canary-Sensitive Automatic Exploitability Evaluation Solution for Vulnerabilities in Binary Programs

We propose CanaryExp, an exploitability evaluation solution for vulnerabilities among binary programs protected by StackGuard. CanaryExp devises three novel techniques, namely canary leakage proof of concept generation, canary leaking analysis time exploitation, and dynamic canary-relocation-based exploitability evaluation. The canary leakage proof of concept input generation mechanism first traces the target program’s execution, transforming the execution state into some canary leaking state, from which some canary leaking input is derived. This input can be deemed as proof that some vulnerability that can lead to canary leakage exists. The canary leaking analysis time exploit generation then performs incremental analysis based on the canary leaking input, crafting analysis time exploit that can complete vulnerability exploitation in the analysis time environment. Based on the analysis time exploit, the dynamic canary-relocation-based exploitability evaluation component collects the necessary metadata, on which an exploitation session is automatically constructed that can not only leak the runtime canary and relocate it in the input stream but also evaluate the exploitability of the desired vulnerability. Using a benchmark containing six test programs, eight challenges from some network challenging events and four real-world applications, we demonstrate that CanaryExp can generate canary leaking samples more effectively than existing test case generation methods and automatically evaluate the exploitability for vulnerabilities among programs where the StackGuard protection mechanism is deployed.

DTester: Diversity-Driven Test Case Generation for Web Applications

Search-based Test Case Generation (TCG) for web applications suffers from unstable performance and suboptimal test suite problems due to diversity loss. However, previous diversity metrics mainly only focus on client-side models or server-side code, which are prone to low robustness and poor generalization in practical applications. We propose a diversity-driven TCG method DTester, which can maximize behavior exploration and minimize the test suite size while covering more server-side vulnerable paths. Three diversity metrics (i.e. phenotypic coupling, intent coupling and competitiveness) are proposed to measure the underlying relationship between test cases from user behavior, code logic and test execution history. Moreover, a 3-dimensional weight graph is designed to model association among metrics, which provides fine-grained guidance for the genetic algorithm to generate diverse test cases from the client-side behavior model. Our empirical evaluation on five web applications shows that DTester can efficiently and robustly generate better test suites than the state-of-the-art TCG method. The maximum improvement is [Formula: see text], [Formula: see text], [Formula: see text] and [Formula: see text] in efficiency, test suite size, diversity and robustness.

Automatic Generation of Test Cases Based on Genetic Algorithm and RBF Neural Network

Software testing plays an important role in improving the quality of software, but the design of test cases requires a lot of manpower, material resources, and time, and designers tend to be subjective when designing test cases. To solve this problem and make the test cases have objectivity and greater coverage, a branch coverage test case automatic generation method based on genetic algorithm and RBF neural network algorithm (GAR) is proposed. In terms of test case generation, based on the genetic algorithm optimized in this paper, a certain number of test case samples are randomly selected to train the RBF neural network to simulate the fitness function and to calculate the individual fitness value. The experiment uses 7 C language codes to automatically generate test cases and compares the experimental data generated by the branch coverage test case generation method based on adaptive genetic algorithm (PDGA), traditional genetic algorithm (SGA), and random test generation method (random) to evaluate the proposed algorithm. The experimental results show that the method is feasible and effective, the branch coverage is increased in the generation of test cases, and the number of iterations of the population is less.

MT-ART: A Test Case Generation Method Based on Adaptive Random Testing and Metamorphic Relation

Most of metamorphic testing (MT) research works focused on the generation and application of metamorphic relations (MRs). There is no clear conclusion about the relationship between test case generation methods and performance of MT. In this article, we introduce a novel method based on adaptive random testing (ART) and MR for MT test case generation. It proposes a family of algorithms for MT test cases generation, named as MT based ART (MT-ART). Three distances are measured to generate the next MT test case. In order to verify the performance of this method, series of experiments on four programs with different numbers of inputs are introduced. The results show that MT-ART performs better than other ART algorithms not only in test effectiveness, but also in test efficiency and test coverage. Based on this article, the following conclusions can be drawn: first, considering the effectiveness of MRs and test cases in MT may lead to better results. In this way, most of the existing research can be improved by this method. This is the most important contribute of our research. Second, not only the source test cases, but also the follow-up test cases can improve the performance of MT. Therefore, they should be considered together during the process of the next test case generation. Third, the average distance performs better than the max distance and the minimum distance in metamorphic test case selection.

A Review of Software Reliability Testing Techniques

In the era of intelligent systems, the safety and reliability of software have received more attention. Software reliability testing is a significant method to ensure reliability, safety and quality of software. The intelligent software technology has not only offered new opportunities but also posed challenges to software reliability technology. The focus of this paper is to explore the software reliability testing technology under the impact of intelligent software technology. In this study, the basic theories of traditional software and intelligent software reliability testing were investigated via related previous works, and a general software reliability testing framework was established. Then, the technologies of software reliability testing were analyzed, including reliability modeling, test case generation, reliability evaluation, testing criteria and testing methods. Finally, the challenges and opportunities of software reliability testing technology were discussed at the end of this paper.

Research on automatic test case generation method of flight control system based on UML state diagram

In order to realize the automatic test of flight control system, the automatic generation method of flight control system test case is studied by using UML state diagram. Using UML extension mechanism, the real-time extension scheme of UML state diagram of flight control system is given. Aiming at the real-time performance of flight control system, the migration equivalence class is constructed by using the time domain equivalence partition method. After the test path is generated according to the corresponding test standard, the test case is generated automatically. Taking a simplified flight control system as an example, the feasibility and practicability of the method are verified by comparing the time of generating test cases with that of generating test cases manually.

Mutation testing with hyperproperties

We present a new method for model-based mutation-driven test case generation. Mutants are generated by making small syntactical modifications to the model or source code of the system under test. A test case kills a mutant if the behavior of the mutant deviates from the original system when running the test. In this work, we use hyperproperties—which allow to express relations between multiple executions—to formalize different notions of killing for both deterministic as well as non-deterministic models. The resulting hyperproperties are universal in the sense that they apply to arbitrary reactive models and mutants. Moreover, an off-the-shelf model checking tool for hyperproperties can be used to generate test cases. Furthermore, we propose solutions to overcome the limitations of current model checking tools via a model transformation and a bounded SMT encoding. We evaluate our approach on a number of models expressed in two different modeling languages by generating tests using a state-of-the-art mutation testing tool.

Test Case Generation Method based on Generative Adversarial Network

Traditional fuzzing tools generate low diversity of test cases and low vulnerability detection efficiency. This paper uses a test case generation model based on a generative confrontation network. The model uses LSTM as the generation network to generate data. The model uses fully connected network to construct a discriminant network for classification, and the test cases with the same protocol format are automatically generated after training. Finally the model uses the generated test cases to attack the protocol system to detect vulnerabilities. This method is compared with the traditional tools AFL, Peach and Sulley on the Modbus protocol, and the validity of the method is verified.

Software test data generation technology based on polymorphic particle swarm evolutionary algorithm

Combinatorial testing is a statute-based software testing method that aims to select a small number of valid test cases from a large combinatorial space of software under test to generate a set of test cases with high coverage and strong error debunking ability. However, combinatorial test case generation is an NP-hard problem that requires solving the combinatorial problem in polynomial time, so a meta-heuristic search algorithm is needed to solve the problem. Compared with other meta-heuristic search algorithms, the particle swarm algorithm is more competitive in terms of coverage table generation scale and execution time. In this paper, we systematically review and summarize the existing research results on generating combinatorial test case sets using particle swarm algorithm, and propose a combinatorial test case generation method that can handle arbitrary coverage strengths by combining the improved one-test-at-a-time strategy and the adaptive particle swarm algorithm for the variable strength combinatorial test problem and the parameter selection problem of the particle swarm algorithm. To address the parameter configuration problem of the particle swarm algorithm, the four parameters of inertia weight, learning factor, population size and iteration number are reasonably set, which makes the particle swarm algorithm more suitable for the generation of coverage tables. For the inertia weights.

Test Case Generation Method Based on User Operation

After the software is published, its users are mainly operators. Although they can find problems with the software, they cannot identify and reproduce software failures. When the software fails, there must be a lot of information related to the failure in the running data of the software to be tested. Therefore, on the basis of obtaining the running data of the software under test, a test case generation method based on user operation is proposed. Based on the generation of software failure, the operation related to the failure is filtered out from the user operation. By taking these operations as the values of the system parameters to be tested, test cases that will inevitably cause software failure are obtained. Finally, the validity of the method is verified by an example, and the test cases obtained are specific to the faults of the software.

Automated Test Case Generation for Effective Spectrum-based Fault Localization

Software testing is the key to ensuring software quality. Test case generation and software fault localization are two important research objects of testing. Spectrum-based fault localization (SBFL) method is a mainstream dynamic fault localization method, which inputs test cases into the program for execution and collects statement coverage information and execution results. By analysing this information, a fault report can be generated. The performance of SBFL is affected by test case quality. Therefore, this paper proposes a test case generation method based on improved genetic algorithm to assist SBFL. This method uses a small number of initial test cases to execute the program, then uses the XGBoost model to pre-process the spectrum information and generates a list of potential fault statements to guide the iterative process of the genetic algorithm. The experimental results show that the test cases generated using the improved genetic algorithm have certain advantages over the test cases generated by general genetic algorithm and the baseline test cases.

Test Case Generation Method for Increasing Software Reliability in Safety-Critical Embedded Systems

Finite-state machines (FSMs) and the W method have been widely used in software testing. However, the W method fails to detect post-processing errors in the implementation under test (IUT) because it ends testing when it encounters a previously visited state. To alleviate this issue, we propose an enhanced fault-detection W method. The proposed method does not stop the test, even if it has reached a previously visited state; it continues to test and check the points that the W method misses. Through various case studies, we demonstrated software testing using the W method and the proposed method. From the results, it can be inferred that the proposed method can more explicitly determine the consistency between design and implementation, and it is a better option for testing larger software. Unfortunately, the testing time of the proposed method is approximately 1.4 times longer than that of the W method because of the added paths. However, our method is more appropriate than the W method for software testing in safety-critical systems, even if this method is time consuming. This is because the error-free characteristics of a safety-critical system are more important than anything else. As a result, our method can be used to increase software reliability in safety-critical embedded systems.

Web service composition on IoT reliability test based on cross entropy

AbstractWeb service has developed the managed IoT application to let connected devices easily and securely interact with cloud applications and other devices. As an important factor for web service, the reliability of web services refers to the probability of web service running success. For modeling web service composition, we should abstract the process of web service composition. Due to the diversity and complexity of web service composition, it is unlikely to do exhaustive testing. In order to improve the quality of web service composition test cases and find out which path leads to the greatest probability of service combination failure, heuristic test case generation method is adopted to obtain the optimal test path. First, the web service composition test is abstracted into the MDP model. The QoS of the web service composition is taken as the software test optimization goal, and the cross‐entropy strategy is used to optimize the test case. The experimental results show that the test profile given by the cross‐strategy is better than the random test strategy. Detect and exclude the same number of software defects. Cross‐entropy strategy can significantly reduce the number of test cases, reduce test costs, and improve defect detection efficiency.

Towards A Test Paths Generation Method for CTCS Level Transition

Test case is an important basis for correctness and safety verification of Chinese Train Control System (CTCS). Focusing on the test cases generation method of UPPAAL which is widely used in CTCS testing activity, the problems are analyzed, and an improved test cases generation method for CTCS is proposed. First, the process and characteristics of UPPAAL test cases generation method are analyzed; then the test requirements of CTCS are studied, and a test cases generation method based on UPPAAL query file is proposed. Finally, taking the level transition function of CTCS as an example, test cases are generated by the proposed method, which shows that this method can meet the test requirements of CTCS.