Test Case Generation Research Articles

A framework to identify failure scenarios in the control mode transition process for autonomous ships with dynamic autonomy

The dynamic autonomy resulting from control mode transitions in the Maritime Autonomous Surface Ships (MASS) poses a significant risk to its navigation safety. While Systems-Theoretic Process Analysis (STPA) has proven effective in the hazard identification of autonomous ships, the safety challenges inherent in the dynamic autonomy of MASS with seafarers onboard controlled by three possible controllers have not been addressed much. This study proposes a three-phase framework that combines STPA, state machines and Sequentially Timed Events Plotting (STEP) diagram aiming at identifying failure scenarios in the transition process. Findings from a case study highlight the need for such a framework to understand the triggering events for mode transition, the transition process, how transition failure may occur, and refine safety constraints to ensure smooth and safe transitions. The results lay the groundwork for future research into test case generation methods that focus on these identified transition failure scenarios, intending to ensure safe and secure navigation for the progressive introduction of MASS.

Random Testing and Evolutionary Testing for Fuzzing GraphQL APIs

The Graph Query Language (GraphQL) is a powerful language for application programming interface (API) manipulation in web services. It has been recently introduced as an alternative solution for addressing the limitations of RESTful APIs. This article introduces an automated solution for GraphQL API testing. We present a full framework for automated API testing, from the schema extraction to test case generation. In addition, we consider two kinds of testing: white-box and black-box testing. The white-box testing is performed when the source code of the GraphQL API is available. Our approach is based on evolutionary search. Test cases are evolved to intelligently explore the solution space while maximizing code coverage and fault-finding criteria. The black-box testing does not require access to the source code of the GraphQL API. It is therefore of more general applicability, albeit it has worse performance. In this context, we use a random search to generate GraphQL data. The proposed framework is implemented and integrated into the open source EvoMaster tool. With enabled white-box heuristics (i.e., white-box mode), experiments on 7 open source GraphQL APIs and three search algorithms show statistically significant improvement of the evolutionary approach compared to the baseline random search. In addition, experiments on 31 online GraphQL APIs reveal the ability of the black-box mode to detect real faults.

CrossFuzz: Cross-contract fuzzing for smart contract vulnerability detection

Context:Smart contracts are computer programs that run on a blockchain. As the functions implemented by smart contracts become increasingly complex, the number of cross-contract interactions within them also rises. Consequently, the combinatorial explosion of transaction sequences poses a significant challenge for smart contract security vulnerability detection. Existing static analysis-based methods for detecting cross-contract vulnerabilities suffer from high false-positive rates and cannot generate test cases, while fuzz testing-based methods exhibit low code coverage and may not accurately detect security vulnerabilities. Objective:The goal of this paper is to address the above limitations and efficiently detect cross-contract vulnerabilities. To achieve this goal, we present CrossFuzz, a fuzz testing-based method for detecting cross-contract vulnerabilities. Method:First, CrossFuzz generates parameters of constructors by tracing data propagation paths. Then, it collects inter-contract data flow information. Finally, CrossFuzz optimizes mutation strategies for transaction sequences based on inter-contract data flow information to improve the performance of fuzz testing. Results:We implemented CrossFuzz, which is an extension of ConFuzzius, and conducted experiments on a real-world dataset containing 396 smart contracts. The results show that CrossFuzz outperforms xFuzz, a fuzz testing-based tool optimized for cross-contract vulnerability detection, with a 10.58% increase in bytecode coverage. Furthermore, CrossFuzz detects 1.82 times more security vulnerabilities than ConFuzzius. Conclusion:Our method utilizes data flow information to optimize mutation strategies. It significantly improves the efficiency of fuzz testing for detecting cross-contract vulnerabilities.

Test Case Generation Evaluator for the Implementation of Test Case Generation Algorithms Based on Learning to Rank

Test Case Generation Evaluator for the Implementation of Test Case Generation Algorithms Based on Learning to Rank

Generating automated test case from sequence diagram using Pre-order Traversal

Automated test case generation is important in software testing to evaluate developed software. The testing process requires test cases that can be based on Software Requirements Specification (SRS) and Software Design Document (SDD). One of UML diagrams included in the previous documentations is sequence diagram. This research proposed a modified pre-order algorithm that can generate test cases automatically from sequence diagrams. Our manual cyclomatic complexity evaluation showed that our method produced accurate test cases based on the sequence of messages in the sequence diagram. This result justifies that the generated test cases match the relevant sequence diagram.

Effective Test Cases Generation with Harmony Search and RBF Neural Network

Software testing is one of the integral activities during development of software products. Generation and selection of the test cases either in static or dynamic form play pivot role for ensuring the quality of software products. There are numerous approaches in the literature for automatic generation of test cases but coverage criteria and fault detection rate are prominent metrics for checking the effectiveness of the software products during testing phase of software development. In the present, a new Harmony Radial Testing (HRT) is proposed by combining the concepts of Harmony Search Algorithm (HSA) and Radial Basis Function-Neural Network (RBF-NN) approaches. The main objective of the proposed HRT method is to generate automatic test cases by considering the criteria of branch coverage with improvement in the Maximum branch Coverage (MaxC), Average Coverage (AC) and Average Percentage Fault Detection (APFD) rates. The proposed approach combined with the Radial Basis Function (RBF), denoted as a HRT approach. The proposed approach is used to optimize harmony search over the randomly selected sample test cases, training the RBF-NN to simulate the fitness function. Seven Python codes have been tested through proposed approach and computed results are compared with Primal-Dual Genetic Algorithm (PDGA), Simple Genetic Algorithm (SGA) and random methods. It is observed that the proposed HRT algorithm optimizes consistently yielded reliable results, which may be used in future for enriching the software testing process by the software industries.

Measuring Software Testability via Automatically Generated Test Cases

Measuring Software Testability via Automatically Generated Test Cases

Non-bio-inspired Metaheuristics in Software Testing: A Systematic Literature Review

The software testing phase usually consumes a lot of the development of software projects time in order to find defects before release. Different strategies have been approached to optimize this phase of the testing stage. Metaheuristics are important in software testing due to their ability to find optimal or near-optimal solutions in complex situations. This research aims to analyze the current status of the application of metaheuristics that assist in software testing phase activities, specifically the most representative Non-bio-inspired algorithms (NBA) are surveyed, being Hill Climbing the most reported. The main activities of the software testing where NBA were implemented, were test case and test data generation and test case prioritization. It was concluded that NBAs used on their own are only viable in some activities of the software testing phase. As future work, it is proposed to investigate the use of hybrid algorithms and approaches in software testing phase.

Test Case Prioritization via Embedded Autoencoder Model for Software Quality Assurance

Software quality can be ensured by passing the process of software testing before the software is released. However, the software testing process involves many phases which leads to more resources and time consumption. Test Case Prioritization (TCP) has gained widespread acceptance because it prioritizes the tasks and reduces the number of phases and also produces results in good quality software free from defects. The coverage-based prioritization can be useful to distinguish each test case and result in a better prioritization process by using some algorithm. In this work, we propose a coverage-based prioritized test case generation using an Embedded Auto Encoder (EAE) algorithm which will produce an ordered sequence of the prioritized test cases. Initially, the code coverage for each benchmark has been extracted from the source code repository and is further processed to eliminate the noise in the data. The processed data will be given to the embedded autoencoder which consists of an autoencoder and a sparse autoencoder. Once the modeling is done, the model has been trained with the data generated by the Keras Data Generator class. The efficiency of the proposed EAE technique has been evaluated by using the APFD metric and the observations clearly show that the EAE framework proves to provide an APFD metric of 0.72 on average which is a good value in comparison to the previously deployed methodologies.

Generation-based Differential Fuzzing for Deep Learning Libraries

Deep learning (DL) libraries have become the key component in developing and deploying DL-based software nowadays. With the growing popularity of applying DL models in both academia and industry across various domains, any bugs inherent in the DL libraries can potentially cause unexpected server outcomes. As such, there is an urgent demand for improving the software quality of DL libraries. Although there are some existing approaches specifically designed for testing DL libraries, their focus is usually limited to one specific domain, such as computer vision (CV). It is still not very clear how the existing approaches perform in detecting bugs of different DL libraries regarding different task domains and to what extent. To bridge this gap, we first conduct an empirical study on four representative and state-of-the-art DL library testing approaches. Our empirical study results reveal that it is hard for existing approaches to generalize to other task domains. We also find that the test inputs generated by these approaches usually lack diversity, with only a few types of bugs. What is worse, the false-positive rate of existing approaches is also high ( up to 58% ). To address these issues, we propose a guided differential fuzzing approach based on generation , namely, Gandalf . To generate testing inputs across diverse task domains effectively, Gandalf adopts the context-free grammar to ensure validity and utilizes a Deep Q-Network to maximize the diversity. Gandalf also includes 15 metamorphic relations to make it possible for the generated test cases to generalize across different DL libraries. Such a design can decrease the false positives because of the semantic difference for different APIs. We evaluate the effectiveness of Gandalf on nine versions of three representative DL libraries, covering 309 operators from computer vision, natural language processing, and automated speech recognition. The evaluation results demonstrate that Gandalf can effectively and efficiently generate diverse test inputs. Meanwhile, Gandalf successfully detects five categories of bugs with only 3.1% false-positive rates. We report all 49 new unique bugs found during the evaluation to the DL libraries’ developers, and most of these bugs have been confirmed. Details about our empirical study and evaluation results are available on our project website. 1

Hierarchical Distribution-aware Testing of Deep Learning

With its growing use in safety/security-critical applications, Deep Learning (DL) has raised increasing concerns regarding its dependability. In particular, DL has a notorious problem of lacking robustness. Input added with adversarial perturbations, i.e., Adversarial Examples (AEs) , are easily mispredicted by the DL model. Despite recent efforts made in detecting AEs via state-of-the-art attack and testing methods, they are normally input distribution–agnostic and/or disregard the perceptual quality of adversarial perturbations. Consequently, the detected AEs are irrelevant inputs in the application context or noticeably unrealistic to humans. This may lead to a limited effect on improving the DL model’s dependability, as the testing budget is likely to be wasted on detecting AEs that are encountered very rarely in its real-life operations. In this article, we propose a new robustness testing approach for detecting AEs that considers both the feature-level distribution and the pixel-level distribution, capturing the perceptual quality of adversarial perturbations. The two considerations are encoded by a novel hierarchical mechanism. First, we select test seeds based on the density of feature-level distribution and the vulnerability of adversarial robustness. The vulnerability of test seeds is indicated by the auxiliary information, which are highly correlated with local robustness. Given a test seed, we then develop a novel genetic algorithm–based local test case generation method, in which two fitness functions work alternatively to control the perceptual quality of detected AEs. Finally, extensive experiments confirm that our holistic approach considering hierarchical distributions is superior to the state-of-the-arts that either disregard any input distribution or only consider a single (non-hierarchical) distribution, in terms of not only detecting imperceptible AEs but also improving the overall robustness of the DL model under testing.

Improving Coverage and Vulnerability Detection in Smart Contract Testing Using Self-Adaptive Learning GA

In the domain of software testing, the generation of test cases is a critical process for detecting system errors and bugs. However, automated test case generation for smart contracts often encounters challenges related to automation, vulnerability diversity, and coverage. This paper presents a novel method, the self-adaptive learning Genetic Algorithm (self-adaptive learning GA), designed to address these issues. Our research methodology incorporates several construction models, namely the Control Dependence Graph (CDG), Control Flow Graph (CFG), and Application Binary Interface (ABI). Initially, the ABI model provides essential information for generating and executing test cases. The CFG model subsequently visualizes potential execution paths through the functions of smart contracts. Ultimately, the CDG model identifies potential vulnerabilities in smart contracts. Using these models, our method enhances automatic test case generation in smart contracts by improving coverage and reducing execution time. We selected a variety of smart contracts from the Decentralized Finance (DeFi) ecosystem for data collection and comparative analysis. The experimental results show superior performance rates, with an average code coverage rate of 98.1%, a total of 3500 vulnerabilities detected, a vulnerability detection rate of 98.7%, a false positive rate of 1.3%, a recall of 98.2%, precision of 98.8%, a path uniqueness rate of 96.4%, false negative rate of 3.5%, an execution time of 25 s, and test case generation time of 16 s. In conclusion, our proposed approach demonstrates a significant improvement over existing methods for test case generation by providing a promising solution for the robustness of smart contracts and security enhancement in the DeFi ecosystem.

Research on Specific Scenario Generation Methods for Autonomous Driving Simulation Tests

In this paper, we propose a method for the generation of simulated test scenarios for autonomous driving. Based on the requirements of standard regulatory test scenarios, we can generate virtually simulated scenarios and functional scenario libraries for autonomous driving, which can be used for the simulated verification of different ADAS functions. Firstly, the operational design domain (ODD) of a functional scenario is selected, and the weight values of the ODD elements are calculated. Then, a combination test algorithm based on parameter weights is improved to generate virtually simulated autonomous driving test cases for the ODD elements, which can effectively reduce the number of generated test cases compared with the traditional combination test algorithm. Then, the traffic participant elements in each test case are sampled and clustered so as to obtain hazard-specific scenarios. Then, the values of the subelements under the traffic participant element in each test case are sampled and clustered to obtain hazard-specific scenarios. Finally, the specific scenarios are applied to the automatic emergency braking (AEB) system on the model-in-the-loop (MIL) testbed to verify the effectiveness of this scenario generation method.

Investigating the robustness of locators in template-based Web application testing using a GUI change classification model

GUI-based test-cases generated by Capture and Replay tools suffer from the well-known fragility problem: they may break even if small layout changes are operated in a Web application, without modifying the app functionality. An approach based on the automatic injection of HTML tag attributes named hooks in the source code of Web templates has been recently proposed to solve this problem. Such hooks allow the unique identification of GUI items to be located during test case execution. This technique showed its effectiveness in a preliminary validation study, where it allowed to significantly reduce the number of test case locator breakages in regression testing of student-made Web applications. This paper presents a further validation study where we compared the robustness of hook-based test cases against state-of-the-art and state-of-the-practice techniques for locating GUI objects. We proposed a three dimensional model for classifying different types of layout changes and used it to define a benchmark of realistic changes. Thanks to the model, we systematically compared the robustness of test cases generated by different techniques with respect to specific types of changes and studied the relationship between fragility issues and types of changes in different test case generation techniques.

Diversity-aware fairness testing of machine learning classifiers through hashing-based sampling

Context:There are growing concerns about algorithmic fairness, as some machine learning (ML)-based algorithms have been found to exhibit biases against protected attributes such as gender, race, age and so on. Individual fairness requires an ML classifier to produce similar outputs for similar individuals. Verification Based Testing (Vbt) is a state-of-the-art black-box testing algorithm for individual fairness that leverages constraint solving to generate test cases. Objective:Generating diverse test cases is expected to facilitate efficient detection of diverse discriminatory data instances (i.e., cases that violate individual fairness). Hashing-based sampling techniques draw a sample approximately uniformly at random from the set of solutions of given Boolean constraints. We propose Vbt-X, which improves Vbt with hashing-based sampling, aiming to improve its testing performance. Method:We realize hashing-based sampling for Vbt. The challenge is that the off-the-shelf hashing-based sampling techniques cannot be integrated in a straightforward manner because the constraints in Vbt are generally not Boolean. Moreover, we propose several enhancement techniques to make Vbt-X more efficient. Results:To evaluate our method, we conduct experiments, where Vbt-X is compared to Vbt, Sg and ExpGA (other well-known fairness testing algorithms) over a set of configurations consisting of several datasets, protected attributes, and ML classifiers. The results show that, with each configuration, Vbt-X detects more discriminatory data instances with higher diversity than Vbt and Sg. Vbt-X detects discriminatory data instances with higher diversity than ExpGA, though the number of discriminatory data instances detected by Vbt-X is lesser than ExpGA. Conclusion:Our proposed method performs better than other state-of-the-art black-box fairness testing algorithms, particularly in terms of diversity. Our method can serve to efficiently identify flaws in ML classifiers with respect to individual fairness for subsequent improvements of an ML classifier. On the other hand, although our method is specific to individual fairness, it could work for testing other aspects of a software system such as security and counterfactual explanations with some technical adaptations, which remains for future work.

Comparison of Subspace Identification Methods for Modal Estimation of Bladed Disks

Abstract Bladed disks are critical parts of aeronautic turbojet engines requiring numerous dynamic analysis tests to fully achieve their design process. This work presents subspace state-space identification techniques for estimation of modal parameters of bladed disks using realistic although numerically generated test case data. Indeed, modal testing of bladed disks can exhibit high modal density leading to modal estimation issues. This study focuses on subspace state-space identification framework as it is an efficient way to process vector valued time series and thus to estimate modal parameters in a context of high modal density. In order to evaluate the techniques of interest, a versatile modal model has been specifically implemented to simulate data representative of full scale rotating aeronautic fan modal tests. This model is easily adjustable allowing to evaluate the identification methods in a more or less severe estimation context. The performance of the investigated methods is discussed and compared with the prescribed parameters of the model. Moreover, different techniques to estimate the order of the associated state-space model are reviewed and tested over several simulated configurations. Finally, a method to evaluate the uncertainties over the model parameters using covariance of estimator results and pseudospectrum computation is proposed and discussed over the investigated test cases.

KVFL: Key-Value-Based Persistent Fuzzing for IoT Web Servers

Abstract As the number of Internet of Thing (IoT) devices increases, attacks against their vulnerabilities have become a serious threat. The web servers (WSs) in IoT devices provide management services for end-users, which are currently the major attack surface. Several fuzzing solutions for identifying vulnerabilities in IoT devices have been proposed, but there is currently no grey-box fuzzer specifically designed for the unique features of WSs in IoT to effectively detect memory corruption vulnerabilities. We design and implement KVFL, an efficient grey-box fuzzer, to address the issues of low throughput and slow exploration of deep code when fuzzing for IoT WSs. Firstly, KVFL employs a delicate hooking technology that heuristically hijacks and emulates hardware-dependent functions, ensuring WSs can be accurately and efficiently emulated in user-mode. On this basis, KVFL fully utilizes the loop parsing HTTP requests feature of WSs through a redesigned fork-server, to minimize nonessential rebooting losses of the target, thereby significantly improving fuzzing throughput. Secondly, KVFL leverages code coverage feedback to automatically infer a set of valid Keys and derive a Key-Value mutation. This enables the generation of high-quality test cases that can facilitate deeper code exploration of WSs. The evaluation results show that compared to the state-of-the-art IoT grey-box fuzzer FIRM-AFL, KVFL improves the throughput by over 2× and explores 4.5× more edges. Additionally, it identifies all 1-day vulnerabilities with over 7× faster speed than the baseline and detects three previously unknown 0-day vulnerabilities. These all indicate that KVFL is effective and efficient at fuzzing IoT WSs.

Software Quality Assurace

Abstract: Software Quality Assurance (SQA) plays a pivotal role in the software development lifecycle, ensuring the delivery of high-quality software products. This abstract provides a comprehensive overview of the latest advancements in Software Quality Assurance methodologies, practices, and tools. The evolving landscape of software development, characterized by agile methodologies, DevOps practices, and continuous integration, has necessitated a reevaluation and enhancement of SQA processes. The abstract begins by elucidating the fundamental principles of SQA, emphasizing its critical role in detecting and preventing defects throughout the software development process. It explores the shift-left approach, where quality assurance is integrated early in the development cycle, and the benefits of this approach in terms of cost reduction and accelerated time-tomarket. Furthermore, the abstract delves into the integration of artificial intelligence (AI) and machine learning (ML) techniques in SQA processes. The use of AI-driven testing tools for automated test case generation, anomaly detection, and predictive analysis is explored as a means to enhance the efficiency and effectiveness of quality assurance activities. The abstract also discusses the challenges and ethical considerations associated with the use of AI in SQA. Additionally, the abstract highlights the importance of comprehensive test automation strategies in modern SQA. It outlines the advantages of continuous testing, test-driven development (TDD), and behavior-driven development (BDD) methodologies in ensuring robust software quality. The abstract also addresses the challenges of maintaining test automation suites and advocates for a balanced approach that combines automated and manual testing. The abstract concludes by emphasizing the need for a holistic and adaptive SQA framework that aligns with the dynamic nature of software development. It calls for ongoing research and collaboration in the SQA domain to address emerging challenges and capitalize on new opportunities for improving software quality

Seed Selection for Testing Deep Neural Networks

Deep learning (DL) has been applied in many applications. Meanwhile, the quality of DL systems is becoming a big concern. To evaluate the quality of DL systems, a number of DL testing techniques have been proposed. To generate test cases, a set of initial seed inputs are required. Existing testing techniques usually construct seed corpus by randomly selecting inputs from training or test dataset. Till now, there is no study on how initial seed inputs affect the performance of DL testing and how to construct an optimal one. To fill this gap, we conduct the first systematic study to evaluate the impact of seed selection strategies on DL testing. Specifically, considering three popular goals of DL testing (i.e., coverage, failure detection, and robustness), we develop five seed selection strategies, including three based on single-objective optimization (SOO) and two based on multi-objective optimization (MOO). We evaluate these strategies on seven testing tools. Our results demonstrate that the selection of initial seed inputs greatly affects the testing performance. SOO-based selection can construct the best seed corpus that can boost DL testing with respect to the specific testing goal. MOO-based selection strategies can construct seed corpus that achieve balanced improvement on multiple objectives.

Proposed Test Case Generation Model using Fuzzy Logic (TCGMFL)

This research addresses the pressing need to enhance software testing, specifically focusing on white-box testing and basis path generation. Software testing is a linchpin in the software development process, ensuring software operates flawlessly and aligns with its intended objectives. However, this phase often incurs substantial time and resource investments. The primary aim of this study is to introduce an efficient and automated approach for basis path generation, a crucial component of white-box testing. The model commences by transforming source code into a tailored control flow graph (CFG), streamlining the automated generation of test paths. Central to this model is an algorithm for generating test paths (AGTP), meticulously traversing CFG nodes from source to destination. The algorithm’s design aims to comprehensively cover all test paths within the CFG. To enhance testing process efficiency, the model employs k-means clustering to generate and cluster inputs. Path coverage is rigorously assessed for each cluster, and fuzzy logic is used to determine the optimal path. The overarching goals of this research are to reduce time and financial costs associated with software testing while maintaining precision and efficiency. The model’s effectiveness in generating test cases is confirmed through the examination of multiple examples, underscoring its valuable contribution to software testing. This study marks a significant advancement toward more effective and cost-efficient software testing methodologies.