Test Data Generation Research Articles

Generate High-Coverage Unit Test Data Using the LLM Tool

Unit testing is a critical phase in the software development lifecycle, essential for ensuring the quality and reliability of code. However, the manual creation of unit test scripts and the preparation of corresponding test data can be a time consuming and labor intensive process. To address these challenges, several automated approaches have been explored, including search based, constraint based, random-based, and symbolic execution based techniques for generating unit tests. In recent years, the rapid advancement of large language models (LLMs) has opened new avenues for automating various tasks, including the automatic generation of unit test scripts and test data. Despite their potential, using LLMs in a straightforward manner to generate unit tests may lead to low test coverage. This means that a significant portion of the source code, including specific statements or branches, may remain untested, which can reduce the effectiveness of the tests. To overcome this limitation, the paper presents a novel approach that not only automates the generation of unit test scripts and test data but also improves test coverage. The proposed solution begins by using an LLM tool (such as ChatGPT) to generate initial unit test scripts and data from the source code. To enhance test coverage, the specification document of the source code is also input into the LLM to generate additional test data. Following this, a coverage checking tool is used to evaluate the test coverage and identify untested statements or branches. The LLM is then applied again to generate new test data aimed specifically at addressing these gaps. The initial experimental results indicate that this method significantly improves test coverage, demonstrating its potential to enhance automated unit testing processes.

Generative AI to Generate Test Data Generators

Generative AI to Generate Test Data Generators

Addressing Low-Cost Methane Sensor Calibration Shortcomings with Machine Learning

Quantifying methane emissions is essential for meeting near-term climate goals and is typically carried out using methane concentrations measured downwind of the source. One major source of methane that is important to observe and promptly remediate is fugitive emissions from oil and gas production sites but installing methane sensors at the thousands of sites within a production basin is expensive. In recent years, relatively inexpensive metal oxide sensors have been used to measure methane concentrations at production sites. Current methods used to calibrate metal oxide sensors have been shown to have significant shortcomings, resulting in limited confidence in methane concentrations generated by these sensors. To address this, we investigate using machine learning (ML) to generate a model that converts metal oxide sensor output to methane mixing ratios. To generate test data, two metal oxide sensors, TGS2600 and TGS2611, were collocated with a trace methane analyzer downwind of controlled methane releases. Over the duration of the measurements, the trace gas analyzer’s average methane mixing ratio was 2.40 ppm with a maximum of 147.6 ppm. The average calculated methane mixing ratios for the TGS2600 and TGS2611 using the ML algorithm were 2.42 ppm and 2.40 ppm, with maximum values of 117.5 ppm and 106.3 ppm, respectively. A comparison of histograms generated using the analyzer and metal oxide sensors mixing ratios shows overlap coefficients of 0.95 and 0.94 for the TGS2600 and TGS2611, respectively. Overall, our results showed there was a good agreement between the ML-derived metal oxide sensors’ mixing ratios and those generated using the more accurate trace gas analyzer. This suggests that the response of lower-cost sensors calibrated using ML could be used to generate mixing ratios with precision and accuracy comparable to higher priced trace methane analyzers. This would improve confidence in low-cost sensors’ response, reduce the cost of sensor deployment, and allow for timely and accurate tracking of methane emissions.

Test data generation using deep reinforcement learning

The process of creating test data for software is one of the most complex and labor-intensive stages in the software development cycle. It requires significant resources and effort, especially when it comes to achieving high test coverage. Search-Based Software Testing (SBST) is an approach that automates this process by using metaheuristic algorithms to generate test data. Metaheuristic algorithms, such as genetic algorithms or simulated annealing, operate on the principle of systematically exploring possible options and selecting the most effective solutions based on feedback from a fitness function. This process is similar to reinforcement learning, where an agent receives rewards for correct actions and penalties for incorrect ones, with the goal of optimizing overall performance. We have proposed a new approach that explores the feasibility of applying reinforcement learning in SBST, with the aim of replacing traditional, human-designed metaheuristic algorithms. To this end, we developed the GunPowder platform, which transforms the software testing process (STP) into a learning environment. In this environment, the agent uses the Double Deep Q-Networks (DDQN) method, which incorporates deep neural networks to learn the optimal strategy for interacting with the software under test. Our study involved a series of empirical experiments to evaluate the effectiveness of the proposed approach. The results indicate that agents trained using GunPowder can effectively learn and apply metaheuristic algorithms, achieving high branch coverage in training functions. In particular, our agents achieved 100% branch coverage in the case of training functions, demonstrating their ability to adapt to different test scenarios. These findings highlight the potential of using deep neural networks and reinforcement learning in SBST, which could significantly improve the quality and efficiency of the software testing process in the future. Thus, our work opens up new opportunities for integrating modern machine learning methods into automated software testing, potentially reducing the time and resources required for testing and improving the quality of final products.

CIEVaD: A Lightweight Workflow Collection for the Rapid and On-Demand Deployment of End-to-End Testing for Genomic Variant Detection.

The identification of genomic variants has become a routine task in the age of genome sequencing. In particular, small genomic variants of a single or few nucleotides are routinely investigated for their impact on an organism's phenotype. Hence, the precise and robust detection of the variants' exact genomic locations and changes in nucleotide composition is vital in many biological applications. Although a plethora of methods exist for the many key steps of variant detection, thoroughly testing the detection process and evaluating its results is still a cumbersome procedure. In this work, we present a collection of easy-to-apply and highly modifiable workflows to facilitate the generation of synthetic test data, as well as to evaluate the accordance of a user-provided set of variants with the test data. The workflows are implemented in Nextflow and are open-source and freely available on Github under the GPL-3.0 license.

Fusion of Multiple Data Sources for Vehicle Crashworthiness Prediction Using CycleGAN and Temporal Convolutional Networks

Abstract Computer-aided engineering (CAE) models play a pivotal role in predicting crashworthiness of vehicle designs. While CAE models continue to advance in fidelity, an inherent discrepancy between CAE model predictions and the responses of physical tests remains inevitable. Machine learning (ML) models have shown promising potential in improving the prediction accuracy of CAE models. Nevertheless, the scarcity of vehicle crash data poses a significant challenge for the training of such models. This paper overcomes these challenges by fusing multiple data sources from two different types of vehicles. More specifically, the Cycle-Consistent Generative Adversarial Neural networks (CycleGAN) are first employed to translate features of time-series test data from one domain to another using cycle consistency loss. Such a translation allows for the generation of synthetic crash test data for the second vehicle type by leveraging existing tests from both vehicle types. In parallel, an initial temporal convolutional network (TCN) model is trained using CAE simulation data and physical test data of the first vehicle type. This pre-trained TCN model is then fine-tuned using three sources of data from the second vehicle type, namely the CAE data, test data, and the augmented virtual test data generated using CycleGAN. Through these data fusion, the crashworthiness prediction accuracy of the second vehicle type can be improved. The proposed method is demonstrated by a real-world case study with a small-size SUV and a medium-size SUV. Results show enhancements in the predictive performance of the medium-size SUV model.

Investigation on the thermal characteristics of electronic system and prediction of chip temperature by machine learning

In this work, the thermal characteristics and steady-state temperatures (SST) of CPU and FPGA of electronic system in nuclear power plant are explored. Finite element analysis is performed to simulate the test process. Furthermore, three machine learning algorithms are used to predict chips temperatures at different operating conditions. It is found that when the ambient temperature is 20 °C and all the fans are power-off, the SST of the CPU and FPGA reaches 75 °C and 72 °C, respectively. While when the fans are power-on, the SST of the CPU and FPGA drops to 37.5 °C and 33 °C. When the ambient temperature increases to 55 °C and all the fans are power-on, the SST of the CPU and FPGA is 72.3 °C and 68.2 °C, respectively. The finite element model is verified and used to generate test data. Three machine learning models are verified by predicting the SST of CPU and FPGA under different operating conditions. It is found that M-SVR has better prediction ability than DT and ANN. The findings can be used for chip reliability evaluation of other electronic system devices, and provide a new method for predicting the possible steady-state temperature of chips under different service conditions.

AI-Based Forecasting of Polymer Properties for High-Temperature Butyl Acrylate Polymerizations.

High-temperature polymerizations involving self-initiation of the monomer are attractive because of high reaction rate, comparable lower viscosities, and no need for an additional initiator. However, the polymers obtained show a more complex microstructure, e.g., with specific branching levels or significant amounts of macromonomer. Simulations of the polymerization processes are powerful tools to gain a deeper understanding of the processes and the elemental reactions at the molecular level. However, simulations can be computationally demanding, requiring significant time and memory resources. Therefore, this study aims at applying AI-based forecasting of tailored polymer properties and using a kinetic Monte Carlo simulator for the generation of training and test data. The applied machine learning (ML) models (random forest and kernel density (KD) regression) predict monomer concentration, macromonomer content, and full molar mass distributions as a function of time, as well as the average branching level with an excellent performance (R 2 (coefficient of determination) > 0.99, MAE (mean absolute error) < 1% for kernel density regression). This study explores the number of training data needed for reliable and accurate predictions in ML models. Explainability methods reveal that the importance of input variables in ML models aligns with expert knowledge.

Enhancing the security of web applications through innovative patterns of integration of artificial intelligence

Ensuring the security of digital operations, especially in the areas of e-commerce and financial transactions, remains increasingly relevant. Therefore, the subject of research is the development of a specialized software library. This library aims to improve the security of web applications. The purpose of this study is to develop a software library that uses artificial intelligence and machine learning methods to analyze and improve the level of security of financial transactions. The use of these advanced technologies helps automate the detection of potentially fraudulent or risky transactions, thereby providing a higher level of user protection. The following tasks are solved in the article: analysis of modern methods of processing financial transactions and identification of possible security threats; development of a UML diagram of library classes for processing and analyzing financial transactions; testing and validation of the developed artificial intelligence model for assessing the security of financial transactions on real financial data. Machine learning methods were defined and applied using the scikit-learn library in Python, the algorithms of which are capable of analyzing large volumes of data and identifying potential risks with high accuracy. This ensures effective integration of artificial intelligence technologies. The following results were obtained in the work: the criteria for assessing the riskiness of financial transactions for the identification of potential risks are defined; the program operation algorithm is described, which includes procedures for determining and classifying transaction risks; pseudocode is presented, which illustrates the structure of classes and methods of the model, opening opportunities for its adaptation and scaling; methods of generating test data reproducing realistic scenarios of financial transactions have been developed; an analysis of the results was carried out to assess the effectiveness of the developed model. In conclusion, the results of research and testing allow us to evaluate the model's response to various data and its effectiveness in real conditions, as the work presents examples of processing various types of transactions. In addition, the study presents not only the development and validation of the developed model, but also the prospects of its use on a larger scale, integration with existing web applications.

A method to enhance Apache Spark performance based on data segmentation and configuration parameters settings

When using modern big data processing tools, there is a problem of increasing the productivity of using modern frameworks in the context of effective setting of various configuration parameters. The object of the research is computational processes of processing big data with the use of technologies of high-performance frameworks. The subject is methods and approaches to the effective setting of configuration parameters of frameworks in the conditions of limitations of virtualization environments and local resources. The purpose of the study is to improve the performance of Apache Spark and Apache Hadoop deployment modes based on a combined approach that includes preprocess segmentation of input data and setting of basic and additional configuration parameters that take into account the limitations of the virtual environment and local resources. Achieving the set goal involves the following tasks: create a synthesized set of WordCount test data for using input data segmentation methods. Determine the composition of general and specific Apache Spark and Apache Hadoop configuration parameters that most affect the performance of frameworks in Spark Standalone and Hadoop Yarn (FIFO) deployment modes. Justify changes in the values of the configuration parameters (accepted by default) by setting the level of parallelism, the number of partitions of the input file according to the number of processor cores, the number of tasks assigned to each core and the system executor. Conduct experimental research to substantiate theoretical results and prove their use in practice. Methods. The research used the following methods: statistical analysis; a method of generating test data based on defined segmentation characteristics with arbitrary volumes of data; a systematic approach for comprehensive evaluation and analysis of performance of frameworks based on selected configuration parameters. The results. On the basis of the developed system of parameters for evaluating the performance of the studied frameworks, experiments were carried out, which include: the application of the method of segmentation of input data based on the division of the input file into paragraphs (lines) for different values of the ranges of the number of words and the number of letters in each word; setting the main parameters and specific ones, in particular, partitioning and parallelism, taking into account the characteristics of the virtual environment and the local resource. According to the obtained results, a detailed analysis of the use of the proposed methods to improve the performance of the studied frameworks with recommendations for choosing the optimal values of data segmentation parameters and configuration parameters was carried out. You are snowmen. The obtained results of the experiments allow us to conclude that the use of the proposed methods of setting the configuration parameters of Spark and Hadoop will increase the processing productivity: for small files (0.5–1 GB) on average up to 25–30%, for large ones (1.5–2.5 GB ) – up to 10–20% on average. At the same time, the average value of the execution time of one task decreased by 10-15% for files of different sizes and with different number of words in a line.

A Control Flow Graph Generation Method for Java Projects

Many software quality assurance methods depend on the control flow graph (CFG) for theanalyzing process. However, existing methods for Java projects have not described details of the CFGgeneration process, which is critical for having a fully automatic analysis method. This paper presentsa novel method, named CFG4J, for generating CFG associated with a given Java unit. The generatedCFG will be used in many other software quality assurance methods such as test data generation forunit testing, unit debugging, code coverage computation, etc. We have implemented CFG4J in a tooland performed some initial experiments with some common units with potential results. Finally, wegive some discussions about the experimental results at the end of the paper.Keywords: Control flow graph, unit testing, test data generation, Java projects.

Automated test data generation and stubbing method for C/C++ embedded projects

Automated test data generation and stubbing method for C/C++ embedded projects

Set evolution based test data generation for killing stubborn mutants

Mutation testing is a fault-based and powerful software testing technique, but the large number of mutations can result in extremely high costs. To reduce the cost of mutation testing, researchers attempt to identify stubborn mutants and generate test data to kill them, in order to achieve the same testing effect. However, existing methods suffer from inaccurate identification of stubborn mutants and low productiveness in generating test data, which will seriously affect the effectiveness and efficiency of mutation testing. Therefore, we propose a new method of generating test data for killing stubborn mutants based on set evolution, namely TDGMSE. We first propose an integrated indicator to identify stubborn mutants. Then, we establish a constrained multi-objective model for generating test data of killing stubborn mutants. Finally, we develop a new genetic algorithm based on set evolution to solve the mathematical model. The results on 14 programs depict that the average false positive (or negative) rate of TDGMSE is decreased about 81.87% (or 32.34%); the success rate of TDGMSE is 99.22%; and the average number of iterations of TDGMSE is 16132.23, which is lowest of all methods. The research highlights several potential research directions for mutation testing.

The Evolving Role of Artificial Intelligence in Software Testing: Prospects and Challenges

Artificial intelligence (AI) refers to a machine's capacity for operations typically performed by human intelligence, such as learning, thinking, solving problems, and making decisions. Machine learning, neural networks, expert systems, and rule-based systems are all used in artificial intelligence. AI employs methods and algorithms to process data, draw conclusions from patterns and laws, and enhance performance over time. A software application or product's intended functionality is evaluated and verified through the process of software testing. The benefits of testing include the prevention of bugs, decreased development costs, and improved performance. Through test generation, test data generation, and automated test script writing, AI can be used in software testing to enhance the quality of our product and the manual testing processes. Software testing is a time-consuming, laborious, and tiresome process. Automation solutions have been created to help with automating some testing process operations in order to increase quality and delivery time. As continuous integration and delivery (CI/CD) pipelines are added, automation systems gradually lose part of their usefulness. The testing community is looking to AI to fill the gap because AI has the capacity to check the code for flaws and defects without the need for any human intervention and much more quickly than humans. In this study, we want to comprehend the effects of AI technology on various STLC tasks or components of software testing. The study also makes an effort to pinpoint and explain some of the biggest challenges faced by software testers when implementing AI in testing. The report also suggests several significant potential contributions of AI to the field of software testing.

Improving Test Data Generation for MPI Program Path Coverage With FERPSO-IMPR and Surrogate-Assisted Models

Improving Test Data Generation for MPI Program Path Coverage With FERPSO-IMPR and Surrogate-Assisted Models

RatInABox, a toolkit for modelling locomotion and neuronal activity in continuous environments.

Generating synthetic locomotory and neural data is a useful yet cumbersome step commonly required to study theoretical models of the brain's role in spatial navigation. This process can be time consuming and, without a common framework, makes it difficult to reproduce or compare studies which each generate test data in different ways. In response, we present RatInABox, an open-source Python toolkit designed to model realistic rodent locomotion and generate synthetic neural data from spatially modulated cell types. This software provides users with (i) the ability to construct one- or two-dimensional environments with configurable barriers and visual cues, (ii) a physically realistic random motion model fitted to experimental data, (iii) rapid online calculation of neural data for many of the known self-location or velocity selective cell types in the hippocampal formation (including place cells, grid cells, boundary vector cells, head direction cells) and (iv) a framework for constructing custom cell types, multi-layer network models and data- or policy-controlled motion trajectories. The motion and neural models are spatially and temporally continuous as well as topographically sensitive to boundary conditions and walls. We demonstrate that out-of-the-box parameter settings replicate many aspects of rodent foraging behaviour such as velocity statistics and the tendency of rodents to over-explore walls. Numerous tutorial scripts are provided, including examples where RatInABox is used for decoding position from neural data or to solve a navigational reinforcement learning task. We hope this tool will significantly streamline computational research into the brain's role in navigation.

Test data generation for covering mutation-based path using MGA for MPI program

Message Passing Interface (MPI) is a communication protocol used for parallel programming in various languages, valued for its reliability and broad applicability. Mutation testing is a software testing method for systematically simulating software faults. However, the significant number of inserted mutation branches in the program escalates the testing cost. To address this issue, we propose the comprehensive PMTDGM framework. The framework first generates mutation-based paths based on the relevancy of mutant branches and the difficulty of mutant branch coverage, followed by the establishment of a multitask model of path coverage. Finally, we employ a multi-population genetic algorithm (MGA) to generate test data. Our experiments, performed on six MPI programs of varying sizes and structures, demonstrate that the mutation-based paths have the small test sets and are easy to cover. Additionally, the multitask model and MGA can significantly improve the efficiency of generating test data and reduce the cost of mutation testing for MPI programs compared to traditional methods.

A Bioinspired Test Generation Method Using Discretized and Modified Bat Optimization Algorithm

The process of software development is incomplete without software testing. Software testing expenses account for almost half of all development expenses. The automation of the testing process is seen to be a technique for reducing the cost of software testing. An NP-complete optimization challenge is to generate the test data with the highest branch coverage in the shortest time. The primary goal of this research is to provide test data that covers all branches of a software unit. Increasing the convergence speed, the success rate, and the stability of the outcomes are other goals of this study. An efficient bioinspired technique is suggested in this study to automatically generate test data utilizing the discretized Bat Optimization Algorithm (BOA). Modifying and discretizing the BOA and adapting it to the test generation problem are the main contributions of this study. In the first stage of the proposed method, the source code of the input program is statistically analyzed to identify the branches and their predicates. Then, the developed discretized BOA iteratively generates effective test data. The fitness function was developed based on the program’s branch coverage. The proposed method was implemented along with the previous one. The experiments’ results indicated that the suggested method could generate test data with about 99.95% branch coverage with a limited amount of time (16 times lower than the time of similar algorithms); its success rate was 99.85% and the average number of required iterations to cover all branches is 4.70. Higher coverage, higher speed, and higher stability make the proposed method suitable as an efficient test generation method for real-world large software.

Evaluation of Clustering Approaches and Proximity Measures for Product Family Identification

Value stream mapping is the most widely used method for analyzing material and information flows in production. The first step of the method is the identification of product families, which can be suitably supported by cluster analysis. In order to evaluate the suitability of different cluster algorithms and proximity measures, this paper conducts a study based on synthetically generated test data. Seven performance measures are used to evaluate the quality of clustering solutions. Based on the results of the study, recommendations are derived for the appropriate use of cluster analysis for product family identification.

Non-bio-inspired Metaheuristics in Software Testing: A Systematic Literature Review

The software testing phase usually consumes a lot of the development of software projects time in order to find defects before release. Different strategies have been approached to optimize this phase of the testing stage. Metaheuristics are important in software testing due to their ability to find optimal or near-optimal solutions in complex situations. This research aims to analyze the current status of the application of metaheuristics that assist in software testing phase activities, specifically the most representative Non-bio-inspired algorithms (NBA) are surveyed, being Hill Climbing the most reported. The main activities of the software testing where NBA were implemented, were test case and test data generation and test case prioritization. It was concluded that NBAs used on their own are only viable in some activities of the software testing phase. As future work, it is proposed to investigate the use of hybrid algorithms and approaches in software testing phase.