Synthetic Training Data Generation Research Articles

Synthetic data at scale: a development model to efficiently leverage machine learning in agriculture.

The rise of artificial intelligence (AI) and in particular modern machine learning (ML) algorithms during the last decade has been met with great interest in the agricultural industry. While undisputedly powerful, their main drawback remains the need for sufficient and diverse training data. The collection of real datasets and their annotation are the main cost drivers of ML developments, and while promising results on synthetically generated training data have been shown, their generation is not without difficulties on their own. In this paper, we present a development model for the iterative, cost-efficient generation of synthetic training data. Its application is demonstrated by developing a low-cost early disease detector for tomato plants (Solanum lycopersicum) using synthetic training data. A neural classifier is trained by exclusively using synthetic images, whose generation process is iteratively refined to obtain optimal performance. In contrast to other approaches that rely on a human assessment of similarity between real and synthetic data, we instead introduce a structured, quantitative approach. Our evaluation shows superior generalization results when compared to using non-task-specific real training data and a higher cost efficiency of development compared to traditional synthetic training data. We believe that our approach will help to reduce the cost of synthetic data generation in future applications.

Deep Generation of Synthetic Training Data for the Automated Extraction of Semantic Knowledge from Historical Maps

Deep Generation of Synthetic Training Data for the Automated Extraction of Semantic Knowledge from Historical Maps

Using convolutional neural networks for stereological characterization of 3D hetero-aggregates based on synthetic STEM data

The 3D nano/microstructure of materials can significantly influence their macroscopic properties. In order to enable a better understanding of such structure-property relationships, 3D microscopy techniques can be deployed, which are however often expensive in both time and costs. Often 2D imaging techniques are more accessible, yet they have the disadvantage that the 3D nano/microstructure of materials cannot be directly retrieved from such measurements. The motivation of this work is to overcome the issues of characterizing 3D structures from 2D measurements for hetero-aggregate materials. For this purpose, a method is presented that relies on machine learning combined with methods of spatial stochastic modeling for characterizing the 3D nano/microstructure of materials from 2D data. More precisely, a stochastic model is utilized for the generation of synthetic training data. This kind of training data has the advantage that time-consuming experiments for the synthesis of differently structured materials followed by their 3D imaging can be avoided. More precisely, a parametric stochastic 3D model is presented, from which a wide spectrum of virtual hetero-aggregates can be generated. Additionally, the virtual structures are passed to a physics-based simulation tool in order to generate virtual scanning transmission electron microscopy (STEM) images. The preset parameters of the 3D model together with the simulated STEM images serve as a database for the training of convolutional neural networks, which can be used to determine the parameters of the underlying 3D model and, consequently, to predict 3D structures of hetero-aggregates from 2D STEM images. Furthermore, an error analysis is performed with respect to structural descriptors, e.g. the hetero-coordination number. The proposed method is applied to image data of TiO2-WO3 hetero-aggregates, which are highly relevant in photocatalysis processes. However, the proposed method can be transferred to other types of aggregates and to different 2D microscopy techniques. Consequently, the method is relevant for industrial or laboratory setups in which product quality is to be quantified by means of inexpensive 2D image acquisition.

Review of methods and systems for generation of synthetic training data

It is impossible to imagine the advancement of modern artificial intelligence systems without neural network technologies. During the design process researchers are often faced with the fact that there is not enough data to train modern neural network models, these data may be unbalanced or highly sparse. Often it happens that real data simply does not exist, as the research field is still emerging. A relevant problem is ensuring the confidentiality of real personal or patient medical data, which is used in the exchange between researchers or in the testing of various neural network systems. In many subject areas, the cost of collecting and marking up real data can be very high. Synthetic data is increasingly being used to solve these problems. The purpose of this publication is to introduce readers to advances in the generation and use of synthetic data. The paper presents a description of various methods, systems and software tools used to generate synthetic data, which can help to improve neural network models. Since an entire industry for synthetic data production has already formed, the leading data synthesis technology platforms are presented. The paper is of an overview nature, so it contains an extensive bibliography. The value of the article lies in the fact that this review will help readers broaden their understanding of the use of synthetic data in solving a wide range of neural network problems, as well as to become more familiar with the methods and tools for their generation.

Collagen fiber centerline tracking in fibrotic tissue via deep neural networks with variational autoencoder-based synthetic training data generation.

The role of fibrillar collagen in the tissue microenvironment is critical in disease contexts ranging from cancers to chronic inflammations, as evidenced by many studies. Quantifying fibrillar collagen organization has become a powerful approach for characterizing the topology of collagen fibers and studying the role of collagen fibers in disease progression. We present a deep learning-based pipeline to quantify collagen fibers' topological properties in microscopy-based collagen images from pathological tissue samples. Our method leverages deep neural networks to extract collagen fiber centerlines and deep generative models to create synthetic training data, addressing the current shortage of large-scale annotations. As a part of this effort, we have created and annotated a collagen fiber centerline dataset, with the hope of facilitating further research in this field. Quantitative measurements such as fiber orientation, alignment, density, and length can be derived based on the centerline extraction results. Our pipeline comprises three stages. Initially, a variational autoencoder is trained to generate synthetic centerlines possessing controllable topological properties. Subsequently, a conditional generative adversarial network synthesizes realistic collagen fiber images from the synthetic centerlines, yielding a synthetic training set of image-centerline pairs. Finally, we train a collagen fiber centerline extraction network using both the original and synthetic data. Evaluation using collagen fiber images from pancreas, liver, and breast cancer samples collected via second-harmonic generation microscopy demonstrates our pipeline's superiority over several popular fiber centerline extraction tools. Incorporating synthetic data into training further enhances the network's generalizability. Our code is available at https://github.com/uw-loci/collagen-fiber-metrics.

Deep Learning Based Vehicle Detection on Real and Synthetic Aerial Images: Training Data Composition and Statistical Influence Analysis

The performance of deep learning based algorithms is significantly influenced by the quantity and quality of the available training and test datasets. Since data acquisition is complex and expensive, especially in the field of airborne sensor data evaluation, the use of virtual simulation environments for generating synthetic data are increasingly sought. In this article, the complete process chain is evaluated regarding the use of synthetic data based on vehicle detection. Among other things, content-equivalent real and synthetic aerial images are used in the process. This includes, in the first step, the learning of models with different training data configurations and the evaluation of the resulting detection performance. Subsequently, a statistical evaluation procedure based on a classification chain with image descriptors as features is used to identify important influencing factors in this respect. The resulting findings are finally incorporated into the synthetic training data generation and in the last step, it is investigated to what extent an increase of the detection performance is possible. The overall objective of the experiments is to derive design guidelines for the generation and use of synthetic data.

Polynomial Chaos-Based Procedural Generation of Synthetic Training Data in Machine Learning for Automated Acoustic Monitoring

In additive manufacturing such as powder bed fusion the acoustic monitoring taking care of timely process termination in case of failure is commonly achieved by ear and therefore highly susceptible to human bias. Solutions based on machine learning algorithms need large datasets for training purposes which are not readily available. Additionally, capturing high-quality audio samples and providing respective material parts are expensive both in terms of time and cost. To overcome this problem, this work proposes a method by which the required synthetic datasets are obtained by way of procedural generation. Here, synthetic data implies the substitution of measured audio data by equivalent virtual and artificial samples from 3D acoustic simulations. In order to cover process variations as well as consider the variability of multiple input parameters, a design-of-experiments based on the theory of generalized polynomial chaos is conducted. Additionally, the polynomialchaos method is extended through use of a decision tree so that the prevalence of specific critical events may be accounted for.

I Spy You

This paper presents iSpyU, a system that shows the feasibility of recognition of natural speech content played on a phone during conference calls (Skype, Zoom, etc) using a fusion of motion sensors such as accelerometer and gyroscope. While microphones require permissions from the user to be accessible by an app developer, the motion sensors are zero-permission sensors, thus accessible by a developer without alerting the user. This allows a malicious app to potentially eavesdrop on sensitive speech content played by the user's phone. In designing the attack, iSpyU tackles a number of technical challenges including: (i) Low sampling rate of motion sensors (500 Hz in comparison to 44 kHz for a microphone). (ii) Lack of availability of large-scale training datasets to train models for Automatic Speech Recognition (ASR) with motion sensors. iSpyU systematically addresses these challenges by a combination of techniques in synthetic training data generation, ASR modeling, and domain adaptation. Extensive measurement studies on modern smartphones show a word level accuracy of 53.3 - 59.9% over a dictionary of 2000-10000 words, and a character level accuracy of 70.0 - 74.8%. We believe such levels of accuracy poses a significant threat when viewed from a privacy perspective.

Array Camera Image Fusion using Physics-Aware Transformers

We demonstrate a physics-aware transformer for feature-based data fusion from cameras with diverse resolution, color spaces, focal planes, focal lengths, and exposure. We also demonstrate a scalable solution for synthetic training data generation for the transformer using open-source computer graphics software. We demonstrate image synthesis on arrays with diverse spectral responses, instantaneous field of view and frame rate.

Effect of Kinematics and Fluency in Adversarial Synthetic Data Generation for ASL Recognition With RF Sensors

RF sensors have been recently proposed as a new modality for sign language processing technology. They are non-contact, effective in the dark, and acquire a direct measurement of signing kinematic via exploitation of the micro-Doppler effect. First, this work provides an in depth, comparative examination of the kinematic properties of signing as measured by RF sensors for both fluent ASL users and hearing imitation signers. Second, as ASL recognition techniques utilizing deep learning requires a large amount of training data, this work examines the effect of signing kinematics and subject fluency on adversarial learning techniques for data synthesis. Two different approaches for the synthetic training data generation are proposed: 1) adversarial domain adaptation to minimize the differences between imitation signing and fluent signing data, and 2) kinematically-constrained generative adversarial networks for accurate synthesis of RF signing signatures. The results show that the kinematic discrepancies between imitation signing and fluent signing are so significant that training on data directly synthesized from fluent RF signers offers greater performance (93% top-5 accuracy) than that produced by adaptation of imitation signing (88% top-5 accuracy) when classifying 100 ASL signs.

Designing a Human-in-the-Loop System for Object Detection in Floor Plans

In recent years, companies in the Architecture, Engineering, and Construction (AEC) industry have started exploring how artificial intelligence (AI) can reduce time-consuming and repetitive tasks. One use case that can benefit from the adoption of AI is the determination of quantities in floor plans. This information is required for several planning and construction steps. Currently, the task requires companies to invest a significant amount of manual effort. Either digital floor plans are not available for existing buildings, or the formats cannot be processed due to lack of standardization. In this paper, we therefore propose a human-in-the-loop approach for the detection and classification of symbols in floor plans. The developed system calculates a measure of uncertainty for each detected symbol which is used to acquire the knowledge of human experts for those symbols that are difficult to classify. We evaluate our approach with a real-world dataset provided by an industry partner and find that the selective acquisition of human expert knowledge enhances the model’s performance by up to 10.5%—resulting in an overall prediction accuracy of 92.1% on average. We further design a pipeline for the generation of synthetic training data that allows the systems to be adapted to new construction projects with minimal manual effort. Overall, our work supports professionals in the AEC industry on their journey to the data-driven generation of business value.

SinGAN-Seg: Synthetic training data generation for medical image segmentation.

Analyzing medical data to find abnormalities is a time-consuming and costly task, particularly for rare abnormalities, requiring tremendous efforts from medical experts. Therefore, artificial intelligence has become a popular tool for the automatic processing of medical data, acting as a supportive tool for doctors. However, the machine learning models used to build these tools are highly dependent on the data used to train them. Large amounts of data can be difficult to obtain in medicine due to privacy reasons, expensive and time-consuming annotations, and a general lack of data samples for infrequent lesions. In this study, we present a novel synthetic data generation pipeline, called SinGAN-Seg, to produce synthetic medical images with corresponding masks using a single training image. Our method is different from the traditional generative adversarial networks (GANs) because our model needs only a single image and the corresponding ground truth to train. We also show that the synthetic data generation pipeline can be used to produce alternative artificial segmentation datasets with corresponding ground truth masks when real datasets are not allowed to share. The pipeline is evaluated using qualitative and quantitative comparisons between real data and synthetic data to show that the style transfer technique used in our pipeline significantly improves the quality of the generated data and our method is better than other state-of-the-art GANs to prepare synthetic images when the size of training datasets are limited. By training UNet++ using both real data and the synthetic data generated from the SinGAN-Seg pipeline, we show that the models trained on synthetic data have very close performances to those trained on real data when both datasets have a considerable amount of training data. In contrast, we show that synthetic data generated from the SinGAN-Seg pipeline improves the performance of segmentation models when training datasets do not have a considerable amount of data. All experiments were performed using an open dataset and the code is publicly available on GitHub.

Generation of synthetic training data for SEEG electrodes segmentation.

Stereoelectroencephalography (SEEG) is a minimally invasive surgical procedure, used to locate epileptogenic zones. An accurate identification of the metallic contacts recording the SEEG signal is crucial to ensure effectiveness of the upcoming treatment. However, due to the presence of metal, post-operative CT scans contain strong streak artefacts that interfere with deep learning segmentation algorithms and require a lot of training data to distinguish from actual contacts. We propose a method to generate synthetic data and use them to train a neural network to precisely locate SEEG electrode contacts. Random electrodes were generated following manufacturer's specifications and dimensions and placed in acceptable regions inside metal-free CT images. Metal artefacts were simulated in the generated data set using radon transform, beam hardening, and filtered back projection. A UNet neural network was trained for the contacts segmentation task using various training set-ups combining real data, basic augmented data, and synthetic data. The results were compared. We reported a higher accuracy when including synthetic data during the network training, while training only on real and basic augmented data more often led to misclassified artefacts or missed contacts. The network segments post-operative CT slices in less than 2 s using 4 GeForce RTX2080 Ti GPUs and in under a minute using a standard PC with GeForce GTX1060. Using synthetic data to train the network significantly improves contact detection and segmentation accuracy.

Clothing Items Classification Based on X-Ray Multi-Shot Imaging for E-Commerce

The mail order business with e-commerce products such as clothing is growing steadily. This trend is associated with an increased effort in package processing of incoming and outgoing goods. These processes can be automated with X-ray technology. Therefore, we describe a method for classifying the contents of e-commerce packages using multiple X-ray projections. A selection of clothing items is classified using a Convolutional and Recurrent Neural Network model (CNN-RNN) trained on an input of multiple X-ray projections. The described workflow allows a fast generation of synthetic training data and is evaluated on real (non-synthetic) data.

Rapid Quantitative Analysis of IR Absorption Spectra for Trace Gas Detection by Artificial Neural Networks Trained with Synthetic Data.

Infrared absorption spectroscopy is a widely used tool to quantify and monitor compositions of gases. The concentration information is often retrieved by fitting absorption profiles to the acquired spectra, utilizing spectroscopic databases. In complex gas matrices an expanded parameter space leads to long computation times of the fitting routines due to the increased number of spectral features that need to be computed for each iteration during the fit. This hinders the capability of real-time analysis of the gas matrix. Here, an artificial neural network (ANN) is employed for rapid prediction of gas concentrations in complex infrared absorption spectra composed of mixtures of CO and N2O. Experimental data is acquired with a mid-infrared dual frequency comb spectrometer. To circumvent the experimental collection of huge amounts of training data, the network is trained on synthetically generated spectra. The spectra are based on simulated absorption profiles making use of the HITRAN database. In addition, the spectrometer’s influence on the measured spectra is characterized and included in the synthetic training data generation. The ANN was tested on measured spectra and compared to a non-linear least squares fitting algorithm. An average evaluation time of 303 µs for a single measured spectrum was achieved. Coefficients of determination were 0.99997 for the predictions of N2O concentrations and 0.99987 for the predictions of CO concentrations, with uncertainties on the predicted concentrations between 0.04 and 0.18 ppm for 0 to 100 ppm N2O and between 0.05 and 0.18 ppm for 0 to 60 ppm CO.

Procedural synthetic training data generation for AI-based defect detection in industrial surface inspection

Supervised machine learning methods are increasingly used for detecting defects in automated visual inspection systems. However, these methods require large quantities of annotated image data of the surface being inspected, including images of defective surfaces. In industrial contexts, it is difficult to collect the latter since acquiring sufficient image data of defective surfaces is costly and time-consuming. Additionally, gathered datasets tend to contain selection-bias, e.g. under representation of certain defect classes, and therefore result in insufficient training data quality. Synthetic training data is a promising alternative as it can be easily generated unbiasedly and in large quantities. In this work, we present a procedural pipeline for generating training data based on physically based renderings of the object under inspection. Defects are being introduced as 3D-models on the surface of the object. The generator provides the ability to randomize object and camera parameters within given intervals, allowing the user to use the domain randomization technique to bridge the domain gap between the synthetic data and the real world. Experiments suggest that the data generated in this way can be beneficial to training defect detection models.

Synthetic Training Data Generation for Visual Object Identification on Load Carriers

With visual AI processes relying on individual and context accurate training data, the existing common object datasets and randomization based synthetic data pipelines can only hardly be transferred or applied on specific and narrow industrial tasks. To enable visual AI applications for intralogistics processes, such as supervision or localization of objects, a domain-knowledge driven implementation for generation of context accurate synthetic training data is introduced. With this consideration of process and domain requirements in the data generation pipeline itself, a data-generator for object identification on load carriers is contributed.

Simulation-Based Generation of Representative and Valid Training Data for Acoustic Resonance Testing

Analyzing eigenfrequencies of serial parts by acoustic resonance testing enables an efficient nondestructive assessment of component quality or structural state. Usually, each application is based on experimentally acquired training data, which represent the typical natural vibration behavior of the component type to be inspected. From the training data, suitable test characteristics are identified according to the inspection objective. The experimental collection of training data, which involves selecting and characterizing numerous representing parts, is often associated with a great amount of effort. Instead, this work focuses on a simulation-based generation of synthetic training data. Within an application example, the eigenfrequencies of a set of virtual parts were calculated with FEM as a function of geometry. The resulting simulation values were adapted using empirical correction factors, which were derived from both calculated and measured eigenfrequencies of machine-made reference parts. The simulation-based data were finally used to form linear regression models within a training procedure. These models enabled the precise estimation of geometric dimensions of further machine-made parts using their measured eigenfrequencies as input data. The novel approach, which requires the experimental characterization of only a few real parts, can thus significantly reduce the effort associated with efficient and reliable acoustic resonance testing.

Generating Human Action Videos by Coupling 3D Game Engines and Probabilistic Graphical Models

Deep video action recognition models have been highly successful in recent years but require large quantities of manually-annotated data, which are expensive and laborious to obtain. In this work, we investigate the generation of synthetic training data for video action recognition, as synthetic data have been successfully used to supervise models for a variety of other computer vision tasks. We propose an interpretable parametric generative model of human action videos that relies on procedural generation, physics models and other components of modern game engines. With this model we generate a diverse, realistic, and physically plausible dataset of human action videos, called PHAV for “Procedural Human Action Videos”. PHAV contains a total of 39,982 videos, with more than 1000 examples for each of 35 action categories. Our video generation approach is not limited to existing motion capture sequences: 14 of these 35 categories are procedurally-defined synthetic actions. In addition, each video is represented with 6 different data modalities, including RGB, optical flow and pixel-level semantic labels. These modalities are generated almost simultaneously using the Multiple Render Targets feature of modern GPUs. In order to leverage PHAV, we introduce a deep multi-task (i.e. that considers action classes from multiple datasets) representation learning architecture that is able to simultaneously learn from synthetic and real video datasets, even when their action categories differ. Our experiments on the UCF-101 and HMDB-51 benchmarks suggest that combining our large set of synthetic videos with small real-world datasets can boost recognition performance. Our approach also significantly outperforms video representations produced by fine-tuning state-of-the-art unsupervised generative models of videos.