Generative Adversarial Networks Training Research Articles

Synthetic flow-based cryptomining attack generation through Generative Adversarial Networks

Due to the growing rise of cyber attacks in the Internet, the demand of accurate intrusion detection systems (IDS) to prevent these vulnerabilities is increasing. To this aim, Machine Learning (ML) components have been proposed as an efficient and effective solution. However, its applicability scope is limited by two important issues: (i) the shortage of network traffic data datasets for attack analysis, and (ii) the data privacy constraints of the data to be used. To overcome these problems, Generative Adversarial Networks (GANs) have been proposed for synthetic flow-based network traffic generation. However, due to the ill-convergence of the GAN training, none of the existing solutions can generate high-quality fully synthetic data that can totally substitute real data in the training of ML components. In contrast, they mix real with synthetic data, which acts only as data augmentation components, leading to privacy breaches as real data is used. In sharp contrast, in this work we propose a novel and deterministic way to measure the quality of the synthetic data produced by a GAN both with respect to the real data and to its performance when used for ML tasks. As a by-product, we present a heuristic that uses these metrics for selecting the best performing generator during GAN training, leading to a novel stopping criterion, which can be applied even when different types of synthetic data are to be used in the same ML task. We demonstrate the adequacy of our proposal by generating synthetic cryptomining attacks and normal traffic flow-based data using an enhanced version of a Wasserstein GAN. The results evidence that the generated synthetic network traffic can completely replace real data when training a ML-based cryptomining detector, obtaining similar performance and avoiding privacy violations, since real data is not used in the training of the ML-based detector.

JoT-GAN: A Framework for Jointly Training GAN and Person Re-Identification Model

To cope with the problem caused by inadequate training data, many person re-identification (re-id) methods exploit generative adversarial networks (GAN) for data augmentation, where the training of GAN is typically independent of that of the re-id model. The coupling relation between them that probably brings in a performance gain of re-id is thus ignored. In this work, we propose a general framework, namely JoT-GAN, to jointly train GAN and the re-id model. It can simultaneously achieve the optima of both the generator and the re-id model, where the training is guided by each other through a discriminator. The re-id model is boosted for two reasons: (1) the adversarial training encourages it to fool the discriminator, and (2) the generated samples augment the training data. Extensive results on benchmark datasets show that for the re-id model trained with the identification loss as well as the triplet loss, the proposed joint training framework outperforms existing methods with separate training and achieves state-of-the-art re-id performance.

Toward a general unsupervised novelty detection framework in structural health monitoring

AbstractThis study proposes an unsupervised, online structural health monitoring framework robust to the sensor configuration, that is, the number and placement of sensors. The proposed methodology leverages generative adversarial networks (GANs). The GAN's discriminator network is the novelty detector, while its generator provides additional data to tune the detection threshold. GAN models are trained with the fast Fourier transform of structural accelerations as input, avoiding the need for any structure‐specific feature extraction. Dense, convolutional (convolutional neural network), and long short‐term memory (LSTM) units are evaluated as discriminators under different GAN training loss patterns, that is, the differences between discriminator and generator training losses. Results show that the LSTM‐based discriminators and the suggested threshold tuning technique to be robust even in loss patterns with overfitted discriminators, a probable outcome of limited training sets. The framework is evaluated on two benchmark datasets. With only 100 s of training data, it achieved 95% novelty detection accuracy, distinguishing between different damage classes and identifying their resurgence under varying sensor configurations. Finally, the majority‐vote‐ensemble of discriminator‐generator pairs at different training epochs is introduced to reduce false alarms, improve novelty detection accuracy and stability.

A Spin Glass Model for the Loss Surfaces of Generative Adversarial Networks

We present a novel mathematical model that seeks to capture the key design feature of generative adversarial networks (GANs). Our model consists of two interacting spin glasses, and we conduct an extensive theoretical analysis of the complexity of the model’s critical points using techniques from Random Matrix Theory. The result is insights into the loss surfaces of large GANs that build upon prior insights for simpler networks, but also reveal new structure unique to this setting which explains the greater difficulty of training GANs.

Validation Methods for Energy Time Series Scenarios From Deep Generative Models

The design and operation of modern energy systems are heavily influenced by time-dependent and uncertain parameters, e.g., renewable electricity generation, load-demand, and electricity prices. These are typically represented by a set of discrete realizations known as scenarios. A popular scenario generation approach uses deep generative models (DGM) that allow scenario generation without prior assumptions about the data distribution. However, the validation of generated scenarios is difficult, and a comprehensive discussion about appropriate validation methods is currently lacking. To start this discussion, we provide a critical assessment of the currently used validation methods in the energy scenario generation literature. In particular, we assess validation methods based on probability density, auto-correlation, and power spectral density. Furthermore, we propose using the multifractal detrended fluctuation analysis (MFDFA) as an additional validation method for non-trivial features like peaks, bursts, and plateaus. As representative examples, we train generative adversarial networks (GANs), Wasserstein GANs (WGANs), and variational autoencoders (VAEs) on two renewable power generation time series (photovoltaic and wind from Germany in 2013 to 2015) and an intra-day electricity price time series form the European Energy Exchange in 2017 to 2019. We apply the four validation methods to both the historical and the generated data and discuss the interpretation of validation results as well as common mistakes, pitfalls, and limitations of the validation methods. Our assessment shows that no single method sufficiently characterizes a scenario but ideally validation should include multiple methods and be interpreted carefully in the context of scenarios over short time periods.

Diagnosis of Scoliosis Using Chest Radiographs with a Semi-Supervised Generative Adversarial Network.

To develop and validate a deep learning-based screening tool for the early diagnosis of scoliosis using chest radiographs with a semi-supervised generative adversarial network (GAN). Using a semi-supervised learning framework with a GAN, a screening tool for diagnosing scoliosis was developed and validated through the chest PA radiographs of patients at two different tertiary hospitals. Our proposed method used training GAN with mild to severe scoliosis only in a semi-supervised manner, as an upstream task to learn scoliosis representations and a downstream task to perform simple classification for differentiating between normal and scoliosis states sensitively. The area under the receiver operating characteristic curve, negative predictive value (NPV), positive predictive value, sensitivity, and specificity were 0.856, 0.950, 0.579, 0.985, and 0.285, respectively. Our deep learning-based artificial intelligence software in a semi-supervised manner achieved excellent performance in diagnosing scoliosis using the chest PA radiographs of young individuals; thus, it could be used as a screening tool with high NPV and sensitivity and reduce the burden on radiologists for diagnosing scoliosis through health screening chest radiographs.

A Data Augmentation Pipeline to Generate Synthetic Labeled Datasets of 3D Echocardiography Images Using a GAN

Due to privacy issues and limited amount of publicly available labeled datasets in the domain of medical imaging, we propose an image generation pipeline to synthesize 3D echocardiographic images with corresponding ground truth labels, to alleviate the need for data collection and for laborious and error-prone human labeling of images for subsequent Deep Learning (DL) tasks. The proposed method utilizes detailed anatomical segmentations of the heart as ground truth label sources. This initial dataset is combined with a second dataset made up of real 3D echocardiographic images to train a Generative Adversarial Network (GAN) to synthesize realistic 3D cardiovascular ultrasound images paired with ground truth labels. To generate the synthetic 3D dataset, the trained GAN uses high resolution anatomical models from Computed Tomography (CT) as input. A qualitative analysis of the synthesized images showed that the main structures of the heart are well delineated and closely follow the labels obtained from the anatomical models. To assess the usability of these synthetic images for DL tasks, segmentation algorithms were trained to delineate the left ventricle, left atrium, and myocardium. A quantitative analysis of the 3D segmentations given by the models trained with the synthetic images indicated the potential use of this GAN approach to generate 3D synthetic data, use the data to train DL models for different clinical tasks, and therefore tackle the problem of scarcity of 3D labeled echocardiography datasets.

BigGAN evaluation for the generation of vehicle interior images

The number of shared autonomous vehicles (SAV) tends to increase in the coming years, highlighting the need to create monitoring systems that safeguard the integrity of the SAV and the safety of passengers. For the creation of monitoring systems, it is necessary to develop algorithms capable of detecting and classifying a multitude of objects (i.e. dangerous, forgotten, damaged), in different types of vehicles. Currently, deep learning (DL) algorithms present themselves as the best option to solve this problem, but require a large amount of data for training. This article focuses on the use of Generative Adversarial Networks (GAN) for the automatic generation of artificial images of vehicle interiors. Specifically, we propose to employ the BigGAN arquitecture, with the combined implementation of two recent techniques that aim to improve training stability and GAN generalization, namely consistency regularization and differential augmentation. With an expanded version of MoLa-VI dataset (made publicly available), satisfactory results were obtained with the proposed. Moreover, CR+BigGAN combination presented the best results, achieving a Fréchet Inception Distance of 28.23 and an Inception Score of 17.19.

Cloud Removal in Remote Sensing Images Using Generative Adversarial Networks and SAR-to-Optical Image Translation

Satellite images are often contaminated by clouds. Cloud removal has received special attention due to the wide range of satellite image applications. As the clouds thicken, the process of removing them becomes more challenging. In such cases, using auxiliary images, such as near-infrared or synthetic aperture radar (SAR), for reconstructing is common. In this study, we attempt to solve the problem using two generative adversarial networks (GANs): the first translates SAR images to optical images and the second removes clouds using the translated images of prior GAN. Also, we propose dilated residual inception blocks (DRIBs) instead of vanilla U-net in the generator networks and use structural similarity index measure (SSIM) in addition to the L1 loss function. Reducing the number of downsamplings and expanding receptive fields by dilated convolutions increased the quality of output images. We used the SEN1-2 dataset to train and test both GANs, and we made cloudy images by adding synthetic clouds to optical images. In addition, we used the SEN12MS-CR dataset to test network performance to remove real clouds. The restored images are evaluated using PSNR, SSIM, SAM, MAE, RMSE, and <inline-formula> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula>. We compared the proposed method with state-of-the-art deep learning models and achieved more accurate results in both SAR-to-optical image translation and cloud removal parts.

Stochastically Flipping Labels of Discriminator’s Outputs for Training Generative Adversarial Networks

Generative Adversarial Networks (GANs) play the adversarial game between two neural networks: the generator and the discriminator. Many studies treat the discriminator’s outputs as an implicit posterior distribution prior to the input image distribution. Thus, increasing the discriminator’s output dimensions can represent richer information than a single output dimension of the discriminator. However, increasing the output dimensions will lead to a very strong discriminator, which can easily surpass the generator and break the balance of adversarial learning. Solving such conflict and elevating the generation quality of GANs remains challenging. Hence, we propose a simple yet effective method to solve this conflict problem based on a stochastic selecting method by extending the flipped and non-flipped non-saturating losses in BipGAN. We organized our experiments based on the famous BigGAN model for comparison. Our experiments successfully validated our approach to strengthening the generation quality within limited output dimensions via several standard evaluation metrics and real-world datasets and achieved competitive results in the Human face generation task.

Tackling stain variability using CycleGAN-based stain augmentation

BackgroundConsiderable inter- and intra-laboratory stain variability exists in pathology, representing a challenge in development and application of deep learning (DL) approaches. Since tackling all sources of stain variability with manual annotation is not feasible, we here investigated and compared unsupervised DL approaches to reduce the consequences of stain variability in kidney pathology. MethodsWe aimed to improve the applicability of a pretrained DL segmentation model to 3 external multi-centric cohorts with large stain variability. In contrast to the traditional approach of training generative adversarial networks (GAN) for stain normalization, we here propose to tackle stain variability by data augmentation. We augment the training data of the pretrained model by the stain variability using CycleGANs and then retrain the model on the stain-augmented dataset. We compared the performance of i/ the unmodified pretrained segmentation model with ii/ CycleGAN-based stain normalization, iii/ a feature-preserving modification to ii/ for improved normalization, and iv/ the proposed stain-augmented model. ResultsThe proposed stain-augmented model showed highest mean segmentation accuracy in all external cohorts and maintained comparable performance on the training cohort. However, the increase in performance was only marginal compared to the pretrained model. CycleGAN-based stain normalization suffered from encoded imperceptible information into the normalizations that confused the pretrained model and thus resulted in slightly worse performance. ConclusionsOur findings suggest that stain variability can be tackled more effectively by augmenting data by it than by following the commonly used approach of normalizing the stain. However, the applicability of this approach providing only a rather slight performance increase has to be weighted against an additional carbon footprint.

ScoreMix: A Scalable Augmentation Strategy for Training GANs With Limited Data.

Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available. To facilitate GAN training, current methods propose to use data-specific augmentation techniques. Despite the effectiveness, it is difficult for these methods to scale to practical applications. In this article, we present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks. We first produce augmented samples using the convex combinations of the real samples. Then, we optimize the augmented samples by minimizing the norms of the data scores, i.e., the gradients of the log-density functions. This procedure enforces the augmented samples close to the data manifold. To estimate the scores, we train a deep estimation network with multi-scale score matching. For different image synthesis tasks, we train the score estimation network using different data. We do not require the tuning of the hyperparameters or modifications to the network architecture. The ScoreMix method effectively increases the diversity of data and reduces the overfitting problem. Moreover, it can be easily incorporated into existing GAN models with minor modifications. Experimental results on numerous tasks demonstrate that GAN models equipped with the ScoreMix method achieve significant improvements.

Generative adversarial networks with bio-inspired primary visual cortex for Industry 4.0

Biologicalization (biological transformation) is an emerging trend in Industry 4.0 affecting digitization of manufacturing and related processes. It brings up the next generation of manufacturing technology and systems that extensively use biological and bio-inspired principles, materials, functions, structures and resources. This research is a contribution to the further convergence of computer and human vision for more robust and accurate automated object recognition and image generation. We present VOneGANs, a novel class of generative adversarial networks (GANs) with the qualitatively updated discriminative component. The new model incorporates a biologically constrained digital primary visual cortex V1. This earliest cortical visual area performs the first stage of human‘s visual processing and is believed to be a reason of its robustness and accuracy. Experiments with the updated architectures confirm the improved stability of GANs training and the higher quality of the automatically generated visual content. The promising results allow considering VOneGANs as providers of high-quality training content and as enablers of future simulation-based decision-making and decision-support tools for condition-monitoring, supervisory control, diagnostics, predictive maintenance, and cybersecurity in Industry 4.0.

DALF: An AI Enabled Adversarial Framework for Classification of Hyperspectral Images

Hyperspectral image classification is very complex and challenging process. However, with deep neural networks like Convolutional Neural Networks (CNN) with explicit dimensionality reduction, the capability of classifier is greatly increased. However, there is still problem with sufficient training samples. In this paper, we overcome this problem by proposing an Artificial Intelligence (AI) based framework named Deep Adversarial Learning Framework (DALF) that exploits deep autoencoder for dimensionality reduction, Generative Adversarial Network (GAN) for generating new Hyperspectral Imaging (HSI) samples that are to be verified by a discriminator in a non-cooperative game setting besides using aclassifier. Convolutional Neural Network (CNN) is used for both generator and discriminator while classifier role is played by Support Vector Machine (SVM) and Neural Network (NN). An algorithm named Generative Model based Hybrid Approach for HSI Classification (GMHA-HSIC) which drives the functionality of the proposed framework is proposed. The success of DALF in accurate classification is largely dependent on the synthesis and labelling of spectra on regular basis. The synthetic samples made with an iterative process and being verified by discriminator result in useful spectra. By training GAN with associated deep learning models, the framework leverages classification performance. Our experimental results revealed that the proposed framework has potential to improve the state of the art besides having an effective data augmentation strategy.

Image Generation with a Unified Generative Adversarial Network Training via Self-Labeling and Self-Attention

Image Generation with a Unified Generative Adversarial Network Training via Self-Labeling and Self-Attention

Real-time super-resolution mapping of locally anisotropic grain orientations for ultrasonic non-destructive evaluation of crystalline material

Estimating the spatially varying microstructures of heterogeneous and locally anisotropic media non-destructively is necessary for the accurate detection of flaws and reliable monitoring of manufacturing processes. Conventional algorithms used for solving this inverse problem come with significant computational cost, particularly in the case of high-dimensional, nonlinear tomographic problems, and are thus not suitable for near-real-time applications. In this paper, for the first time, we propose a framework which uses deep neural networks (DNNs) with full aperture, pitch-catch and pulse-echo transducer configurations, to reconstruct material maps of crystallographic orientation. We also present the first application of generative adversarial networks (GANs) to achieve super-resolution of ultrasonic tomographic images, providing a factor-four increase in image resolution and up to a 50% increase in structural similarity. The importance of including appropriate prior knowledge in the GAN training data set to increase inversion accuracy is demonstrated: known information about the material’s structure should be represented in the training data. We show that after a computationally expensive training process, the DNNs and GANs can be used in less than 1 second (0.9 s on a standard desktop computer) to provide a high-resolution map of the material’s grain orientations, addressing the challenge of significant computational cost faced by conventional tomography algorithms.

Automatic expression recognition of face image sequence based on key-frame generation and differential emotion feature

Using two key-frames with the neutral expression and maximum expression intensity respectively in face image sequence to alleviate inter-subject variations for Facial Expression Recognition (FER) has become the research focus of computer vision. To determine the two key-frames, an automatic Generation Difference Convolutional Neural Network (GDCNN) framework was presented to reduce the influences of high inter-subject variations caused by individual differences. Firstly, for any given input expression, a trained conditional Generative Adversarial Network (cGAN) is utilized to generate the frame with neutral expression while keeping the identity-related information. Secondly, the fine-tuned triplet distance model is used to detect the frame with the maximum expression intensity. Finally, an optimized two-stream CNN model is designed to reduce the influences of inter-subject variations and trained to extract the differential emotion features for FER. Extensive comparisons were performed on the CK+, MMI, and Beihang University databases and the results demonstrated the superiority of the proposed GDCNN framework over the state-of-the-art methods.

Image Augmentation based on GAN deep learning approach with Textual Content Descriptors

Computer vision, also known as computational visual perception, is a branch of artificial intelligence that allows computers to interpret digital pictures and videos in a manner comparable to biological vision. It entails the development of techniques for simulating biological vision. The aim of computer vision is to extract more meaningful information from visual input than that of a biological vision. Computer vision is exploding due to the avalanche of data being produced today. Powerful generative models, such as Generative Adversarial Networks (GANs), are responsible for significant advances in the field of picture creation. The focus of this research is to concentrate on textual content descriptors in the images used by GANs to generate synthetic data from the MNIST dataset to either supplement or replace the original data while training classifiers. This can provide better performance than other traditional image enlarging procedures due to the good handling of synthetic data. It shows that training classifiers on synthetic data are as effective as training them on pure data alone, and it also reveals that, for small training data sets, supplementing the dataset by first training GANs on the data may lead to a significant increase in classifier performance.

Multi-class Generative Adversarial Networks: Improving One-class Classification of Pneumonia Using Limited Labeled Data.

From generating never-before-seen images to domain adaptation, applications of Generative Adversarial Networks (GANs) spread wide in the domain of vision and graphics problems. With the remarkable ability of GANs in learning the distribution and generating images of a particular class, they have been used for semi-supervised disease detection in medical images such as COVID-19 and Pneumonia in X-rays. However, the challenge is that if two classes of images share similar characteristics, the GAN might learn to generalize and hinder the classification of the two classes. In this paper, first we use MNIST and Fashion-MNIST datasets that are easy to visually inspect, to illustrate how similar images cause the GAN to generalize, leading to the poor classification of images. We then show how this generalization can misclassify pneumonia X-rays as healthy cases when using GANs for semi-supervised detection of pneumonia. We propose a modification to the traditional training of GANs that, using small sets of labeled data, allows for improved classification in similar classes of images in a semi-supervised learning framework.

Automated Processing and Phenotype Extraction of Ovine Medical Images Using a Combined Generative Adversarial Network and Computer Vision Pipeline.

The speed and accuracy of phenotype detection from medical images are some of the most important qualities needed for any informed and timely response such as early detection of cancer or detection of desirable phenotypes for animal breeding. To improve both these qualities, the world is leveraging artificial intelligence and machine learning against this challenge. Most recently, deep learning has successfully been applied to the medical field to improve detection accuracies and speed for conditions including cancer and COVID-19. In this study, we applied deep neural networks, in the form of a generative adversarial network (GAN), to perform image-to-image processing steps needed for ovine phenotype analysis from CT scans of sheep. Key phenotypes such as gigot geometry and tissue distribution were determined using a computer vision (CV) pipeline. The results of the image processing using a trained GAN are strikingly similar (a similarity index of 98%) when used on unseen test images. The combined GAN-CV pipeline was able to process and determine the phenotypes at a speed of 0.11 s per medical image compared to approximately 30 min for manual processing. We hope this pipeline represents the first step towards automated phenotype extraction for ovine genetic breeding programmes.