Adversarial Data Augmentation Research Articles

Learning a Generalized Gaze Estimator from Gaze-Consistent Feature

Gaze estimator computes the gaze direction based on face images. Most existing gaze estimation methods perform well under within-dataset settings, but can not generalize to unseen domains. In particular, the ground-truth labels in unseen domain are often unavailable. In this paper, we propose a new domain generalization method based on gaze-consistent features. Our idea is to consider the gaze-irrelevant factors as unfavorable interference and disturb the training data against them, so that the model cannot fit to these gaze-irrelevant factors, instead, only fits to the gaze-consistent features. To this end, we first disturb the training data via adversarial attack or data augmentation based on the gaze-irrelevant factors, i.e., identity, expression, illumination and tone. Then we extract the gaze-consistent features by aligning the gaze features from disturbed data with non-disturbed gaze features. Experimental results show that our proposed method achieves state-of-the-art performance on gaze domain generalization task. Furthermore, our proposed method also improves domain adaption performance on gaze estimation. Our work provides new insight on gaze domain generalization task.

Do Invariances in Deep Neural Networks Align with Human Perception?

An evaluation criterion for safe and trustworthy deep learning is how well the invariances captured by representations of deep neural networks (DNNs) are shared with humans. We identify challenges in measuring these invariances. Prior works used gradient-based methods to generate identically represented inputs (IRIs), ie, inputs which have identical representations (on a given layer) of a neural network, and thus capture invariances of a given network. One necessary criterion for a network's invariances to align with human perception is for its IRIs look 'similar' to humans. Prior works, however, have mixed takeaways; some argue that later layers of DNNs do not learn human-like invariances yet others seem to indicate otherwise. We argue that the loss function used to generate IRIs can heavily affect takeaways about invariances of the network and is the primary reason for these conflicting findings. We propose an adversarial regularizer on the IRI generation loss that finds IRIs that make any model appear to have very little shared invariance with humans. Based on this evidence, we argue that there is scope for improving models to have human-like invariances, and further, to have meaningful comparisons between models one should use IRIs generated using the regularizer-free loss. We then conduct an in-depth investigation of how different components (eg architectures, training losses, data augmentations) of the deep learning pipeline contribute to learning models that have good alignment with humans. We find that architectures with residual connections trained using a (self-supervised) contrastive loss with l_p ball adversarial data augmentation tend to learn invariances that are most aligned with humans. Code: github.com/nvedant07/Human-NN-Alignment

When Neural Networks Fail to Generalize? A Model Sensitivity Perspective

Domain generalization (DG) aims to train a model to perform well in unseen domains under different distributions. This paper considers a more realistic yet more challenging scenario, namely Single Domain Generalization (Single-DG), where only a single source domain is available for training. To tackle this challenge, we first try to understand when neural networks fail to generalize? We empirically ascertain a property of a model that correlates strongly with its generalization that we coin as "model sensitivity". Based on our analysis, we propose a novel strategy of Spectral Adversarial Data Augmentation (SADA) to generate augmented images targeted at the highly sensitive frequencies. Models trained with these hard-to-learn samples can effectively suppress the sensitivity in the frequency space, which leads to improved generalization performance. Extensive experiments on multiple public datasets demonstrate the superiority of our approach, which surpasses the state-of-the-art single-DG methods by up to 2.55%. The source code is available at https://github.com/DIAL-RPI/Spectral-Adversarial-Data-Augmentation.

Adversarial and Random Transformations for Robust Domain Adaptation and Generalization.

Data augmentation has been widely used to improve generalization in training deep neural networks. Recent works show that using worst-case transformations or adversarial augmentation strategies can significantly improve accuracy and robustness. However, due to the non-differentiable properties of image transformations, searching algorithms such as reinforcement learning or evolution strategy have to be applied, which are not computationally practical for large-scale problems. In this work, we show that by simply applying consistency training with random data augmentation, state-of-the-art results on domain adaptation (DA) and generalization (DG) can be obtained. To further improve the accuracy and robustness with adversarial examples, we propose a differentiable adversarial data augmentation method based on spatial transformer networks (STNs). The combined adversarial and random-transformation-based method outperforms the state-of-the-art on multiple DA and DG benchmark datasets. Furthermore, the proposed method shows desirable robustness to corruption, which is also validated on commonly used datasets.

Features kept generative adversarial network data augmentation strategy for hyperspectral image classification

In recent years, significant breakthroughs have been achieved in hyperspectral image (HSI) processing using deep learning techniques, including classification, object detection, and anomaly detection. However, the practical application of deep learning in HSI processing is limited by challenges such as small-sample size and sample imbalance issues. To mitigate these limitations, we propose a novel data augmentation strategy called Feature-Preserving Generative Adversarial Network Data Augmentation (FPGANDA). What sets our data augmentation strategy apart from existing generative model-based approaches is that we preserve the main spectral bands of HSI data using a newly designed band selection method. Additionally, our proposed generative model generates synthetic spectral bands, which are combined with the real spectral bands using a mixture strategy to create augmented data. This approach ensures that the augmented data retain the main features of the original data while also incorporating diverse features from the generated data. We evaluate our method on three different HSI datasets, comparing it with state-of-the-art techniques. Experimental results demonstrate that our proposed method significantly improves classification performance in most scenes and exhibits remarkable compatibility.

Deep Convolutional Generative Adversarial Networks-Based Data Augmentation Method for Classifying Class-Imbalanced Defect Patterns in Wafer Bin Map

In the semiconductor industry, achieving a high production yield is a very important issue. Wafer bin maps (WBMs) provide critical information for identifying anomalies in the manufacturing process. A WBM forms a certain defect pattern according to the error occurring during the process, and by accurately classifying the defect pattern existing in the WBM, the root causes of the anomalies that have occurred during the process can be inferred. Therefore, WBM defect pattern recognition and classification tasks are important for improving yield. In this paper, we propose a deep convolutional generative adversarial network (DCGAN)-based data augmentation method to improve the accuracy of a convolutional neural network (CNN)-based defect pattern classifier in the presence of extremely imbalanced data. The proposed method forms various defect patterns compared to the data augmentation method by using a convolutional autoencoder (CAE), and the formed defect patterns are classified into the same pattern as the original pattern through a CNN-based defect pattern classifier. Here, we introduce a new quantitative index called PGI to compare the effectiveness of the augmented models, and propose a masking process to refine the augmented images. The proposed method was tested using the WM-811k dataset. The proposed method helps to improve the classification performance of the pattern classifier by effectively solving the data imbalance issue compared to the CAE-based augmentation method. The experimental results showed that the proposed method improved the accuracy of each defect pattern by about 5.31% on average compared to the CAE-based augmentation method.

Adversarial Data Augmentation and Transfer Net for Scrap Metal Identification Using Laser-Induced Breakdown Spectroscopy Measurement of Standard Reference Materials.

In this study, we propose a transfer learning-based classification model for identifying scrap metal using an augmented training dataset consisting of laser-induced breakdown spectroscopy (LIBS) measurement of standard reference material (SRMs) samples, considering varying experimental setups and environmental conditions. LIBS provides unique spectra for identifying unknown samples without complicated sample preparation. Thus, LIBS systems combined with machine learning methods have been actively studied for industrial applications such as scrap metal recycling. However, in machine learning models, a training set of the used samples may not cover the diversity of the scrap metal encountered in field measurements. Moreover, differences in experimental configuration, where laboratory standards and real samples are analyzed in situ, may lead to a wider gap in the distribution of training and test sets, dramatically reducing the performance of the LIBS-based fast classification system for real samples. To address these challenges, we propose a two-step Aug2Tran model. First, we augment the SRM dataset by synthesizing spectra of unobserved types through attenuation of dominant peaks corresponding to sample composition and generating spectra depending on the target sample using a generative adversarial network. Second, we used the augmented SRM dataset to build a robust real-time classification model with a convolutional neural network, which is further customized for the target scrap metal with limited measurements through transfer learning. For evaluation, SRMs of five representative metal types, including aluminum, copper, iron, stainless steel, and brass, are measured with a typical setup to form the SRM dataset. For testing, scrap metal from actual industrial fields is experimented with three different configurations, resulting in eight different test datasets. The experimental results show that the proposed scheme produces an average classification accuracy of 98.25% for the three experimental conditions, as high as the results of the conventional scheme with three separately trained and executed models. Additionally, the proposed model improves the classification accuracy of arbitrarily shaped static or moving samples with various surface contaminations and compositions, and even for differing ranges of charted intensities and wavelengths. Therefore, the proposed Aug2Tran model can be used as a systematic model for scrap metal classification with generalizability and ease of implementation.

Wind turbine anomaly detection based on SCADA: A deep autoencoder enhanced by fault instances

An increasing number of deep autoencoder-based algorithms for intelligent condition monitoring and anomaly detection have been reported in recent years to improve wind turbine reliability. However, most existing studies have only focused on the precise modeling of normal data in an unsupervised manner; few studies have utilized the information of fault instances in the learning process, which results in suboptimal detection performance and low robustness. To this end, we first developed a deep autoencoder enhanced by fault instances, that is, a triplet-convolutional deep autoencoder (triplet-Conv DAE), jointly integrating a convolutional autoencoder and deep metric learning. Aided by fault instances, triplet-Conv DAE can not only capture normal operation data patterns but also acquire discriminative deep embedding features. Moreover, to overcome the difficulty of scarce fault instances, we adopted an improved generative adversarial network-based data augmentation method to generate high-quality synthetic fault instances. Finally, we validated the performance of the proposed anomaly detection method using a multitude of performance measures. The experimental results show that our method is superior to three other state-of-the-art methods. In addition, the proposed augmentation method can efficiently improve the performance of the triplet-Conv DAE when fault instances are insufficient.

A contrastive learning based universal representation for time series forecasting

Time series forecasting has wide applications in our daily lives, such as meteorological warnings and decision-making. However, traditional supervised models do not perform well on forecasting tasks due to the lack of annotated training data available in real time series. Recently, researchers have proposed self-supervised methods, especially the contrastive learning approach, which can build a universal representation framework for downstream tasks through data augmentation and contrastive loss. Although it alleviates the shortage of labeled data, the data augmentation approach directly transferred from computer vision is not appropriate for the time domain due to noise vectors and unrelated variables that may interfere with the accuracy of representation. In this paper, we propose a novel time series forecasting model based on disentangled seasonal-trend representation named ACST. It employs an improved cycle generative adversarial data augmentation method to generate samples close to real data for contrastive loss. Moreover, we apply gated residual networks and a noise decomposition module to reduce the impact of different noise vectors and feature variable weights on the results. Extensive experiments show that ACST achieves an average improvement of 26.8% on six benchmarks.

Towards better long-tailed oracle character recognition with adversarial data augmentation

Deciphering oracle bone script is of great significance to the study of ancient Chinese culture as well as archaeology. Although recent studies on oracle character recognition have made substantial progress, they still suffer from the long-tailed data situation that results in a noticeable performance drop on the tail classes. To mitigate this issue, we propose a generative adversarial framework to augment oracle characters in the problematic classes. In this framework, the generator produces synthetic data through convex combinations of all the available samples in the corresponding classes, and is further optimized through adversarial learning with the classifier and simultaneously the discriminator. Meanwhile, we introduce Repatch to generalize samples in the generator. Since tail classes do not have sufficient data for convex combinations, we propose the TailMix mechanism to generate suitable tail class samples from other classes. Experimental results show that our proposed algorithm obtains remarkable performance in oracle character recognition and achieves new state-of-the-art average (total) accuracy with 86.03% (89.46%), 86.54% (93.86%), 95.22% (96.17%) on the three datasets Oracle-AYNU, OBC306 and Oracle-20K, respectively.

Adversarial liveness detector: Leveraging adversarial perturbations in fingerprint liveness detection

AbstractThe widespread use of fingerprint authentication systems (FASs) in consumer electronics opens for the development of advanced presentation attacks, that is, procedures designed to bypass a FAS using a forged fingerprint. As a consequence, FAS are often equipped with a fingerprint presentation attack detection (FPAD) module, to recognise live fingerprints from fake replicas. In this work, a novel FPAD approach based on Convolutional Neural Networks (CNNs) and on an ad hoc adversarial data augmentation strategy designed to iteratively increase the considered detector robustness is proposed. In particular, the concept of adversarial fingerprint, that is, fake fingerprints disguised by using ad hoc fingerprint adversarial perturbation algorithms was leveraged to help the detector focus only on salient portions of the fingerprints. The procedure can be adapted to different CNNs, adversarial fingerprint algorithms and fingerprint scanners, making the proposed approach versatile and easily customisable todifferent working scenarios. To test the effectiveness of the proposed approach, the authors took part in the LivDet 2021 competition, an international challenge gathering experts to compete on fingerprint liveness detection under different scanners and fake replica generation approach, achieving first place out of 23 participants in the ‘Liveness Detection in Action track’.

Hydrogen yield prediction for supercritical water gasification based on generative adversarial network data augmentation

Machine learning methods can accurately predict the hydrogen yield of supercritical water gasification (SCWG), which can provide a reference for SCWG optimization. However, the time cost of obtaining SCWG data by experiments is high, and the prediction accuracy of the SCWG model is low when the sample number is small. For this problem, a novel sample generation method based on the generative adversarial network (GAN) and the hybrid strategy is proposed to achieve data augmentation for SCWG hydrogen yield prediction. The hybrid strategy that combines the cross-validation and looping iteration methods can reduce the randomness of the GAN, thereby improving the generated sample quality. In addition, the least squares support vector regression is adopted to construct a hydrogen yield prediction model. A case study was conducted on a small sample SCWG dataset of Yimin lignite with reaction temperatures in the range of 650–850 °C. The results show that after expanding the small sample dataset by the single GAN, the average RMSE and MAE of the prediction model are reduced by 20.43% and 11.58%, respectively. The proposed sample generation method is applied to the SCWG hydrogen yield prediction. The average RMSE and MAE are as low as 1.1737 mol/kg and 1.0171 mol/kg, respectively, while the average R2 reaches 0.9457. The average RMSE and MAE are further reduced by 18.44% and 15.29% based on the single GAN model, respectively. It indicates that the hybrid strategy improves the quality and reliability of the generated samples, and the proposed method can effectively achieve data augmentation for SCWG hydrogen yield prediction, which can provide ideas for the small sample SCWG modeling.

ADASSM: Adversarial Data Augmentation in Statistical Shape Models From Images.

Statistical shape models (SSM) have been well-established as an excellent tool for identifying variations in the morphology of anatomy across the underlying population. Shape models use consistent shape representation across all the samples in a given cohort, which helps to compare shapes and identify the variations that can detect pathologies and help in formulating treatment plans. In medical imaging, computing these shape representations from CT/MRI scans requires time-intensive preprocessing operations, including but not limited to anatomy segmentation annotations, registration, and texture denoising. Deep learning models have demonstrated exceptional capabilities in learning shape representations directly from volumetric images, giving rise to highly effective and efficient Image-to-SSM networks. Nevertheless, these models are data-hungry and due to the limited availability of medical data, deep learning models tend to overfit. Offline data augmentation techniques, that use kernel density estimation based (KDE) methods for generating shape-augmented samples, have successfully aided Image-to-SSM networks in achieving comparable accuracy to traditional SSM methods. However, these augmentation methods focus on shape augmentation, whereas deep learning models exhibit image-based texture bias resulting in sub-optimal models. This paper introduces a novel strategy for on-the-fly data augmentation for the Image-to-SSM framework by leveraging data-dependent noise generation or texture augmentation. The proposed framework is trained as an adversary to the Image-to-SSM network, augmenting diverse and challenging noisy samples. Our approach achieves improved accuracy by encouraging the model to focus on the underlying geometry rather than relying solely on pixel values.

Augmented Multi-Scale Spatiotemporal Inconsistency Magnifier for Generalized DeepFake Detection

Recently, realistic DeepFake videos have raised severe security concerns in society. Existing video-based detection methods observe local spatial regions with the coarse temporal view, thus it is difficult to obtain subtle spatiotemporal information, resulting in limited generalization ability. In this paper, we propose a novel Augmented Multi-scale Spatiotemporal Inconsistency Magnifier (AMSIM) with a Global Inconsistency View (GIV) and a more meticulous Multi-timescale Local Inconsistency View (MLIV), focusing on mining comprehensive and more subtle spatiotemporal cues. Firstly, the GIV that includs the global spatial and long-term temporal views is established to ensure comprehensive spatiotemporal clues are captured. Then, the MLIV with the critical local spatial and multi-timescale local temporal views is designed for magnifying the indetectable spatiotemporal abnormality. Subsequently, GIV is utilized to guide MLIV to dynamically find local spatiotemporal anomalies that are highly relevant to the overall video. Finally, to further obtain a generalized framework, the adversarial data augmentation is specially designed to expand source domains and simulate unseen forgery domains. Extensive experiments on six large-scale datasets show that our AMSIM outperforms state-of-the-art detection methods and remains effective when applied to unseen forgery techniques and datasets.

A generative adversarial network-based data augmentation approach with transient vibration data

In order to perform fault classification using machine learning algorithms, sufficient and balanced data is required. The paper presents a novel approach for data augmentation using a Generative Adversarial Network (GAN) with transient, time-dependent vibration data. The proposed approach aims to generate synthetic data in form of spectrograms that closely resemble the characteristics of the real data sets. Synthetic data generation can be used to improve the training performance of neural networks for vibration analysis if not enough or unbalanced real data is available. The authors demonstrate the effectiveness of this approach by training a vibration-based fault detection model using synthetic data and comparing its performance to a model trained on real data only as well as with at-distributed stochastic neighbour embedding (t-SNE). Real data is acquired on a bearing test rig and measurements are carried out on bearings with four different system states. The results show that the model trained with the synthetic data set outperforms the model trained with real data only, indicating that the synthetic data generated by the proposed approach can improve the training performance and accuracy of the machine learning model. Overall, the paper highlights the potential of GAN-based data augmentation approach via spectrograms for vibration analysis and offers insights into its practical application to bearings.

A Fuzzy-Boundary Enhanced Trident Network for Parcel Extraction in the Urban–Rural Area

As the basic unit of farmland, parcel is crucial for remote sensing tasks, such as urban management. Previous studies of farmland parcels extraction are based on boundary detection and instance segmentation methods. However, these methods perform poorly in the parcels with complex shape and fuzzy boundary due to the insufficient feature extraction capability. Moreover, for the lack of multi-scale features extraction and fusion, they are difficult to extract different scale farmland parcels accurately. Based on these issues, we propose a Fuzzy-Boundary Enhanced Trident Network, named FBETNet, to enhance the feature of fuzzy boundary and generate multi-scale parcels. First, a semantic-guided multi-task strategy is introduced in order to enhance the feature of fuzzy boundary. Second, we design a multi-scale trident module to further improve the performance of multi-scale feature extraction. Finally, a adversarial data augmentation strategy is employed in the training phase to strengthen the robustness and stability of out proposed method. Experiments show that our proposed method improves significantly in both accuracy and visualization, especially for the parcels with fuzzy boundary and complex shape.

Automatic tunnel lining crack detection via deep learning with generative adversarial network-based data augmentation

Aiming at solving the challenges of insufficient data samples and low detection efficiency in tunnel lining crack detection methods based on deep learning, a novel detection approach for tunnel lining crack was proposed, which is based on pruned You Look Only Once v4 (YOLOv4) and Wasserstein Generative Adversarial Network enhanced by Residual Block and Efficient Channel Attention Module (WGAN-RE). In this study, a data augmentation method named WGAN-RE was proposed, which can achieve the automatic generation of crack images to enrich data set. Furthermore, YOLOv4 was selected as the basic model for training, and a pruning algorithm was introduced to lighten the model size, thereby effectively improving the detection speed. Average Precision (AP), F1 Score (F1), model size, and Frames Per Second (FPS) were selected as evaluation indexes of the model performance. Results indicate that the storage space of the pruned YOLOv4 model is only 49.16 MB, which is 80% compressed compared with the model before pruning. In addition, the FPS of the model reaches 40.58f/s, which provides a basis for the real-time detection of tunnel lining cracks. Findings also demonstrate that the F1 score and AP of the pruned YOLOv4 are only 0.77% and 0.50% lower than that before pruning, respectively. Besides, the pruned YOLOv4 is superior in both model accuracy and efficiency compared with YOLOv3, SSD, and Faster RCNN, which indicated that the pruned YOLOv4 model can realize the accurate, fast and intelligent detection of tunnel lining cracks in practical tunnel engineering.

Enhancing MR image segmentation with realistic adversarial data augmentation.

The success of neural networks on medical image segmentation tasks typically relies on large labeled datasets for model training. However, acquiring and manually labeling a large medical image set is resource-intensive, expensive, and sometimes impractical due to data sharing and privacy issues. To address this challenge, we propose AdvChain, a generic adversarial data augmentation framework, aiming at improving both the diversity and effectiveness of training data for medical image segmentation tasks. AdvChainaugments data with dynamic data augmentation, generating randomly chained photo-metric and geometric transformations to resemble realistic yet challenging imaging variations to expand training data. By jointly optimizing the data augmentation model and a segmentation network during training, challenging examples are generated to enhance network generalizability for the downstream task. The proposed adversarial data augmentation does not rely on generative networks and can be used as a plug-in module in general segmentation networks. It is computationally efficient and applicable for both low-shot supervised and semi-supervised learning. We analyze and evaluate the method on two MR image segmentation tasks: cardiac segmentation and prostate segmentation with limited labeled data. Results show that the proposed approach can alleviate the need for labeled data while improving model generalization ability, indicating its practical value in medical imaging applications.

VideoDG: Generalizing Temporal Relations in Videos to Novel Domains.

This paper introduces video domain generalization where most video classification networks degenerate due to the lack of exposure to the target domains of divergent distributions. We observe that the global temporal features are less generalizable, due to the temporal domain shift that videos from other unseen domains may have an unexpected absence or misalignment of the temporal relations. This finding has motivated us to solve video domain generalization by effectively learning the local-relation features of different timescales that are more generalizable, and exploiting them along with the global-relation features to maintain the discriminability. This paper presents the VideoDG framework with two technical contributions. The first is a new deep architecture named the Adversarial Pyramid Network, which improves the generalizability of video features by capturing the local-relation, global-relation, and cross-relation features progressively. On the basis of pyramid features, the second contribution is a new and robust approach of adversarial data augmentation that can bridge different video domains by improving the diversity and quality of augmented data. We construct three video domain generalization benchmarks in which domains are divided according to different datasets, different consequences of actions, or different camera views, respectively. VideoDG consistently outperforms the combinations of previous video classification models and existing domain generalization methods on all benchmarks.

Multimodal CT Image Synthesis Using Unsupervised Deep Generative Adversarial Networks for Stroke Lesion Segmentation

Deep learning-based techniques can obtain high precision for multimodal stroke segmentation tasks. However, the performance often requires a large number of training examples. Additionally, existing data extension approaches for the segmentation are less efficient in creating much more realistic images. To overcome these limitations, an unsupervised adversarial data augmentation mechanism (UTC-GAN) is developed to synthesize multimodal computed tomography (CT) brain scans. In our approach, the CT samples generation and cross-modality translation differentiation are accomplished simultaneously by integrating a Siamesed auto-encoder architecture into the generative adversarial network. In addition, a Gaussian mixture translation module is further proposed, which incorporates a translation loss to learn an intrinsic mapping between the latent space and the multimodal translation function. Finally, qualitative and quantitative experiments show that UTC-GAN significantly improves the generation ability. The stroke dataset enriched by the proposed model also provides a superior improvement in segmentation accuracy, compared with the performance of current competing unsupervised models.