Adversarial Data Augmentation Research Articles

Swin transformer adaptation into YOLOv7 for road damage detection

Highways are an important component of any country. However, some highways in Indonesia endanger users while maintaining road safety. Crack detection early in the deterioration process can prevent further damage and lower maintenance costs. A recent study sought to develop a method for detecting road damage by combining the road damage detection (RDD) dataset with generative adversarial network technology and data augmentation to improve training. The current study aims to broaden the you only look once (YOLO) framework by incorporating the Swin Transformer into the chiral stationary phases (CSP) component of YOLOv7, with the goal of improving object detection accuracy in a variety of visual scenarios. The study compares the performance of various object detection models with varying parameters and configurations, such as YOLOv5l, YOLOv6l, YOLOv7-tiny, YOLOv7, and YOLOv7x. YOLOv5l has 46 million parameters and 108 billion floating point operations per second (FLOPS), whereas YOLOv6l has 59.5 million parameters and 150 billion FLOPS. With 31 million parameters and 140 billion FLOPS, the YOLOv7-swin model performs best with mean average precision (mAP), mAP_0.50 of 0.47. and mAP_0.5:0.95 of 0.232. The experimental results show that our YOLOv7-swin model outperforms both YOLOv7x and YOLOv7-tiny. The proposed model significantly improves object detection accuracy while keeping complexity and performance in balance.

Automatic cybersickness detection by deep learning of augmented physiological data from off-the-shelf consumer-grade sensors

Cybersickness is still a prominent risk factor potentially affecting the usability of virtual reality applications. Automated real-time detection of cybersickness promises to support a better general understanding of the phenomena and to avoid and counteract its occurrence. It could be used to facilitate application optimization, that is, to systematically link potential causes (technical development and conceptual design decisions) to cybersickness in closed-loop user-centered development cycles. In addition, it could be used to monitor, warn, and hence safeguard users against any onset of cybersickness during a virtual reality exposure, especially in healthcare applications. This article presents a novel real-time-capable cybersickness detection method by deep learning of augmented physiological data. In contrast to related preliminary work, we are exploring a unique combination of mid-immersion ground truth elicitation, an unobtrusive wireless setup, and moderate training performance requirements. We developed a proof-of-concept prototype to compare (combinations of) convolutional neural networks, long short-term memory, and support vector machines with respect to detection performance. We demonstrate that the use of a conditional generative adversarial network-based data augmentation technique increases detection performance significantly and showcase the feasibility of real-time cybersickness detection in a genuine application example. Finally, a comprehensive performance analysis demonstrates that a four-layered bidirectional long short-term memory network with the developed data augmentation delivers superior performance (91.1% F1-score) for real-time cybersickness detection. To encourage replicability and reuse in future cybersickness studies, we released the code and the dataset as publicly available.

Adversarial Learning-Based Data Augmentation for Palm-Vein Identification

Adversarial Learning-Based Data Augmentation for Palm-Vein Identification

Enhancing recognition performance of vortex arrays through conditional generative adversarial network-based data augmentation

Enhancing recognition performance of vortex arrays through conditional generative adversarial network-based data augmentation

Rethinking Cost-Sensitive Classification in Deep Learning via Adversarial Data Augmentation

Cost-sensitive classification is critical in applications where misclassification errors widely vary in cost. However, overparameterization poses fundamental challenges to the cost-sensitive modeling of deep neural networks (DNNs). The ability of a DNN to fully interpolate a training data set can render a DNN, evaluated purely on the training set, ineffective in distinguishing a cost-sensitive solution from its overall accuracy maximization counterpart. This necessitates rethinking cost-sensitive classification in DNNs. To address this challenge, this paper proposes a cost-sensitive adversarial data augmentation (CSADA) framework to make overparameterized models cost sensitive. The overarching idea is to generate targeted adversarial examples that push the decision boundary in cost-aware directions. These targeted adversarial samples are generated by maximizing the probability of critical misclassifications and used to train a model with more conservative decisions on costly pairs. Experiments on well-known data sets and a pharmacy medication image (PMI) data set, made publicly available, show that our method can effectively minimize the overall cost and reduce critical errors while achieving comparable overall accuracy. Funding: Research reported in this publication was supported by the National Library of medicine of the National Institutes of Health in the United States under award number R01LM013624. Data Ethics & Reproducibility Note: This paper abides by data ethics requirements. Data used are publicly available online at https://deepblue.lib.umich.edu/data/concern/data_sets/6d56zw997 . Codes to replicate the results of this paper are available at https://doi.org/10.24433/CO.2139841.v1 .

Generative adversarial network-based data augmentation for improving hypoglycemia prediction: A proof-of-concept study

Background and objectiveHypoglycemia is one of the major barriers for intensive insulin treatment to achieve optimal glycemic control for people with diabetes. Accurate prediction of hypoglycemia became an important factor for advancing insulin therapy, and thus numerous studies have proposed data-driven models. However, the data-driven models still suffer from performance degradation due to severe data imbalance between hypoglycemia and non-hypoglycemia. To overcome this problem, we propose a generative adversarial network (GAN) based data augmentation method, generating realistic continuous glucose monitoring (CGM) time series labeled hypoglycemia. MethodsHaving acquired a large-scale CGM time series dataset, we compared the performance of various models before and after five data augmentation methods. ResultsThe GAN-based data augmentation method improved the hypoglycemia prediction performance when combined with ML models and we found that the data augmentation method was comparable to conventional data augmentation method. Through visualization, it was found that successfully generated CGM time series satisfied a given condition, and the generated CGM time series were visually separated according to the given condition in an embedding space. These results suggest that GAN-based data augmentation is a promising approach for solving the severe data imbalance of hypoglycemia prediction. ConclusionsWe believe that the combination of more accurate hypoglycemia prediction models and intensive insulin therapy will result in better glycemic control for people with diabetes.

Improving Breast Tumor Multi-Classification from High-Resolution Histological Images with the Integration of Feature Space Data Augmentation

To support pathologists in breast tumor diagnosis, deep learning plays a crucial role in the development of histological whole slide image (WSI) classification methods. However, automatic classification is challenging due to the high-resolution data and the scarcity of representative training data. To tackle these limitations, we propose a deep learning-based breast tumor gigapixel histological image multi-classifier integrated with a high-resolution data augmentation model to process the entire slide by exploring its local and global information and generating its different synthetic versions. The key idea is to perform the classification and augmentation in feature latent space, reducing the computational cost while preserving the class label of the input. We adopt a deep learning-based multi-classification method and evaluate the contribution given by a conditional generative adversarial network-based data augmentation model on the classifier’s performance for three tumor classes in the BRIGHT Challenge dataset. The proposed method has allowed us to achieve an average F1 equal to 69.5, considering only the WSI dataset of the Challenge. The results are comparable to those obtained by the Challenge winning method (71.6), also trained on the annotated tumor region dataset of the Challenge.

Image segmentation of exfoliated two-dimensional materials by generative adversarial network-based data augmentation

Mechanically cleaved two-dimensional materials are random in size and thickness. Recognizing atomically thin flakes by human experts is inefficient and unsuitable for scalable production. Deep learning algorithms have been adopted as an alternative, nevertheless a major challenge is a lack of sufficient actual training images. Here we report the generation of synthetic two-dimensional materials images using StyleGAN3 to complement the dataset. DeepLabv3Plus network is trained with the synthetic images which reduces overfitting and improves recognition accuracy to over 90%. A semi-supervisory technique for labeling images is introduced to reduce manual efforts. The sharper edges recognized by this method facilitate material stacking with precise edge alignment, which benefits exploring novel properties of layered-material devices that crucially depend on the interlayer twist-angle. This feasible and efficient method allows for the rapid and high-quality manufacturing of atomically thin materials and devices.

Distill-SODA: Distilling Self-Supervised Vision Transformer for Source-Free Open-Set Domain Adaptation in Computational Pathology.

Developing computational pathology models is essential for reducing manual tissue typing from whole slide images, transferring knowledge from the source domain to an unlabeled, shifted target domain, and identifying unseen categories. We propose a practical setting by addressing the above-mentioned challenges in one fell swoop, i.e., source-free open-set domain adaptation. Our methodology focuses on adapting a pre-trained source model to an unlabeled target dataset and encompasses both closed-set and open-set classes. Beyond addressing the semantic shift of unknown classes, our framework also deals with a covariate shift, which manifests as variations in color appearance between source and target tissue samples. Our method hinges on distilling knowledge from a self-supervised vision transformer (ViT), drawing guidance from either robustly pre-trained transformer models or histopathology datasets, including those from the target domain. In pursuit of this, we introduce a novel style-based adversarial data augmentation, serving as hard positives for self-training a ViT, resulting in highly contextualized embeddings. Following this, we cluster semantically akin target images, with the source model offering weak pseudo-labels, albeit with uncertain confidence. To enhance this process, we present the closed-set affinity score (CSAS), aiming to correct the confidence levels of these pseudo-labels and to calculate weighted class prototypes within the contextualized embedding space. Our approach establishes itself as state-of-the-art across three public histopathological datasets for colorectal cancer assessment. Notably, our self-training method seamlessly integrates with open-set detection methods, resulting in enhanced performance in both closed-set and open-set recognition tasks.

Quantitative analysis of heavy metals in soil by X-ray fluorescence: Fusion, intelligentization and Nonstandard-sample calculation

Qualitative and quantitative analysis of heavy metal elements in soil by X-ray fluorescence (XRF) has received widespread attention and research from scholars as an important method for assessing environmental pollution. As a detection sample for multi-component systems, the study of matrix correction has always been the key to XRF data analysis for geological samples. In this study, we reviewed the calculation and analysis methods of XRF data used for geological samples since the Sherman equation was proposed, and divided the development of XRF data processing for soil samples into three stages based on the changes in the matrix correction methods used. By reviewing the processing ideas from past research, this paper summarizes the process of quantitative analysis of geological samples into seven stages and reviews the commonly used methods for each stage. Due to limitations in instrument and standard sample costs, as well as methodological constraints, geological samples currently face three challenges: a shortage of standard samples, insufficient generalization ability of established models, and large measurement errors in low-content element determination. With the further cross-penetration of multiple fields and disciplines and the summary of past research trends, we propose three research trends that may break through these limitations: fusion, intelligentization, and nonstandard-sample calculation. We also discuss the technical solutions related to these three research trends. We extensively discussed the feasibility and advantages of using spectral co-use, knowledge engineering, and adversarial data augmentation techniques to address problems. Our review provides insights into the XRF spectral data processing methods and frameworks for evaluating geological samples, and provides technical solutions to address the current challenges faced by XRF analysis of geological samples.

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition

Disentangling different levels of GAN fingerprints for task-specific forensics

Image generation using generative adversarial networks (GANs) has raised new security challenges recently. One promising forensic solution is verifying whether or not a suspicious image contains a GAN fingerprint, a unique trace left behind by the source GAN. Previous methods mainly focused on GAN fingerprint extraction while underestimating the downstream forensic applications, and the fingerprints are often single-level which only supports one specific forensic task. In this study, we investigate the problem of disentangling different levels of GAN fingerprints to satisfy the need for varying forensics tasks. Based on an analysis of fingerprint dependency revealing the existence of two levels of fingerprints in different signal domains, we proposed a decoupling representation framework to separate and extract two types of GAN fingerprints from different domains. An adversarial data augmentation strategy plus a transformation-invariant loss is added to the framework to enhance the robustness of fingerprints to image perturbations. Then we elaborated on three typical forensics tasks and the task-specific fingerprinting using different GAN fingerprints. Extensive experiments have verified our dependency analysis, the effectiveness and robustness of the proposed fingerprint extraction framework, and the applicability of task-specific fingerprinting in real-world and simulated scenarios.

Adversarial Data Augmentation for HMM-based Anomaly Detection.

In this work, we concentrate on the detection of anomalous behaviors in systems operating in the physical world and for which it is usually not possible to have a complete set of all possible anomalies in advance. We present a data augmentation and retraining approach based on adversarial learning for improving anomaly detection. In particular, we first define a method for generating adversarial examples for anomaly detectors based on Hidden Markov Models (HMMs). Then, we present a data augmentation and retraining technique that uses these adversarial examples to improve anomaly detection performance. Finally, we evaluate our adversarial data augmentation and retraining approach on four datasets showing that it achieves a statistically significant performance improvement and enhances the robustness to adversarial attacks. Key differences from the state-of-the-art on adversarial data augmentation are the focus on multivariate time series (as opposed to images), the context of one-class classification (in contrast to standard multi-class classification, and the use of HMMs (in contrast to neural networks).

A driver stress detection model via data augmentation based on deep convolutional recurrent neural network

Excessive stress generally leads to degraded driving performance, which increases the risk of road accidents. Therefore, real-time driver stress detection is of great significance to improve road safety. To this end, this paper proposes a deep convolutional recurrent neural network with Wasserstein generative adversarial network-based data augmentation (WGAN-DCRNN) for real-time driver stress detection, where only the pupillary response data are used. The proposed model can learn the sequence features from the input data more effectively and address the class-imbalance problem of the dataset. First, two types of DCRNN models (i.e., CNN-LSTM and CNN-GRU) are established and compared with baseline models (i.e., CNN, LSTM and GRU). The best baseline model (i.e., CNN) achieves 93.02% accuracy on the overall samples of the test set, while the CNN-LSTM and CNN-GRU improve the accuracy to 94.68% and 94.71%, respectively. This suggests that the performance of the CNN can be improved by integrating the RNN to consider the temporal dependencies of the pupillary response data. Second, deep generative model-based data augmentation is used to address the class-imbalance problem of the dataset. Specifically, the WGAN is trained to generate new samples, which are used to expand and balance the training set. To validate the effect of the proposed data augmentation method, comparative experiments are further conducted. The results show that the accuracies of CNN-LSTM and CNN-GRU are further improved to 95.39% and 95.30%, respectively. More importantly, the recall rates of minority categories achieve significant improvement. This indicates that the proposed model overcomes the class-imbalance problem and that the generalization ability of the recognition model has been further improved by using data augmentation. In addition, the WGAN-DCRNN outperforms models in existing driver stress detection studies. The proposed model can be used for different classification tasks to reduce road accident risks caused by drivers.

Stochastic Gradient Perturbation: An Implicit Regularizer for Person Re-Identification

Generalization of the person re-identification (ReID) model plays an important role in practical application, and we discuss a simple yet effective regularizer to improve it inspired by Adversarial Training (AT). AT has been indicated as an advanced regularizer due to its adversarial mechanism, ability to mine hard samples, and nature of data augmentation. However, serving as an augmentation-based regularizer, AT shows low diversity of the perturbation, excessive computational cost, and the optimization dilemma between adversarial robustness and accuracy for ReID task, and is thus suboptimal. To tackle these limitations and get a more effective regularizer for ReID, we rethink the nature of AT and unveil that the adversarial data augmentation is essentially reflected by gradients. Based on this, a novel implicit regularizer, named Stochastic Gradient Perturbation (SGP), is proposed, which naturally brings three merits: 1) Better diversity of the perturbation due to the proposed non-directional stochastic perturbations rather than directional adversarial perturbations. 2) Lower computational cost due to the proposed implicit gradient augmentation rather than explicitly additional data. 3) The optimization dilemma of the adversarial robustness and generalization is naturally overcome since SGP contains the adversarial gradient perturbation. Further, we put forward a perspective that the generalization and adversarial robustness may have an inter unity. Experiments on the baseline and SOTA models demonstrate powerful performances of the plugged-played SGP, and both generalization and adversarial robustness can be guaranteed.

Simulation-driven Bearing Fault Diagnosis for Condition Monitoring without Faulty Data

The failure of rolling element bearings in complex mechanical systems is a significant cause of mechanical failures, leading to decreased productivity and safety risks. Deep learning has shown promising results in bearing fault diagnosis, but the predictive performance depends on highquality data. Domain adaptation has been studied to solve this problem, but it still has limitations when applied to real-world industrial applications. In this study, we propose a deep learning-based domain generalization framework for bearing fault diagnosis using the bearing simulation model and adversarial data augmentation method. The proposed framework was validated on a real bearing fault dataset and showed promising results in improving diagnostic performance in cases where fault data cannot be obtained or when dealing with unlearned target domains. This approach has the potential to improve industrial maintenance systems by obtaining improved generalization performance in the absence of fault datasets.

AdvMask: A sparse adversarial attack-based data augmentation method for image classification

Data augmentation has been an essential technique for improving the generalization ability of deep neural networks in image classification tasks. However, intensive changes in appearance and different degrees of occlusion in images are the key factors that severely affect the generalization ability of image classification models. Therefore, in order to enhance the generalization performance and robustness of deep models, data augmentation approaches by providing models with more diverse training data in various scenarios are widely applied. Although many existing data augmentation methods simulate occlusion in the augmented images to enhance the generalization of models, these methods randomly delete some areas in images without considering the semantic information of images. In this work, we propose a novel data augmentation method named AdvMask for image classification based on sparse adversarial attack techniques. AdvMask first identifies the key points that have the greatest influence on the classification results via a proposed end-to-end sparse adversarial attack module. During the data augmentation process, AdvMask efficiently generates diverse augmented data with structured occlusions based on the key points. By doing so, AdvMask can force deep models to seek other relevant content while the most discriminative content is hidden. Extensive experimental results on various benchmark datasets and deep models demonstrate that our proposed method can effectively improve the generalization performance of deep models and significantly outperforms previous data augmentation methods. Code for reproducing our results is available at https://github.com/Jackbrocp/AdvMask.

Illumination Adaptive Multi-Scale Water Surface Object Detection with Intrinsic Decomposition Augmentation

Visual object detection is an essential task for the intelligent navigation of an Unmanned Surface Vehicle (USV), which can sense the obstacles while navigating. However, the harsh illumination conditions and large scale variation of the objects significantly harm the performance of object detection methods. To address the above problems, we propose a robust water surface object detection method named multi-scale feature fusion network with intrinsic decomposition generative adversarial network data augmentation (MFFDet-IDGAN). We introduce intrinsic decomposition as data augmentation for the object detection to achieve illumination adapting. And an intrinsic decomposition generative adversarial network (IDGAN) is proposed to achieve unsupervised intrinsic decomposition. Moreover, the multi-scale feature fusion network (MFFDet) adopts an improved bidirectional feature pyramid network (BiFPN) and spatial pyramid pooling (SPP) blocks to fuse features of different resolution for better multi-scale detection. And an improved weighted stochastic weight averaging (SWA) is proposed and applied in the training process to improve the generalization performance. We conduct extensive experiments on the Water Surface Object Detection Dataset (WSODD), and the results show that the proposed method can achieve 44% improvement over the baseline. And we further test our method on a real USV in the sailing process, the results show that our method can exceeding the baseline by 4.5%.

Deepfake Detection Algorithm Based on Dual-Branch Data Augmentation and Modified Attention Mechanism

Mainstream deepfake detection algorithms generally fail to fully extract forgery traces and have low accuracy when detecting forged images with natural corruptions or human damage. On this basis, a new algorithm based on an adversarial dual-branch data augmentation framework and a modified attention mechanism is proposed in this paper to improve the robustness of detection models. First, this paper combines the traditional random sampling augmentation method with the adversarial sample idea to enhance and expand the forged images in data preprocessing. Then, we obtain training samples with diversity and hardness uniformity. Meanwhile, a new attention mechanism is modified and added to the ResNet50 model. The improved model serves as the backbone, effectively increasing the weight of forged traces in the multi-scale feature maps. The Jensen–Shannon divergence loss and cosine annealing algorithms are introduced into the training process to improve the model’s accuracy and convergence speed. The proposed algorithm is validated on standard and corrupted datasets. The experiments show that the algorithm proposed in this paper significantly improves effectiveness and robustness, with accuracies of 4.16%, 7.37%, and 3.87% better than the baseline model on DeepFakes, FaceSwap, and FaceShifer, respectively. Most importantly, its detection performance on the corrupted datasets DeepFakes-C, FaceSwap-C, and FaceShifer-C is much higher than that of mainstream methods.

Self-Attention-Based Convolutional GRU for Enhancement of Adversarial Speech Examples

Recent research has identified adversarial examples which are the challenges to DNN-based ASR systems. In this paper, we propose a new model based on Convolutional GRU and Self-attention U-Net called [Formula: see text] to improve adversarial speech signals. To represent the correlation between neighboring noisy speech frames, a two-Layer GRU is added in the bottleneck of U-Net and an attention gate is inserted in up-sampling units to increase the adversarial stability. The goal of using GRU is to combine the weights sharing technique with the use of gates to control the flow of data across multiple feature maps. As a result, it outperforms the original 1D convolution used in [Formula: see text]. Especially, the performance of the model is evaluated by explainable speech recognition metrics and its performance is analyzed by the improved adversarial training. We used adversarial audio attacks to perform experiments on automatic speech recognition (ASR). We saw (i) the robustness of ASR models which are based on DNN can be improved using the temporal features grasped by the attention-based GRU network; (ii) through adversarial training, including some additive adversarial data augmentation, we could improve the generalization power of Automatic Speech Recognition models which are based on DNN. The word-error-rate (WER) metric confirmed that the enhancement capabilities are better than the state-of-the-art [Formula: see text]. The reason for this enhancement is the ability of GRU units to extract global information within the feature maps. Based on the conducted experiments, the proposed [Formula: see text] increases the score of Speech Transmission Index (STI), Perceptual Evaluation of Speech Quality (PESQ), and the Short-term Objective Intelligibility (STOI) with adversarial speech examples in speech enhancement.