Generative Adversarial Training Research Articles

Enhancing Image Captioning Using Deep Convolutional Generative Adversarial Networks

Introduction:: Introduction: Image caption generation has long been a fundamental challenge in the area of computer vision (CV) and natural language processing (NLP). In this research, we present an innovative approach that harnesses the power of Deep Convolutional Generative Adversarial Networks (DCGAN) and adversarial training to revolutionize the generation of natural and contextually relevant image captions. Method:: Our method significantly improves the fluency, coherence, and contextual relevance of generated captions and showcases the effectiveness of RL reward-based fine-tuning. Through a comprehensive evaluation of COCO datasets, our model demonstrates superior performance over baseline and state-of-the-art methods. On the COCO dataset, our model outperforms current state-of-the-art (SOTA) models across all metrics, achieving BLEU-4 (0.327), METEOR (0.249), Rough (0.525) and CIDEr (1.155) scores. Result:: The integration of DCGAN and adversarial training opens new possibilities in image captioning, with applications spanning from automated content generation to enhanced accessibility solutions. Conclusion:: This research paves the way for more intelligent and context-aware image understanding systems, promising exciting future exploration and innovation prospects.

Evaluating Transferability of Attacks across Generative Models

The need for adversarial sample transferability is to attack black-box deep learning models. Whereas much recent work focuses on making untargeted adversarial attacks more transferable, there has been scarce research on the creation of transferable targeted adversarial instances that can trick models into believing they are of a particular class. The present transferable targeted adversarial attacks are not transferable since they cannot sufficiently define the distribution of target classes. In this paper, we propose a generative adversarial training system consisting of a feature-label dual discriminator to identify the adversarial instances formed from the target class images and a generator to construct targeted adversarial examples. It is concluded that adversarial scenarios have significant real-world applications in safety-critical fields like biometrics and autonomous driving. In addition, it is demonstrated that the current networks' susceptibility to hostile attacks, even under the worst black-box conditions has far-reaching societal consequences. We intend to further encourage more research into the inner workings of neural networks in the face of adversarial attacks, whereby people might use this knowledge to build robust defense mechanisms.

GAN-based self-supervised message passing graph representation learning

Graph representation learning aims to learn embedding representations of nodes in graph-structured data through supervised or unsupervised methods. However, for unlabeled or sparsely labeled graph datasets, conventional supervised approaches fail to perform adequately, and existing unsupervised methods have difficulty achieving high accuracy on multiple downstream tasks such as node classification and link prediction at lower embedding dimensions. To address this issue, we propose a novel self-supervised graph representation learning approach called GAN-based Message Passing Graph Representation Learning. This approach significantly improves the accuracy of node embeddings without relying on label information. Specifically, GMP-GL has developed a generator based on neighbor-weighted random walk to obtain positive and negative node sample pairs, as well as a discriminator that distinguishes whether a real connection exists between node pairs. Through adversarial learning, GMP-GL automatically learns node embeddings. Furthermore, GMP-GL optimizes the training outcomes of the generative adversarial model through self-supervised embedding aggregation learning, thus introducing rich structural information from the graph for the subsequent rounds of generative adversarial training. In addition, GMP-GL employs multiple autoencoders to generate and filter original embeddings, ensuring that the pre-embeddings preserve a significant amount of original feature information for formal learning. The study selects six real-world datasets and conducts comparative experiments with ten SOTA baseline methods. The experimental results demonstrate that the proposed model not only surpasses previous unsupervised methods but also matches or even exceeds the performance of supervised methods. This indicates the great potential of the proposed approach for representation learning tasks on unlabeled graph datasets and its effective applicability to real-world problems.

Positive Effect of Super-Resolved Structural Magnetic Resonance Imaging for Mild Cognitive Impairment Detection.

This paper presents a novel approach to improving the detection of mild cognitive impairment (MCI) through the use of super-resolved structural magnetic resonance imaging (MRI) and optimized deep learning models. The study introduces enhancements to the perceptual quality of super-resolved 2D structural MRI images using advanced loss functions, modifications to the upscaler part of the generator, and experiments with various discriminators within a generative adversarial training setting. It empirically demonstrates the effectiveness of super-resolution in the MCI detection task, showcasing performance improvements across different state-of-the-art classification models. The paper also addresses the challenge of accurately capturing perceptual image quality, particularly when images contain checkerboard artifacts, and proposes a methodology that incorporates hyperparameter optimization through a Pareto optimal Markov blanket (POMB). This approach systematically explores the hyperparameter space, focusing on reducing overfitting and enhancing model generalizability. The research findings contribute to the field by demonstrating that super-resolution can significantly improve the quality of MRI images for MCI detection, highlighting the importance of choosing an adequate discriminator and the potential of super-resolution as a preprocessing step to boost classification model performance.

Spatiotemporal gated traffic trajectory simulation with semantic-aware graph learning

Traffic trajectories of various vehicles, bicycles and pedestrians can help understand the traffic dynamics in a fine-grained manner like traffic flow, traffic congestion and ride-hailing demand. The comprehensive usage of traffic trajectory data has not been fully investigated due to the prevalent privacy concerns and commercial limitations. The traffic trajectory simulation task has emerged to generate high-fidelity trajectories in demand for downstream tasks to fill the gap between the scarce trajectory data and the widespread applications. Previous state-of-the-art methods build the spatiotemporal dependencies of trajectories with Graph Neural Networks (GNNs) under generative adversarial training, yielding better yet unstable trajectory quality. We observe that the unsatisfied synthetic trajectories are caused by the insufficient spatiotemporal modeling of road networks and trajectory semantics. In this paper, we propose a novel SpatioTEmporal GAted (STEGA) framework equipped with semantic-aware graph learning for traffic trajectory simulation to enable the explicit modeling of spatiotemporal dependencies throughout the learning pipeline. On the one hand, STEGA employs a graph encoder with the semantics of road networks for the spatial points of a trajectory, together with a time encoder for the time points. On the other hand, STEGA devises two spatiotemporal gates with the semantic graphs for the predictions of the future trajectory. Boosted by the semantic-aware graph learning, the proposed STEGA outperforms the state-of-the-art counterparts consistently at both macro- and micro-level metrics on two datasets. Elaborate ablation studies and downstream tasks of the synthetic trajectories further demonstrate the superiority of STEGA. Our code is available at https://github.com/Star607/STEGA.

The Application of Radiomics and AI to Molecular Imaging for Prostate Cancer.

Molecular imaging is a key tool in the diagnosis and treatment of prostate cancer (PCa). Magnetic Resonance (MR) plays a major role in this respect with nuclear medicine imaging, particularly, Prostate-Specific Membrane Antigen-based, (PSMA-based) positron emission tomography with computed tomography (PET/CT) also playing a major role of rapidly increasing importance. Another key technology finding growing application across medicine and specifically in molecular imaging is the use of machine learning (ML) and artificial intelligence (AI). Several authoritative reviews are available of the role of MR-based molecular imaging with a sparsity of reviews of the role of PET/CT. This review will focus on the use of AI for molecular imaging for PCa. It will aim to achieve two goals: firstly, to give the reader an introduction to the AI technologies available, and secondly, to provide an overview of AI applied to PET/CT in PCa. The clinical applications include diagnosis, staging, target volume definition for treatment planning, outcome prediction and outcome monitoring. ML and AL techniques discussed include radiomics, convolutional neural networks (CNN), generative adversarial networks (GAN) and training methods: supervised, unsupervised and semi-supervised learning.

Attention Mechanism-based Anti-Jamming Detector with Generative Adversarial Training

Attention Mechanism-based Anti-Jamming Detector with Generative Adversarial Training

MERF: A Practical HDR-Like Image Generator via Mutual-Guided Learning Between Multi-Exposure Registration and Fusion.

In this paper, we present a novel high dynamic range (HDR)-like image generator that utilizes mutual-guided learning between multi-exposure registration and fusion, leading to promising dynamic multi-exposure image fusion. The method consists of three main components: the registration network, the fusion network, and the dual attention network which seamlessly integrates registration and fusion processes. Initially, within the registration network, the estimation of deformation fields among multi-exposure image sequences is conducted following an exposure-invariant feature extraction phase. This leads to enhanced accuracy by mitigating discrepancies across domains. Subsequently, the fusion network utilizes a progressive frequency fusion module in two distinct stages, addressing color correction and detail preservation within low and high-frequency domains, respectively. To facilitate the mutual enhancement of the registration and fusion networks, we undertake a mutual-guided learning strategy encompassing their physical connection and constraint paradigm. Firstly, a dual attention network bridges the registration and fusion networks, addressing ghosting, which is beyond the scope of registration and facilitates information exchange between input images. Secondly, a meticulously designed generative adversarial network-like iterative training schema guides the overall network framework, thereby yielding high-quality HDR-like images through mutual enhancement. Comprehensive experiments on publicly available datasets validate the superiority of our method over existing state-of-the-art approaches.

Introducing SPINE: A Holistic Approach to Synthetic Pulmonary Imaging Evaluation Through End-to-End Data and Model Management.

In the evolving field of medical imaging and machine learning (ML), this paper introduces a novel framework for evaluating synthetic pulmonary imaging aiming to assess synthetic data quality and applicability. Our study concentrates on synthetic X-ray chest images, crucial for diagnosing respiratory diseases. We employ SPINE (Synthetic Pulmonary Imaging Evaluation) framework, a threefold synthetic images evaluation method including expert domain assessment, statistical data analysis and adversarial evaluation. In order to replicate and validate our methodology, we followed an End-to-End data and model management process which begins with a dataset of Normal and Pneumonia chest X-rays, generating synthetic images using Generative Adversarial Networks (GANs) and training a baseline classifier, essential in the adversarial evaluation axis, testing synthetic images against real data assessing their predictive value. The critical outcome of our approach is the post-market analysis of synthetic images. This innovative method evaluates synthetic images using clinical, statistical, and scientific criteria independently from traditional generation performance metrics. This independent evaluation provides deep insights into the clinical and research effectiveness of the synthetic data. By ensuring these images mirror real data's statistical properties and maintain clinical accuracy, our framework establishes a new standard for the ethical and reliable use of synthetic data in medical imaging and research.

Leveraging Activation Maximization and Generative Adversarial Training to Recognize and Explain Patterns in Natural Areas in Satellite Imagery

Leveraging Activation Maximization and Generative Adversarial Training to Recognize and Explain Patterns in Natural Areas in Satellite Imagery

Measures for the Integration of Digital Media Art and Film and Animation Creation Based on Virtual Reality Technology

Abstract In this paper, we propose a scene animation network (FSGAN) method that uses spectral features to perceive the deep style of an image, which uses three loss functions for perceiving the distribution of image style features in the same style domain to generate anime style-consistent images. The recurrent consistency adversarial network is supplemented with a local reinforcement module based on self-attention, adapted to the animalization of portrait scenes based on their characteristics and needs. This module guides the network to pay attention to the salient regions in the image and combines layer normalization and instance normalization methods through the weighting of the attention weight coefficients to construct a portrait animation network based on local reinforcement perception. Then, based on the digital media technology to complete the virtual reality scene construction, the constructed space for three-dimensional division, and the use of generative adversarial network training model, the proposed key events and random sampling technology combined with the control of virtual animation scene generation method, to achieve the controllability and diversity of the virtual animation scene generation. The first-stage design efficiency, the second-stage design efficiency and the overall design efficiency of the animation creation method in this paper are all basically stabilized between 0.8-1. Compared with the traditional method, they are higher by 0.3932, 0.3596, and 0.5635, respectively. The extreme pixel percentage, standard deviation and contrast of the animation images created by this paper’s method are 0.0349, 0.0037 and 0.3382, respectively, and the overall color performance is better. Thus, the method presented in this paper has the best overall effect on processing anime images using digital media art effects.

ScDREAMER for atlas-level integration of single-cell datasets using deep generative model paired with adversarial classifier

Integration of heterogeneous single-cell sequencing datasets generated across multiple tissue locations, time, and conditions is essential for a comprehensive understanding of the cellular states and expression programs underlying complex biological systems. Here, we present scDREAMER (https://github.com/Zafar-Lab/scDREAMER), a data-integration framework that employs deep generative models and adversarial training for both unsupervised and supervised (scDREAMER-Sup) integration of multiple batches. Using six real benchmarking datasets, we demonstrate that scDREAMER can overcome critical challenges including skewed cell type distribution among batches, nested batch-effects, large number of batches and conservation of development trajectory across batches. Our experiments also show that scDREAMER and scDREAMER-Sup outperform state-of-the-art unsupervised and supervised integration methods respectively in batch-correction and conservation of biological variation. Using a 1 million cells dataset, we demonstrate that scDREAMER is scalable and can perform atlas-level cross-species (e.g., human and mouse) integration while being faster than other deep-learning-based methods.

Foggy image restoration using deep sub‐pixel reconstruction network

AbstractLight undergoes attenuation due to scattering and refraction when propagating through aerosols. In foggy conditions, Aerosol particles in the troposphere exhibit high mobility introducing intricate non‐linear noise into images. Foggy image restoration represents an ill‐posed problem, where traditional physical models and image enhancement techniques often prove inadequate in delivering effective solutions. This paper introduces a novel deep sub‐pixel reconstruction algorithm for foggy image restoration, pioneering the application of sub‐pixel reconstruction modules to this domain. This model employs convolutional layers to extract low‐level features and dense‐connected layers for high‐level feature extraction. Furthermore, a specialized sub‐pixel reconstruction module tailored for the task of foggy image restoration is designed, with the purpose of reconstructing dehazed images from latent vectors. During training, a generative adversarial training framework is adopted, incorporating a purpose‐designed discriminator. Additionally, a fusion loss is implemented to facilitate model refinement. Quantitative and qualitative evaluation experiments conducted on synthetic and real‐world image datasets demonstrate the effectiveness of the proposed method in preserving finer details. The Structural Similarity Index (SSIM) is observed to improve by 2.5%, attesting to enhanced perceptual quality for grayscale foggy images.

Stochastic differential equation approximations of generative adversarial network training and its long-run behavior

AbstractThis paper analyzes the training process of generative adversarial networks (GANs) via stochastic differential equations (SDEs). It first establishes SDE approximations for the training of GANs under stochastic gradient algorithms, with precise error bound analysis. It then describes the long-run behavior of GAN training via the invariant measures of its SDE approximations under proper conditions. This work builds a theoretical foundation for GAN training and provides analytical tools to study its evolution and stability.

Open-set federated adversarial domain adaptation based cross-domain fault diagnosis

Abstract Data-driven fault diagnosis techniques utilizing deep learning have achieved widespread success. However, their diagnostic capability and application possibility are significantly reduced in real-world scenarios where fault modes are not fully covered and labels are lacking. Owing to potential conflicts of interest and legal risks, industrial equipment fault data usually exist in the form of isolated islands, making it difficult to carry out large-scale centralized model training. This paper proposes open-set federated adversarial domain adaptation (OS-FADA) to achieve collaborative evolution of fault diagnosis capabilities among cross-domain data owners while protecting privacy. The OS-FADA is a general fault diagnosis framework that employs two-phase adversarial learning. First, faced with the data distribution shift caused by variable working conditions, a generative adversarial feature extractor training strategy is designed to achieve domain-invariant fault feature extraction by approximating the feature distributions of clients to a unified generated distribution. Second, considering the label distribution shift of unknown faults occurring in the target client, an adversarial learning method is proposed to establish decision boundaries between known and unknown faults. Ultimately, the co-evolution of fault diagnosis models between clients is achieved by combining two-phase adversarial learning and federated aggregation. Results from an industrial gearbox case demonstrate that our proposed method achieves over 20% diagnostic accuracy improvement and has excellent potential for cross-domain fault diagnosis tasks with unknown faults when the data silos problem cannot be ignored.

Enhancing Spatial Variability Representation of Radar Nowcasting with Generative Adversarial Networks

Weather radar plays an important role in accurate weather monitoring and modern weather forecasting, as it can provide timely and refined weather forecasts for the public and for decision makers. Deep learning has been applied in radar nowcasting tasks and has exhibited a better performance than traditional radar echo extrapolation methods. However, current deep learning-based radar nowcasting models are found to suffer from a spatial “blurry” effect that can be attributed to a deficiency in spatial variability representation. This study proposes a Spatial Variability Representation Enhancement (SVRE) loss function and an effective nowcasting model, named the Attentional Generative Adversarial Network (AGAN), to alleviate this blurry effect by enhancing the spatial variability representation of radar nowcasting. An ablation experiment and a comparison experiment were implemented to assess the effect of the generative adversarial (GA) training strategy and the SVRE loss, as well as to compare the performance of the AGAN and SVRE loss function with the current advanced radar nowcasting models. The performances of the models were validated on the whole test set and inspected in two storm cases. The results showed that both the GA strategy and SVRE loss function could alleviate the blurry effect by enhancing the spatial variability representation, which helps the AGAN to achieve better nowcasting performance than the other competitor models. Our study provides a feasible solution for high-precision radar nowcasting applications.

A cGAN-based tumor segmentation method for breast ultrasound images

Objective. This paper proposes a conditional GAN (cGAN)-based method to perform data enhancement of ultrasound images and segmentation of tumors in breast ultrasound images, which improves the reality of the enhenced breast ultrasound image and obtains a more accurate segmentation result. Approach. We use the idea of generative adversarial training to accomplish the following two tasks: (1) in this paper, we use generative adversarial networks to generate a batch of samples with labels from the perspective of label-generated images to expand the dataset from a data enhancement perspective. (2) In this paper, we use adversarial training instead of postprocessing steps such as conditional random fields to enhance higher-level spatial consistency. In addition, this work proposes a new network, EfficientUNet, based on U-Net, which combines ResNet18, an attention mechanism and a deep supervision technique. This segmentation model uses the residual network as an encoder to retain the lost information in the original encoder and can avoid the gradient disappearance problem to improve the feature extraction ability of the model, and it also uses deep supervision techniques to speed up the convergence of the model. The channel-by-channel weighting module of SENet is then used to enable the model to capture the tumor boundary more accurately. Main results. The paper concludes with experiments to verify the validity of these efforts by comparing them with mainstream methods on Dataset B. The Dice score and IoU score reaches 0.8856 and 0.8111, respectively. Significance. This study successfully combines cGAN and optimized EfficientUNet for the segmentation of breast tumor ultrasound images. The conditional generative adversarial network has a good performance in data enhancement, and the optimized EfficientUNet makes the segmentation more accurate.

Learning domain-invariant representation for generalizing face forgery detection

With the improvement of face forgery techniques, people can easily generate various forgery face images, which brings serious challenges to public confidence. Numerous face forgery detection methods have been proposed, few of them can gain satisfying performance when training and testing forgery face images are generated by different forgery methods. To address this problem, we propose a domain-invariant representation learning (DIRL) method, which include feature distribution discrepancy minimization (FDDM) and optimal classier distance minimization (OCDM). FDDM adopts the idea of joint disentanglement and generative adversarial training to separate the irrelevant information and constrains the generator to reduce the discrepancy in feature distribution between source and target domains to obtain domain-shared features. OCDM uses the meta-learning method obtain a domain-invariant representation by minimizing the optimal classier distance between the domain-shared features of source and target domains. With FDDM and OCDM, we can learn an effective domain-invariant representation space in which there exists an ideal classifier equivalent to the optimal source domain classifier and the optimal target domain classifier, and the minimum classifier error can be achieved in both the source and target domains. Ablation experiments on the FaceForensics++ dataset show that compared to the baseline, our method can bring a decrease of 3.56% to 8.56% on Half Total Error Rate (HTER) when the test and training sets are generated by different face forgery methods. And the performance comparison experiments with the state-of-the-art methods also show that our method outperforms other methods.

Synthesizing Audio from Tongue Motion During Speech Using Tagged MRI Via Transformer.

Investigating the relationship between internal tissue point motion of the tongue and oropharyngeal muscle deformation measured from tagged MRI and intelligible speech can aid in advancing speech motor control theories and developing novel treatment methods for speech related-disorders. However, elucidating the relationship between these two sources of information is challenging, due in part to the disparity in data structure between spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio waveforms. In this work, we present an efficient encoder-decoder translation network for exploring the predictive information inherent in 4D motion fields via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder is based on 3D convolutional spatial modeling and transformer-based temporal modeling. The extracted features are processed by an asymmetric 2D convolution decoder to generate spectrograms that correspond to 4D motion fields. Furthermore, we incorporate a generative adversarial training approach into our framework to further improve synthesis quality on our generated spectrograms. We experiment on 63 paired motion field sequences and speech waveforms, demonstrating that our framework enables the generation of clear audio waveforms from a sequence of motion fields. Thus, our framework has the potential to improve our understanding of the relationship between these two modalities and inform the development of treatments for speech disorders.

GAT–GMM: Generative Adversarial Training for Gaussian Mixture Models

Generative adversarial networks (GANs) learn the distribution of observed samples through a zero-sum game between two machine players, a generator and a discriminator. While GANs achieve great success in learning the complex distribution of image, sound, and text data, they perform suboptimally in learning multimodal distribution-learning benchmarks such as Gaussian mixture models (GMMs). In this paper, we propose Generative Adversarial Training for Gaussian Mixture Models (GAT-GMM), a minimax GAN framework for learning GMMs. Motivated by optimal transport theory, we design the zero-sum game in GAT-GMM using a random linear generator and a softmax-based quadratic discriminator architecture, which leads to a nonconvex concave minimax optimization problem. We show that a gradient descent ascent (GDA) method converges to an approximate stationary minimax point of the GAT-GMM optimization problem. In the benchmark case of a mixture of two symmetric, well-separated Gaussians, we further show that this stationary point recovers the true parameters of the underlying GMM. We discuss the application of the proposed GAT-GMM framework for learning GMMs in the distributed federated learning setting, where the widely used expectation-maximization (EM) algorithm can incur great computational and communication costs. On the other hand, we show that GAT-GMM provides a scalable learning approach and a distributed GDA algorithm can still solve the GAT-GMM minimax problem without incurring extra computation costs. We numerically support our theoretical results by performing experiments which show that our minimax framework is successful in centralized learning tasks and can outperform standard EM-type algorithms in the federated setting.