Unified Generative Adversarial Networks Research Articles

A Unified Generative Adversarial Network With Convolution and Transformer for Remote Sensing Image Fusion

A Unified Generative Adversarial Network With Convolution and Transformer for Remote Sensing Image Fusion

UniQGAN: Unified Generative Adversarial Networks for Augmented Modulation Classification

Deep learning has been widely applied to automatic modulation classification (AMC), and there have been many studies on data augmentation techniques using deep generative models to improve performance. However, existing solutions need to train different models independently for each SNR, which leads to undeniable overhead. This letter presents <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">UniQGAN</i> , Unified Generative Adversarial Networks for IQ constellations of various SNRs, requiring a single model training. The proposed method introduces <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-conditions embedding</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-domains classification</i> to leverage both conditions, i.e., modulation type and SNR. Experimental results show that UniQGAN effectively improves the AMC performance, while the training time is reduced.

SRDN: A Unified Super-Resolution and Motion Deblurring Network for Space Image Restoration

Space target super-resolution (SR) is a domain-specific single image SR problem aiming to help distinguish the satellite and spacecrafts from numerous space debris. Compared to the other object SR problem, images for space target are always in low quality with varies of degradation condition, as a result of long distance and motion blur, which significantly reduces the manual classification reliability, especially for these small targets, e.g., satellite payloads. To address this challenge, we present an end-to-end SR and deblurring network (SRDN). Concretely, focusing on the low-resolution (LR) space target images with blind motion blur, we integrate the SR and deblur function together, improving the image quality by a unified generative adversarial network (GAN)-based framework. We implement a deblur module by using contrastive learning to extract degradation feature and add symmetrical downsampling and upsampling modules to the SR network in order to restore texture information, while shortcut connections are redesigned to maintain the global similarity. Extensive experiments on the public satellite dataset, BUAA-SID-share1.5, demonstrate that our network outperforms the state-of-the-art SR and deblur methods.

Image Generation with a Unified Generative Adversarial Network Training via Self-Labeling and Self-Attention

Image Generation with a Unified Generative Adversarial Network Training via Self-Labeling and Self-Attention

Unified Generative Adversarial Networks for Multidomain Fingerprint Presentation Attack Detection.

With the rapid growth of fingerprint-based biometric systems, it is essential to ensure the security and reliability of the deployed algorithms. Indeed, the security vulnerability of these systems has been widely recognized. Thus, it is critical to enhance the generalization ability of fingerprint presentation attack detection (PAD) cross-sensor and cross-material settings. In this work, we propose a novel solution for addressing the case of a single source domain (sensor) with large labeled real/fake fingerprint images and multiple target domains (sensors) with only few real images obtained from different sensors. Our aim is to build a model that leverages the limited sample issues in all target domains by transferring knowledge from the source domain. To this end, we train a unified generative adversarial network (UGAN) for multidomain conversion to learn several mappings between all domains. This allows us to generate additional synthetic images for the target domains from the source domain to reduce the distribution shift between fingerprint representations. Then, we train a scale compound network (EfficientNetV2) coupled with multiple head classifiers (one classifier for each domain) using the source domain and the translated images. The outputs of these classifiers are then aggregated using an additional fusion layer with learnable weights. In the experiments, we validate the proposed methodology on the public LivDet2015 dataset. The experimental results show that the proposed method improves the average classification accuracy over twelve classification scenarios from 67.80 to 80.44% after adaptation.

Multimodal MRI synthesis using unified generative adversarial networks.

Complementary information obtained from multiple contrasts of tissue facilitates physicians assessing, diagnosing and planning treatment of a variety of diseases. However, acquiring multiple contrasts magnetic resonance images (MRI) for every patient using multiple pulse sequences is time-consuming and expensive, where, medical image synthesis has been demonstrated as an effective alternative. The purpose of this study is to develop a unified framework for multimodal MR image synthesis. A unified generative adversarial network consisting of only a single generator and a single discriminator was developed to learn the mappings among images of four different modalities. The generator took an image and its modality label as inputs and learned to synthesize the image in the target modality, while the discriminator was trained to distinguish between real and synthesized images and classify them to their corresponding modalities. The network was trained and tested using multimodal brain MRI consisting of four different contrasts which are T1-weighted (T1), T1-weighted and contrast-enhanced (T1c), T2-weighted (T2), and fluid-attenuated inversion recovery (Flair). Quantitative assessments of our proposed method were made through computing normalized mean absolute error (NMAE), peak signal-to-noise ratio (PSNR), structural similarity index measurement (SSIM), visual information fidelity (VIF), and naturalness image quality evaluator (NIQE). The proposed model was trained and tested on a cohort of 274 glioma patients with well-aligned multi-types of MRI scans. After the model was trained, tests were conducted by using each of T1, T1c, T2, Flair as a single input modality to generate its respective rest modalities. Our proposed method shows high accuracy and robustness for image synthesis with arbitrary MRI modality that is available in the database as input. For example, with T1 as input modality, the NMAEs for the generated T1c, T2, Flair respectively are 0.034±0.005, 0.041±0.006, and 0.041±0.006, the PSNRs respectively are 32.353±2.525dB, 30.016±2.577dB, and 29.091±2.795dB, the SSIMs are 0.974±0.059, 0.969±0.059, and 0.959±0.059, the VIF are 0.750±0.087, 0.706±0.097, and 0.654±0.062, and NIQE are 1.396±0.401, 1.511±0.460, and 1.259±0.358, respectively. We proposed a novel multimodal MR image synthesis method based on a unified generative adversarial network. The network takes an image and its modality label as inputs and synthesizes multimodal images in a single forward pass. The results demonstrate that the proposed method is able to accurately synthesize multimodal MR images from a single MR image.

Unified generative adversarial networks for multimodal segmentation from unpaired 3D medical images.

To fully define the target objects of interest in clinical diagnosis, many deep convolution neural networks (CNNs) use multimodal paired registered images as inputs for segmentation tasks. However, these paired images are difficult to obtain in some cases. Furthermore, the CNNs trained on one specific modality may fail on others for images acquired with different imaging protocols and scanners. Therefore, developing a unified model that can segment the target objects from unpaired multiple modalities is significant for many clinical applications. In this work, we propose a 3D unified generative adversarial network, which unifies the any-to-any modality translation and multimodal segmentation in a single network. Since the anatomical structure is preserved during modality translation, the auxiliary translation task is used to extract the modality-invariant features and generate the additional training data implicitly. To fully utilize the segmentation-related features, we add a cross-task skip connection with feature recalibration from the translation decoder to the segmentation decoder. Experiments on abdominal organ segmentation and brain tumor segmentation indicate that our method outperforms the existing unified methods.

Unified Generative Adversarial Networks for Multiple-Choice Oriented Machine Comprehension

In this article, we address the multiple-choice machine comprehension (MC) problem in natural language processing. Existing approaches for MC are usually designed for general cases; however, we specially develop a novel method for solving the multiple-choice MC problem. We take the inspiration generative adversarial networks (GANs) and first propose an adversarial framework for multiple-choice oriented MC, named McGAN . Specifically, our approach is designed as a GAN-based method that unifies both generative and discriminative MC models. Working together, the generative model focuses on predicting relevant answer given a passage (text) and a question; the discriminative model focuses on predicting their relevancy given an answer-passage-question set. Based on the competition via adversarial training in a minimize-maximize game, the proposed method takes advantages from both models. To evaluate the performance, we test our McGAN model on three well-known datasets for multiple-choice MC. Our results show that McGAN can achieve a significant increase in accuracy compared to existing models based on all three datasets, and it consistently outperforms all tested baselines, including state-of-the-art techniques.

Unified Generative Adversarial Networks for Controllable Image-to-Image Translation.

We propose a unified Generative Adversarial Network (GAN) for controllable image-to-image translation, i.e., transferring an image from a source to a target domain guided by controllable structures. In addition to conditioning on a reference image, we show how the model can generate images conditioned on controllable structures, e.g., class labels, object keypoints, human skeletons, and scene semantic maps. The proposed model consists of a single generator and a discriminator taking a conditional image and the target controllable structure as input. In this way, the conditional image can provide appearance information and the controllable structure can provide the structure information for generating the target result. Moreover, our model learns the image-to-image mapping through three novel losses, i.e., color loss, controllable structure guided cycle-consistency loss, and controllable structure guided self-content preserving loss. Also, we present the Fr´echet ResNet Distance (FRD) to evaluate the quality of the generated images. Experiments on two challenging image translation tasks, i.e., hand gesture-to-gesture translation and cross-view image translation, show that our model generates convincing results, and significantly outperforms other state-of-the-art methods on both tasks. Meanwhile, the proposed framework is a unified solution, thus it can be applied to solving other controllable structure guided image translation tasks such as landmark guided facial expression translation and keypoint guided person image generation. To the best of our knowledge, we are the first to make one GAN framework work on all such controllable structure guided image translation tasks. Code is available at https://github.com/Ha0Tang/GestureGAN.

UGAN: Unified Generative Adversarial Networks for Multidirectional Text Style Transfer

Recently, text style transfer has become a very hot research topic in the field of natural language processing. However, the conventional text style transfer is unidirectional, and it is not possible to obtain a model with multidirectional transformations through training once. To address this limitation, we propose a new task called multidirectional text style transfer. It aims to use a single model to transfer the underlying style of text among multiple style attributes and keep its main content unchanged. In this paper, we propose Unified Generative Adversarial Networks (UGAN), a practical approach that combines target vector and generative adversarial techniques to perform multidirectional text style transfer. Our model allows simultaneous training of multi-attribute data on a single network. Such unified structure makes our model more efficient and flexible than existing approaches. We demonstrate the superiority of our approach on three benchmark datasets. Experimental results show that our method not only outperforms other baselines, but also reduces training time by an average of 13%.

InjectionGAN: Unified Generative Adversarial Networks for Arbitrary Image Attribute Editing

Existing image-to-image translation methods usually incorporate encoder-decoder and generative adversarial networks to generate images. The encoder compresses an entire image into a static representation using a sequence of convolution layers until a bottleneck, and then, the intermediate features are decoded to the target image. However, the existence of bottleneck layer in those approaches still has limitations in the sharpness of details, distinct image translation and identity preservation, since different domain translations may be related to the global or local region in the input image. To address these issues, we propose a new model, InjectionGAN, based on a novel generative adversarial network (GAN) for arbitrary attribute transfer. Specifically, conditional on the target domain label, an auto-encoder-like network with multiple linear transformation and refinement connections are trained to translate the input image into the target domain. The connections block better shuttle the low-level information in the encoder to the decoder, which helps to preserve the structural information while modify the appearance slightly at the pixel level through adversarial training. The results on two popular datasets suggest that InjectionGAN achieves a better performance.

Visual Haze Removal by a Unified Generative Adversarial Network

Existence of haze significantly degrades visual quality and hence negatively affects the performance of visual surveillance, video analysis, and human–machine interaction. To remove haze from a visual signal, in this paper, we propose a generative adversarial network for visual haze removal called HRGAN. HRGAN consists of a generator network and a discriminator network. A unified network jointly estimating transmission maps, atmospheric light, and haze-free images (called UNTA) is proposed as the generator network of HRGAN. Instead of being optimized by minimizing the pixel-wise loss, HRGAN is optimized by minimizing a novel loss function consisting of pixel-wise loss, perceptual loss, and adversarial loss produced by a discriminator network. Classical model-based image dehazing algorithms consist of three separate stages: 1) estimating transmission map; 2) estimating atmospheric light; and 3) restoring haze-free image by using an atmospheric scattering model to process the transmission map and atmospheric light. Such a separate scheme is not guaranteed to achieve optimal results. On the contrary, UNTA performs transmission map estimation and atmospheric light estimation simultaneously to obtain joint optimal solutions. The experimental results on both synthetic and real-world image databases demonstrate that HRGAN outperforms the state-of-the-art algorithms in terms of both effectiveness and efficiency.

Unsupervised Learning for Cell-Level Visual Representation in Histopathology Images With Generative Adversarial Networks.

The visual attributes of cells, such as the nuclear morphology and chromatin openness, are critical for histopathology image analysis. By learning cell-level visual representation, we can obtain a rich mix of features that are highly reusable for various tasks, such as cell-level classification, nuclei segmentation, and cell counting. In this paper, we propose a unified generative adversarial networks architecture with a new formulation of loss to perform robust cell-level visual representation learning in an unsupervised setting. Our model is not only label-free and easily trained but also capable of cell-level unsupervised classification with interpretable visualization, which achieves promising results in the unsupervised classification of bone marrow cellular components. Based on the proposed cell-level visual representation learning, we further develop a pipeline that exploits the varieties of cellular elements to perform histopathology image classification, the advantages of which are demonstrated on bone marrow datasets.