Attentive Generative Adversarial Network Research Articles

Underwater Image Enhancement Based on Pyramid Attention Mechanism and Generative Adversarial Network

Underwater Image Enhancement Based on Pyramid Attention Mechanism and Generative Adversarial Network

Enhanced Attentive Generative Adversarial Network for Single-Image Deraining

The problem of image rain removal has drawn widespread attention as the blurry images caused by rain streaks can degrade the performance of many computer vision tasks. Although exiting deep learning-based methods outperform most traditional methods, there are still unresolved issues in terms of performance. In this paper, we propose a novel enhanced attentive generative adversarial network named EAGAN to effectively remove the rain streaks and restore the image structural details at the same time. As rain streaks have different sizes and shapes, EAGAN utilizes a multiscale aggregation attention module (MAAM) to produce an attention map to guide the subsequent network to put conscious attention to rain regions. A symmetrical autoencoder with long-range skip-connections, squeeze-and-excitation (SE) modules, and non-local operation is further utilized to enhance the representation of the network. Finally, spectral normalization and a relativistic generative adversarial network (GAN) are further applied to improve the training stability and deraining performance. Both qualitative and quantitative validations on synthetic and real-world datasets demonstrate that the proposed approach can achieve a competitive performance in comparison with the state-of-the-art methods.

Spatial orthogonal attention generative adversarial network for MRI reconstruction.

Recent studies have witnessed that self-attention modules can better solve the vision understanding problems by capturing long-range dependencies. However, there are very few works designing a lightweight self-attention module to improve the quality of MRI reconstruction. Furthermore, it can be observed that several important self-attention modules (e.g., the non-local block) cause high computational complexity and need a huge number of GPU memory when the size of the input feature is large. The purpose of this study is to design a lightweight yet effective spatial orthogonal attention module (SOAM) to capture long-range dependencies, and develop a novel spatial orthogonal attention generative adversarial network, termed as SOGAN, to achieve more accurate MRI reconstruction. We first develop a lightweight SOAM, which can generate two small attention maps to effectively aggregate the long-range contextual information in vertical and horizontal directions, respectively. Then, we embed the proposed SOAMs into the concatenated convolutional autoencoders to form the generator of the proposed SOGAN. The experimental results demonstrate that the proposed SOAMs improve the quality of the reconstructed MR images effectively by capturing long-range dependencies. Besides, compared with state-of-the-art deep learning-based CS-MRI methods, the proposed SOGAN reconstructs MR images more accurately, but with fewer model parameters. The proposed SOAM is a lightweight yet effective self-attention module to capture long-range dependencies, thus, can improve the quality of MRI reconstruction to a large extent. Besides, with the help of SOAMs, the proposed SOGAN outperforms the state-of-the-art deep learning-based CS-MRI methods.

Single-channel blind source separation based on attentional generative adversarial network

Blind single-channel source separation is a long-standing machine learning and signal processing problem. Traditional blind source separation (BSS) algorithms were proposed to solve this task utilizing multiple signal constraints. Generative adversarial network (GAN) are free from statistical constraints and samples, but the role of adversarial training in the task of BSS has not been fully demonstrated. Therefore, a new separation network model that enables to learn the known separated signal distribution from the stepwise fine estimation of the unknown mixture distribution was presented in this paper, and a self-attention mechanism was introduced to solve the problem of the blurring details of the generated image by the generator which preserves image details in the process of image separation. Compared with the existing single-channel blind source separation algorithm based on generative adversarial network-neural egg separation (NES), the detailed information of this new separation algorithm is more prominent, and the source signal in the mixed image has been separated more effectively, and has better separation performance than the classic blind source separation algorithms.

Blind image separation based on attentional generative adversarial network

Mixing signal separation is an important field of image processing. However, traditional blind source separation (BSS) algorithms were proposed to solve this task utilizing multiple signal constraints such as independent, non-Gaussian, low rank, sparsity, temporal continuity etc. What’s more, as a case of ill-conditioned signal mixing, the single-channel blind source separation (SCBSS) is more difficult. Because neural networks have strong adaptability and self-organization capability, neural network methods based on training and learning ideas are favored by researchers. However, most BSS methods based on neural network are limited by small sample sizes. Among various neural network, generative adversarial network (GAN) has emerged as an interesting candidate because it is free from statistical constraints and samples. Therefore, we present a single-channel blind image separation algorithm based on attention mechanism GAN, coined AGAN, which uses an end-to-end manner, and it will have more hopeful prospects in the blind image separation task. The network with feature extraction, as well as edge guidance to data creates a new way to iteratively separate mixing images. The experimental results show that AGAN can effectively separate the source signal in the mixing images compared with the neural egg separation (NES) algorithm, which is a neural network separation algorithm. Compared with the classical blind source separation algorithms, this method has better separation performance.

AUTOMATIC CLOUD DETECTION METHOD BASED ON GENERATIVE ADVERSARIAL NETWORKS IN REMOTE SENSING IMAGES

Abstract. Clouds in optical remote sensing images seriously affect the visibility of background pixels and greatly reduce the availability of images. It is necessary to detect clouds before processing images. In this paper, a novel cloud detection method based on attentive generative adversarial network (Auto-GAN) is proposed for cloud detection. Our main idea is to inject visual attention into the domain transformation to detect clouds automatically. First, we use a discriminator (D) to distinguish between cloudy and cloud free images. Then, a segmentation network is used to detect the difference between cloudy and cloud-free images (i.e. clouds). Last, a generator (G) is used to fill in the different regions in cloud image in order to confuse the discriminator. Auto-GAN only requires images and their labels (1 for a cloud-free image, 0 for a cloudy image) in the training phase which is more time-saving to acquire than existing methods based on CNNs that require pixel-level labels. Auto-GAN is applied to cloud detection in Sentinel-2A Level 1C imagery. The results indicate that Auto-GAN method performs well in cloud detection over different land surfaces.

Single Image De-raining Based on a Novel Enhanced Attentive Generative Adversarial Network

With rapid development of deep learning in artificial intelligence and computer vision, generative adversarial network plays an important role in single image de-raining. Attentive generative adversarial network (AttGAN) has problems about complicated network structure and distortion of background color. Considering the relative complex background in the real image with raindrops, a new single image de-raining algorithm based on enhanced attentive generative adversarial network (EAttGAN) is proposed to retain the original background of the blurred image. In order to restore more complete background and accelerate network training, a generator enhanced by residual scaling and a Markovian discriminator are fused effectively in the network. Compared with AttGAN, experimental results indicate that EAttGAN can not only achieve higher sharpness of a single image, but also take less time in the training process.

Semantically consistent text to fashion image synthesis with an enhanced attentional generative adversarial network

Recent advancements in Generative Adversarial Networks (GANs) have led to significant improvements in various image generation tasks including image synthesis based on text descriptions. In this paper, we present an enhanced Attentional Generative Adversarial Network (e-AttnGAN) with improved training stability for text-to-image synthesis. e-AttnGAN’s integrated attention module utilizes both sentence and word context features and performs feature-wise linear modulation (FiLM) to fuse visual and natural language representations. In addition to multimodal similarity learning for text and image features of AttnGAN [1], similarity and feature matching losses between real and generated images are included while employing classification losses for “significant attributes”. In order to improve the stability of the training and solve the mode collapse issue, spectral normalization and two-time scale update rule are used for the discriminator together with instance noise. Our experiments show that e-AttnGAN outperforms state-of-the-art methods using the FashionGen and DeepFashion-Synthesis datasets in terms of inception score, R-precision and classification accuracy. A detailed ablation study has been conducted to observe the effect of each component.

Dilated Residual Spatial Attentive Generative Adversarial Network for Single Image De-raining

This paper propose a dilated residual spatial attentive generative adversarial network for single image de-raining. The improved spatial attentive mechanism is combined with the dilated convolution residual module to optimize the Condition Generative Adversarial Network(CGAN) structure, to solve two problems still exist in the task of removing rain from a single image: First, the rain streaks contained in the dataset we can use are limited, and in the case of real rainy days, the rain streak density is diverse, it is impossible to simulate them completely and accurately. Then, the existing rain removal models cannot remove rain streaks properly for images with different rain streak density which attend to over or under rain removal. In our methods, Firstly, the dilated convolution module is used to enhance the feature extraction of rain streaks, and then an attention map is generated by the spatial attention mechanism module to accurately locate the position of rain streaks, and the rain streaks are extracted as the foreground information by combining the two parts of the network to guide the subsequent rain removal operation; Then the rain streaks is removed through the Contextual auto-encoder, combining with PatchGAN discriminator. The experimental results on two synthetic dataset and real-world rain image show that the network model proposed in this paper has high generalization ability under different rainfall conditions. it improves the rain removal effect of the model on the most realistic public dataset, and improves the recovery ability of image details while effectively removing rain on other public datasets, which verifies the effectiveness of the model in improving the generalization ability and improving the image quality after rain removal. At the same time, the effect of rain removal is better than other methods on the real rain images, which proves that the method proposed in this paper has high practical application value.

A Novel Attentive Generative Adversarial Network for Waterdrop Detection and Removal of Rubber Conveyor Belt Image

The lens for monitoring the rubber conveyor belt is easy to adhere to a large number of water droplets, which seriously affects the image quality and then affects the effect of fault monitoring. In this paper, a new method for detecting and removing water droplets on rubber conveyor belts based on the attentive generative adversarial network is proposed to solve this problem. First, the water droplet image of the rubber conveyor belt is input into the generative network composed of a cyclic visual attentive network and an autoencoder with skip connections, and an image of removing water droplets and an attention map for detecting the position of the water droplet are generated. Then, the generated image of removing water droplets is evaluated by the attentive discriminant network to assess the local consistency of the water droplet recovery area. In order to better learn the water droplet regions and the surrounding structures during the training, the image morphology is added to the precise water droplet regions. A dewatered rubber conveyor belt image is generated by increasing the number of circular visual attention network layers and the number of skip connection layers of the autoencoder. Finally, a large number of comparative experiments prove the effectiveness of the water droplet image removal algorithm proposed in this paper, which outperforms of Convolutional Neural Network (CNN), Discriminative Sparse Coding (DSC), Layer Prior (LP), and Attention Generative Adversarial Network (ATTGAN).

Multi-Scale Attention Generative Adversarial Networks for Video Frame Interpolation

Video frame interpolation is a fundamental task in computer vision. Recent methods usually apply convolutional neural networks to generate intermediate frame with two consecutive frames as inputs. But sometimes existing methods fail to handle with complex motion and long-range dependencies. In this paper, a multi-scale dense attention generative adversarial network is proposed. First, a multi-scale generative adversarial framework is established for video frame interpolation. Generators from coarse to fine can better combine global and local information. Second, an attention module introduced to generator makes network accurately focus on moving objects. Third, a sequence discriminator is designed to improve the ability of capturing spatial and temporal consistency in frame sequence. Experimental results of ablation study prove the effectiveness of our three contributions. And results on several datasets demonstrate that our approach attains higher performance and produce more photo-realistic in-between frame comparing with previous works.

Multi-Scale Weighted Fusion Attentive Generative Adversarial Network for Single Image De-Raining

With the rapid development of outdoor vision system, removing rain streaks from a single image has attracted considerable attention as rain streaks can affect the quality of the image taken in rainy days, and interrupt the key information, which will greatly reduce the use value of the image, thus affecting the performance of traffic, safety monitoring and other facilities. Although the deep learning methods have achieved satisfying performance in single image de-raining, there are still two problems: First, the rain streaks contained in one dataset we can use are limited, and in the case of real rainy days, the rain streak density is diverse, it is impossible to accurately classify them. Therefore, the existing rain removal models cannot remove rain streaks properly for images with different rain streak density which attend to over or under rain removal. Secondly, the results of single image after rain removal model often appear the phenomenon of variegated spots, image contrast saturation change and even unsmooth rain streak after rain removal. We use a three-way multi-scale weighted fusion module to enhance the feature extraction, and then generate an attention map through the improved spatial attentive module to accurately locate the location of the rain streaks. After the combination of the two, we will obtain the foreground information of the rain streaks. Through the characteristic of mutual game in the training mechanism of GAN, we can enhance the rain streak location recognition and effectively remove the rain at the same time. Through the training mechanism of the GAN network game, we can enhance the rain line location recognition and effectively remove the rain at the same time. Experiments show that our network achieves superior performance, it has high generalization for different rain streak density, and ensures that the contrast and saturation of the image are not changed.

Multi-Scale Feature Channel Attention Generative Adversarial Network for Face Sketch Synthesis

Face sketch synthesis for photos is an applied research topic and it is critical for criminal investigation. However, sketch synthesis remains some challenges because of the blur and artifacts in the generated face sketches. To mitigate these problems in face sketch synthesis, we propose a fast Generative Adversarial Network with fast Multi-scale feature channel Attention, namely MAGAN. In the generator network, multi-scale features are extracted by proposed multi-scale feature extraction to produce detailed sketches. Then, a channel attention mechanism is applied to emphasize the significance of important feature channels, further enhancing the synthesized sketches. Besides, the loss of patch-wise high-layer features from the VGG-19 network is applied to supervise the generator to synthesize more realistic sketches. To accelerate the training process, the features from the pooling layers are adopted to calculate the pseudo sketch feature loss. The experimental results demonstrate that our MAGAN can achieve better performance in both visual evaluations and quantitative evaluations (in terms of feature similarity and learned perceptual image patch similarity), compared with the state-of-the-art methods.

CA-GAN: Class-Condition Attention GAN for Underwater Image Enhancement

Underwater images suffer from serious color distortion and detail loss because of the wavelength-dependent light absorption and scattering, which seriously influences the subsequent underwater object detection and recognition. The latest methods for underwater image enhancement are based on deep models, which focus on finding a mapping function from the underwater image subspace to a ground-truth image subspace. They neglect the diversity of underwater conditions which leads to different background colors of underwater images. In this paper, we propose a Class-condition Attention Generative Adversarial Network (CA-GAN) to enhance an underwater image. We build an underwater image dataset which contains ten categories generated by the simulator with different water attenuation coefficient and depth. Relying on the underwater image classes, CA-GAN creates a many-to-one mapping function for an underwater image. Moreover, in order to generate the realistic image, attention mechanism is utilized. In the channel attention block, the feature maps in the front-end layers and the back-end layers are fused along channels, and in the spatial attention block, feature maps are pixel-wise fused. Extensive experiments are conducted on synthetic and real underwater images. The experimental results demonstrate that CA-GAN can effectively recover color and detail of various scenes of underwater images and is superior to the state-of-the-art methods.

Semi-supervised segmentation of lesion from breast ultrasound images with attentional generative adversarial network

Background and objectiveAutomatic segmentation of breast lesion from ultrasound images is a crucial module for the computer aided diagnostic systems in clinical practice. Large-scale breast ultrasound (BUS) images remain unannotated and need to be effectively explored to improve the segmentation quality. To address this, a semi-supervised segmentation network is proposed based on generative adversarial networks (GAN). MethodsIn this paper, a semi-supervised learning model, denoted as BUS-GAN, consisting of a segmentation base network—BUS-S and an evaluation base network—BUS-E, is proposed. The BUS-S network can densely extract multi-scale features in order to accommodate the individual variance of breast lesion, thereby enhancing the robustness of segmentation. Besides, the BUS-E network adopts a dual-attentive-fusion block having two independent spatial attention paths on the predicted segmentation map and leverages the corresponding original image to distill geometrical-level and intensity-level information, respectively, so that to enlarge the difference between lesion region and background, thus improving the discriminative ability of the BUS-E network. Then, through adversarial training, the BUS-GAN model can achieve higher segmentation quality because the BUS-E network guides the BUS-S network to generate more accurate segmentation maps with more similar distribution as ground truth. ResultsThe counterpart semi-supervised segmentation methods and the proposed BUS-GAN model were trained with 2000 in-house images, including 100 annotated images and 1900 unannotated images, and tested on two different sites, including 800 in-house images and 163 public images. The results validate that the proposed BUS-GAN model can achieve higher segmentation accuracy on both the in-house testing dataset and the public dataset than state-of-the-art semi-supervised segmentation methods. ConclusionsThe developed BUS-GAN model can effectively utilize the unannotated breast ultrasound images to improve the segmentation quality. In the future, the proposed segmentation method can be a potential module for the automatic breast ultrasound diagnose system, thus relieving the burden of a tedious image annotation process and alleviating the subjective influence of physicians’ experiences in clinical practice. Our code will be made available on https://github.com/fiy2W/BUS-GAN.

Multi-stream attentive generative adversarial network for dynamic scene deblurring

Image deblurring methods based on deep learning have been proved to be effective for dynamic scene deblurring. Nevertheless, the most existing methods usually fail to deal with severe and complex dynamic scene blurs. In this paper, we propose a novel multi-stream attentive generative adversarial network (MSA-GAN) for solving the problems. Firstly, we design a novel attentive guidance module for generating an attention map which can guide to pay more attention on the severe blur regions or objects and their edge or texture structures in the feature extraction and deblurring process. Secondly, we propose the multi-stream and multi-scale feature extraction strategy and design an attentive and multi-scale feature extraction module, in which we design a multi-scale residual block for extracting the multi-scale features and capturing more information of complex blurs. Thirdly, we propose a multi-scale feature fusion strategy and the strategy is implemented in the multi-stream and multi-scale feature fusion module through the designed gate structure, which can adaptively learn and fuse multi-scale features from different streams. Finally, The extensive experiments are performed on the GoPro dynamic scene dataset and real image data, and the experimental results both quantitatively and qualitatively have demonstrated that the proposed method outperform the recent state-of-the-art methods.

Attention GANs: Unsupervised Deep Feature Learning for Aerial Scene Classification

With the development of deep learning, supervised feature learning methods have achieved prominent performance in the field of aerial scene classification. However, supervised feature learning methods require a large amount of labeled training data. To address this limitation, in this article, a novel unsupervised deep feature learning method, namely, Attention generative adversarial networks (Attention GANs), is proposed for aerial scene classification. First, Attention GANs integrates the attention mechanism into GANs to enhance the representation power of the discriminator. Then, to obtain contextual information, a context-aggregation-based feature fusion architecture is designed in the discriminator. Furthermore, the generator and discriminator losses are improved on basis of the Relativistic GAN. At the same time, a content loss is formed by using the feature representations from the context-aggregation-based feature fusion architecture. In the experiments, our Attention GANs is evaluated via comprehensive experiments with four publicly available remote sensing scene data sets, i.e., the UC-Merced data set with 21 scene classes, the RSSCN7 data set with 7 scene classes, the AID data set with 30 scene classes, and the NWPU-RESISC45 data set with 45 scene classes. Experimental results demonstrate that our Attention GANs can obtain the best performance compared with the state-of-the-art methods.

Regional attention generative adversarial network

In this Letter, the authors propose a novel attention mechanism combined with a classical generative adversarial network (GAN) model to improve the visual quality of generated samples. This novel attention model is named regional attention GAN. The proposed mechanism can build dependencies between the high-level representations extracted from attention regions of real images and corresponding feature maps of the generative network. By modelling these dependencies, the generative network can be facilitated to learn feature mapping and fit the distribution of real data. They conduct extensive experiments on widely used datasets CIFAR-10, STL-10, and CelebA. The quantitative and qualitative performance improvement over state-of-the-art methods demonstrates the validity of the proposed attention mechanism in improving the quality of generated images.

Attention GAN-Based Method for Designing Intelligent Making System

Design of the intelligent making system (iMAS) poses a big technical challenge to recognize both printed and handwriting characters from the scanned images. In this paper, we propose an iMAS method based on the attention generative adversarial network (GAN), which innovatively applies the idea of removing raindrops through attention GAN to remove the printed text first, and then generates the image of handwriting text. The proposed method introduces a classical attention model in the visual field and generates the attention maps which focus only on the important areas through recurrent neural network (RNN), and adopts you only look once at object detection (YOLOv3) method to recognize the characters. Experimental results show that the structural similarity (SSIM) of the image generated by attention GAN is 0.89 and the accuracy of recognition is 91.34%.

Dualattn-GAN: Text to Image Synthesis With Dual Attentional Generative Adversarial Network

Recent generative adversarial network based methods have shown promising results for the charming but challenging task of synthesizing images from text descriptions. These approaches can generate images with general shape and color but often produce distorted global structures with unnatural local semantic details. It is due to ineffectiveness of convolutional neural networks in capturing the high-level semantic information for pixel-level image synthesis. In this paper, we propose a Dual Attentional Generative Adversarial Network (DualAttn-GAN) in which the dual attention modules are introduced to enhance local details and global structures by attending to related features from relevant words and different visual regions. As one of the dual modules, the textual attention module is designed to explore the fine-grained interaction between vision and language. On the other hand, visual attention module models internal representations of vision from channel and spatial axes, which can better capture the global structures. Meanwhile, we apply an attention embedding module to merge multi-path features. Furthermore, we present an inverted residual structure to boost representation power of CNNs and apply spectral normalization to stabilize GAN training. With extensive experimental validation on two benchmark datasets, our method significantly improves state-of-the-art models over the evaluation metrics of inception score and Frechet inception distance.