Advantage Of Generative Adversarial Network Research Articles

Aggregated Contextual Transformations for High-Resolution Image Inpainting.

Image inpainting that completes large free-form missing regions in images is a promising yet challenging task. State-of-the-art approaches have achieved significant progress by taking advantage of generative adversarial networks (GAN). However, these approaches can suffer from generating distorted structures and blurry textures in high-resolution images (e.g., 512×512). The challenges mainly drive from (1) image content reasoning from distant contexts, and (2) fine-grained texture synthesis for a large missing region. To overcome these two challenges, we propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN), for high-resolution image inpainting. Specifically, to enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. The AOT blocks aggregate contextual transformations from various receptive fields, allowing to capture both informative distant image contexts and rich patterns of interest for context reasoning. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task. Such a training objective forces the discriminator to distinguish the detailed appearances of real and synthesized patches, and in turn facilitates the generator to synthesize clear textures. Extensive comparisons on Places2, the most challenging benchmark with 1.8 million high-resolution images of 365 complex scenes, show that our model outperforms the state-of-the-art. A user study including more than 30 subjects further validates the superiority of AOT-GAN. We further evaluate the proposed AOT-GAN in practical applications, e.g., logo removal, face editing, and object removal. Results show that our model achieves promising completions in the real world. We release codes and models in https://github.com/researchmm/AOT-GAN-for-Inpainting.

Multivariate time series data imputation using attention-based mechanism

As the widely deployment of different sensors and Internet of Things, a large volume of multivariate time series has been collected. However, there are many missing values in the multivariate time series due to different reasons, such as sensor damage, environmental intrusion and machine failure. These missing values bring challenges to further analysis of multivariate time series. The existing methods for multivariate time series imputation either destroy the properties of the original data or fail to capture the correlations of the data effectively. In order to solve the problems in existing methods, we propose a novel imputation method for multivariate time series based on Generative Adversarial Networks (GAN). Specifically, we use Auto-Encoder (AE) as the generator (G), Recurrent Neural Networks (RNN) as the discriminator (D). Taking advantages of generative adversarial networks to impute the incomplete multivariate time series data. In addition, Attention Mechanism is integrated to the structure of the Encoder and Decoder to keep the correlations among data. By integrating the advantages of Attention Mechanism, it enables the generator to take advantage of the internal features of the original data when imputing the data. Therefore, the imputed multivariate time series are more reasonable and reliable. Furthermore, we also improve our model by integrating Self-Attention Mechanism to improve the accuracy for multivariate time series imputation. Experimental results on real datasets demonstrate that the performance of the proposed model exceeds existing models and achieves state-of-the-art performance.

Generative Adversarial Networks for Video Summarization Based on Key-frame Selection

Video summarization based on generative adversarial networks (GANs) has been shown to easily produce more realistic results. However, most summary videos are composed of multiple key components. If the selection of some video frames changes during the training process, the information carried by these frames may not be reasonably reflected in the identification results. In this paper, we propose a video summarization method based on selecting keyframes over GANs. The novelty of the proposed method is the discriminator not only identifies the completeness of the video, but also takes into account the value judgment of the candidate keyframes, thus enabling the influence of keyframes on the result value. Given GANs are mainly designed to generate continuous real values, it is generally challenging to generate discrete symbol sequences during the summarization process directly. However, if the generated sample is based on discrete symbols, the slight guidance change of the discrimination network may be meaningless. To better use the advantages of GANs, the study also adopts the video summarization optimization method of GANs under a collaborative reinforcement learning strategy. Experimental results show the proposed method gets a significant summarization effect and character compared with the existing cutting-edge methods.

Real‐time coloring method of laser surgery video based on generative adversarial network

AbstractLasers have become an important technology for the medical treatment of cancer. However, the high power of visible laser knives makes conventional endoscopic imaging light pollution severe. This phenomenon can cause the physician to be unable to determine the condition of the lesion site. In this article, a colorization method for laser surgery videos is proposed. The method performs real‐time colorization of the laser surgery video under black‐and‐white imaging. The main contribution of this work is to propose a new idea for solving laser contamination in medical surgery videos, along with real‐time generative adversarial networks (RTGANs). The network takes advantage of generative adversarial networks and residual networks for fast colorization of black and white images. To evaluate the method, we conducted experiments using three datasets. The experimental results show that the RTGAN network takes only 0.026 s to colorize a single frame image. Coloring speed up to 67 times faster than advanced methods, and the Peak Signal to Noise Ratio reached a maximum of 35.736.

TP-GAN: Simple Adversarial Network With Additional Player for Dense Depth Image Estimation

We present a simple yet robust monocular depth estimation technique by synthesizing a depth map image from a single RGB input image using the advantage of generative adversarial networks (GAN). We employ an additional sub-model termed refiner to extract local depth features, then combine it with the global scene information from the generator to improve the GAN’s performance compared to the standard GAN architectural scheme. Notably, the generator is the first player to learn to synthesize depth images. The second player, the discriminator, classifies the generated depth. In the meantime, the third player, the refiner, enhances the final reconstructed depth. Complementing the GAN model, we apply a conditional generative network (cGAN) to lead the generator in mapping the input image to the respective depth representation. We further incorporate a structured similarity (SSIM) as our loss function for the generator and refiner in GAN training. Through extensive experiment validation, we confirmed the performance of our strategy on the publicly indoor NYU Depth v2 and KITTI outdoor data. Experiment results on the NYU depth v2 dataset show that our proposed approach achieves the best performance by 96.0% on threshold accuracy ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\delta &lt; 1.25^{2}$ </tex-math></inline-formula> ) and the second-best accuracy on all thresholds on the KITTI dataset. We discovered that our proposed method compares favorably to numerous existing monocular depth estimation strategies and demonstrates a considerable improvement in the accuracy of image depth estimation despite its simple network architecture.

HKGAIL: Policy shaping via integrating human knowledge with generative adversarial imitation learning

AbstractGenerative adversarial imitation learning (GAIL) has shown good results in several research areas by taking advantage of generative adversarial networks. However, GAIL lacks a reward mechanism and usually adopts a model‐free approach based on stochastic policies, which is not ideal for solving complex, dynamically uncertain population intelligence problems, especially in the face of autonomous driving environments. In this paper, a policy framework is shaped by combining the human knowledge with GAIL (HKGAIL). HKGAIL embeds human decision models into the learning process to infer the underlying structure of expert demonstrations. The skills learned from expert demonstrations can directly guide the actions (policies) of the learning process of the agents, and the policies can be optimized through the feedback function of the discriminator. Experiments on both driving and landing tasks show that HKGAIL was able to better fit the policy close to the expert, and was 16.2% safer than GAIL for the driving task.

Group behavior dissemination model of social hotspots based on data enhancement and data representation

The emergence and dissemination of hotspots in social networks mainly depend on the participation of group users. In this paper, considering the sparsity and complexity of effective data, we propose a group behavior dissemination model on the basis of data enhancement and data representation. First, given the inaccurate prediction results caused by the sparsity of valid data and the advantages of Generative Adversarial Networks (GAN) in learning data distribution and enhancing data, GAN is introduced to generate homomorphic data. We found that the accuracy improved by at least 6%. Second, with the diversity and complexity of the hotspot feature space and the ability of representation learning to mine hidden features of the hotspot, we designed a new method, called HP2vec, convert the feature space to a low rank and dense vector. Finally, considering the dynamic time limit of hotspot spreading, we create time slices to discretize the life of a hotspot and propose a dynamic dissemination method based on CNN. The experimental section shows that the accuracy of this method is as high as 86% in Weibo dataset and as high as 93% in twitter dataset. The model alleviates data sparseness and effectively predicts the group behavior in hotspots.

Multilayer Perceptron Generative Model via Adversarial Learning for Robust Visual Tracking

Visual tracking is an open and exciting field of research. The researchers introduced great efforts to be close to the ideal state of stable tracking of objects regardless of different appearances or circumstances. Owing to the attractive advantages of generative adversarial networks (GANs), they have been a promising area of research in many fields. However, GAN network architecture has not been thoroughly investigated in the visual tracking research community. Inspired by visual tracking via adversarial learning (VITAL), we present a novel network to generate randomly initialized masks for building augmented feature maps using multilayer perceptron (MLP) generative models. To obtain more robust tracking these augmented masks can extract robust features that do not change over a long temporal span. Some models such as deep convolutional generative adversarial networks (DCGANs) have been proposed to obtain powerful generator architectures by eliminating or minimizing the use of fully connected layers. This study demonstrates that the use of MLP architecture for the generator is more robust and efficient than the convolution-only architecture. Also, to realize better performance, we used one-sided label smoothing to regularize the discriminator in the training stage and the label smoothing regularization (LSR) method to reduce the overfitting of the classifier in the online tracking stage. The experiments show that the proposed model is more robust than the DCGAN model and offers satisfactory performance compared with the state-of-the-art deep visual trackers on OTB-100, VOT2019 and LaSOT datasets.

A Tutorial on Generative Adversarial Networks with Application to Classification of Imbalanced Data.

A challenge unique to classification model development is imbalanced data. In a binary classification problem, class imbalance occurs when one class, the minority group, contains significantly fewer samples than the other class, the majority group. In imbalanced data, the minority class is often the class of interest (e.g., patients with disease). However, when training a classifier on imbalanced data, the model will exhibit bias towards the majority class and, in extreme cases, may ignore the minority class completely. A common strategy for addressing class imbalance is data augmentation. However, traditional data augmentation methods are associated with overfitting, where the model is fit to the noise in the data. In this tutorial we introduce an advanced method for data augmentation: Generative Adversarial Networks (GANs). The advantages of GANs over traditional data augmentation methods are illustrated using the Breast Cancer Wisconsin study. To promote the adoption of GANs for data augmentation, we present an end-to-end pipeline that encompasses the complete life cycle of a machine learning project along with alternatives and good practices both in the paper and in a separate video. Our code, data, full results and video tutorial are publicly available in the paper's github repository.

An unsupervised generative adversarial network for single image deraining

As the basis of image processing, single image deraining has always been a significant and challenging issue. Due to the lack of real rainy images and corresponding clean images, most deraining networks are trained by synthetic datasets, which makes the output images unsatisfactory in real applications. Besides, note that a heavy rainfall is typically accompanied with some fog. Although some deraining networks have been proposed to remove the rain streaks in the rainy images, the output images may still be blurred due to the accompanied fog. In this paper, these problems existing in single image deraining is comprehensively considered, and propose a Cycle-Derain network based on an unsupervised attention-guided mechanism. Specifically, the Cycle-Derain network takes advantage of generative adversarial networks with two mappings and the cycle consistency loss to train both unpaired rainy images and rain-free images. Moreover, it introduces an unsupervised attention-guided mechanism and exploits the loop-search positioning algorithm to deal with the details of rain and fog in images. Extensive experiments have been carried out, and the results show that the proposed Cycle-Derain network is preferable compared with other deraining networks, especially in term of rainy image restoration.

Edge-guided multispectral image fusion algorithm

Most existing multispectral fusion algorithms often suffer from spectral or spatial information distortion. Driven by this motivation, we propose an edge-guided multispectral (MS) image fusion algorithm. In particular, it combines the advantages of generative adversarial networks and improved fusion frameworks, so the merged image can better preserve the spectral information of the original multispectral image while injecting spatial detail information. Specifically, first, an MS image with more image detail is generated using the generated confrontation network for preliminary reconstruction. The panchromatic image edge information and the antagonistic learning strategy are introduced for the robust multispectral image reconstruction. Then, using the reconstructed MS image and the general component substitution image fusion framework, the whole fusion system of this paper is constructed. An enhancement operator is introduced to inject spatial details. Our extensive dataset evaluations show that our approach performs better in terms of high objective quality and human visual perception than several of the most advanced fusion methods.

De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks

Machine learning techniques have been widely applied to various applications. However, they are potentially vulnerable to data poisoning attacks, where sophisticated attackers can disrupt the learning procedure by injecting a fraction of malicious samples into the training dataset. Existing defense techniques against poisoning attacks are largely attack-specific : they are designed for one specific type of attacks but do not work for other types, mainly due to the distinct principles they follow. Yet few general defense strategies have been developed. In this paper, we propose De-Pois, an attack-agnostic defense against poisoning attacks. The key idea of De-Pois is to train a mimic model the purpose of which is to imitate the behavior of the target model trained by clean samples. We take advantage of Generative Adversarial Networks (GANs) to facilitate informative training data augmentation as well as the mimic model construction. By comparing the prediction differences between the mimic model and the target model, De-Pois is thus able to distinguish the poisoned samples from clean ones, without explicit knowledge of any ML algorithms or types of poisoning attacks. We implement four types of poisoning attacks and evaluate De-Pois with five typical defense methods on different realistic datasets. The results demonstrate that De-Pois is effective and efficient for detecting poisoned data against all the four types of poisoning attacks, with both the accuracy and F1-score over 0.9 on average.

Bidirectional generative transductive zero-shot learning

Most zero-shot learning (ZSL) methods aim to learn a mapping from visual feature space to semantic feature space or from both visual and semantic feature spaces to a common joint space and align them. However, in these methods the visual and semantic information are not utilized sufficiently and the useless information is not excluded. Moreover, there exists a strong bias problem that the instances from unseen classes always tend to be predicted as some seen classes in most ZSL methods. In this paper, combining the advantages of generative adversarial networks (GANs), a method based on bidirectional projections between the visual and semantic feature spaces is proposed. GANs are used to perform bidirectional generations and alignments between the visual and semantic features. In addition, cycle mapping structure ensures that the important information are kept in the alignments. Furthermore, in order to better solve the bias problem, pseudo-labels are generated for unseen instances and the model is adjusted along with them iteratively. We conduct extensive experiments at traditional ZSL and generalized ZSL settings, respectively. Experiment results confirm that our method achieves the state-of-the-art performances on the popular datasets AWA2, aPY and SUN.

TrajVAE: A Variational AutoEncoder model for trajectory generation

Large-scale trajectory dataset is always required for self-driving and many other applications. In this paper, we focus on the trajectory generation problem, which aims to generate qualified trajectory dataset that is indistinguishable from real trajectories, for fulfilling the needs of large-scale trajectory data by self-driving simulation and traffic analysis tasks in data sparse cities or regions. We propose two advanced solutions, namely TrajGAN and TrajVAE, which utilize LSTM to model the characteristics of trajectories first, and then take advantage of Generative Adversarial Network (GAN) and Variational AutoEncoder (VAE) frameworks respectively to generate trajectories. In order of compare the similarity of existing trajectories in our dataset and the generated trajectories, we utilize multiple trajectory similarity metrics. Through several experiments, we demonstrate that our method is more accurate and stable than the baseline.

End-to-End Image Super-Resolution via Generative Adversarial Network

Image super-resolution is to extract information from single or more low- resolution images and use this information to get the corresponding high-resolution image. The ability to capture high texture details of low-resolution images is one of the impressive advantages of generative adversarial networks (GANs). This paper mainly studies the image reconstruction method based on a single low-resolution image, and builds an end-to-end image Super-Resolution method via generative adversarial networks to improve the image resolution. Convolution neural networks use the mean square error (MSE) as loss function. For such loss function, structural similarity (SSIM) and peak signal to noise ratio (PSNR) can be obtained, but high texture details of original images are often lost. The generative adversarial network mainly contains two parts: generator network and discriminator network. Adversarial loss uses a discriminator network to push our solution to a natural image manifold, which is trained to distinguish super-resolution images from photo-realistic images. Additionally, content loss of our network was indicated by perceptual similarity rather than similarity of pixel-wised space.

Fully\u2011automated deep\u2011learning segmentation of pediatric cardiovascular magnetic resonance of patients with complex congenital heart diseases

BackgroundFor the growing patient population with congenital heart disease (CHD), improving clinical workflow, accuracy of diagnosis, and efficiency of analyses are considered unmet clinical needs. Cardiovascular magnetic resonance (CMR) imaging offers non-invasive and non-ionizing assessment of CHD patients. However, although CMR data facilitates reliable analysis of cardiac function and anatomy, clinical workflow mostly relies on manual analysis of CMR images, which is time consuming. Thus, an automated and accurate segmentation platform exclusively dedicated to pediatric CMR images can significantly improve the clinical workflow, as the present work aims to establish.MethodsTraining artificial intelligence (AI) algorithms for CMR analysis requires large annotated datasets, which are not readily available for pediatric subjects and particularly in CHD patients. To mitigate this issue, we devised a novel method that uses a generative adversarial network (GAN) to synthetically augment the training dataset via generating synthetic CMR images and their corresponding chamber segmentations. In addition, we trained and validated a deep fully convolutional network (FCN) on a dataset, consisting of 64 pediatric subjects with complex CHD, which we made publicly available. Dice metric, Jaccard index and Hausdorff distance as well as clinically-relevant volumetric indices are reported to assess and compare our platform with other algorithms including U-Net and cvi42, which is used in clinics.ResultsFor congenital CMR dataset, our FCN model yields an average Dice metric of 91.0mathrm{%} and 86.8mathrm{%} for LV at end-diastole and end-systole, respectively, and 84.7mathrm{%} and 80.6mathrm{%} for RV at end-diastole and end-systole, respectively. Using the same dataset, the cvi42, resulted in 73.2mathrm{%}, 71.0mathrm{%}, 54.3mathrm{%} and 53.7mathrm{%} for LV and RV at end-diastole and end-systole, and the U-Net architecture resulted in 87.4mathrm{%}, 83.9mathrm{%}, 81.8mathrm{%} and 74.8mathrm{%} for LV and RV at end-diastole and end-systole, respectively.ConclusionsThe chambers’ segmentation results from our fully-automated method showed strong agreement with manual segmentation and no significant statistical difference was found by two independent statistical analyses. Whereas cvi42 and U-Net segmentation results failed to pass the t-test. Relying on these outcomes, it can be inferred that by taking advantage of GANs, our method is clinically relevant and can be used for pediatric and congenital CMR segmentation and analysis.

Fault Diagnosis of Distribution Terminal Units’ Measurement System Based on Generative Adversarial Network Combined with Convolutional Neural Network

As the condition monitoring and control device in the distribution automation system, the abnormal or fault state of distribution terminal units’ measurement system will negatively affect the quality of measured electrical quantities, and therefore, the fast and accurate discrimination of the abnormal state’s data will improve the reliability of distribution automation system. This paper proposes a method, which is based on generative adversarial network (GAN) combined with convolutional neural network (CNN), to discriminate the specific fault category of distribution terminals’ measuring electrical data. Firstly, four fault state characteristic time-frequency domain graph models of terminals’ AC voltage sampling data are established, which based on Fourier transform (STFT). Then, take advantage of GAN’s reconstruction of input graph data, to generate additional time-frequency sample graphs expanding the sample size of training set, which will be used to train a CNN to diagnosis and classify the fault state of terminals from measuring data. Finally, three training sets with different capacity expansion modes are set up to compare and verify that the method of GAN combined with CNN proposed in this paper improves the discrimination accuracy on the fault data, and the validation of diagnosis terminals’ measurement system.

Task-Oriented GAN for PolSAR Image Classification and Clustering.

Based on a generative adversarial network (GAN), a novel version named Task-Oriented GAN is proposed to tackle difficulties in PolSAR image interpretation, including PolSAR data analysis and small sample problem. Besides two typical parts in GAN, i.e., generator (G-Net) and discriminator (D-Net), there is a third part named TaskNet (T-Net) in the Task-Oriented GAN, where T-Net is employed to accomplish a certain task. Two tasks, PolSAR image classification and clustering, are studied in this paper, where T-Net acts as a Classifier and a Clusterer, respectively. The learning procedure of Task-Oriented GAN consists of two main stages. In the first stage, G-Net and D-Net vie with each other like that in a general GAN; in the second stage, G-Net is adjusted and oriented by T-Net so that more samples, which are benefit for the task and called fake data, are generated. As a result, Task-Oriented GAN not only has the advantage of GAN (no-assumption data modeling) but also overcomes the disadvantage of GAN (task-free). After learning, fake data are employed to enrich training set and avoid overfitting; so Task-Oriented GAN performs well even if the manual-labeled data are small. To verify the effectiveness of T-Net, a visualized comparison is provided, where some fake digits generated from Task-Oriented GAN are illustrated along with that from GAN. What is more, considering that there is a great difference between PolSAR data and general data, in our PolSAR image classification and clustering tasks, the specific PolSAR information is inserted into the structure of the Task-Oriented GAN. This enables researchers to mine inherent information in PolSAR data without any data hypothesis and find ways for small sample problem at the same time. Experiment results tested on three PolSAR images show that the proposed method performs well in dealing with PolSAR image classification and clustering.

From Night to Day: GANs Based Low Quality Image Enhancement

Nighttime images have lower contrast and higher noise than their daytime counterparts of the same scene. The aim of nighttime image enhancement is to improve the visual quality of nighttime images, so that they are visually as close as possible to daytime images. This problem is still challenging because of the deteriorated conditions of illumination lack and uneven lighting. In this paper, we propose a generative adversarial networks (GANs) based framework for nighttime image enhancement. To take advantage of GANs’ powerful ability of generating image from real data distribution, we make the established network well constrained by combining several loss functions including adversarial loss, perceptual loss, and total variation loss. Particularly, a pre-trained network is applied to leverage the perceptual loss which is beneficial to generate high-quality images. Meanwhile, for tackling the light-at-night effect, we present a fusion network in which the dark channel prior based illumination compensation is employed for the training of generator network. Experimental results have demonstrated the effectiveness of the proposed nighttime image enhancement network.