Generative Adversarial Networks Training Research Articles

Environment-Aware Multiscene Image Enhancement for Internet of Things Enabled Edge Cameras

With the development of the Internet of Things, it is feasible to leverage as much computing resources on an IoT-enabled camera to realize edge intelligence (hereafter referred to as an edge camera) for providing various human-centered services. However, environmental interference can significantly degrade the visual quality of an image into edge cameras, thereby affecting the overall image and service quality. In order to maintain image stability, image enhancement using complex deep networks has been widely utilized. As edge cameras often have restricted computing power and cannot run complex neural networks efficiently, we design a lightweight residual enhancement network, which incorporates highly efficient convolution operations as basic building blocks. With this modular design on a generative adversarial network training framework, the resultant network can be easily optimized according to hardware specifications of a given edge camera. To accommodate various weather conditions for an edge camera, we further design an environment-aware model deployer that can detect the weather conditions and deploy the most suitable image-enhancement model. The experimental results show that the lightweight residual enhancement network can produce image quality close to what past research demonstrated while increasing running speeds by more than 30%. In a streaming video to simulate changing weather, the results also demonstrate the effectiveness of the proposed environment-aware model deployer to maintain the quality of the images under a dynamic environment.

Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN

Training of Generative Adversarial Network (GAN) on a video dataset is a challenge because of the sheer size of the dataset and the complexity of each observation. In general, the computational cost of training GAN scales exponentially with the resolution. In this study, we present a novel memory efficient method of unsupervised learning of high-resolution video dataset whose computational cost scales only linearly with the resolution. We achieve this by designing the generator model as a stack of small sub-generators and training the model in a specific way. We train each sub-generator with its own specific discriminator. At the time of the training, we introduce between each pair of consecutive sub-generators an auxiliary subsampling layer that reduces the frame-rate by a certain ratio. This procedure can allow each sub-generator to learn the distribution of the video at different levels of resolution. We also need only a few GPUs to train a highly complex generator that far outperforms the predecessor in terms of inception scores.

Early prediction for mode anomaly in generative adversarial network training: An empirical study

Mode anomaly (MA for short) significantly blocks the application of generative adversarial networks (GANs). Although diverse metrics have been proposed to measure the MA, and a lot of efforts have been made to resolve the MA, none of them gives a quantitative definition for MA detection. Moreover, very few studies concentrate on the early-stage prediction of MA. In this paper, we make the first effort to this field with a systematic empirical study. To this end, we first give a fine-grained definition where the MA is categorized into three typical sub-patterns. Afterwards, traditional MA metrics are studied with extensive experiments on numbers of representative combinations of subjects (including 13 GANs and 3 datasets) to explore their sensitivity for the MA across different training steps. We find that in most of cases, the MA can be reasonably predicted in very early training stage through our sensitivity studies. Under the insight, we propose a novel prediction strategy using conception of “anomaly sign”. The evaluation results on diverse experimental subjects demonstrate the feasibility and high accuracy for the early prediction of MA. We also discuss the prediction efficiency, as well as analyze the prediction effectiveness from human perception.

Single image super resolution via wavelet transform fusion and SRFeat network

Image super resolution is a vital research topic in the field of computer vision. It aims to reconstruct high resolution images from low resolution images. Although the conventional image super resolution methods have achieved good performance and effect, there are still have some issues, e.g., the high-frequency details information is insufficient, and the reconstruction process will bring additional noise, and most basic interpolation techniques produce blurry results. To settle the problems mentioned above, we consider combining the deep learning method with the frequency domain fusion method. In this paper, a novel single image super resolution method based on SRFeat network and wavelet fusion is proposed. First, the training image is taken as the input of the backbone SRFeat network, then the generative adversarial network training is carried out. Then, the up-sampling is utilized to obtain the coarse super resolved image. Finally, the output image after the network training is combined with the up-sampling image of the low-resolution image by Wavelet fusion to obtain the final result. Without increasing the depth of the network and the redundant parameters, the proposed method can achieve better reconstruct result. The experimental results show that the proposed method can not only reduce the probability of image distortion, but recover the global information of the reconstructed image and remove the noise brought by the reconstruction process. The PSNR value of the proposed method is improved 0.3 dB, and the SSIM is improved 0.02.

Speaker recognition based on short utterance compensation method of generative adversarial networks

On the basis of gaussian mixture model–universal background model (GMM–UBM) in the speaker recognition system, the paper proposes a short utterance sample compensation method based on the generative adversarial network (GAN) to solve the problem of the inadequate corpus data caused by short utterance, which has led to a serious reduction of recognition rate. The presented method compensates the short utterance samples into the speech samples with sufficient speaker identity information by completing the antagonistic training of generator network and discriminator network. In order to avoid the model crash and gradient instability in the process of GAN training, this paper adopts the condition information in the conditional GAN to guide the compensation process of the generator network, and proposes the generator compensation performance measurement training task and the feature tag training task of the discriminator to stabilize training process. Finally, the proposed short utterance compensation method is evaluated on the speaker recognition system based on GMM–UBM. The experimental results indicate that the presented method can effectively reduce the equal error rate of the speaker recognition system in short utterance environment.

Privacy-aware offloading for training tasks of generative adversarial network in edge computing

Currently, the generative adversarial network (GAN), with complex training processes in the physical machine (PM), has achieved great priority in image generation, audio conversion, image translation, etc. To improve the training efficiency of GAN, the edge computing paradigm is accepted as an alternative of the PMs to accommodate the training tasks, that is, the training tasks are migrated to the edge nodes (ENs) for hosting. However, it is still a key challenge to keep the overall network performance (i.e., load balance, transmission time) and privacy protection of training tasks at the same time. To address this challenge, a privacy-aware task offloading method, named POM, is developed accordingly in this paper. First, improving the strength pareto evolutionary algorithm (SPEA2) is fully investigated to obtain the offloading strategies for collaboratively improving the training performance and privacy preservation. Then, the most balanced offloading strategy is acquired for training GAN. Eventually, systematic experiments indicate that POM achieves an optimal performance efficiently among the other representative benchmark methods.

Multiautonomous underwater vehicle consistent collaborative hunting method based on generative adversarial network

The time-varying ocean currents and the delay of underwater acoustic communication have caused the uncertainty of single autonomous underwater vehicle (AUV) tracking target and the inconsistency of multi-AUV coordination, which make it difficult for multiple AUVs to form a hunting alliance. To solve the above problems, this article proposes the multi-AUV consistent collaborative hunting method based on generative adversarial network (GAN). Firstly, the three-dimensional (3D) kinematic model of AUV is established for the underwater 3D environment. Secondly, combined with the Laplacian matrix, the topology of the hunting alliance in the ideal environment is established, and the control rate of AUV is calculated. Finally, using the GAN network model, the control relationship after environmental interference is used as the input of the generative model. The control rate in the ideal environment is used as the comparison object of the discriminative model. Using the iterative training of GAN to generate a control rate that adapts to the current interference environment and combining multi-AUV topological hunting model to achieve successful hunting of noncooperative target, the experimental results show that the algorithm reduces the average hunting time to 62.53 s and the success rate of hunting is increased to 84.69%, which is 1.17% higher than the particle swarm optimization-constant modulus algorithm (PSO-CMA) algorithm.

Training GANs with centripetal acceleration

ABSTRACT Training generative adversarial networks (GANs) often suffers from cyclic behaviours of iterates. Based on a simple intuition that the direction of centripetal acceleration of an object moving in uniform circular motion is toward the centre of the circle, we present the Simultaneous Centripetal Acceleration (SCA) method and the Alternating Centripetal Acceleration (ACA) method to alleviate the cyclic behaviours. Under suitable conditions, gradient descent methods with either SCA or ACA are shown to be linearly convergent for bilinear games. Numerical experiments are conducted by applying ACA to existing gradient-based algorithms in a GAN setup scenario, which demonstrate the superiority of ACA.

Utilizing Amari-Alpha Divergence to Stabilize the Training of Generative Adversarial Networks.

Generative Adversarial Nets (GANs) are one of the most popular architectures for image generation, which has achieved significant progress in generating high-resolution, diverse image samples. The normal GANs are supposed to minimize the Kullback–Leibler divergence between distributions of natural and generated images. In this paper, we propose the Alpha-divergence Generative Adversarial Net (Alpha-GAN) which adopts the alpha divergence as the minimization objective function of generators. The alpha divergence can be regarded as a generalization of the Kullback–Leibler divergence, Pearson divergence, Hellinger divergence, etc. Our Alpha-GAN employs the power function as the form of adversarial loss for the discriminator with two-order indexes. These hyper-parameters make our model more flexible to trade off between the generated and target distributions. We further give a theoretical analysis of how to select these hyper-parameters to balance the training stability and the quality of generated images. Extensive experiments of Alpha-GAN are performed on SVHN and CelebA datasets, and evaluation results show the stability of Alpha-GAN. The generated samples are also competitive compared with the state-of-the-art approaches.

Rank3DGAN: Semantic Mesh Generation Using Relative Attributes

In this paper, we investigate a novel problem of using generative adversarial networks in the task of 3D shape generation according to semantic attributes. Recent works map 3D shapes into 2D parameter domain, which enables training Generative Adversarial Networks (GANs) for 3D shape generation task. We extend these architectures to the conditional setting, where we generate 3D shapes with respect to subjective attributes defined by the user. Given pairwise comparisons of 3D shapes, our model performs two tasks: it learns a generative model with a controlled latent space, and a ranking function for the 3D shapes based on their multi-chart representation in 2D. The capability of the model is demonstrated with experiments on HumanShape, Basel Face Model and reconstructed 3D CUB datasets. We also present various applications that benefit from our model, such as multi-attribute exploration, mesh editing, and mesh attribute transfer.

Virtual optofluidic time-stretch quantitative phase imaging

Optofluidic time-stretch quantitative phase imaging (OTS-QPI) is a potent tool for biomedical applications as it enables high-throughput QPI of numerous cells for large-scale single-cell analysis in a label-free manner. However, there are a few critical limitations that hinder OTS-QPI from being widely applied to diverse applications, such as its costly instrumentation and inherent phase-unwrapping errors. Here, to overcome the limitations, we present a QPI-free OTS-QPI method that generates “virtual” phase images from their corresponding bright-field images by using a deep neural network trained with numerous pairs of bright-field and phase images. Specifically, our trained generative adversarial network model generated virtual phase images with high similarity (structural similarity index >0.7) to their corresponding real phase images. This was also supported by our successful classification of various types of leukemia cells and white blood cells via their virtual phase images. The virtual OTS-QPI method is highly reliable and cost-effective and is therefore expected to enhance the applicability of OTS microscopy in diverse research areas, such as cancer biology, precision medicine, and green energy.

Personalized Sentence Generation using Generative Adversarial Networks with Author-Specific Word Usage

The author-specific word usage is a vital feature to let readers perceive the writing style of the author. In this work, a personalized sentence generation method based on generative adversarial networks (GANs) is proposed to cope with this issue. The frequently used function word and content word are incorporated not only as the input features but also as the sentence structure constraint for the GAN training. For the sentence generation with the related topics decided by the user, the Named Entity Recognition (NER) information of the input words is also used in the network training. We compared the proposed method with the GAN-based sentence generation methods, and the experimental results showed that the generated sentences using our method are more similar to the original sentences of the same author based on the objective evaluation such as BLEU and SimHash score.

Personalized Sentence Generation using Generative Adversarial Networks with Author-Specific Word Usage

The author-specific word usage is a vital feature to let readers perceive the writing style of the author. In this work, a personalized sentence generation method based on generative adversarial networks (GANs) is proposed to cope with this issue. The frequently used function word and content word are incorporated not only as the input features but also as the sentence structure constraint for the GAN training. For the sentence generation with the related topics decided by the user, the Named Entity Recognition (NER) information of the input words is also used in the network training. We compared the proposed method with the GAN-based sentence generation methods, and the experimental results showed that the generated sentences using our method are more similar to the original sentences of the same author based on the objective evaluation such as BLEU and SimHash score.

ApGAN: Approximate GAN for Robust Low Energy Learning From Imprecise Components

A Generative Adversarial Network (GAN) is an adversarial learning approach which empowers conventional deep learning methods by alleviating the demands of massive labeled datasets. However, GAN training can be computationally-intensive limiting its feasibility in resource-limited edge devices. In this paper, we propose an approximate GAN (ApGAN) for accelerating GANs from both algorithm and hardware implementation perspectives. First, inspired by the binary pattern feature extraction method along with binarized representation entropy, the existing Deep Convolutional GAN (DCGAN) algorithm is modified by binarizing the weights for a specific portion of layers within both the generator and discriminator models. Further reduction in storage and computation resources is achieved by leveraging a novel hardware-configurable in-memory addition scheme, which can operate in the accurate and approximate modes. Finally, a memristor-based processing-in-memory accelerator for ApGAN is developed. The performance of the ApGAN accelerator on different data-sets such as Fashion-MNIST, CIFAR-10, STL-10, and celeb-A is evaluated and compared with recent GAN accelerator designs. With almost the same Inception Score (IS) to the baseline GAN, the ApGAN accelerator can increase the energy-efficiency by ~28.6× achieving 35-fold speedup compared with a baseline GPU platform. Additionally, it shows 2.5× and 5.8× higher energy-efficiency and speedup over CMOS-ASIC accelerator subject to an 11 percent reduction in IS.

Incremental focal loss GANs

Generative Adversarial Networks (GANs) have achieved inspiring performance in both unsupervised image generation and conditional cross-modal image translation. However, how to generate quality images at an affordable cost is still challenging. We argue that it is the vast number of easy examples that disturb training of GANs, and propose to address this problem by down-weighting losses assigned to easy examples. Our novel Incremental Focal Loss (IFL) progressively focuses training on hard examples and prevents easy examples from overwhelming the generator and discriminator during training. In addition, we propose an enhanced self-attention (ESA) mechanism to boost the representational capacity of the generator. We apply IFL and ESA to a number of unsupervised and conditional GANs, and conduct experiments on various tasks, including face photo-sketch synthesis, map↔aerial-photo translation, single image super-resolution reconstruction, and image generation on CelebA, LSUN, and CIFAR-10. Results show that IFL boosts learning of GANs over existing loss functions. Besides, both IFL and ESA make GANs produce quality images with realistic details in all these tasks, even when no task adaptation is involved.

Lessons Learned From the Training of GANs on Artificial Datasets

Generative Adversarial Networks (GANs) have made great progress in synthesizing realistic images in recent years. However, they are often trained on image datasets with either too few samples or too many classes belonging to different data distributions. Consequently, GANs are prone to underfitting or overfitting, making the analysis of them difficult and constrained. Therefore, in order to conduct a thorough study on GANs while obviating unnecessary interferences introduced by the datasets, we train them on artificial datasets where there are infinitely many samples and the real data distributions are simple, high-dimensional and have structured manifolds. Moreover, the generators are designed such that optimal sets of parameters exist. Empirically, we find that under various distance measures, the generator fails to learn such parameters with the GAN training procedure. We also find that training mixtures of GANs leads to more performance gain compared to increasing the network depth or width when the model complexity is high enough. Our experimental results demonstrate that a mixture of generators can discover different modes or different classes automatically in an unsupervised setting, which we attribute to the distribution of the generation and discrimination tasks across multiple generators and discriminators. As an example of the generalizability of our conclusions to realistic datasets, we train a mixture of GANs on the CIFAR-10 dataset and our method significantly outperforms the state-of-the-art in terms of popular metrics, i.e., Inception Score (IS) and Frechet Inception Distance (FID).

On Gradient-Based Learning in Continuous Games

We formulate a general framework for competitive gradient-based learning that encompasses a wide breadth of multi-agent learning algorithms, and analyze the limiting behavior of competitive gradient-based learning algorithms using dynamical systems theory. For both general-sum and potential games, we characterize a non-negligible subset of the local Nash equilibria that will be avoided if each agent employs a gradient-based learning algorithm. We also shed light on the issue of convergence to non-Nash strategies in general- and zero-sum games, which may have no relevance to the underlying game, and arise solely due to the choice of algorithm. The existence and frequency of such strategies may explain some of the difficulties encountered when using gradient descent in zero-sum games as, e.g., in the training of generative adversarial networks. To reinforce the theoretical contributions, we provide empirical results that highlight the frequency of linear quadratic dynamic games (a benchmark for multi-agent reinforcement learning) that admit global Nash equilibria that are almost surely avoided by policy gradient.

Adversarial Learning for Invertible Steganography

Deep neural networks have revolutionised the research landscape of steganography. However, their potential has not been explored in invertible steganography, a special class of methods that permits the recovery of distorted objects due to steganographic perturbations to their pristine condition. In this paper, we revisit the regular-singular (RS) method and show that this elegant but obsolete invertible steganographic method can be reinvigorated and brought forwards to modern generation via neuralisation. Towards developing a renewed RS method, we introduce adversarial learning to capture the regularity of natural images automatically in contrast to handcrafted discrimination functions based on heuristic image prior. Specifically, we train generative adversarial networks (GANs) to predict bit-planes that have been used to carry hidden information. We then form a synthetic image and use it as a reference to provide guidance on data embedding and image recovery. Experimental results showed a significant improvement over the prior implementation of the RS method based on large-scale statistical evaluations.

Semi-Supervised Texture Filtering With Shallow to Deep Understanding

This work proposed a semi-supervised method for automatic texture filtering. Our method leveraged a limited amount of labeled data and a large amount of unlabeled data to train Generative Adversarial Networks (GANs). Separate loss functions were designed for both labeled and unlabeled datasets. Our main contribution is the introduction of knowledge extracted from shallow and deep layers in neural networks. Loss defined within shallow layers preserves the edge, while loss defined within the deep layers identifies the semantic content and conversely removes the small-scale texture variations. This contribution directly addresses the major challenge for texture filtering, distinguishing the structural content from non-structural textures at the pixel level. The extracted information, in our study, improved the content and color consistency before and after the process of filtering, for unlabeled samples in particular. The proposed method offers twofold benefits: first, significant reductions in the amounts of time and effort expended in reconstructing the labeled dataset, especially given the delicate operations required at the pixel level; second, a reduction in over-fitting, in supervised learning with a small amount of labeled data, by utilizing a large amount of unlabeled data. The results confirm that our method can perform comparably with non-learning-based methods, alleviating the demand for the determination of optimal parameter values.

Spectral bounding: Strictly satisfying the 1-Lipschitz property for generative adversarial networks

Imposing the 1-Lipschitz constraint is a problem of key importance in the training of Generative Adversarial Networks (GANs), which has been proved to productively improve stability of GAN training. Although some interesting alternative methods have been proposed to enforce the 1-Lipschitz property, these existing approaches (e.g., weight clipping, gradient penalty (GP), and spectral normalization (SN)) are only partially successful. In this paper, we propose a novel method, which we refer to as spectral bounding (SB) to strictly enforce the 1-Lipschitz constraint. Our method adopts very cost-effective terms of both 1-norm and ∞-norm, and yet allows us to efficiently approximate the upper bound of spectral norms. In this way, our method provide important insights to the relationship between an alternative of strictly satisfying the Lipschitz property and explainable training stability improvements of GAN. Our proposed method thus significantly enhances the stability of GAN training and the quality of generated images. Extensive experiments are conducted, showing that the proposed method outperforms GP and SN on both CIFAR-10 and ILSVRC2015 (ImagetNet) dataset in terms of the standard inception score.