Original Generative Adversarial Networks Research Articles

Closing the Gap Between Theory and Practice During Alternating Optimization for GANs.

Synthesizing high-quality and diverse samples is the main goal of generative models. Despite recent great progress in generative adversarial networks (GANs), mode collapse is still an open problem, and mitigating it will benefit the generator to better capture the target data distribution. This article rethinks alternating optimization in GANs, which is a classic approach to training GANs in practice. We find that the theory presented in the original GANs does not accommodate this practical solution. Under the alternating optimization manner, the vanilla loss function provides an inappropriate objective for the generator. This objective forces the generator to produce the output with the highest discriminative probability of the discriminator, which leads to mode collapse in GANs. To address this problem, we introduce a novel loss function for the generator to adapt to the alternating optimization nature. When updating the generator by the proposed loss function, the reverse Kullback-Leibler divergence between the model distribution and the target distribution is theoretically optimized, which encourages the model to learn the target distribution. The results of extensive experiments demonstrate that our approach can consistently boost model performance on various datasets and network structures.

Side-Scan Sonar Image Augmentation Method Based on CC-WGAN

The utilization of deep learning algorithms for side-scan sonar target detection is impeded by the restricted quantity and representativeness of side-scan sonar (SSS) samples. To address this issue, this paper proposes a method for image augmentation using a CC-WGAN network. First, the generator incorporates the Convolutional Block Attention Module (CBAM) to enhance the assimilation of global information and local features in the input images. This integration also improves stability and avoids mode collapse problems associated with the original Generative Adversarial Network. Subsequently, the CBAM is incorporated into the discriminator to facilitate a better understanding of the relevance and significance of input data, thereby enhancing the model’s generalization ability. Finally, based on this model, existing few-sample SSS images are augmented, and we utilize the augmented images for discrimination and detection with YOLOv5. The experimental results show that following training with the SSS dataset that is augmented by this network, the accuracy of target detection increased by 7.6%, validating the feasibility of our proposed method. This method presents a novel solution to the problem of low model accuracy in underwater target detection with side-scan sonar due to limited samples.

Super-resolution reconstruction method of the optical synthetic aperture image using generative adversarial network

Abstract In order to solve the contradiction between large aperture elements and high-resolution images, in this study, we propose an improved image-resolution method based on generative adversarial network (GAN). First, we analyze the imaging principle of the optical synthetic aperture. Further, we improve a super-resolution GAN; especially, this network uses a multi-scale convolutional cascade to obtain global features of the image, and a multi-scale receptive field block and residual in residual dense block are built to obtain image details. In addition, this study uses the Mish function as the activation function of the discriminator to solve the problems of neuron extreme, gradient explosion, and poor generalization ability of the model. Through simulation, the results show that the proposed method can achieve a peak signal-to-noise ratio (PSNR) of 30 dB compared with traditional image super-resolution reconstruction methods for synthetic aperture image. The method proposed has an improvement of 2 dB in the PSNR and 0.016 in structure similarity index measure compared with the original super-resolution GAN. Therefore, this method can effectively reduce the image distortion and improve the quality of image reconstruction.

A Unifying Generator Loss Function for Generative Adversarial Networks.

A unifying α-parametrized generator loss function is introduced for a dual-objective generative adversarial network (GAN) that uses a canonical (or classical) discriminator loss function such as the one in the original GAN (VanillaGAN) system. The generator loss function is based on a symmetric class probability estimation type function, Lα, and the resulting GAN system is termed Lα-GAN. Under an optimal discriminator, it is shown that the generator's optimization problem consists of minimizing a Jensen-fα-divergence, a natural generalization of the Jensen-Shannon divergence, where fα is a convex function expressed in terms of the loss function Lα. It is also demonstrated that this Lα-GAN problem recovers as special cases a number of GAN problems in the literature, including VanillaGAN, least squares GAN (LSGAN), least kth-order GAN (LkGAN), and the recently introduced (αD,αG)-GAN with αD=1. Finally, experimental results are provided for three datasets-MNIST, CIFAR-10, and Stacked MNIST-to illustrate the performance of various examples of the Lα-GAN system.

A time series continuous missing values imputation method based on generative adversarial networks

Generative adversarial networks (GANs) have been widely utilized in time series analysis and modeling, wherein generators and discriminators interact to generate realistic data. However, when addressing the challenge of imputing continuous missing values in time series, the generator struggles to learn meaningful features due to the loss of local information, and the discriminator’s loss function exhibits significant deviations in determining the probability, making it difficult to effectively update model parameters during training. In response to these issues, this study presents a novel imputation model called cue Wasserstein generative adversarial network with gradient penalty (CWGAIN-GP). CWGAIN-GP incorporates the contextual cue information matrixs into the generator, enabling obtain and capture the potential data evolution features that hidden beyond the missing positions. This approach constrains the generator’s generation results to be closer to the true values probability distribution. Meanwhile, it optimizes the generator and discriminator neural network structure and loss function computation strategy of the original GAN model, which not only improves the accuracy of continuous missing data interpolation, but also improves the training stability of the model. Finally, We used three imputation accuracies on two real-world datasets, including experiments on imputation accuracy for eight comparison models, ablation study, and experiments on the effect of consecutive missing lengths. The experimental results show that CWGAIN-GP achieves impressive performance in imputing continuous missing time series.

Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis

Generative Adversarial Networks (GAN) is a model for data synthesis, which creates plausible data through the competition of generator and discriminator. Although GAN application to image synthesis is extensively studied, it has inherent limitations to natural language generation. Because natural language is composed of discrete tokens, a generator has difficulty updating its gradient through backpropagation; therefore, most text-GAN studies generate sentences starting with a random token based on a reward system. Thus, the generators of previous studies are pre-trained in an autoregressive way before adversarial training, causing data memorization that synthesized sentences reproduce the training data. In this paper, we synthesize sentences using a framework similar to the original GAN. More specifically, we propose Text Embedding Space Generative Adversarial Networks (TESGAN) which generate continuous text embedding spaces instead of discrete tokens to solve the gradient backpropagation problem. Furthermore, TESGAN conducts unsupervised learning which does not directly refer to the text of the training data to overcome the data memorization issue. By adopting this novel method, TESGAN can synthesize new sentences, showing the potential of unsupervised learning for text synthesis. We expect to see extended research combining Large Language Models with a new perspective of viewing text as an continuous space.

FontFusionGAN: Refinement of Handwritten Fonts by Font Fusion

Handwritten fonts possess unique expressive qualities; however, their clarity often suffers because of inconsistent handwriting. This study introduces FontFusionGAN (FFGAN), a novel method that enhances handwritten fonts by mixing them with printed fonts. The proposed approach leverages a generative adversarial network (GAN) to synthesize fonts that combine the desirable features of both handwritten and printed font styles. Training a GAN on a comprehensive dataset of handwritten and printed fonts enables it to produce legible and visually appealing font samples. The methodology was applied to a dataset of handwriting fonts, showing substantial enhancements in the legibility of the original fonts, while retaining their unique aesthetic essence. Unlike the original GAN setting where a single noise vector is used to generate a sample image, we randomly selected two noise vectors, z1 and z2, from a Gaussian distribution to train the generator. Simultaneously, we input a real image into the fusion encoder for exact reconstruction. This technique ensured the learning of style mixing during training. During inference, we provided the encoder with two font images, one handwritten and the other printed font, to obtain their respective latent vectors. Subsequently, the latent vector of the handwritten font image was injected into the first five layers of the generator, whereas the latent vector of the printed font image was injected into the last two layers to obtain a refined handwritten font image. The proposed method has the potential to improve the readability of handwritten fonts, offering benefits across diverse applications, such as document composition, letter writing, and assisting individuals with reading and writing difficulties.

Understanding the mathematical background of Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs) have gained widespread attention since their introduction, leading to numerous extensions and applications of the original GAN idea. A thorough understanding of GANs' mathematical foundations is necessary to use and build upon these techniques. However, most studies on GANs are presented from a computer science or engineering perspective, which can be challenging for beginners to understand fully. Therefore, this paper aims to provide an overview of the mathematical background of GANs, including detailed proofs of optimal solutions for vanilla GANs and boundaries for $f$-GANs that minimize a variational approximation of the $f$-divergence between two distributions. These contributions will enhance the understanding of GANs for those with a mathematical background and pave the way for future research.

Handwritten Digit Image Generation Using Improved Generative Neural Network

Generative adversarial network (GAN) is a new and effective neural network model. GANs are used in a lot of areas such as images and visual, speech and language, composing music, and creating 3D shapes. Although the GAN has a lot of applications, there are still some problems with it. Because not only the generator but also the discriminator needs to be trained, this model is easy to be unstable. And the GAN may not generate different images in human vision to make the generated images can safely pass the discriminator. Some methods will be shown to solve the problems of unstable and lack of diversity in this article by an example that generates handwritten digits. The methods are changing the optimizer from SGD to Adam or SGD with Adam, adding a Batchnorm, and combining GAN with CNN or RNN, each of the improvements will be contrasted with others by the chart of loss rate and the generated images. The research finds that each of the improvements has some advantages over the original GAN in the area of speed or stability. Especially, GAN with the optimizer Adam and DCGAN have the most stable consequence according to the image from the generator.

ATLAS: GAN-Based Differentially Private Multi-Party Data Sharing

In this paper, we study the problem of differentially private multi-party data sharing, where the involved parties assisted by a semi-honest curator collectively generate a shared dataset while satisfying differential privacy. Inspired by the success of data synthesis with the generative adversarial network (GAN), we propose a novel GAN-based differentially private multi-party data sharing approach named ATLAS. In ATLAS, we extend the original GAN to multiple discriminators, and let each party hold a discriminator while the curator holds a generator. To update the generator without compromising each party's privacy, we decompose the calculation of the generator's gradient and selectively sanitize the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">discriminators' responses</i> . Additionally, we propose two methods to improve the utility of shared data, i.e., the collaborative discriminator filtering (CDF) method and the adaptive gradient perturbation (AGP) method. Specifically, the CDF method utilizes trained discriminators to refine synthetic records, while the AGP method adaptively adjusts the noise scale during training to reduce the impact of deferentially private noise on the final shared data. Extensive experiments on real-world datasets validate the superiority of our ATLAS approach.

Single‐image snow removal algorithm based on generative adversarial networks

AbstractThe effect of snowfall on an image is not only the interference of snow particles but also snow streaks and masking effects (similar to haze). Snowy weather severely reduces the accuracy of computer vision systems. There is a lot of interest in how to effectively remove snow while preserving as much of the original image information as possible. Based on this, the authors propose an effective Generative Adversarial Network (GAN) snow removal algorithm for single images to solve the snow removal failure problem caused by irregular snow particles and snow streaks. Specifically, the authors improve the original GAN network as follows: A novel Transformer module, the Contextual Transformer (CoT) module, is adopted in the residual modules based generator. It effectively uses the contextual information of the snow streaks neighbourhood to restore the texture and information in the noisy image as much as possible based on the focus on snow streaks features. Also, using learnable Regionalized Normalization (RN‐L), potentially corrupted and undamaged regions are automatically detected for separate normalization, and global affine transformations are performed to enhance their fusion. In addition, a multi‐scale discriminator is used in the discriminator to make the discrimination more adequate and retain more details. Extensive experiments have shown that the authors’ GAN network snow removal algorithm outperforms various current networks on snow removal studies in terms of evaluation metrics on both synthetic and real datasets.

IDGAN: Information-Driven Generative Adversarial Network of Coverless Image Steganography

Traditional image steganography techniques complete the steganography process by embedding secret information into cover images, but steganalysis tools can easily detect detectable pixel changes that lead to the leakage of confidential information. The use of a generative adversarial network (GAN) makes it possible to embed information using a combination of information and noise in generating images to achieve steganography. However, this approach is usually accompanied by issues such as poor image quality and low steganography capacity. To address these challenges, we propose a steganography model based on a novel information-driven generative adversarial network (IDGAN), which fuses a GAN, attention mechanisms, and image interpolation techniques. We introduced an attention mechanism on top of the original GAN model to improve image accuracy. In the generation model, we replaced some transposed convolution operations with image interpolation for better quality of dense images. In contrast to traditional steganographic methods, the IDGAN generates images containing confidential information without using cover images and utilizes GANs for information embedding, thus having better anti-detection capability. Moreover, the IDGAN uses an attention mechanism to improve the image details and clarity and optimizes the steganography effect through an image interpolation algorithm. Experimental results demonstrate that the IDGAN achieves an accuracy of 99.4%, 95.4%, 93.2%, and 100% on the MNIST, Intel Image Classification, Flowers, and Face datasets, respectively, with an embedding rate of 0.17 bpp. The model effectively protects confidential information while maintaining high image quality.

A Systematic Survey of Regularization and Normalization in GANs

Generative Adversarial Networks (GANs) have been widely applied in different scenarios thanks to the development of deep neural networks. The original GAN was proposed based on the non-parametric assumption of the infinite capacity of networks. However, it is still unknown whether GANs can fit the target distribution without any prior information. Due to the overconfident assumption, many issues remain unaddressed in GANs training, such as non-convergence, mode collapses, and gradient vanishing. Regularization and normalization are common methods of introducing prior information to stabilize training and improve discrimination. Although a handful number of regularization and normalization methods have been proposed for GANs, to the best of our knowledge, there exists no comprehensive survey that primarily focuses on objectives and development of these methods, apart from some incomprehensive and limited-scope studies. In this work, we conduct a comprehensive survey on the regularization and normalization techniques from different perspectives of GANs training. First, we systematically describe different perspectives of GANs training and thus obtain the different objectives of regularization and normalization. Based on these objectives, we propose a new taxonomy. Furthermore, we compare the performance of the mainstream methods on different datasets and investigate the applications of regularization and normalization techniques that have been frequently employed in state-of-the-art GANs. Finally, we highlight potential future directions of research in this domain. Code and studies related to the regularization and normalization of GANs in this work are summarized at https://github.com/iceli1007/GANs-Regularization-Review .

An improved generative adversarial networks for remote sensing image super-resolution reconstruction via multi-scale residual block

Existing image super-resolution algorithms still suffer from the problems of not extracting rich image features and losing realistic high-frequency details. In order to solve these problems, this paper proposes an improved generative adversarial network algorithm for super-resolution reconstruction of remote sensing images by multi-scale residual blocks. The original generative adversarial network (GAN) structure is improved and multi-scale residual blocks are introduced in the generator to fuse features at different scales. After extracting the parallel information of multi-scale features, information is exchanged between multi-resolution information streams to obtain contextual information through spatial and channel attention mechanisms, and multi-scale features are fused according to the attention mechanism. In the discriminator, the concept of relative average GAN (RaGAN) is introduced, and the loss function of the network is redesigned so that the discriminator can predict relative probabilities instead of absolute probabilities thus enabling clear learning of edge and texture details. Experimental results show that the proposed method in this paper significantly outperforms state-of-the-art (SOTA) methods in terms of both subjective and objective metrics.In three test datasets, compared with SOTA methods, the Peak Signal to Noise Ratio(PSNR) is improved by a maximum of 1.18 dB, 0.84 dB and 1.29 dB respectively, and the Structural Similarity Index (SSIM) is improved by 0.0264, 0.0077 and 0.0109 respectively in scale of 2, 3 and 4 times images super-resolution.The model proposed in this paper effectively improved the super-resolution re-construction results of remote sensing images.

Face Generation Using General Adversial Networks

Face generation is a difficult task with numerous uses in security (for example, creating portraits from descriptions), styling, and entertainment. In this study, we investigate generating faces conditioned on identification using modifications to generative adversarial networks (GANs). In the realm of artificial intelligence, deep learning has had significant success, and several deep learning models have been created. One of the deep learning models, Generative Adversarial Networks (GAN), which was proposed based on zero-sum game theory and has become a new research hotspot. The importance of the GAN model variation is to provide more accurate and realistic data by using unsupervised learning to obtain the data distribution. Due to their vast potential for use in areas including language processing, video processing, and image and vision computing, GANs are currently the subject of extensive research. This project introduces the history of the GAN, theoretical models, and extensional variants of GANs, where the variants can either modify the fundamental structures of the original GAN or further optimise it. The typical uses of GANs are then described. Finally, a summary of the current GAN issues and a description of the models' upcoming work are provided.

A Novel Model to Generate Heterogeneous and Realistic Time-Series Data for Post-Stroke Rehabilitation Assessment.

The application of machine learning-based tele-rehabilitation faces the challenge of limited availability of data. To overcome this challenge, data augmentation techniques are commonly employed to generate synthetic data that reflect the configurations of real data. One such promising data augmentation technique is the Generative Adversarial Network (GAN). However, GANs have been found to suffer from mode collapse, a common issue where the generated data fails to capture all the relevant information from the original dataset. In this paper, we aim to address the problem of mode collapse in GAN-based data augmentation techniques for post-stroke assessment. We applied the GAN to generate synthetic data for two post-stroke rehabilitation datasets and observed that the original GAN suffered from mode collapse, as expected. To address this issue, we propose a Time Series Siamese GAN (TS-SGAN) that incorporates a Siamese network and an additional discriminator. Our analysis, using the longest common sub-sequence (LCSS), demonstrates that TS-SGAN generates data uniformly for all elements of two testing datasets, in contrast to the original GAN. To further evaluate the effectiveness of TS-SGAN, we encode the generated dataset into images using Gramian Angular Field and classify them using ResNet-18. Our results show that TS-SGAN achieves a significant accuracy increase of classification accuracy (35.2%-42.07%) for both selected datasets. This represents a substantial improvement over the original GAN.

GAN-CL: Generative Adversarial Networks for Learning From Complementary Labels.

Learning from complementary labels (CLs) is a useful learning paradigm, where the CL specifies the classes that the instance does not belong to, instead of providing the ground truth as in the ordinary supervised learning scenario. In general, although it is less laborious and more efficient to collect CLs compared with ordinary labels, the less informative signal in the complementary supervision is less helpful to learn competent feature representation. Consequently, the final classifier's performance greatly deteriorates. In this article, we leverage generative adversarial networks (GANs) to derive an algorithm GAN-CL to effectively learn from CLs. In addition to the role in original GAN, the discriminator also serves as a normal classifier in GAN-CL, with the objective constructed partly with the complementary information. To further prove the effectiveness of our schema, we study the global optimality of both generator and discriminator for the GAN-CL under mild assumptions. We conduct extensive experiments on benchmark image datasets using deep models, to demonstrate the compelling improvements, compared with state-of-the-art CL learning approaches.

Small sample classification based on data enhancement and its application in flip chip defection

With the development of chip integration, scanning acoustic microscopy (SAM) technology is widely used in the intelligent inspection of chips. However, only a few SAM images of the chip are acquired for the SAM test, which will lead to a lack of sufficient data for intelligent recognition of the chip defects. To solve this problem, a generative adversarial network with the addition of a residual attention module, RA-GAN, was proposed and applied to a small sample. Firstly, the original generative adversarial network (GAN) was constructed and the residual attention module was added to GAN. Secondly, the part of the generator network was improved for the SAM images of flip chip, which increases the perceptual field and enriches the details of the generated images. Then, the appropriate learning rate was adjusted during the network training phase, which can prevent the overfitting of the model. Finally, an enhanced dataset with improved image quality and quantity was obtained. To evaluate the effectiveness of the reconstructed dataset, deep learning algorithms were used for SAM image classification. The results validate the effectiveness and advancement of combining RA-GAN with traditional image data enhancement, which can be applied for small sample classification in engineering applications.

Feedback-AVPGAN: Feedback-guided generative adversarial network for generating antiviral peptides.

In this study, we propose Feedback-AVPGAN, a system that aims to computationally generate novel antiviral peptides (AVPs). This system relies on the key premise of the Generative Adversarial Network (GAN) model and the Feedback method. GAN, a generative modeling approach that uses deep learning methods, comprises a generator and a discriminator. The generator is used to generate peptides; the generated proteins are fed to the discriminator to distinguish between the AVPs and non-AVPs. The original GAN design uses actual data to train the discriminator. However, not many AVPs have been experimentally obtained. To solve this problem, we used the Feedback method to allow the discriminator to learn from the existing as well as generated synthetic data. We implemented this method using a classifier module that classifies each peptide sequence generated by the GAN generator as AVP or non-AVP. The classifier uses the transformer network and achieves high classification accuracy. This mechanism enables the efficient generation of peptides with a high probability of exhibiting antiviral activity. Using the Feedback method, we evaluated various algorithms and their performance. Moreover, we modeled the structure of the generated peptides using AlphaFold2 and determined the peptides having similar physicochemical properties and structures to those of known AVPs, although with different sequences.

A literature review: Generative adversarial networks for civil structural health monitoring

Structural Health Monitoring (SHM) of civil structures has been constantly evolving with novel methods, advancements in data science, and more accessible technology to address issues related to structural safety, operations, and resiliency. Research and development in the civil SHM field during the last few decades have been progressive due to the increasing use of Artificial Intelligence (AI) methods such as Machine Learning (ML) and Deep Learning (DL). Particularly, Generative Adversarial Networks (GAN), which is a subfamily of Deep Learning has been highly favored in the SHM community within the last couple of years. After its release in 2014, GANs (original GAN and other GAN variants) have been in use for a wide variety of applications in various disciplines, and it has been one of the most popular research topics in the AI-ML domain. While there has not been a review study on the applications of GAN in the civil SHM field, this paper aims to fill this gap by presenting a literature review of the studies that employed GAN specifically in civil SHM applications from 2014 to date, in a condensed format. This study intends to inform SHM practitioners and researchers about GANs and present the highlights of the published work on GANs in the civil SHM field.