DANCE: Distributed Generative Adversarial Networks with Communication Compression

Optimization algorithms and objective functions play an important role in the training of deep learning networks. This paper explores the impact of using various optimization algorithms and objective functions for the training of different Generative Adversarial Networks. The paper first summarizes various Generative Adversarial Networks available in the literature. Various Generative Adversarial Networks are then evaluated for different objective functions and optimization algorithms. Empirically, Generative Adversarial Networks are analyzed here based on generator loss, discriminator loss, and accuracy metrics. The training of various Generative Adversarial Networks is analyzed on the MNIST dataset. The results indicate that Adam optimization algorithm and conditional objective function is a good choice for improved training of Generative Adversarial Networks.

Generative Adversarial Networks (GANs) have generated realistic and diverse facial images with promising results. This work demonstrates a technique for creating facial pictures using GANs and evaluates the effectiveness of several GAN designs and training approaches. The CelebA dataset is leveraged for training and evaluating the GAN model, and employ a variety of evaluation metrics, such as the Structural Similarity Index (SSIM) to assess the quality and diversity of the generated images. Progressive GAN outperforms Deep Convolutional GAN in terms of image quality and diversity, and conditional GAN training is more effective than standard GAN training for generating facial images with specific attributes. The combination of Progressive GAN and conditional GAN training produces facial images of the utmost quality and diversity. The findings contribute to a broader comprehension of the use of GANs for generating facial images and have ramifications for a variety of applications, from facial recognition to virtual reality.

As a new unsupervised learning algorithm framework, generative adversarial networks(GAN) have been favored by more and more researchers, and it has become a research hotspot now. GAN is inspired by the two-person zero-sum game theory in game theory. Its unique adversarial training idea can generate high-quality samples and has more powerful feature learning and feature expression capabilities than traditional machine learning algorithms. At present, GAN has achieved remarkable success in the field of computer vision, especially in the field of sample generation. Every year, a large number of GAN-related research papers are produced, reflecting the fiery degree of research on GAN model. Aiming at the hot model of GAN, first introduce the research status of GAN; then introduce the theory and framework of GAN, which analyzes in detail why the gradient disappears and the mode collapses during the training of GAN; then discussed some typical GAN improvement models, and summarized their theoretical improvements, advantages, limitations, application scenarios and implementation costs; Finally, the application results of GAN in data generation, image super-resolution, and image style conversion are shown, and the current challenges and future research directions of GAN are discussed.

<p>High data collection costs and complicated data access regulations increase the demand for synthetic data. Generative Adversarial Networks (GANs) are a novel generative framework with great potential for high quality synthetic data generation. GANs formulate the true distribution of data implicitly, and the success of GANs are often measured based on the similarity of generated data to this true distribution. GANs were originally designed to work with continuous data. However, many important real-world datasets such as medical images involve discontinuous distributions. GAN training for discontinuous distributions is relatively more challenging, as the training procedure often suffers from instability and mode collapse issues. This dissertation focuses on designing novel GAN architectures to generate representative synthetic image data, and proposes new structures to alleviate GANs' mode collapse issue. As part of this thesis, novel applications of image data generation with GANs have been also investigated for important problems arising in the telecommunication industry and medical domain. Specifically, we first explore various GAN structures to generate engineered electromagnetic surfaces. We consider the continuous approximation of the data and explore the capabilities of feed-forward and convolutional GANs for synthetic data generation. Next, we introduce a novel GAN architecture to address the problem of mode collapse in GAN training. The proposed structure incorporates a third network that penalizes the generator for generating low diversity samples. Lastly, we study the challenging problem of object generation in 3D space using GANs, and we propose extensions to existing 3D GAN structures to generate connected 3D volumes. Additionally, we explore a more challenging version of this 3D volume generation problem by generating connected volumes packed with spheres. This research has applications in radiosurgery treatment planning, and the proposed 3D GAN structure can help generate rare, unseen 3D tumor volumes and information on how to treat these tumors. Accordingly, our analysis contributes to overcoming data scarcity issues (e.g., due to privacy considerations) for an important practical problem in the medical domain.</p>

Optical proximity correction becomes more and more critical since the technology nodes shrinks nowadays. It usually costs a lot of computational power and days are needed to finish this process. Increasing its speed has become an important research topic. Machine learning technology has been applied to achieve this goal. Generative modelling such as generative adversarial networks appears to be beneficial and applicable in doing the optical proximity correction. We prepare the paired target layout and post OPC layout. The target layout is input into the U net type generator and its output is the calculated post OPC layout. The calculated post OPC layout and corresponding post OPC layout are input into the discriminator of the generative adversarial networks. The discriminator is trained to maximize the discriminative loss function, while the generator is trained to minimize the discriminative loss function. When the whole conditional generative adversarial networks converge, the generator can generate the calculated post OPC layouts quite similar to the prepared ones. The generalization capability of the deep neural network is important here. The generator can also provide good post OPC layout for unseen target layouts. However, the training of generative adversarial networks is difficult and often unstable. To improve this, we use Wasserstein distance as the loss function and stabilize the training and convergence of the conditional generative adversarial networks. We can obtain better results easier this way.

With increase in urbanization and Earth Sciences research into urban areas, the need to quickly and accurately segment urban rooftop maps has never been greater. Current machine learning techniques struggle to produce high accuracy maps in dense urban zones where there is high image noise and foot print overlap. In this paper, we evaluate a training methodology for pixel-wise segmentation for high-resolution satellite imagery using progressive growing of generative adversarial networks as a solution. We apply our model to segmenting building rooftops and compare these results to conventional methods for rooftop segmentation. We evaluate our approach using the SpaceNet version 2 and xView datasets. Our experiments show that for SpaceNet, progressive Generative Adversarial Network (GAN) training achieved a test accuracy of 93% compared to 89% for traditional GAN training and 87% for U-Net architecture, while for xView, we achieved 71% accuracy using progressive GAN training compared to 69% through traditional GAN training and 65% using U-Net.

In order to stabilize the training of generative adversarial networks, several recent works advocate spectral normalization in the discriminator. However, the method ignores the influence of the generator, and the quality of the images generated in practice is unstable. We propose L2 norm regularization in the generator based on the spectral normalization, which can solve the above shortcomings. Our method directly makes the generated data close to real data in Euclidean space, and indirectly helps the spectral normalization achieve tighter Lipschitz constraint during the training of generative adversarial networks. Our experiments on CIFAR-10 and STL-10 dataset confirm that our method can not only stable the quality of the images generated by spectral normalization, but also improve the quality of generated images.

Recent successes in Generative Adversarial Networks (GAN) have affirmed the importance of using more data in GAN training. Yet it is expensive to collect data in many domains such as medical applications. Data Augmentation (DA) has been applied in these applications. In this work, we first argue that the classical DA approach could mislead the generator to learn the distribution of the augmented data, which could be different from that of the original data. We then propose a principled framework, termed Data Augmentation Optimized for GAN (DAG), to enable the use of augmented data in GAN training to improve the learning of the original distribution. We provide theoretical analysis to show that using our proposed DAG aligns with the original GAN in minimizing the Jensen-Shannon (JS) divergence between the original distribution and model distribution. Importantly, the proposed DAG effectively leverages the augmented data to improve the learning of discriminator and generator. We conduct experiments to apply DAG to different GAN models: unconditional GAN, conditional GAN, self-supervised GAN and CycleGAN using datasets of natural images and medical images. The results show that DAG achieves consistent and considerable improvements across these models. Furthermore, when DAG is used in some GAN models, the system establishes state-of-the-art Fréchet Inception Distance (FID) scores. Our code is available (https://github.com/tntrung/dag-gans).

Generative adversarial networks (GANs) have been advancing and gaining tremendous interests from both academia and industry. With the development of wireless technologies, a huge amount of data generated at the network edge provides an unprecedented opportunity to develop GANs applications. However, due to the constraints such as bandwidth, privacy, and legal issues, it is inappropriate to collect and send all data to the cloud or servers for analysis, training, and mining. Thus, deploying and training GANs at the edge becomes a promising alternative solution. The instability of GANs introduced by non-independent and identical data (Non-IID) poses significant challenges to training GANs. To address these challenges, this paper presents a novel federated learning framework for GANs, namely, Collaborated gAme Parallel Learning (CAP). CAP supports parallel training of data and models for GANs, breaking the isolated training among generators that exists in the previous distributed algorithms, and achieving collaborative learning among cloud, edge servers, and devices. Then, to further enhance the ability of CAP-GAN for addressing Non-IID issues, we propose a Mix-Generator module (Mix-G) which divides a generator into the sharing layer and personalizing layer. The Mix-G module extracts the generic and personalization features and improves the performance of CAP GAN on extremely personalizing datasets. Experimental results and analysis substantiate the usefulness and superiority of our proposed CAP-GAN scheme which can achieve better results in the Non-IID scenarios compared with the state-of-the-art algorithms.

Our team is reviewing tools and techniques that enable rapid prototyping. Generative Adversarial Networks (GANs) have been shown to reduce training requirements for detection problems. GANs compete generative and discriminative classifiers to improve detection performance. This paper expands the use of GANs from detection (k=2) to classification (k>2) problems. Several GAN network structures and training set sizes were compared to the baseline discriminative network and Bayes' classifiers. The results show no significant performance differences among any of the network configurations or training set size trials. However, the GANs trained with fewer network nodes and iterations than needed by the discriminator classifiers alone.

Generative adversarial networks (GANs) achieved relevant results regarding the production of realistic samples. However, the training of GANs has issues that affect the stability and convergence of the algorithm. One line of research to tackle these issues uses Evolutionary Algorithms to drive the training and evolution of GANs, such as COEGAN. In this work, we propose COEGAN-v2, an extension of COEGAN that allows the use of spectral normalization and upsampling layers through the variation operators. Additionally, we use the loss functions of RaSGAN in training and also as fitness. Results show that COEGAN-v2 is more efficient and achieves better outcome quality when compared to the original COEGAN version and also regular GANs.

<h2>ABSTRACT</h2> <p><strong>Objectives:</strong> To provide insight into deep generative models and review the most prominent and efficient deep generative models, including Variational Auto-encoder (VAE) and Generative Adversarial Networks (GANs). <strong>Methods:</strong> We provide a comprehensive overview of VAEs and GANs along with their advantages and disadvantages. This paper also surveys the recently introduced Attention-based GANs and the most recently introduced Transformer based GANs. <strong>Findings:</strong> GANs have been intensively researched because of their significant advantages over VAE. Furthermore, GANs are powerful generative models that have been widely employed in a variety of fields. Though GANs have a number of advantages over VAEs, but, despite their immense popularity and success, training GANs is still difficult and has experienced a lot of setbacks. These failures include mode collapse, where the generator produces the same set of outputs for various inputs, ultimately resulting in the loss of diversity; non-convergence due to oscillatory and diverging behaviors of the generator and discriminator during the training phase; and vanishing or exploding gradients, where learning either ceases to occur or occurs very slowly. Recently, some attention-based GANs and Transformer-based GANs have also been proposed for high-fidelity image generation. <strong>Novelty:</strong> Unlike previous survey articles, which often focus on all DGMs and dive into their complicated aspects, this work focuses on the most prominent DGMs, VAEs, and GANs and provides a theoretical understanding of them. Furthermore, because GAN is now the most extensively used DGM being studied by the academic community, the literature on it needs to be explored more. Moreover, while numerous articles on GANs are available, none have analyzed the most recent attention-based GANs and Transformer-based GANs. So, in this study, we review the recently introduced attention-based GANs and Transformer-based GANs, the literature related to which has not been reviewed by any survey paper.</p> <p><strong>Keywords:</strong> Variational Autoencoder; Generative Adversarial Networks; Autoencoder; Transformer; Self-Attention</p>

Training generative adversarial networks (GANs) are known to be difficult, especially for financial time series. This paper first analyzes the well‐posedness problem in GANs minimax games and the widely recognized convexity issue in GANs objective functions. It then proposes a stochastic control framework for hyper‐parameters tuning in GANs training. The weak form of dynamic programming principle and the uniqueness and the existence of the value function in the viscosity sense for the corresponding minimax game are established. In particular, explicit forms for the optimal adaptive learning rate and batch size are derived and are shown to depend on the convexity of the objective function, revealing a relation between improper choices of learning rate and explosion in GANs training. Finally, empirical studies demonstrate that training algorithms incorporating this adaptive control approach outperform the standard ADAM method in terms of convergence and robustness. From GANs training perspective, the analysis in this paper provides analytical support for the popular practice of “clipping,” and suggests that the convexity and well‐posedness issues in GANs may be tackled through appropriate choices of hyper‐parameters.

Generative Adversarial Networks (GAN) is an adversarial model that became relevant in the last years, displaying impressive results in generative tasks. A GAN combines two neural networks, a discriminator and a generator, trained in an adversarial way. The discriminator learns to distinguish between real samples of an input dataset and fake samples. The generator creates fake samples aiming to fool the discriminator. The training progresses iteratively, leading to the production of realistic samples that can mislead the discriminator. Despite the impressive results, GANs are hard to train, and a trial-and-error approach is generally used to obtain consistent results. Since the original GAN proposal, research has been conducted not only to improve the quality of the generated results but also to overcome the training issues and provide a robust training process. However, even with the advances in the GAN model, stability issues are still present in the training of GANs. Neuroevolution, the application of evolutionary algorithms in neural networks, was recently proposed as a strategy to train and evolve GANs. These proposals use the evolutionary pressure to guide the training of GANs to build robust models, leveraging the quality of results, and providing a more stable training. Furthermore, these proposals can automatically provide useful architectural definitions, avoiding the manual discovery of suitable models for GANs. We show the current advances in the use of evolutionary algorithms and GANs, presenting the state-of-the-art proposals related to this context. Finally, we discuss perspectives and possible directions for further advances in the use of evolutionary algorithms and GANs.

CosSIF: Cosine similarity-based image filtering to overcome low inter-class variation in synthetic medical image datasets