Generative Adversarial Network Architecture Research Articles

WCGAN: Robust portrait watercolorization with adaptive hierarchical localized constraints

Deep learning has enabled image style transfer to make great strides forward. However, unlike many other styles, transferring the watercolor style to portraits is significantly challenging in image synthesis and style transfer. Pixel-correlation-based methods do not produce satisfactory watercolors. This is because portrait watercolors exhibit the sophisticated fusion of various painting techniques in local areas, which poses a problem for convolutional neural networks to accurately handle fine-grained features. Moreover, the common but problematic way of coping with multiple scales greatly impedes the performance of existing style transfer methods with fixed receptive fields. Although it is possible to develop an image processing pipeline mimicking various watercolor effects, such algorithms are slow and fragile, especially for inputs of different scales. As a remedy, this paper proposes WCGAN, a generative adversarial network (GAN) architecture dedicated to watercolorization of portraits. Specifically, a novel localized style loss suitable for watercolorization is proposed to deal with local details. To handle portraits of different scales and improve robustness, a novel discriminator architecture with three parallel branches of varying sizes of receptive fields is introduced. In addition, the application of WCGAN is expanded to video style transfer where a novel kind of video training data based on random crops is developed to efficiently capture temporal consistency. Extensive experimental results from qualitative and quantitative analyses demonstrate that WCGAN generates state-of-the-art, high quality watercolors from portraits.

URLGEN—Toward Automatic URL Generation Using GANs

URLs play an essential role on the Internet, allowing access to Web resources. Automatically generating URLs is helpful in various tasks, such as application debugging, API testing, and blocklist creation for security applications. Current testing suites deeply embed experts’ domain knowledge to generate suitable URLs, resulting in an ad-hoc solution for each given application. These tools thus require heavy manual intervention, with the expensive coding of rules that are hard to maintain. We here introduce URLGEN, a system that uses Generative Adversarial Networks (GANs) to tackle the automatic URL generation problem. URLGEN is designed for web API testing and generates URL samples for an application without any system expertise, complementing the existing tools. It leverages Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) architectures, augmented by an embedding layer that simplifies the URL learning and generation process. We show that URLGEN learns to generate new valid URLs from samples of real URLs without requiring any domain knowledge and following a purely data-driven approach. We compare the GAN architecture of URLGEN against other design options and show that the LSTM architecture can better capture the correlation among URL characters, outperforming previously proposed solutions. Finally, we show that the URLGEN approach can be extended to other scenarios, which we illustrate with two use cases, i.e., cybersquatting domain prediction and URL classification.

Unsupervised Domain Adaptation with CycleGAN: Adapting Image Style and Content for Improved Cross-Domain Performance

Abstract: In order to improve model performance while transferring information from a source domain to an unlabelled target domain, unsupervised domain adaptation is a crucial problem in deep learning. The CycleGAN design, a well-known generative adversarial network architecture, is used in this study to explore the domain adaptation challenge. Improved cross-domain performance is the goal, and both picture style and content adaptation are being focused on. In this study, we explore the use of CycleGAN to modify images from a source domain to more closely reflect the look and feel of a target domain. With no need for paired data, the architecture's built-in capacity to learn domain mappings makes it easier to transfer knowledge between dissimilar domains. By placing a strong emphasis on content and style adaptation, we hope to get beyond the problems of domain shift and distribution mismatch, which will improve generalization and classification accuracy in the target domain. We extensively experiment on benchmark datasets to show the efficacy of the suggested approach. Comparing cross-domain performance to baseline models and other adaption strategies, quantitative evaluation using known measures demonstrates the significant improvement attained. Additionally, qualitative assessments demonstrate the successful alteration of images, demonstrating CycleGAN's competence in adjusting both visual appeal and semantic information. By highlighting the need of simultaneously addressing visual style and content, this study makes a contribution to the field of unsupervised domain adaptation. The outcomes highlight CycleGAN's potential as a formidable domain adaptation tool, opening the door for improved knowledge transfer and performance in real-world settings across several domains

Data-driven modeling of noise time series with convolutional generative adversarial networks ∗ ∗U.S. government work not protected by U.S. copyright.

Random noise arising from physical processes is an inherent characteristic of measurements and a limiting factor for most signal processing and data analysis tasks. Given the recent interest in generative adversarial networks (GANs) for data-driven modeling, it is important to determine to what extent GANs can faithfully reproduce noise in target data sets. In this paper, we present an empirical investigation that aims to shed light on this issue for time series. Namely, we assess two general-purpose GANs for time series that are based on the popular deep convolutional GAN architecture, a direct time-series model and an image-based model that uses a short-time Fourier transform data representation. The GAN models are trained and quantitatively evaluated using distributions of simulated noise time series with known ground-truth parameters. Target time series distributions include a broad range of noise types commonly encountered in physical measurements, electronics, and communication systems: band-limited thermal noise, power law noise, shot noise, and impulsive noise. We find that GANs are capable of learning many noise types, although they predictably struggle when the GAN architecture is not well suited to some aspects of the noise, e.g. impulsive time-series with extreme outliers. Our findings provide insights into the capabilities and potential limitations of current approaches to time-series GANs and highlight areas for further research. In addition, our battery of tests provides a useful benchmark to aid the development of deep generative models for time series.

A Weakly Supervised Deep Learning Framework for Whole Slide Classification to Facilitate Digital Pathology in Animal Study.

The pathology of animal studies is crucial for toxicity evaluations and regulatory assessments, but the manual examination of slides by pathologists remains time-consuming and requires extensive training. One inherent challenge in this process is the interobserver variability, which can compromise the consistency and accuracy of a study. Artificial intelligence (AI) has demonstrated its ability to automate similar examinations in clinical applications with enhanced efficiency, consistency, and accuracy. However, training AI models typically relies on costly pixel-level annotation of injured regions and is often not available for animal pathology. To address this, we developed the PathologAI system, a "weakly" supervised approach for WSI classification in rat images without explicit lesion annotation at the pixel level. Using rat liver imaging data from the Open TG-GATEs system, PathologAI was applied to predict necrosis of n = 816 WSIs (377 controls). TG-GATEs studied 170 compounds at three dose levels (low, middle, and high) for four time points (3, 7, 14, and 28 days). PathologAI first preprocessed WSIs at the tile level to generate a high-level representation with a Generative Adversarial Network architecture. The prediction of liver necrosis relied on an ensemble model of 5 CNN classifiers trained on 335 WSIs. The ensemble model achieved notable classification accuracy on the holdout test set: 87% among 87 control slides free of findings, 83% among 120 controls with spontaneous necrosis, 67% among 147 treated animals with spontaneous minimal or slight necrosis, and 59% among 127 treated animals with minimal or slight necrosis caused by the treatment. Importantly, PathologAI was able to discriminate WSIs with spontaneous necrosis from those with treatment related necrosis and discriminated mild lesion level findings (slight vs minimal) and between treatment dose levels. PathologAI could provide an inexpensive and rapid screening tool to assist the digital pathology analysis in preclinical applications and general toxicological studies.

An Augmented Modulated Deep Learning Based Intelligent Predictive Model for Brain Tumor Detection Using GAN Ensemble.

Brain tumor detection in the initial stage is becoming an intricate task for clinicians worldwide. The diagnosis of brain tumor patients is rigorous in the later stages, which is a serious concern. Although there are related pragmatic clinical tools and multiple models based on machine learning (ML) for the effective diagnosis of patients, these models still provide less accuracy and take immense time for patient screening during the diagnosis process. Hence, there is still a need to develop a more precise model for more accurate screening of patients to detect brain tumors in the beginning stages and aid clinicians in diagnosis, making the brain tumor assessment more reliable. In this research, a performance analysis of the impact of different generative adversarial networks (GAN) on the early detection of brain tumors is presented. Based on it, a novel hybrid enhanced predictive convolution neural network (CNN) model using a hybrid GAN ensemble is proposed. Brain tumor image data is augmented using a GAN ensemble, which is fed for classification using a hybrid modulated CNN technique. The outcome is generated through a soft voting approach where the final prediction is based on the GAN, which computes the highest value for different performance metrics. This analysis demonstrated that evaluation with a progressive-growing generative adversarial network (PGGAN) architecture produced the best result. In the analysis, PGGAN outperformed others, computing the accuracy, precision, recall, F1-score, and negative predictive value (NPV) to be 98.85, 98.45%, 97.2%, 98.11%, and 98.09%, respectively. Additionally, a very low latency of 3.4 s is determined with PGGAN. The PGGAN model enhanced the overall performance of the identification of brain cell tissues in real time. Therefore, it may be inferred to suggest that brain tumor detection in patients using PGGAN augmentation with the proposed modulated CNN technique generates the optimum performance using the soft voting approach.

Energy consumption auditing based on a generative adversarial network for anomaly detection of robotic manipulators

This paper presents a generative adversarial network (GAN)-based energy auditing and anomaly detection framework for the health and security monitoring of robotic manipulators. The system features two key components: a side-channel measurement of the energy consumption and a GAN model to learn the data patterns during normal operations only. Anomalies in energy consumption during online testing are then discovered as outliers to the normal data by scrutinizing the anomaly score produced by the discriminator of the GAN model. This semi-supervised network circumvents the need for vast data labeling and exploits the overfitting during the training stage, hence magnifying the difference between the normal patterns learned in training and the unobserved abnormal energy profiles in testing. Moreover, considering the physical aspects of the robotic manipulator, a modified GAN architecture containing multiple discriminators to account for distinct contributions from individual joints/motors. A dynamic thresholding approach that continuously updates the statistical characteristics of the anomaly score is also developed to detect outliers and mitigate the effect of environmental variations during system operation. The proposed framework is tested on the custom dataset for performance evaluation. The results demonstrate the feasibility of the proposed method, especially for detecting physical attacks, which achieves an accuracy of approximately 0.93 in instant-level detection and over 0.84 in event-level detection.

3DCascade-GAN: Shape completion from single-view depth images

Depth images can be easily acquired using depth cameras. However, these images only contain partial information about the shape due to unavoidable self-occlusion. Thanks to the availability of large datasets of shapes, it is possible to use a learning-based approach to produce complete shapes from single depth images. State-of-the-art generative adversarial network (GAN) architectures can produce reasonable results. However, the use of relatively local convolutions restricts GAN architectures from producing globally plausible shapes. In this study, we develop a novel dynamic latent code selection mechanism in which the model learns to select only important codes from the latent space. Furthermore, a novel 3D self-attention (3DSA) layer is introduced that is able to capture non-local relationships across the 3D space. We further design a GAN architecture that uses a multistage encoder–decoder to recover the shape, where our 3DSA layer is introduced to the discriminator to help attend to global features, which stabilizes the model learning and encourages shape refinement, making our reconstruction more structurally plausible. Through extensive experiments, we demonstrate that our method outperforms other state-of-the-art methods for single depth image 3D reconstruction.

FMGAN: A Filter-Enhanced MLP Debias Recommendation Model Based on Generative Adversarial Network

In recommendation models, bias can distort the distribution of user-generated data, leading to inaccurate representation of user preferences. Failure to filter out biased data can result in significant learning errors, ultimately reducing the accuracy of the recommendation model. To address this issue, this paper proposes a Generative Adversarial Network (GAN) model comprising a filter-enhanced Multi-Layer Perceptron (MLP) generator and a linear discriminator to mitigate bias and improve the accuracy of the recommendation. The proposed model leverages the GAN architecture, where the filter structure in the generator enhances the data distribution before model training, allowing for the generation of more precise recommendation lists. The discriminator learns from the skew-corrected user review list to extract user features, which are then used alongside the recommendation list generated by G in an adversarial process. This adversarial process enables each component to optimize and improve itself while strengthening the correction effect. To enhance the accuracy of G generation, we evaluate the influence of three different input lists on the filter effect. Finally, we validate our model on two real-world datasets by comparing the effect of filter-augmented MLP and pure MLP generators. Our results demonstrate the effectiveness of filters, and our model achieves better recommendation accuracy than other baseline models.

Systematic survey on generative adversarial networks for brain tumor segmentation and classification

SummaryBrain cancer is one of the leading diseases of death in the world caused due to unwanted proliferation of cells. If left untreated, this growth can spread into other human body parts. So early detection helps to enhance long‐term survival and reduce the mortality rate. A lot of research has been done in the field of brain tumors. There have been numerous previous studies conducted using traditional approaches. So far, few review studies are available on brain tumor segmentation and classification and moreover, they were quite limited in explaining the connections of Generative Adversarial Networks (GAN) for brain tumor segmentation and classification. This review mainly focuses on the GAN architectures for brain tumor segmentation and classification. Therefore, first, an overview of brain tumors, GAN‐based publications, approaches to brain tumor segmentation and classification, data sets, software platforms, and performance parameters, are presented. Finally, GAN applications and current trends are discussed. This thorough survey will aid researchers and beginners in the future in creating a better decision support system.

Predicting extreme events in the stock market using generative adversarial networks

Accurately predicting extreme stock market fluctuations at the right time will allow traders and investors to make better-informed investment decisions and practice more efficient financial risk management. However, extreme stock market events are particularly hard to model because of their scarce and erratic nature. Moreover, strong trading strategies, market stress tests, and portfolio optimization largely rely on sound data. While the application of generative adversarial networks (GANs) for stock forecasting has been an active area of research, there is still a gap in the literature on using GANs for extreme market movement prediction and simulation. In this study, we proposed a framework based on GANs to efficiently model stock prices’ extreme movements. By creating synthetic real-looking data, the framework simulated multiple possible market-evolution scenarios, which can be used to improve the forecasting quality of future market variations. The fidelity and predictive power of the generated data were tested by quantitative and qualitative metrics. Our experimental results on S&P 500 and five emerging market stock data show that the proposed framework is capable of producing a realistic time series by recovering important properties from real data. The results presented in this work suggest that the underlying dynamics of extreme stock market variations can be captured efficiently by some state-of-the-art GAN architectures. This conclusion has great practical implications for investors, traders, and corporations willing to anticipate the future trends of their financial assets. The proposed framework can be used as a simulation tool to mimic stock market behaviors.

DAEGAN: Generative adversarial network based on dual-domain attention-enhanced encoder-decoder for low-dose PET imaging

BackgroundPositron emission tomography (PET) is a valuable medical imaging modality utilized in both clinical and preclinical settings. There is a growing concern regarding the potential radiation exposure associated with the administration of radiotracers. This concern has led to a focus on improving the quality of PET images obtained from low-dose radiotracer injections. MethodsIn this study, we propose a novel generative adversarial network (GAN) architecture called dual-domain attention-enhanced encoder-decoder GAN (DAEGAN) for low-dose PET imaging. The DAEGAN architecture incorporates a dual-domain encoder-decoder-based generator, which enables the network to focus simultaneously on the structure information and content features. Additionally, attention modules are integrated into the discriminator and the generator to effectively aggregate meaningful features. A significant contribution of this study is the incorporation of the classification activation map (CAM) loss as a component of the generator loss function for the denoising task.Experiments: Experiments on datasets of small animal and human brain PET scans including normal-dose images and low-dose images for the proposed method and other published methods were carried out. A cross-dataset validation based on a homemade Derenzo phantom was also implemented. ResultsIn the qualitative evaluation experiments, the proposed model can generate denoised images approaching normal-dose images with better visual structural details. Meanwhile, the proposed method achieved the best score in metrics on the test datasets and cross-dataset validation. ConclusionsThe results suggest that the proposed DAEGAN architecture is a promising approach for improving the quality of low-dose PET images compared to state-of-the-art methods.

AutoInfo GAN: Toward a better image synthesis GAN framework for high-fidelity few-shot datasets via NAS and contrastive learning

Background:Generative adversarial networks (GANs) are vital techniques for synthesizing high-fidelity images. Recent studies have applied them to generation tasks under small-data scenarios. Most studies do not directly train GANs on few-shot datasets, which have small data samples; instead, they borrow methods to transfer knowledge from large datasets to GANs with small ones. Partial fine-tuning of GANs is difficult to ensure the transfer performance, especially when the image domains are of great difference. FastGAN firstly trains GAN with small data samples by a carefully designed skip layer exception (SLE) connection to improve synthesis and an unsupervised discriminator to avoid overfitting. Problem. However, in FastGAN, different designs of SLE connections and operation settings would lead to great differences in synthesis performance. It is necessary to find the most appropriate ways of architecture design. Meanwhile, FastGAN merely improves discriminator learning, but ignores that the generator learning process is also insufficient due to small data samples. Objective:Based on FastGAN, this study aims to find the best generator designs and then improve the training process of it via unsupervised learning. Methods. This work applies a reinforcement learning neural architecture search method to find the optimal GAN architecture and an unsupervised contrastive loss function assisted by a discriminator to optimize generator learning. These two methods constitute our AutoInfoGAN. Results:Experiments were conducted on 11 datasets using AutoInfoGAN, covering a wide range of image domains, achieving better results than state-of-the-art (SOTA) models. Conclusion:The experimental results demonstrate the SOTA performance of our proposed AutoInfo GAN on few-shot datasets, and we are cautioned that although instance normalization (IN) improves synthesized image quality, it performed poorly in our mode-collapse test.

Non-local generative machine learning-based inverse design for scattering properties.

Metamaterials are created by arranging small scatterers in a regular array throughout a space to manipulate electromagnetic waves. However, current design methods view metasurfaces as independent meta-atoms, which limits the range of geometrical structures and materials used, and prevents the generation of arbitrary electric field distributions. To address this issue, we propose an inverse design method based on generative adversarial networks (GANs), which includes both a forward model and an inverse algorithm. The forward model utilizes dyadic Green's function to interpret the expression of non-local response, realizing the mapping from scattering properties to generated electric fields. The inverse algorithm innovatively transforms the scattering properties and electric fields into images and generates datasets with methods in computer vision (CV), proposing an architecture of GAN with ResBlock to achieve the design for the target electric field pattern. Our algorithm improves upon traditional methods, as it achieves greater time efficiency and generates higher quality electric fields. From a metamaterial perspective, our method can find optimal scattering properties for specific generated electric fields. Training results and extensive experiments demonstrate the algorithm's validity.

Epiphany: predicting Hi-C contact maps from 1D epigenomic signals

Recent deep learning models that predict the Hi-C contact map from DNA sequence achieve promising accuracy but cannot generalize to new cell types and or even capture differences among training cell types. We propose Epiphany, a neural network to predict cell-type-specific Hi-C contact maps from widely available epigenomic tracks. Epiphany uses bidirectional long short-term memory layers to capture long-range dependencies and optionally a generative adversarial network architecture to encourage contact map realism. Epiphany shows excellent generalization to held-out chromosomes within and across cell types, yields accurate TAD and interaction calls, and predicts structural changes caused by perturbations of epigenomic signals.

Wor(l)d-GAN: Toward Natural-Language-Based PCG in Minecraft

This work presents Wor(l)d-GAN, a method to perform data-driven Procedural Content Generation via Machine Learning in Minecraft from a single example. Based on a 3D Generative Adversarial Network (GAN) architecture, we are able to create arbitrarily sized world snippets from a given sample. Our method applies dense representations used in Natural Language Processing (NLP) in two ways. First, we propose block2vec representations based on word2vec. Secondly, we use the pretrained large language model BERT to generate representations directly from the token names. These representations make Wor(l)d-GAN independent from the number of different blocks, which can vary a lot in Minecraft, and enable the generation of larger levels. We evaluate our approach on creations from the community as well as structures generated with the Minecraft World Generator under several metrics. Wor(l)d-GAN enables its users to generate Minecraft worlds based on parts of their creations.

Synthetic Data Generation by Artificial Intelligence to Accelerate Research and Precision Medicine in Hematology.

Synthetic data are artificial data generated without including any real patient information by an algorithm trained to learn the characteristics of a real source data set and became widely used to accelerate research in life sciences. We aimed to (1) apply generative artificial intelligence to build synthetic data in different hematologic neoplasms; (2) develop a synthetic validation framework to assess data fidelity and privacy preservability; and (3) test the capability of synthetic data to accelerate clinical/translational research in hematology. A conditional generative adversarial network architecture was implemented to generate synthetic data. Use cases were myelodysplastic syndromes (MDS) and AML: 7,133 patients were included. A fully explainable validation framework was created to assess fidelity and privacy preservability of synthetic data. We generated MDS/AML synthetic cohorts (including information on clinical features, genomics, treatment, and outcomes) with high fidelity and privacy performances. This technology allowed resolution of lack/incomplete information and data augmentation. We then assessed the potential value of synthetic data on accelerating research in hematology. Starting from 944 patients with MDS available since 2014, we generated a 300% augmented synthetic cohort and anticipated the development of molecular classification and molecular scoring system obtained many years later from 2,043 to 2,957 real patients, respectively. Moreover, starting from 187 MDS treated with luspatercept into a clinical trial, we generated a synthetic cohort that recapitulated all the clinical end points of the study. Finally, we developed a website to enable clinicians generating high-quality synthetic data from an existing biobank of real patients. Synthetic data mimic real clinical-genomic features and outcomes, and anonymize patient information. The implementation of this technology allows to increase the scientific use and value of real data, thus accelerating precision medicine in hematology and the conduction of clinical trials.

CLGAN: a generative adversarial network (GAN)-based video prediction model for precipitation nowcasting

Abstract. The prediction of precipitation patterns up to 2 h ahead, also known as precipitation nowcasting, at high spatiotemporal resolutions is of great relevance in weather-dependent decision-making and early warning systems. In this study, we are aiming to provide an efficient and easy-to-understand deep neural network – CLGAN (convolutional long short-term memory generative adversarial network) – to improve the nowcasting skills of heavy precipitation events. The model constitutes a generative adversarial network (GAN) architecture, whose generator is built upon a u-shaped encoder–decoder network (U-Net) and is equipped with recurrent long short-term memory (LSTM) cells to capture spatiotemporal features. The optical flow model DenseRotation and the competitive video prediction models ConvLSTM (convolutional LSTM) and PredRNN-v2 (predictive recurrent neural network version 2) are used as the competitors. A series of evaluation metrics, including the root mean square error, the critical success index, the fractions skill score, and object-based diagnostic evaluation, are utilized for a comprehensive comparison against competing baseline models. We show that CLGAN outperforms the competitors in terms of scores for dichotomous events and object-based diagnostics. A sensitivity analysis on the weight of the GAN component indicates that the GAN-based architecture helps to capture heavy precipitation events. The results encourage future work based on the proposed CLGAN architecture to improve the precipitation nowcasting and early warning systems.

A review on Generative Adversarial Networks for image generation

Generative Adversarial Networks (GANs) are a type of deep learning architecture that uses two networks namely a generator and a discriminator that, by competing against each other, pursue to create realistic but previously unseen samples. They have become a popular research topic in recent years, particularly for image processing and synthesis, leading to many advances and applications in various fields. With the profusion of published works and interest from professionals of different areas, surveys on GANs are necessary, mainly for those who aim starting on this topic. In this work, we cover the basics and notable architectures of GANs, focusing on their applications in image generation. We also discuss how the challenges to be addressed in GANs architectures have been faced, such as mode coverage, stability, convergence, and evaluating image quality using metrics.

Zero-shot transfer learning for structural health monitoring using generative adversarial networks and spectral mapping

Gathering properly labeled, adequately rich, and case-specific data for successfully training a purely data-driven or hybrid model for structural health monitoring (SHM) applications is a challenging task. We posit that a Transfer Learning (TL) method that utilizes available data in any relevant source domain and directly applies to the target domain through domain adaptation can provide substantial remedies to address this issue. Accordingly, we present a novel TL method that differentiates between the source’s no-damage and damage cases and utilizes a domain adaptation (DA) technique. The DA module transfers the accumulated knowledge in contrasting no-damage and damage cases in the source domain to the target domain, given only the target’s no-damage case. High-dimensional features allow employing signal processing domain knowledge to devise a generalizable DA approach. The Generative Adversarial Network (GAN) architecture is adopted for learning since its optimization process accommodates high-dimensional inputs in a zero-shot setting. At the same time, its training objective conforms seamlessly with the case of no-damage and damage data in SHM since its discriminator network differentiates between real (no-damage) and fake (possibly unseen damage) data. An extensive set of experimental results demonstrates the method’s success in transferring knowledge on differences between no-damage and damage cases across three strongly heterogeneous independent target structures. The area under the Receiver Operating Characteristics curves (Area Under the Curve — AUC) is used to evaluate the differentiation between no-damage and damage cases in the target domain, reaching values as high as 0.95. With no-damage and damage cases discerned from each other, zero-shot structural damage detection is carried out. The mean F1 scores for all damages in the three independent datasets are 0.971, 0.986, and 0.975. The success of the proposed TL approach is expected to pave the way for further improvements in the accuracy and generalizability of data-driven SHM applications, thereby offering new possibilities for large-scale SHM applications in urban settings.