Generative Adversarial Network Architecture Research Articles

Generative Adversarial Network Based Music Generation

Music has been an integral part of human civilization personally and culturally. Historically, music has been generated using various instruments, or natural sounds like water drops, or unconventional musical instruments like metal or glass-wares. At present, technologies like Musical Instrument Digital Interface (MIDI) are used to generate music electronically. This research investigates the use of Generative Adversarial Networks (GANs) for beginner-friendly music production. This model uses Long Short-Term Memory (LSTM) generator and Patch GAN as discriminator for the GAN architecture. The generator consists of input layer, embedding layer, LSTM layer and generates output with a SoftMax function. Similarly, the discriminator consists of a convolution layer, the output of which is averaged by the global average pooling layer, and output is generated by the sigmoid function. The model is trained on Maestro MIDI dataset. We make the process understandable by delving into the implementation specifics and outlining the fundamental concepts of music. Our effective model highlights the potential of GANs in music composition by producing cohesive music. After training for 50 epochs, the model exhibited a remarkable precision of 91.82 percent. This project uses the combination of Artificial Intelligence (AI) with music theory to provide intriguing new opportunities in the field of music. The model can be beneficial for different industries like gaming, music, entertainment, education, etc.

From simulation to reality: tackling data mismatches in speech enhancement with unsupervised pre-training

In this study, we introduce an innovative speech enhancement methodology that ingeniously combines unsupervised pre-training with supervised fine-tuning. This hybrid approach directly addresses the prevalent data mismatch challenge inherent in traditional supervised speech enhancement methods. Our technique distinctly utilizes unpaired noisy and clean speech data and incorporates varied noises during the pre-training phase. This strategy effectively simulates the benefits of supervised learning, eliminating the need for paired data. Inspired by contrastive learning techniques prevalent in computer vision, our model is adept at preserving essential speech features amidst noise interference. At the heart of our method lies a sophisticated Generative Adversarial Network (GAN) architecture. This includes a generator that proficiently processes both magnitude and complex-domain features, alongside a discriminator meticulously designed to optimize specific evaluation metrics. Through rigorous experimental evaluations, we validate the robustness and versatility of our approach. It consistently delivers superior speech quality, demonstrating remarkable efficacy in real-world scenarios, which are often characterized by complex and unpredictable noise environments.

Generative AI: Shaping the Future While Disrupting the Present

Generative AI, a transformative technology in the world of artificial intelligence, is reshaping how we create and interact with digital content across various fields like art, business, and healthcare. This paper delves into the historical journey of generative AI, starting from early neural networks to recent developments like GPT-4 and diffusion-based models. By exploring pivotal technologies such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer architectures, we offer a detailed analysis of how these models have revolutionized content generation. While these advancements open new doors for creativity and innovation, they also introduce significant challenges. Issues of bias, ethical concerns, and the environmental costs of AI particularly the growing water consumption for data centers are discussed at length. The paper further examines the dual impact of generative AI: its ability to enhance productivity while also causing disruptions in traditional industries and human interactions. As the use of AI scales, this research highlights the urgent need for sustainable and ethical approaches to its development and deployment. By examining both the potential and the pitfalls of generative AI, this study aims to provide a balanced outlook on the future of this influential technology.

Predicting somatic mutation origins in cell-free DNA by semi-supervised GAN models

MotivationDistinguishing between pathogenic cancer-associated mutations and other somatic variants present in cell-free DNA (cfDNA) is one of the challenges in the field of liquid biopsy. This distinction is critical, since the misclassification of mutations stemming from clonal hematopoiesis (CH) as tumor-derived and vice versa could result in inaccurate diagnoses and inappropriate therapeutic interventions for patients. ResultsWe addressed this by developing a specialized machine learning technique to differentiate tumor- or CH-related mutations in cfDNA. We established a comprehensive in-house reference catalog, comprising approximately 25,000 single nucleotide variants (SNVs), each linked to either tumor or CH origin. This reference serves as a foundation for training a deep learning model, which is structured on the semi-supervised generative adversarial network (SSGAN) architecture. By analyzing genomic coordinates and nucleotide composition of cfDNA variants, our model attains 95 % area under the curve (AUC) in classifying uncharacterized variants as CH or tumor-derived. In conclusion, our research emphasizes the potential of genomic feature prediction, using cfDNA data, to stand as a robust alternative to conventional multi-analyte sequencing methods. This approach not only enhances the accuracy of distinguishing CH from tumor mutations in liquid biopsy data, but also highlights the potential of advanced data analysis techniques and machine learning in genomics and personalized medicine. Availability: https://github.com/FPalizban/SSGAN.

ACGAN: adaptive conditional generative adversarial network architecture predicting skin lesion using collaboration of transfer learning models

Skin cancer has become a serious disease which has the potential to scale up if it is not identified earlier. It is imperative to detect and give treatment to skin cancer promptly. Diagnosing skin cancer manually takes a lot of time and it is costly, and the probability of false diagnosis has increased due to the outstanding resemblances among various skin lesions. Enhancing the classification of multi-class lesions of skin needs the development of investigative systems which should be automated. Data augmentation with GANs and Adaptive Conditional Generative Adversarial Network strategies improves performance. The performance is tested using balanced and unbalanced datasets. Using a proper process of augmentation of data, the suggested system attains a 94% accuracy for the VGG16, 93% for the ResNet50 and 94.25% for ResNet101. The process of collaboration of all such methods improves accuracy further to 95%. In summary, the novelty of the work lies in its holistic approach to automated skin lesion classification, incorporating advanced deep learning models, novel data augmentation techniques and comprehensive performance evaluation on real-world datasets. These contributions collectively advance the field of computer-aided diagnosis for the detection of skin cancer and treatment.

Parent GAN: image generation model for creating parent’s images using children’s images

AbstractIn the past decade, several applications have emerged in predicting children’s images using their parents via Generative Adversarial Networks (GANs). However, no one has tackled the problem of predicting one of the parents using the other parent and their children or answering the question of the possibility of deducing the parent images from the children and other parent image features. It could be used in parental identification cases. Moreover, it could help children who don’t know one of their parents to have a visual representation of their images. To perform this task, several obstacles were overcome, like the small number of parent pairs in the dataset and stabilizing the GANs to produce good-looking images. The proposed method depends on dual GAN architecture in addition to adaptive instance normalization layers and introducing a triple loss function to stabilize further and improve the resulting images. The results were proven using a kinship verification model, a face verification model, and other well-known evaluation metrics, which showed that the generated parent images are of decent quality compared to real parents’ images with affordable computational hardware. As a result, a novel method is developed that could produce unknown parent images.

A Hybrid GAN-Inception Deep Learning Approach for Enhanced Coordinate-Based Acoustic Emission Source Localization

In this paper, we propose a novel approach to coordinate-based acoustic emission (AE) source localization to address the challenges of limited and imbalanced datasets from fiber-optic AE sensors used for structural health monitoring (SHM). We have developed a hybrid deep learning model combining four generative adversarial network (GAN) variants for data augmentation with an adapted inception neural network for regression-based prediction. The experimental setup features a single fiber-optic AE sensor based on a tightly coiled fiber-optic Fabry-Perot interferometer formed by two identical fiber Bragg gratings. AE signals were generated using the Hsu-Nielsen pencil lead break test on a grid-marked thin aluminum plate with 35 distinct locations, simulating real-world structural monitoring conditions in bounded isotropic plate-like structures. It is demonstrated that the single-sensor configuration can achieve precise localization, avoiding the need for a multiple sensor array. The GAN-based signal augmentation expanded the dataset from 900 to 4500 samples, with the Wasserstein distance between the original and synthetic datasets decreasing by 83% after 2000 training epochs, demonstrating the high fidelity of the synthetic data. Among the GAN variants, the standard GAN architecture proved the most effective, outperforming other variants in this specific application. The hybrid model exhibits superior performance compared to non-augmented deep learning approaches, with the median error distribution comparisons revealing a significant 50% reduction in prediction errors, accompanied by substantially improved consistency across various AE source locations. Overall, this developed hybrid approach offers a promising solution for enhancing AE-based SHM in complex infrastructures, improving damage detection accuracy and reliability for more efficient predictive maintenance strategies.

The 3D tree dataset: an artistic experiment using a voxel-based GAN

AbstractIn visual culture, Generative Adversarial Networks (GANs) have been used to generate two-dimensional images, video, and three-dimensional forms. Whilst there are a relatively large number of conditional datasets of two-dimensional images, there are fewer datasets of three-dimensional objects available for training, evaluation, and practical use. In this paper, we introduce a synthetic dataset of 3D trees that provide novel geometric challenges for machine learning and describe our use of this dataset in training and qualitative evaluation via a public exhibition and survey. The 3D Tree Dataset was made using random variations of 76 bespoke tree templates based on art historical references to favor interesting, beautiful, and varied trunk and branch shapes. In generating and experimenting with this dataset, we wanted to know how a geometrically complex series of organic forms would challenge voxel-GANs compared to existing datasets comprising industrial objects such as chairs and cars. As an interdisciplinary project between visual arts and computer science, we used 3D printing and animation to visualize and communicate our outputs to non-specialist audiences. In this paper, we describe and share the 3D Tree Dataset, outline the process of generating the dataset, describe the GAN architecture and application used to test the dataset and discuss the results relative to our artistic and cultural goals.

Towards a Deep Learning Approach for IoT Attack Detection Based on a New Generative Adversarial Network Architecture and Gated Recurrent Unit

Towards a Deep Learning Approach for IoT Attack Detection Based on a New Generative Adversarial Network Architecture and Gated Recurrent Unit

Creating realistic anterior segment optical coherence tomography images using generative adversarial networks

AimsTo develop a generative adversarial network (GAN) capable of generating realistic high-resolution anterior segment optical coherence tomography (AS-OCT) images.MethodsThis study included 142 628 AS-OCT B-scans from the American University of...

Exploring Generative Adversarial Networks for Augmenting Network Intrusion Detection Tasks

The advent of generative networks and their adoption in numerous domains and communities have led to a wave of innovation and breakthroughs in artificial intelligence and machine learning. Generative Adversarial Networks (GANs) have expanded the scope of what is possible with machine learning, allowing for new applications in areas such as computer vision, natural language processing, and creative AI. GANs, in particular, have been used for a wide range of tasks, including image and video generation, data augmentation, style transfer, and anomaly detection. They have also been used for medical imaging and drug discovery, where they can generate synthetic data to augment small datasets, reduce the need for expensive experiments, and lower the number of real patients that must be included in medical trials. Given these developments, we propose using the power of generative adversarial networks to create and augment flow-based network traffic datasets. We evaluate a series of GAN architectures, including Wasserstein, conditional, energy-based, gradient penalty, and LSTM GANs. We evaluate their performance on a set of flow-based network traffic data collected from 16 subjects who used their computers for home, work, and study purposes. The performance of these GAN architectures is described according to metrics that involve networking principles, data distribution among a collection of flows, and temporal data distribution. Given the tendency of network intrusion detection datasets to have a very imbalanced data distribution, i.e., a large number of samples in the “normal traffic” category and a comparatively low number of samples assigned to the “intrusion” categories, we test our GANs by augmenting the intrusion data and checking whether this helps intrusion detection neural networks in their task. We publish the resulting UPBFlow dataset and code on GitHub 1 .

Mastering Deepfake Detection: A Cutting-edge Approach to Distinguish GAN and Diffusion-model Images

Detecting and recognizing deepfakes is a pressing issue in the digital age. In this study, we first collected a dataset of pristine images and fake ones properly generated by nine different Generative Adversarial Network (GAN) architectures and four Diffusion Models (DM). The dataset contained a total of 83,000 images, with equal distribution between the real and deepfake data. Then, to address different deepfake detection and recognition tasks, we proposed a hierarchical multi-level approach. At the first level, we classified real images from AI-generated ones. At the second level, we distinguished between images generated by GANs and DMs. At the third level (composed of two additional sub-levels), we recognized the specific GAN and DM architectures used to generate the synthetic data. Experimental results demonstrated that our approach achieved more than 97% classification accuracy, outperforming existing state-of-the-art methods. The models obtained in the different levels turn out to be robust to various attacks such as JPEG compression (with different quality factor values) and resize (and others), demonstrating that the framework can be used and applied in real-world contexts (such as the analysis of multimedia data shared in the various social platforms) for support even in forensic investigations to counter the illicit use of these powerful and modern generative models. We are able to identify the specific GAN and DM architecture used to generate the image, which is critical in tracking down the source of the deepfake. Our hierarchical multi-level approach to deepfake detection and recognition shows promising results in identifying deepfakes allowing focus on underlying task by improving (about 2% on the average) standard multiclass flat detection systems. The proposed method has the potential to enhance the performance of deepfake detection systems, aid in the fight against the spread of fake images, and safeguard the authenticity of digital media.

Coarse-to-Fine Structure and Semantic Learning for Single-Sample SAR Image Generation

Synthetic Aperture Radar (SAR) enables the acquisition of high-resolution imagery even under severe meteorological and illumination conditions. Its utility is evident across a spectrum of applications, particularly in automatic target recognition (ATR). Since SAR samples are often scarce in practical ATR applications, there is an urgent need to develop sample-efficient augmentation techniques to augment the SAR images. However, most of the existing generative approaches require an excessive amount of training samples for effective modeling of the SAR imaging characteristics. Additionally, they show limitations in augmenting the interesting target samples while maintaining image recognizability. In this study, we introduce an innovative single-sample image generation approach tailored to SAR data augmentation. To closely approximate the target distribution across both the spatial layout and local texture, a multi-level Generative Adversarial Network (GAN) architecture is constructed. It comprises three distinct GANs that independently model the structural, semantic, and texture patterns. Furthermore, we introduce multiple constraints including prior-regularized noise sampling and perceptual loss optimization to enhance the fidelity and stability of the generation process. Comparative evaluations against the state-of-the-art generative methods demonstrate the superior performance of the proposed method in terms of generation diversity, recognizability, and stability. In particular, its advantages over the baseline method are up to 0.2 and 0.22 in the SIFID and SSIM, respectively. It also exhibits stronger robustness in the generation of images across varying spatial sizes.

Spatial–Temporal Similarity Fusion Graph Adversarial Convolutional Networks for traffic flow forecasting

Traffic flow forecasting is integral to the advancement of intelligent transportation systems and the development of smart cities. This paper introduces a novel model, the Spatial–Temporal Similarity Fusion Graphs Adversarial Convolutional Networks (STSF-GACN), which leverages advanced data preprocessing techniques to enhance the predictive accuracy and efficiency of traffic flow forecasting.The innovation of our approach lies in the meticulous construction of the spatial–temporal similarity matrix through the precise calculation of temporal and spatial similarities. This matrix forms the backbone of our model, serving as the generator in the integrated Generative Adversarial Network (GAN) architecture. The Spatial–Temporal Similarity Fusion Adaptive Graph Convolutional Network, developed as part of our GAN’s generator, utilizes cutting-edge techniques such as the Wasserstein distance and Dynamic Time Warping to optimize the adaptive adjacency matrix, enabling the model to capture latent spatial–temporal correlations with unprecedented depth and precision.The discriminator of the GAN further refines the model by evaluating the accuracy of the traffic predictions, ensuring that the generative model produces results that are not only accurate but also robust against varying traffic conditions. This cohesive integration of GAN into the model architecture allows for a significant improvement in prediction accuracy and convergence speed, moving beyond traditional forecasting methods.

Generative Adversarial Networks with Radiomics Supervision for Lung Lesion Generation.

Data-driven methods for lesion generation are quickly emerging due to the need for realistic imaging targets for image quality assessment and virtual clinical trials. We proposed a generative adversarial network (GAN) architecture for conditional generation of lung lesions based on user-specified classes of lesion size and solidity. The network consists of two discriminators, one for volumetric lesion data, and one for radiomics features derived from the lesion volume. A Wasserstein loss with gradient penalty was adopted for each discriminator. Training data were drawn from contoured and annotated lesions from a public lung CT database. Four quantitative evaluation methods were devised to assess the network performance: 1) overfitting (similarity between generated and real lesions), 2) diversity (similarity among generated lesions), 3) conditional consistency (capability of generating lesions according to user-specified classes), and 4) similarity in distributions of various lesion properties between the generated and real lesions. Ablation studies were also performed to investigate the importance of individual network component. The proposed network was found to generate lesions that resemble real lesions by visual inspection. Solid lesions are distinct from non-solid ones, and lesion sizes largely correspond to their specified classes. With a classifier trained on real lesions, the classification accuracies of generated and real lesions in both solid and non-solid classes are similar. Radiomics features of generated and real lesions were found to have similar distributions, indicated by the relatively low Kullback-Leibler (KL) divergence values. Furthermore, the correlations between pairwise radiomics features in generated lesions were comparable to those of real lesions. The proposed network presents a promising approach for generating realistic lesions with clinically relevant features crucial for the comprehensive assessment of medical imaging systems.

Tunable Privacy Risk Evaluation of Generative Adversarial Networks.

Generative machine learning models such as Generative Adversarial Networks (GANs) have been shown to be especially successful in generating realistic synthetic data in image and tabular domains. However, it has been shown that such generative models, as well as the generated synthetic data, can reveal information contained in their privacy-sensitive training data, and therefore must be carefully evaluated before being used. The gold standard method through which such privacy leakage can be estimated is simulating membership inference attacks (MIAs), in which an attacker attempts to learn whether a given sample was part of the training data of a generative model. The state-of-the art MIAs against generative models, however, rely on strong assumptions (knowledge of the exact training dataset size), or require a lot of computational power (to retrain many "surrogate" generative models), which make them hard to use in practice. In this work, we propose a technique for evaluating privacy risks in GANs which exploits the outputs of the discriminator part of the standard GAN architecture. We evaluate our attacks in terms of performance in two synthetic image generation applications in radiology and ophthalmology, showing that our technique provides a more complete picture of the threats by performing worst-case privacy risk estimation and by identifying attacks with higher precision than the prior work.

Learning-Based Super-Resolution Imaging of Turbulent Flames in Both Time and 3D Space Using Double GAN Architectures

This article presents a spatiotemporal super-resolution (SR) reconstruction model for two common flame types, a swirling and then a jet flame, using double generative adversarial network (GAN) architectures. The approach develops two sets of generator and discriminator networks to learn topographic and temporal features and infer high spatiotemporal resolution turbulent flame structure from supplied low-resolution counterparts at two time points. In this work, numerically simulated 3D turbulent swirling and jet flame structures were used as training data to update the model parameters of the GAN networks. The effectiveness of our model was then thoroughly evaluated in comparison to other traditional interpolation methods. An upscaling factor of 2 in space, which corresponded to an 8-fold increase in the total voxel number and a double time frame acceleration, was used to verify the model’s ability on a swirling flame. The results demonstrate that the assessment metrics, peak signal-to-noise ratio (PSNR), overall error (ER), and structural similarity index (SSIM), with average values of 35.27 dB, 1.7%, and 0.985, respectively, in the spatiotemporal SR results, can reach acceptable accuracy. As a second verification to highlight the present model’s potential universal applicability to flame data of diverse types and shapes, we applied the model to a turbulent jet flame and had equal success. This work provides a different method for acquiring high-resolution 3D structure and further boosting repeat rate, demonstrating the potential of deep learning technology for combustion diagnosis.

Limited Sample Radar HRRP Recognition Using FWA-GAN

In radar High-Resolution Range Profile (HRRP) target recognition, the targets of interest are always non-cooperative, posing a significant challenge in acquiring sufficient samples. This limitation results in the prevalent issue of limited sample availability. To mitigate this problem, researchers have sought to integrate handcrafted features into deep neural networks, thereby augmenting the information content. Nevertheless, existing methodologies for fusing handcrafted and deep features often resort to simplistic addition or concatenation approaches, which fail to fully capitalize on the complementary strengths of both feature types. To address these shortcomings, this paper introduces a novel radar HRRP feature fusion technique grounded in the Feature Weight Assignment Generative Adversarial Network (FWA-GAN) framework. This method leverages the generative adversarial network architecture to facilitate feature fusion in an innovative manner. Specifically, it employs the Feature Weight Assignment Model (FWA) to adaptively assign attention weights to both handcrafted and deep features. This approach enables a more efficient utilization and seamless integration of both feature modalities, thereby enhancing the overall recognition performance under conditions of limited sample availability. As a result, the recognition rate increases by over 4% compared to other state-of-the-art methods on both the simulation and experimental datasets.

Perfusion parameter map generation from TOF-MRA in stroke using generative adversarial networks

PurposeTo generate perfusion parameter maps from Time-of-flight magnetic resonance angiography (TOF-MRA) images using artificial intelligence to provide an alternative to traditional perfusion imaging techniques. Materials and methodsThis retrospective study included a total of 272 patients with cerebrovascular diseases; 200 with acute stroke (from 2010 to 2018), and 72 with steno-occlusive disease (from 2011 to 2014). For each patient the TOF MRA image and the corresponding Dynamic susceptibility contrast magnetic resonance imaging (DSC-MRI) were retrieved from the datasets. The authors propose an adapted generative adversarial network (GAN) architecture, 3D pix2pix GAN, that generates common perfusion maps (CBF, CBV, MTT, TTP, Tmax) from TOF-MRA images. The performance was evaluated by the structural similarity index measure (SSIM). For a subset of 20 patients from the acute stroke dataset, the Dice coefficient was calculated to measure the overlap between the generated and real hypoperfused lesions with a time-to-maximum (Tmax) > 6 s. ResultsThe GAN model exhibited high visual overlap and performance for all perfusion maps in both datasets: acute stroke (mean SSIM 0.88–0.92, mean PSNR 28.48–30.89, mean MAE 0.02–0.04 and mean NRMSE 0.14–0.37) and steno-occlusive disease patients (mean SSIM 0.83–0.98, mean PSNR 23.62–38.21, mean MAE 0.01–0.05 and mean NRMSE 0.03–0.15). For the overlap analysis for lesions with Tmax>6 s, the median Dice coefficient was 0.49. ConclusionOur AI model can successfully generate perfusion parameter maps from TOF-MRA images, paving the way for a non-invasive alternative for assessing cerebral hemodynamics in cerebrovascular disease patients. This method could impact the stratification of patients with cerebrovascular diseases. Our results warrant more extensive refinement and validation of the method.

Synthetic Medical Imaging Generation with Generative Adversarial Networks for Plain Radiographs

In medical imaging, access to data is commonly limited due to patient privacy restrictions, and it can be difficult to acquire enough data in the case of rare diseases. The purpose of this investigation was to develop a reusable open-source synthetic image-generation pipeline, the GAN Image Synthesis Tool (GIST), that is easy to use as well as easy to deploy. The pipeline helps to improve and standardize AI algorithms in the digital health space by generating high quality synthetic image data that is not linked to specific patients. Its image generation capabilities include the ability to generate imaging of pathologies or injuries with low incidence rates. This improvement of digital health AI algorithms could improve diagnostic accuracy, aid in patient care, decrease medicolegal claims, and ultimately decrease the overall cost of healthcare. The pipeline builds on existing Generative Adversarial Networks (GANs) algorithms, and preprocessing and evaluation steps were included for completeness. For this work, we focused on ensuring the pipeline supports radiography, with a focus on synthetic knee and elbow X-ray images. In designing the pipeline, we evaluated the performance of current GAN architectures, studying the performance on available X-ray data. We show that the pipeline is capable of generating high-quality and clinically relevant images based on a lay person’s evaluation and the Fréchet Inception Distance (FID) metric.