GAN-based Methods Research Articles

Super-resolution of magnetic resonance images using Generative Adversarial Networks.

Magnetic Resonance Imaging (MRI) typically comes at the cost of small spatial coverage, high expenses and long scan times. Accelerating MRI acquisition by taking less measurements yields the potential to relax these inherent forfeits. Recent breakthroughs in the field of Machine Learning have shown high-resolution (HR) images could be recovered from low-resolution (LR) signals via super-resolution (SR). In particular, a novel class of neural networks named Generative Adversarial Networks (GAN) has manifested an alternative way of conceiving models capable of generating data. GANs can learn to infer details based on some prior information, subsequently recovering missing data. Accordingly, they manifest huge potential in MRI reconstruction and acceleration tasks. This paper conducts a review on GAN-based SR methods, exhibiting the immersive ability of GANs on upscaling MRIs by a scale factor of ×4 while at the same time maintaining trustworthy and high-frequency details. Despite quantitative results suggesting SRResCycGAN outperforms other popular deep learning methods in recovering ×4 downgraded images, qualitative results show Beby-GAN holds the best perceptual quality and proves GAN-based methods hold the capacity to reduce medical costs, distress patients and even enable new MRI applications where it is currently too slow or expensive.

IGAN: A collaborative filtering model based on Improved Generative Adversarial Networks for recommendation

In recent years, Generative Adversarial Networks (GANs) have been successfully applied to collaborative filtering for implicit feedback. However, the ability of GAN to learn user interest distributions is greatly reduced due to its difficulty in characterizing the features. In this paper, a collaborative filtering model based on Improved Generative Adversarial Networks (IGAN) is proposed to address the above issue. In IGAN, we first introduce the independent encoder and generator to learn features representation during adversarial training. Then, the Kullback–Leibler (KL) loss and reconstruction loss are added as the penalty term to further fit the interest distribution of users and achieve higher recommendation accuracy. Finally, this paper conducts extensive experiments on three real public datasets, showing that the IGAN method has a maximum improvement of 5.26% and 5.43% in P@5 and NDCG@5 compared with the current GAN-based methods.

Time series imputation with GAN inversion and decay connection

Time series imputation is essential for real-world applications. Though the emergence of Generative Adversarial Networks (GANs) and Graph Convolution Networks (GCNs) provides more possibilities to improve imputation performance, how to achieve the optimal latent code and precisely model the properties of incomplete time series remain a challenge. In GAN-based methods, an effective latent code of incomplete time series is necessary for precise reconstruction. To acquire the optimal latent code, we introduce GAN inversion to invert the input to the latent space of a pretrained GAN. The inverted latent code contains rich properties of original observations and thus can better reconstruct the target sample. To model the temporal irregularity due to the presence of missing values, the decay connection is exploited to quantify the influence that dependencies between adjacent observations should decrease as the time lags between them increase. We incorporate the quantification into the adjacent matrix of the GCN to better aggregate adjacent information of incomplete time series. With the adoption of decay connection, the resulting latent code through GAN inversion can further produce faithful reconstruction. Quantitative and qualitative experiments conducted on several time series datasets show that our proposal achieves state-of-the-art or competitive imputation performance.

UDF-GAN: Unsupervised dense optical-flow estimation using cycle Generative Adversarial Networks

In estimating optical flow using convolutional neural networks, a gap in accuracy exists between supervised and unsupervised training because supervised methods can obtain more explicit clues from GT data. In this study, we bridge this disparity by introducing a cycle adversarial mechanism to provide additional flow priors. Specifically, this mechanism eliminates the requirement for paired data and enables unsupervised training. Furthermore, it provides more explicit clues for the network and optimizes the flow performance by adversarial training between the generator and the discriminator. Additionally, we propose a redesigned generator named the Recursion-Attention Flow Generator (RAFG) and Homology Isomerism Discriminator (HID) to adapt the UDF-GAN to flow estimation tasks. Finally, we use SpyNet as a teacher network to direct the UDF-GAN to improve the robustness to mode collapse in the early training phase. The experiments were conducted on various public flow datasets, in contrast to several state-of-the-art flow models for flow estimation, that involve CNN- and GAN-based methods trained in unsupervised and supervised manners, respectively.

DCT-GAN: Dilated Convolutional Transformer-Based GAN for Time Series Anomaly Detection

Time series anomaly detection (TSAD) is an essential problem faced in several fields, e.g., fault detection, fraud detection, and intrusion detection, etc. Although TSAD is a crucial problem in anomaly detection, few solutions in anomaly detection are suitable for it at present. Recently, some researchers use GAN-based methods such as TAnoGAN and TadGAN to solve TSAD problem. However, problems such as model collapse, low generalization capability and poor accuracy still exist. In this article, we proposed a Dilated Convolutional Transformer-based GAN (DCT-GAN) to enhance accuracy and improve generalization capability of the model. Specifically, DCT-GAN utilize several generators and a single discriminator to alleviate the mode collapse problem. Each generator consists of a dilated convolutional neural network and a Transformer block to obtain fine-grained and coarse-grained information of the time series, which is a useful component to improve generalization capability. We also use weight-based mechanism to balance these generators. Experiments verify the effectiveness of our method and each part of DCT-GAN.

Style attention based global-local aware GAN for personalized facial caricature generation.

Caricature is an exaggerated pictorial representation of a person, which is widely used in entertainment and political media. Recently, GAN-based methods achieved automatic caricature generation through transferring caricature style and performing shape exaggeration simultaneously. However, the caricature synthesized by these methods cannot perfectly reflect the characteristics of the subject, whose shape exaggeration are not reasonable and requires facial landmarks of caricature. In addition, the existing methods always produce the bad cases in caricature style due to the simpleness of their style transfer method. In this paper, we propose a Style Attention based Global-local Aware GAN to apply the characteristics of a subject to generate personalized caricature. To integrate the facial characteristics of a subject, we introduce a landmark-based warp controller for personalized shape exaggeration, which employs the facial landmarks as control points to warp image according to its facial features, without requirement of the facial landmarks of caricature. To fuse the facial feature with caricature style appropriately, we introduce a style-attention module, which adopts an attention mechanism, instead of the simple Adaptive Instance Normalization (AdaIN) for style transfer. To reduce the bad cases and increase the quality of generated caricatures, we propose a multi-scale discriminator to both globally and locally discriminate the synthesized and real caricature, which improves the whole structure and realistic details of the synthesized caricature. Experimental results on two publicly available datasets, the WebCaricature and the CaVINet datasets, validate the effectiveness of our proposed method and suggest that our proposed method achieves better performance than the existing methods. The caricatures generated by the proposed method can not only preserve the identity of input photo but also the characteristic shape exaggeration for each person, which are highly close to the real caricatures drawn by real artists. It indicates that our method can be adopted in the real application.

DualDiscWaveGAN-Based Data Augmentation Scheme for Animal Sound Classification.

Animal sound classification (ASC) refers to the automatic identification of animal categories by sound, and is useful for monitoring rare or elusive wildlife. Thus far, deep-learning-based models have shown good performance in ASC when training data is sufficient, but suffer from severe performance degradation if not. Recently, generative adversarial networks (GANs) have shown the potential to solve this problem by generating virtual data. However, in a multi-class environment, existing GAN-based methods need to construct separate generative models for each class. Additionally, they only consider the waveform or spectrogram of sound, resulting in poor quality of the generated sound. To overcome these shortcomings, we propose a two-step sound augmentation scheme using a class-conditional GAN. First, common features are learned from all classes of animal sounds, and multiple classes of animal sounds are generated based on the features that consider both waveforms and spectrograms using class-conditional GAN. Second, we select data from the generated data based on the confidence of the pretrained ASC model to improve classification performance. Through experiments, we show that the proposed method improves the accuracy of the basic ASC model by up to 18.3%, which corresponds to a performance improvement of 13.4% compared to the second-best augmentation method.

Fast underwater image enhancement based on a generative adversarial framework

Underwater image enhancement is a fundamental requirement in the field of underwater vision. Along with the development of deep learning, underwater image enhancement has made remarkable progress. However, most deep learning-based enhancement methods are computationally expensive, restricting their application in real-time large-size underwater image processing. Furthermore, GAN-based methods tend to generate spatially inconsistent styles that decrease the enhanced image quality. We propose a novel efficiency model, FSpiral-GAN, based on a generative adversarial framework for large-size underwater image enhancement to solve these problems. We design our model with equal upsampling blocks (EUBs), equal downsampling blocks (EDBs) and lightweight residual channel attention blocks (RCABs), effectively simplifying the network structure and solving the spatial inconsistency problem. Enhancement experiments on many real underwater datasets demonstrate our model's advanced performance and improved efficiency.

Blind Image Restoration Based on Cycle-Consistent Network

This paper studies the blind image restoration where the ground truth is unavailable and the downsampling process is unknown. This complicated setting makes supervised learning and accurate kernel estimation impossible. Inspired by the recent success of image-to-image translation, this paper resorts to the unsupervised Cycle-consistent based framework to tackle this challenging problem. Different from the image-to-image task, the fidelity of reconstructed image is important for image restoration. Therefore, to improve the reconstruction ability of the Cycle-consistent network, we make explorations from the following aspects. First, we constrain low-frequency content in data to preserve the content of output from LR input. Second, we impose constraint on the content of training data to provide better supervision for discriminator, helping to suppress high-frequency artifacts or fake textures. Third, we average model parameters to further improve the generated image quality and help with model selection for GAN-based methods. Since GAN-based methods tend to produce various artifacts with different models, model average could realize a smoother control of balancing artifacts and fidelity. We have conducted extensive experiments on real noise and super resolution datasets to validate the effectiveness of the above techniques. The proposed ECycleGAN also demonstrates superior performance to SOTA methods in two applications -- blind SR and blind denoising.

Data Augmentation and Classification of Sea–Land Clutter for Over-the-Horizon Radar Using AC-VAEGAN

In the sea-land clutter classification of sky-wave over-the-horizon-radar (OTHR), the imbalanced and scarce data leads to a poor performance of the deep learning-based classification model. To solve this problem, this paper proposes an improved auxiliary classifier generative adversarial network (AC-GAN) architecture, namely auxiliary classifier variational autoencoder generative adversarial network (AC-VAEGAN). AC-VAEGAN can synthesize higher quality sea-land clutter samples than AC-GAN and serve as an effective tool for data augmentation. Specifically, a one-dimensional convolutional AC-VAEGAN architecture is designed to synthesize sea-land clutter samples. Additionally, an evaluation method combining both traditional evaluation of GAN domain and statistical evaluation of signal domain is proposed to evaluate the quality of synthetic samples. Using a dataset of OTHR sea-land clutter, both the quality of the synthetic samples and the performance of data augmentation of AC-VAEGAN are verified. Further, the effect of AC-VAEGAN as a data augmentation method on the classification performance of imbalanced and scarce sea-land clutter samples is validated. The experiment results show that the quality of samples synthesized by AC-VAEGAN is better than those synthesized by the state-of-the-art GAN-based methods, and the data augmentation method with AC-VAEGAN is able to improve the classification performance in the case of imbalanced and scarce sea-land clutter samples.

FTGAN: A Novel GAN-Based Data Augmentation Method Coupled Time–Frequency Domain for Imbalanced Bearing Fault Diagnosis

For imbalanced bearing fault diagnosis, generative adversarial networks (GANs) are a common data augmentation (DA) approach. Nevertheless, current GAN-based methods cannot update the generator from time–frequency domain simultaneously, downgrading the authenticity of signal time–frequency character. In this article, Fourier-like transform GAN (FTGAN), a novel GAN method, is proposed by introducing a Fourier-like transformer (FLT) based on autoencoder (AE) to improve synthetic data quality. FLT approximates the discrete Fourier transform (DFT) by the neural network, learning a universal map from time to frequency domain during training. FTGAN with FLT can decouple input into a time–frequency domain, fitting the distribution of time and frequency of data simultaneously. Multidomain distribution is manipulated in FTGAN without introducing additional signal transformation means. Furthermore, train on real, test on synthetic (TRTS) and train on synthetic, test on real (TSTR) analyses of 1-D data are introduced to evaluate data quality. Real and synthetic data are applied as training or test sets of diagnostic classifiers by turns so that data quality can be analyzed through diagnosis results. Experiment results show that the proposed method can generate bearing fault signals closer to real data in the time and frequency domains, effectively improving the performance under an imbalanced dataset.

An Efficient Lightweight Generative Adversarial Network for Compressed Sensing Magnetic Resonance Imaging Reconstruction

Compressed-sensing-based magnetic resonance imaging (CS-MRI) methods can significantly shorten scanning time while ensuring reconstructed image quality. Recently, deep learning methods, particularly generative adversarial networks (GAN), have been introduced into CS-MRI. However, these GAN-based methods suffer from their heavy learning parameters and ignore long-range dependency, which degrades the reconstructed image quality. Thus, the objective of this study is to design an efficient lightweight GAN to achieve more accurate MRI reconstruction. The proposed framework, named SepGAN, utilises depthwise separable convolution as the basic component to reduce the number of learning parameters. Two modules, the dilated depthwise separable convolution dense block and a squeeze-and-excitation lightweight self-attention module were proposed to extract the long-range dependency and improve the representational ability. The focal frequency loss was also involved in assisting the model to focus on high-frequency information. To evaluate the performance of the three proposed methods and SepGAN, two brain datasets were used in our experiment. From the results of our comparison analysis, SepGAN possesses the minimum number of parameters and multiply–accumulate operations (i.e., 7.32 M and 13.62G) and outperforms other methods in a variety of evaluation metrics, especially in Frechet inception distance, proving that reconstructed images of our method have better visual effects. For unseen pathological data, SepGAN can also perform effective reconstruction with explicit tumour textures and boundaries. The experimental results demonstrate that SepGAN can reconstruct high quality images with fewer parameters and exhibit remarkable generalisation ability.

ICoT-GAN: Integrated Convolutional Transformer GAN for Rolling Bearings Fault Diagnosis Under Limited Data Condition

Limited operating data resulting from complex and changeable working conditions significantly undermines the performance of deep learning-based methods for rolling bearing fault diagnosis. Generally, this problem can be solved by using the generative adversarial network (GAN) to augment data. However, most GAN-based methods seldom comprehensively consider the global interactions and local dependencies in raw vibration signals during data generation, leading to a decline in the quality of the generated data and compromising the diagnostic accuracy. To address this problem, a fault diagnosis method based on integrated convolutional transformer GAN (ICoT-GAN) is proposed to improve the diagnostic performance under limited data condition by generating high-quality signals. Firstly, a new data augmentation model ICoT-GAN is developed. In ICoT-GAN, a novel ICoT block is designed to construct the generator and discriminator. The ICoT block achieves the integration of attention-based global information capture and convolution-based local feature extraction through the incorporation of convolution within the transformer encoder. This design allows the ICoT-GAN to comprehensively extract both global and local time-series features of raw signals and generate high-quality signals. Secondly, a novel data evaluation indicator, referred to as the Multiple Time-Domain Features Indicator (MTFI), is designed to quantitatively evaluate generated signals’ quality by calculating the similarity of time-domain features between the real and generated signals. The MTFI can complement probability distribution indicators and provide a comprehensive evaluation of the data augmentation model’s generation ability by considering both time-series feature similarity and probability distribution differences. Finally, the effectiveness of our proposed method has been successfully demonstrated under limited data condition using the CWRU and IMS bearing datasets. With only 16 training samples per class, our proposed method achieves diagnostic accuracy of 99.99% and 99.70%, respectively. Additionally, the data generation time is only 1163.31s, indicating the efficiency of our proposed method.

SPN2D-GAN: Semantic Prior Based Night-to-Day Image-to-Image Translation

Existing image-to-image translation approaches can deal with simple scenes or styles effectively, such as summer-to-winter, horses-to-zebra, and photo-to-map. Although a great progress has been made by GAN-based methods recently, the performance of night-to-day (N2D) translation remains unsatisfactory due to imbalanced/poor visibility, and thus leading to translation ambiguity. To improve the quality of N2D translation, we propose an unpaired translation scheme based on a semantic prior generator, namely SPN2D-GAN, in a weakly-supervised manner with consideration of both image and semantic information. Specifically, we design a novel N2D generator, which can adopt the semantic information of images as prior knowledge to generate more reasonable and realistic results. Also, we suggest adjusting the brightness of nighttime images to boost the visibility, so that the generator can better extract content information. Moreover, the proposed SPN2D-GAN translates images by enforcing the distribution of daytime images in both image and semantic domains on final outputs. Besides, the cycle consistency is employed to preserve the fidelity between translations from two directions. Extensive experimental results are provided to reveal the effectiveness of our design, and demonstrate its superior performance over other state-of-the-art N2D translation approaches both quantitatively and qualitatively. Our code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/brightknight177/SPN2DGAN</uri> .

Structure-Preserving Image Super-Resolution.

Structures matter in single image super-resolution (SISR). Benefiting from generative adversarial networks (GANs), recent studies have promoted the development of SISR by recovering photo-realistic images. However, there are still undesired structural distortions in the recovered images. In this paper, we propose a structure-preserving super-resolution (SPSR) method to alleviate the above issue while maintaining the merits of GAN-based methods to generate perceptual-pleasant details. First, we propose SPSR with gradient guidance (SPSR-G) by exploiting gradient maps of images to guide the recovery in two aspects. On the one hand, we restore high-resolution gradient maps by a gradient branch to provide additional structure priors for the SR process. On the other hand, we propose a gradient loss to impose a second-order restriction on the super-resolved images, which helps generative networks concentrate more on geometric structures. Second, since the gradient maps are handcrafted and may only be able to capture limited aspects of structural information, we further extend SPSR-G by introducing a learnable neural structure extractor (NSE) to unearth richer local structures and provide stronger supervision for SR. We propose two self-supervised structure learning methods, contrastive prediction and solving jigsaw puzzles, to train the NSEs. Our methods are model-agnostic, which can be potentially used for off-the-shelf SR networks. Experimental results on five benchmark datasets show that the proposed methods outperform state-of-the-art perceptual-driven SR methods under LPIPS, PSNR, and SSIM metrics. Visual results demonstrate the superiority of our methods in restoring structures while generating natural SR images. Code is available at https://github.com/Maclory/SPSR.

Dual Wasserstein generative adversarial network condition: A generative adversarial network-based acoustic impedance inversion method

Deep learning neural networks offer some advantages over conventional methods in acoustic impedance inversion. Because labeled data may be difficult to obtain in realistic field data settings, it can be difficult to obtain high-accuracy inversion results. Some generative adversarial network (GAN)-based acoustic impedance inversion methods have been proposed to solve this problem. However, due to the existence of lateral discontinuity in inversion results of these GAN-based methods, inversion accuracy of these proposals is still not fully satisfactory. Therefore, to tackle the shortcomings of GAN, we develop an acoustic impedance inversion method based on dual Wasserstein GAN condition (dual-WGAN-c). Dual-WGAN-c can perform seismic inversion and forward calculation simultaneously and take the low-frequency information of acoustic impedance as conditional input to obtain accurate acoustic impedance inversion results in the case of insufficient labeled data. Through a data simulation experiment, compared with GAN-based seismic acoustic impedance inversion methods, dual-WGAN-c can obtain more accurate inversion results, with high robustness when the seismic data are noisy. Dual-WGAN-c also can obtain accurate results in acoustic impedance inversion of the recorded seismic data when one has insufficient labeled well-log data.

Synthetic augmentation for semantic segmentation of class imbalanced biomedical images: A data pair generative adversarial network approach

In recent years, deep learning (DL) has been recognized very useful in the semantic segmentation of biomedical images. Such an application, however, is significantly hindered by the lack of pixel-wise annotations. In this work, we propose a data pair generative adversarial network (DPGAN) for the purpose of synthesizing concurrently the diverse biomedical images and the segmentation labels from random latent vectors. First, a hierarchical structure is constructed consisting of three variational auto-encoder generative adversarial networks (VAEGANs) with an extra discriminator. Subsequently, to alleviate the influence from the imbalance between lesions and non-lesions areas in biomedical segmentation data sets, we divide the DPGAN into three stages, namely, background stage, mask stage and advanced stage, with each stage deploying a VAEGAN. In such a way, a large number of new segmentation data pairs are generated from random latent vectors and then used to augment the original data sets. Finally, to validate the effectiveness of the proposed DPGAN, experiments are carried out on a vestibular schwannoma data set, a kidney tumor data set and a skin cancer data set. The results indicate that, in comparison to other state-of-the-art GAN-based methods, the proposed DPGAN shows better performance in the generative quality, and meanwhile, gains an effective boost on semantic segmentation of class imbalanced biomedical images.

Fairness and privacy preservation for facial images: GAN-based methods

Facial images are widely adopted for computer vision tasks such as face recognition or attribute classifications. Consequently, the adoption of mass real facial images leads to significant identification privacy leakage concerns. Meanwhile, the model classification results suffer unfair predictions towards features such as genders due to biased training data distributions. Although methods have been proposed to resolve the privacy and fairness issues separately, simultaneous protection methods are merely studied. In this study, for facial attributes classifications, we propose one unified framework with GAN models to generate synthetic images for privacy protections and contrastive learning based loss designs to enforce fairness protections simultaneously. Meanwhile, unlike other privacy or fairness protection methods, the proposed methods can maintain high data and model utilities. We evaluate our approaches with the high image resolution dataset CelebA-HD, and the results show our methods meet both privacy and fairness requirements.

Integrated Schematic Design Method for Shear Wall Structures: A Practical Application of Generative Adversarial Networks

The intelligent design method based on generative adversarial networks (GANs) represents an emerging structural design paradigm where design rules are not artificially defined but are directly learned from existing design data. GAN-based methods have exhibited promising potential compared to conventional methods in the schematic design phase of reinforced concrete (RC) shear wall structures. However, for the following reasons, it is challenging to apply GAN-based approaches in the industry and to integrate them into the structural design process. (1) The data form of GAN-based methods is heterogeneous from that of the widely used computer-aided design (CAD) methods, and (2) GAN-based methods have high requirements on the hardware and software environment of the user’s computer. As a result, this study proposes an integrated schematic design method for RC shear wall structures, providing a workable GAN application strategy. Specifically, (1) a preprocessing method of architectural CAD drawings is proposed to connect the GAN with the upstream architectural design; (2) a user-friendly cloud design platform is built to reduce the requirements of the user’s local computer environment; and (3) a heterogeneous data transformation method and a parametric modeling procedure are proposed to automatically establish a structural analysis model based on GAN’s design, facilitating downstream detailed design tasks. The proposed method makes it possible for the entire schematic design phase of RC shear wall structures to be intelligent and automated. A case study reveals that the proposed method has a heterogeneous data transformation accuracy of 97.3% and is capable of generating shear wall layout designs similar to the designs of a competent engineer, with 225 times higher efficiency.

TR-GAN: Multi-Session Future MRI Prediction With Temporal Recurrent Generative Adversarial Network

Magnetic Resonance Imaging (MRI) has been proven to be an efficient way to diagnose Alzheimer's disease (AD). Recent dramatic progress on deep learning greatly promotes the MRI analysis based on data-driven CNN methods using a large-scale longitudinal MRI dataset. However, most of the existing MRI datasets are fragmented due to unexpected quits of volunteers. To tackle this problem, we propose a novel Temporal Recurrent Generative Adversarial Network (TR-GAN) to complete missing sessions of MRI datasets. Unlike existing GAN-based methods, which either fail to generate future sessions or only generate fixed-length sessions, TR-GAN takes all past sessions to recurrently and smoothly generate future ones with variant length. Specifically, TR-GAN adopts recurrent connection to deal with variant input sequence length and flexibly generate future variant sessions. Besides, we also design a multiple scale & location (MSL) module and a SWAP module to encourage the model to better focus on detailed information, which helps to generate high-quality MRI data. Compared with other popular GAN architectures, TR-GAN achieved the best performance in all evaluation metrics of two datasets. After expanding the Whole MRI dataset, the balanced accuracy of AD vs. cognitively normal (CN) vs. mild cognitive impairment (MCI) and stable MCI vs. progressive MCI classification can be increased by 3.61% and 4.00%, respectively.