Dual Generative Adversarial Network Research Articles

A learnable full-frequency transformer dual generative adversarial network for underwater image enhancement

Underwater applications present unique challenges such as color deviation, noise, and low contrast, which can degrade image quality. Addressing these issues, we propose a novel approach called the learnable full-frequency transformer dual generative adversarial network (LFT-DGAN). Our method comprises several key innovations. Firstly, we introduce a reversible convolution-based image decomposition technique. This method effectively separates underwater image information into low-, medium-, and high-frequency domains, enabling more thorough feature extraction. Secondly, we employ image channels and spatial similarity to construct a learnable full-frequency domain transformer. This transformer facilitates interaction between different branches of information, enhancing the overall image processing capabilities. Finally, we develop a robust dual-domain discriminator capable of learning spatial and frequency domain characteristics of underwater images. Extensive experimentation demonstrates the superiority of the LFT-DGAN method over state-of-the-art techniques across multiple underwater datasets. Our approach achieves significantly improved quality and evaluation metrics, showcasing its effectiveness in addressing the challenges posed by underwater imaging. The code can be found at https://github.com/zhengshijian1993/LFT-DGAN.

Dual generative adversarial networks for merging ocean transparency from satellite observations

ABSTRACT Satellite ocean transparency data have low spatial coverage due to cloud shading, sun glint, swath width, and temporal revisit. Merging multiple satellite ocean transparency data can improve spatial coverage and create a high-accuracy data set. This study proposed a new satellite ocean transparency merging model based on dual generative adversarial networks (ZSD-merging GAN), and the products of full-coverage and high-accuracy ocean transparency were produced. The ZSD-merging GAN comprises the guess GAN and the merging GAN. The guess GAN is used to generate the guess of the ocean transparency merged product, while the merging GAN combines the guess and satellite ocean transparency data to produce the merged product. The experiments show that the spatial coverage of the ZSD-merging GAN product is 100%. The root-mean-square error (RMSE) and average relative error (ARE) between the ZSD-merging GAN product and unmerged ocean transparency data from the Visible Infrared Imaging Radiometer Suite (VIIRS) on JPSS1 are 4.31 m and 11%, respectively, which are better than 5.59 m and 17% for historical average, 5.55 m and 19% for guess product, 5.08 m and 17% for Poisson blending product, and 5.12 m and 21% for Kriging interpolation product.

PadGAN: An End-to-End dMRI Data Augmentation Method for Macaque Brain

Currently, an increasing number of macaque brain MRI datasets are being made publicly accessible. Unlike human, publicly accessible macaque brain datasets suffer from data quality in diffusion magnetic resonance imaging (dMRI) data. Typically, dMRI data require a minimum ratio of 1:10 between low b-value (b < 10) volumes and high b-value (b > 300) volumes. However, the currently accessible macaque datasets do not meet this ratio. Due to site differences in macaque brain images, traditional human brain image-to-image translation models struggle to perform well on macaque brain images. Our work introduces a novel end-to-end primary-auxiliary dual generative adversarial network (PadGAN) for generating low b-value images. The auxiliary generator in the PadGAN is responsible for extracting the latent space features from peak information maps and transmitting them to the primary generator, enabling the primary generator to generate images with rich details. Experimental results demonstrate that PadGAN outperforms existing methods both qualitatively and quantitatively (mean SSIM increased by 0.1139). Diffusion probabilistic tractography using dMRI data augmented by our method yields superior results.

PECT Composite Defect Detection Algorithm Based on DualGAN

To address the problems of insufficient accuracy and slow reconstruction speed of Planar Electrical Capacitance Tomography (PECT) detection of damaged specimens, a Dual Generative Adversarial Networks (DualGAN)-based PECT image defect detection method is proposed in this paper. The improved particle swarm algorithm with adaptive particle number and L2-norm is used to optimize the sensitivity field, combined with the parallel Landweber algorithm to solve the PECT inverse problem to obtain the dielectric constant distribution map. In the DualGAN network, the Unet generator utilizes an Adam-based local attention mechanism to adjust module weights, facilitating feature extraction and the generation of high-quality transformation images of the Landweber dielectric constant distribution. A PatchGAN discriminator is employed to distinguish between transformation images and real images, using the generated transformation images as target images. Experimental results demonstrate that the sensitivity field, enhanced by the improved particle swarm algorithm and L2-norm normalization, achieves better balance. Furthermore, the addition of a network transformation using the Adam-based local attention weight mechanism on the DualGAN network reduces artifacts in the reconstructed images, resulting in more accurate PECT reconstructions. The PECT image defect detection method, integrating DualGAN, an improved particle swarm optimization algorithm, and a local attention mechanism, has made significant strides in addressing challenges related to image reconstruction accuracy and speed. This technological advancement has enhanced the precision and efficiency of defect detection in carbon fiber composite materials, thereby fostering the broader utilization of planar capacitance tomography technology in industrial damage detection and material defect analysis.

Lifelong Dual Generative Adversarial Nets Learning in Tandem.

Continually capturing novel concepts without forgetting is one of the most critical functions sought for in artificial intelligence systems. However, even the most advanced deep learning networks are prone to quickly forgetting previously learned knowledge after training with new data. The proposed lifelong dual generative adversarial networks (LD-GANs) consist of two generative adversarial networks (GANs), namely, a Teacher and an Assistant teaching each other in tandem while successively learning a series of tasks. A single discriminator is used to decide the realism of generated images by the dual GANs. A new training algorithm, called the lifelong self knowledge distillation (LSKD) is proposed for training the LD-GAN while learning each new task during lifelong learning (LLL). LSKD enables the transfer of knowledge from one more knowledgeable player to the other jointly with learning the information from a newly given dataset, within an adversarial playing game setting. In contrast to other LLL models, LD-GANs are memory efficient and does not require freezing any parameters after learning each given task. Furthermore, we extend the LD-GANs to being the Teacher module in a Teacher-Student network for assimilating data representations across several domains during LLL. Experimental results indicate a better performance for the proposed framework in unsupervised lifelong representation learning when compared to other methods.

Dual generative adversarial networks based on regression and neighbor characteristics.

Imbalanced data is a problem in that the number of samples in different categories or target value ranges varies greatly. Data imbalance imposes excellent challenges to machine learning and pattern recognition. The performance of machine learning models leans to be partially towards the majority of samples in the imbalanced dataset, which will further affect the effect of the model. The imbalanced data problem includes an imbalanced categorical problem and an imbalanced regression problem. Many studies have been developed to address the issue of imbalanced classification data. Nevertheless, the imbalanced regression problem has not been well-researched. In order to solve the problem of unbalanced regression data, we define an RNGRU model that can simultaneously learn the regression characteristics and neighbor characteristics of regression samples. To obtain the most comprehensive sample information of regression samples, the model uses the idea of confrontation to determine the proportion between the regression characteristics and neighbor characteristics of the original samples. According to the regression characteristics of the regression samples, an index ccr (correlation change rate) is proposed to evaluate the similarity between the generated samples and the original samples. And on this basis, an RNGAN model is proposed to reduce the similarity between the generated samples and the original samples by using the idea of confrontation.

DA-DAN: A Dual Adversarial Domain Adaption Network for Unsupervised Non-overlapping Cross-domain Recommendation

Unsupervised Non-overlapping Cross-domain Recommendation (UNCR) is the task that recommends source domain items to the target domain users, which is more challenging as the users are non-overlapped, and its learning process is unsupervised. Unsupervised Non-overlapping Cross-domain Recommendation UNCR is still unsolved due to the following: (1) Previous studies need extra auxiliary information to learn transferable features when aligning two domains, which is unrealistic and hard to obtain due to privacy concerns. (2) Since the adoption of the shared network, existing works cannot well eliminate the domain-specific features in the common feature space, which may incorporate domain noise and harm the cross-domain recommendation. In this work, we propose a domain adaption-based method, namely DA-DAN, to address the above challenges. Specifically, to let DA-DAN be free of auxiliary information, we learn users’ preferences by only exploring their sequential patterns, and propose an improved self-attention layer to model them. To well eliminate the domain-specific features from the common feature space, we resort to a dual generative adversarial network with a multi-target adversarial loss, where two generators and discriminators are leveraged to model each domain separately. Experimental results on three real-world datasets demonstrate the advantage of DA-DAN compared with the state-of-the-art recommendation baselines. Moreover, our source codes have been publicly released. 1

End-to-end translation of human neural activity to speech with a dual–dual generative adversarial network

In a recent study of auditory evoked potential (AEP) based brain–computer interface (BCI), it was shown that, with an encoder–decoder framework, it is possible to translate human neural activity to speech (T-CAS). Current encoder–decoder-based methods achieve T-CAS often with a two-step approach where the information is passed between the encoder and decoder with a shared vector of reduced dimension, which, however, may result in information loss. In this paper, we propose an end-to-end model to translate human neural activity to speech (ET-CAS) by introducing a dual–dual generative adversarial network (Dual-DualGAN) for cross-domain mapping between electroencephalogram (EEG) and speech signals. In this model, we bridge the EEG and speech signals by introducing transition signals which are obtained by cascading the corresponding EEG and speech signals in a certain proportion. We then learn the mappings between the speech/EEG signals and the transition signals. We also develop a new EEG dataset where the attention of the participants is detected before the EEG signals are recorded to ensure that the participants have good attention in listening to speech utterances. The proposed method can translate word-length and sentence-length sequences of neural activity to speech. Experimental results show that the proposed method significantly outperforms state-of-the-art methods on both words and sentences of auditory stimulus.

Super resolution of historic Landsat imagery using a dual generative adversarial network (GAN) model with CubeSat constellation imagery for spatially enhanced long-term vegetation monitoring

Detailed spatial representations of terrestrial vegetation are essential for precision agricultural applications and the monitoring of land cover changes in heterogeneous landscapes. The advent of satellite-based remote sensing has facilitated daily observations of the Earth’s surface with high spatial resolution. In particular, a data fusion product such as Planet Fusion has realized the delivery of daily, gap-free surface reflectance data with 3-m pixel resolution through full utilization of relatively recent (i.e., 2018-) CubeSat constellation data. However, the spatial resolution of past satellite sensors (i.e., 30–60 m for Landsat) has restricted the detailed spatial analysis of past changes in vegetation. In order to overcome the spatial resolution constraint of Landsat data for long-term vegetation monitoring, we propose a dual remote-sensing super-resolution generative adversarial network (dual RSS-GAN) approach combining Planet Fusion and Landsat 8 data to simulate spatially enhanced long-term time-series of the normalized difference vegetation index (NDVI) and near-infrared reflectance from vegetation (NIRv). We evaluated the performance of the dual RSS-GAN against in situ tower-based continuous measurements (up to 8 years) and remotely piloted aerial system-based maps of cropland and deciduous forest in the Republic of Korea. The dual RSS-GAN enhanced spatial representations in Landsat 8 images and captured seasonal variation in vegetation indices (R2 > 0.90, for the dual RSS-GAN maps vs in situ data from all sites). Overall, the dual RSS-GAN reduced Landsat 8 vegetation index underestimations compared with in situ measurements; relative bias values of NDVI ranged from − 5.8 % to 0.3 % and − 12.4 % to − 3.7 % for the dual RSS-GAN and Landsat 8, respectively. This improvement was caused by spatial enhancement through the dual RSS-GAN, which reflects fine-scale information from Planet Fusion. Finally, the dual RSS-GAN maps showed both spatial enhancement and reducing the underestimation of vegetation index in historic Landsat dataset from 1984. This study presents a new approach for resolving sub-pixel spatial information in Landsat images.

Dual generative adversarial networks combining conditional assistance and feature enhancement for imbalanced fault diagnosis

The dataset in the application scenario of existing fault diagnosis methods is often balanced, while the data collected under actual working conditions are often imbalanced. Directly applying existing fault diagnosis methods to this scenario will lead to poor diagnosis effect. In view of the above problems, we proposed a method called dual generative adversarial networks (DGANs) combining conditional assistance and feature enhancement. The method uses data augmentation as a basic strategy to supplement imbalanced datasets by generating high-quality data. Firstly, a new generator is designed to build the basic framework by sharing the dual-branch deconvolutional neural networks, and combining the label auxiliary information and the coral distance loss function to ensure the diversity of generated samples. Secondly, a new discriminator was designed, which is based on deep convolutional neural networks and embedded with auxiliary classifiers, further expanding the function of the discriminator. Thirdly, the self-attention module is introduced into both the generator and the discriminator to enhance deep feature learning and improve the quality of generated samples; finally, the proposed method is experimentally validated on datasets of two different testbeds. The experimental results show that the proposed method can generate fake samples with rich diversity and high quality, using these samples to supplement the imbalanced dataset, the effect of imbalanced fault diagnosis has been substantially improved. This method can be used to solve the problem of fault diagnosis in the case of sample imbalance, which often exists in actual working conditions.

ARU-DGAN: A dual generative adversarial network based on attention residual U-Net for magneto-acousto-electrical image denoising.

Magneto-Acousto-Electrical Tomography (MAET) is a multi-physics coupling imaging modality that integrates the high resolution of ultrasound imaging with the high contrast of electrical impedance imaging. However, the quality of images obtained through this imaging technique can be easily compromised by environmental or experimental noise, thereby affecting the overall quality of the imaging results. Existing methods for magneto-acousto-electrical image denoising lack the capability to model local and global features of magneto-acousto-electrical images and are unable to extract the most relevant multi-scale contextual information to model the joint distribution of clean images and noise images. To address this issue, we propose a Dual Generative Adversarial Network based on Attention Residual U-Net (ARU-DGAN) for magneto-acousto-electrical image denoising. Specifically, our model approximates the joint distribution of magneto-acousto-electrical clean and noisy images from two perspectives: noise removal and noise generation. First, it transforms noisy images into clean ones through a denoiser; second, it converts clean images into noisy ones via a generator. Simultaneously, we design an Attention Residual U-Net (ARU) to serve as the backbone of the denoiser and generator in the Dual Generative Adversarial Network (DGAN). The ARU network adopts a residual mechanism and introduces a linear Self-Attention based on Cross-Normalization (CNorm-SA), which is proposed in this paper. This design allows the model to effectively extract the most relevant multi-scale contextual information while maintaining high resolution, thereby better modeling the local and global features of magneto-acousto-electrical images. Finally, extensive experiments on a real-world magneto-acousto-electrical image dataset constructed in this paper demonstrate significant improvements in preserving image details achieved by ARU-DGAN. Furthermore, compared to the state-of-the-art competitive methods, it exhibits a 0.3 dB increase in PSNR and an improvement of 0.47% in SSIM.

Rendering the image of glare effect based on paired and unpaired dual generative adversarial network

It is a great challenge to rendering glare on image as the current rendering algorithms did not consider well the refraction of human eyes, thus the effect of rendering, in some critical application such as vehicle headlamps, is not real and may affect the safety evaluation. The traditional glare rendering algorithm relies on a large number of hand-designed wave optics processing operators, not only cannot complete the rendering work online in real time, but also cannot cope with the complex and changeable imaging conditions in reality. The mainstream generative adversarial network based algorithms in the field of image style translation are introduced to generate glare effect, which could be rendering online in a real time, however they still fail to render some effects such as detail distortion. In this work, we present a novel glare simulation generation method which is the first algorithm to apply a generative model based style transfer method to glare rendering. In a nutshell, a new method named Glare Generation Network is proposed to aggregate the benefits of content diversity and style consistency, which combines both paired and unpaired branch in a dual generative adversarial network. Our approach increase the structural similarity index measure by at least 0.039 on the custom darkroom vehicle headlamp dataset. We further show our method significantly improve the inference speed.

A High-Quality Rice Leaf Disease Image Data Augmentation Method Based on a Dual GAN

Deep learning models need sufficient training samples to support them in the training process; otherwise, overfitting occurs, resulting in model failure. However, in the field of smart agriculture, there are common problems, such as difficulty in obtaining high-quality disease samples and high cost. To solve this problem, this paper proposed a high-quality image augmentation (HQIA) method for generating high-quality rice leaf disease images based on a dual generative adversarial network (GAN). First, the original samples were used to train Improved Training of Wasserstein GANs (WGAN-GP) to generate pseudo-data samples. The pseudo-data samples were put into the Optimized-Real-ESRGAN (Opt-Real-ESRGAN) to generate high-quality pseudo-data samples. Finally, the high-quality pseudo-data samples were put into the disease classification convolutional neural network, and the effectiveness of the method was verified by indicators. Experimental results showed that this method can generate high-quality rice leaf disease images, and the recognition accuracy of high-quality rice disease image samples augmented by this method was 4.57% higher than that of using only the original training set on ResNet18 and 4.1% higher on VGG11. Compared with the data augmentation method only by WGAN-GP, the accuracy of ResNet18 increased by 3.08%, and the accuracy of VGG11 increased by 3.55%. The results demonstrate the effectiveness of the proposed method with limited training datasets.

A Many-Objective Optimization Based Federal Deep Generation Model for Enhancing Data Processing Capability in IoT

The rapid progress of artificial intelligence expands its wide applicability in Internet of Things (IoT). Meanwhile, data insufficient and data source privacy are key supply chain challenges facing IoT especially in the healthcare industry. To address this problem in healthcare IoT, in this article, we propose a skin cancer detection model based on federated learning integrated with deep generation model. First, we employ dual generative adversarial networks to address the problem of insufficient data. In addition, to improve the quality of generated images, we synchronously optimize the sharpness of images, Frechet inception distance, image diversity, and loss using knee point-driven evolutionary algorithm (KnEA). Then, we protect patient information privacy by training federated skin cancer framework. Finally, we employ the ISIC 2018 dataset to test the performance of the proposed training model under different situations, including using identically distributed data, nonidentically distributed data, a sparse convolutional neural network, and a fully connected convolutional neural network. The experiment results demonstrate that the accuracy and area under the curve reach 91% and 88%, respectively. This model can help resolve problems of insufficient data in smart medicine of IoT and protect the privacy of user data while also providing an excellent detection rate.

Image restoration fabric defect detection based on the dual generative adversarial network patch model

The training of supervised learning requires the use of ground truth, which is difficult to obtain in large quantities in production practice. Unsupervised learning requires only flawless and anomalous images of fabrics, but inevitably generates a great deal of background noise when performing result generation, which reduces the quality of results. To overcome these limitations, we propose a new approach: image restoration fabric defect detection based on the dual generative adversarial network patch model (DGPM). We train with a modified generative adversarial network using only flawless and anomalous images of the fabric. We propose the patch model to directly obtain specific information about fabric defects and add a self-attentive model to reduce the generation of background noise. The performance of the DGPM is evaluated on box-, star-, and dot-patterned fabric databases. The true positive rate (TPR) of the box type is 81.56% and the f-measure is 62.69%, the TPR of the dot type is 83.72% and the f-measure is 67.33%, and the TPR of the star type is 79.79% and the f-measure is 64.65%.

Dual generative adversarial networks for automated component layout design of steel frame-brace structures

With the development of artificial intelligence (AI), it gains in popularity to use AI to solve problems in civil engineering. However, the research on AI is mainly focused on the field of structural health monitoring, and less on the field of structural design. As one new direction in the AI domain, the generative adversarial network (GAN) method has developed rapidly, which is able to synthesize high-quality images based on demand. Therefore, it opens a new window for AI-aided automatic structure design. In this paper, a novel GAN-based method, namely FrameGAN, is proposed to realize automated component layout design of steel frame-brace structures. By collecting and processing drawings designed by senior structural engineers, FrameGAN and two mainstream GAN models (pix2pix and pix2pixHD) are tested and compared, which demonstrates the superiority of the proposed FrameGAN. In addition, the design results of FrameGAN are compared and analyzed with those of senior structural engineers based on two unique evaluation metrics, i.e., expert grading and objective comparison. The results show that the design of FrameGAN is close to that of structural engineers, which indicates the availability of FrameGAN in the component layout design of steel frame-brace structures.

One to multiple mapping dual learning: Learning multiple signals from one mixture

Single channel blind source separation (SCBSS) refers to separating multiple sources from a mixture collected by a single sensor. Existing methods for SCBSS have limited performance in separating multiple sources and generalization. To address these problems, an algorithm is proposed in this paper to separate multiple sources from a mixture by designing a parallel dual generative adversarial network (PDualGAN) that can build the relationship between a mixture and the corresponding multiple sources to achieve one-to-multiple cross-domain mapping. This algorithm can be applied to a variety of mixtures including both instantaneous and convolutive mixtures. In addition, new datasets for single channel source separation are created which include the mixtures and corresponding sources for this study. Experiments were performed on four different datasets including both one-dimensional and two-dimensional signals. Experimental results show that the proposed algorithm outperforms state-of-the-art algorithms, measured with peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), source-to-distortion ratio (SDR), source-to-interferences ratio (SIR), relative root mean squared error (RRMSE) and correlation.

Scope of generative adversarial networks (GANs) in image processing

Generative Adversarial Network is the topic of interest in today’s research in the field of image processing and computer vision. A basic GAN model was introduced by Ian Goodfellow et al. in 2014. After that advancement in the field of research in GAN models has been application specific. In computer vision and image to image translation GANs are playing very effective role either in the case of face detection and recognition or in image resolution enhancement and image augmentation. This paper represents a concise overview of various GAN models along with their features and applications. Pix2Pix and conditional GAN models work upon paired datasets while other models like cycle GAN, discover GAN, dual GAN, info GAN, deep convolutional GAN etc. work upon unpaired datasets. Various image datasets which are commonly used for training of generator and discriminator networks are also discussed in this paper. Since partial mode collapse is a common problem to occur during training process for all models, therefore various normalization techniques are also preferred during the training of generator and discriminator networks. As the advancements in GAN models are increasing at a very fast rate, soon these models will also be preferred in commercial applications.

An encoder–decoder based thermo-visible image translation for disguised and undisguised faces

Thermal cameras can capture images even in low light conditions. However, humans cannot recognize human faces in thermal images. Translation of thermal images to visible domain is one solution to the problem of face recognition in thermal images. Most of the research works have proposed Generative Adversarial Networks (GANs) based solutions for thermal to visible image translation. However, GAN is a heavy network that consumes huge amount of resource for thermal to visible image translation. In this paper, we propose an encoder–decoder architecture for thermal to visible image translation of human faces. Since our proposed architecture is not based on GANs, it is lightweight. The proposed method works well for both disguised and non-disguised thermal facial images. Standard comparison parameters such as Peak Signal-to-noise Ratio (PSNR), Structural Similarity Index (SSIM), and Multiscale Structural Similarity Index (MS-SSIM) are used to evaluate the quality of the generated visible images with respect to the ground truth. It has been found that our proposed architecture outperforms the current state-of-the-art image translator architectures namely pix2pix, Cycle-GAN, modified thermal to visible GAN and Dual GAN by a considerable margin for both disguised as well as non-disguised dataset.

Spatiotemporal and kinematic characteristics augmentation using Dual-GAN for ankle instability detection.

Obtaining massive amounts of training data is often crucial for computer-assisted diagnosis using deep learning. Unfortunately, patient data is often small due to varied constraints. We develop a new approach to extract significant features from a small clinical gait analysis dataset to improve computer-assisted diagnosis of Chronic Ankle Instability (CAI) patients. In this paper, we present an approach for augmenting spatiotemporal and kinematic characteristics using the Dual Generative Adversarial Networks (Dual-GAN) to train a series of modified Long Short-Term Memory (LSTM) detection models making the training process more data-efficient. Namely, we use LSTM-, LSTM-Fully Convolutional Networks (FCN)-, and Convolutional LSTM-based detection models to identify the patients with CAI. The Dual-GAN enables the synthesized data to approximate the real data distribution visualized by the t-distributed Stochastic Neighbor Embedding (t-SNE) algorithm. Then we trained the proposed detection models using real data collected from a controlled laboratory study and mixed data from real and synthesized gait features. The detection models were tested in real data to validate the positive role in data augmentation as well as to demonstrate the capability and effectiveness of the modified LSTM algorithm for CAI detection using spatiotemporal and kinematic characteristics in walking. Dual-GAN generated efficient spatiotemporal and kinematic characteristics to augment the training set promoting the performance of CAI detection and the modified LSTM algorithm yielded an enhanced classification outcome to identify those CAI patients from a group of control subjects based on gait analysis data than any previous reports.