Cycle Generative Adversarial Network Research Articles

Enhancing haze removal and super-resolution in real-world images: a cycle generative adversarial network-based approach for synthesizing paired hazy and clear images

Haze significantly impacts various fields, such as autonomous driving, smart cities, and security monitoring. Deep learning has been proven effective in removing haze from images. However, obtaining pixel-aligned hazy and clear paired images in the real world can be challenging. Therefore, synthesized hazed images are often used for training deep networks. These images are typically generated based on parameters such as depth information and atmospheric scattering coefficient. However, this approach may cause the loss of important haze details, leading to color distortion or incomplete dehazed images. To address this problem, this paper proposes a method for synthesizing hazed images using a cycle generative adversarial network (CycleGAN). The CycleGAN is trained with unpaired hazy and clear images to learn the features of the hazy images. Then, the real haze features are added to clear images using the trained CycleGAN, resulting in well-pixel-aligned synthesized hazy and clear paired images that can be used for dehaze training. The results demonstrate that the dataset synthesized using this method efficiently solves the problem associated with traditional synthesized datasets. Furthermore, the dehazed images are restored using a super-resolution algorithm, enabling the obtainment of high-resolution clear images. This method has broadened the applications of deep learning in haze removal, particularly highlighting its potential in the fields of autonomous driving and smart cities.

Simulation of Facial Palsy Using Cycle GAN with Skip-Layer Excitation Module and Self-Supervised Discriminator

The Yanagihara method is used to evaluate facial nerve palsy based on visual examinations by physicians. Examples of scored images are important for educational purposes and as references, however, due to patient privacy concern, actual facial images of real patients cannot be used for educational purposes. In this paper, we propose a solution to this problem by generating facial images of a virtual patient with facial nerve palsy, that can be shared and utilized by physicians. To reproduce the patient facial expression in a public face image, we propose a method to generate a swapped face image using the improved Cycle Generative Adversarial Networks (Cycle GAN) with a skiplayer excitation module and a self-supervised discriminator. Experimental results demonstrate that the proposed model can generate more coherent swapped faces that are similar to the public face identity and patient facial expressions. The proposed method also improves the quality of generated swapped face images while keeping them identical to the source (genuine) face image.

Leveraging generative adversarial networks to create realistic scanning transmission electron microscopy images

The rise of automation and machine learning (ML) in electron microscopy has the potential to revolutionize materials research through autonomous data collection and processing. A significant challenge lies in developing ML models that rapidly generalize to large data sets under varying experimental conditions. We address this by employing a cycle generative adversarial network (CycleGAN) with a reciprocal space discriminator, which augments simulated data with realistic spatial frequency information. This allows the CycleGAN to generate images nearly indistinguishable from real data and provide labels for ML applications. We showcase our approach by training a fully convolutional network (FCN) to identify single atom defects in a 4.5 million atom data set, collected using automated acquisition in an aberration-corrected scanning transmission electron microscope (STEM). Our method produces adaptable FCNs that can adjust to dynamically changing experimental variables with minimal intervention, marking a crucial step towards fully autonomous harnessing of microscopy big data.

Defogging computational ghost imaging via eliminating photon number fluctuation and a cycle generative adversarial network

Imaging through fluctuating scattering media such as fog is of challenge since it seriously degrades the image quality. We investigate how the image quality of computational ghost imaging is reduced by fluctuating fog and how to obtain a high-quality defogging ghost image. We show theoretically and experimentally that the photon number fluctuations introduced by fluctuating fog is the reason for ghost image degradation. An algorithm is proposed to process the signals collected by the computational ghost imaging device to eliminate photon number fluctuations of different measurement events. Thus, a high-quality defogging ghost image is reconstructed even though fog is evenly distributed on the optical path. A nearly 100% defogging ghost image is obtained by further using a cycle generative adversarial network to process the reconstructed defogging image.

Dual-scale similarity-guided cycle generative adversarial network for unsupervised low-dose CT denoising

Removing the noise in low-dose CT (LDCT) is crucial to improving the diagnostic quality. Previously, many supervised or unsupervised deep learning-based LDCT denoising algorithms have been proposed. Unsupervised LDCT denoising algorithms are more practical than supervised ones since they do not need paired samples. However, unsupervised LDCT denoising algorithms are rarely used clinically due to their unsatisfactory denoising ability. In unsupervised LDCT denoising, the lack of paired samples makes the direction of gradient descent full of uncertainty. On the contrary, paired samples used in supervised denoising allow the parameters of networks to have a clear direction of gradient descent. To bridge the gap in performance between unsupervised and supervised LDCT denoising, we propose dual-scale similarity-guided cycle generative adversarial network (DSC-GAN). DSC-GAN uses similarity-based pseudo-pairing to better accomplish unsupervised LDCT denoising. We design a Vision Transformer-based global similarity descriptor and a residual neural network-based local similarity descriptor for DSC-GAN to effectively describe the similarity between two samples. During training, pseudo-pairs, i.e., similar LDCT samples and normal-dose CT (NDCT) samples, dominate parameter updates. Thus, the training can achieve equivalent effect as training with paired samples. Experiments on two datasets demonstrate that DSC-GAN beats the state-of-the-art unsupervised algorithms and reaches a level close to supervised LDCT denoising algorithms.

Improved Nighttime Traffic Detection Using Day-To-Night Image Transfer

With the rapid development of convolutional neural networks (CNNs), real-time traffic monitoring, which benefits both traffic optimization and management, has been widely applied with smart cameras on streets. However, compared with the daytime traffic detection, nighttime traffic detection is limited by labeling annotations and is thus unstable, inaccurate, and inefficient in practice. To address this issue, this study proposes to use the augmented nighttime traffic images generated by cycle generative adversarial networks (CycleGAN) for better nighttime detection performance. Using CycleGAN, transferred nighttime traffic images are generated by using the daytime traffic images, as both share the same annotations of traffic instances in the daytime. For comparison, different learning rates and crop sizes are adopted for day-to-night traffic image transfer. The previously proposed detection network, dense traffic detection network (DTDNet), is adopted to train the prepared image data set. The indicators of mean average precision (mAP), precision, and recall are adopted for training performance evaluation and comparison. Based on the visualization results, CycleGAN, with a learning rate of 2e-5 and a crop size of 64, has better performance on day-to-night traffic image transfer with our proposed image data set. Considering the indicators of training performance, DTDNet, with 60% of transferred nighttime images and 40% of the original images, has better accuracy in four categories. Overall, this study provides a possible solution for addressing the issue of training data limitation in nighttime traffic detection and demonstrates the potential of GAN-based data augmentation in the transportation domain.

CBCT-based synthetic CT generated using CycleGAN with HU correction for adaptive radiotherapy of nasopharyngeal carcinoma

This study aims to utilize a hybrid approach of phantom correction and deep learning for synthesized CT (sCT) images generation based on cone-beam CT (CBCT) images for nasopharyngeal carcinoma (NPC). 52 CBCT/CT paired images of NPC patients were used for model training (41), validation (11). Hounsfield Units (HU) of the CBCT images was calibrated by a commercially available CIRS phantom. Then the original CBCT and the corrected CBCT (CBCT_cor) were trained separately with the same cycle generative adversarial network (CycleGAN) to generate SCT1 and SCT2. The mean error and mean absolute error (MAE) were used to quantify the image quality. For validations, the contours and treatment plans in CT images were transferred to original CBCT, CBCT_cor, SCT1 and SCT2 for dosimetric comparison. Dose distribution, dosimetric parameters and 3D gamma passing rate were analyzed. Compared with rigidly registered CT (RCT), the MAE of CBCT, CBCT_cor, SCT1 and SCT2 were 346.11 ± 13.58 HU, 145.95 ± 17.64 HU, 105.62 ± 16.08 HU and 83.51 ± 7.71 HU, respectively. Moreover, the average dosimetric parameter differences for the CBCT_cor, SCT1 and SCT2 were 2.7% ± 1.4%, 1.2% ± 1.0% and 0.6% ± 0.6%, respectively. Using the dose distribution of RCT images as reference, the 3D gamma passing rate of the hybrid method was significantly better than the other methods. The effectiveness of CBCT-based sCT generated using CycleGAN with HU correction for adaptive radiotherapy of nasopharyngeal carcinoma was confirmed. The image quality and dose accuracy of SCT2 were outperform the simple CycleGAN method. This finding has great significance for the clinical application of adaptive radiotherapy for NPC.

Logistics box recognition in robotic industrial de-palletising procedure with systematic RGB-D image processing supported by multiple deep learning methods

In an automated box de-palletisation system that utilises robots, vision-based box recognition on the pallet plays the main role in providing picking guidelines. The complexity of the working condition and the target object, particularly the cluttered arrangement and various outer surfaces of the boxes, significantly affect the quality of the outcome. Typically, a large-scale vision dataset is required to train a deep learning object-detection model. However, considerable effort and time is required to achieve this. Therefore, this study proposes a Mask R-CNN-based detection approach for box objects, which is supported by a cycle generative adversarial network (Cycle GAN). The purpose of the Cycle-GAN is to optimise the outer surfaces of boxes by automatically erasing tags, stickers, labels, and symbols that exist on the boxes before loading them to the Mask R-CNN for detection. Subsequently, the obtained result was combined with the output from the developed boundary-enhancing technique that was applied to a depth map. Consequently, the box detection performance was significantly improved, and it was confirmed through experiments with a practical robot system in picking tasks. In the experiments, the success rate of the proposed method was validated using 200 cases of orderly and disorderly arrangements of boxes, respectively. Furthermore, the metric of the mean absolute error between the predicted picking point and the ground truth values for the test cases in the implementation process for the robot operation was also researched.

Improved CycleGAN for super-resolution of engineering drawings

In intelligent manufacturing, the quality of machine translation engineering drawings directly affects manufacturing accuracy. Currently, most engineering drawing translation work is done manually, which greatly reduces production efficiency. This paper proposes an automatic translation method for welded structure engineering drawings based on the Cycle Generative Adversarial Network (CycleGAN). The CycleGAN model of unpaired transfer learning is used to learn the feature mapping of real welding engineering drawings to realize their automatic translation. U-Net and PatchGAN are used as the generator and discriminator, respectively. Based on the idea of removing the identity mapping function, a high-dimensional sparse network is proposed to replace the traditional dense network for the CycleGAN generator in order to improve noise robustness. It is found that increasing the residual block hidden layer increases the resolution of the generated graph. The improved and fine-tuned network models are experimentally validated by computing the gap between real and generated data. The results meet welding engineering precision standards and solve the problem of low drawing recognition efficiency in the welding manufacturing process. After training with our model, the peak signal-to-noise ratio, structural similarity and mean squared error of welding engineering drawings reach about 44.89%, 99.58% and 2.11, respectively, which are superior to the results achieved using traditional networks in terms of both training speed and accuracy.

A Method based on Evolutionary Algorithms and Channel Attention Mechanism to Enhance Cycle Generative Adversarial Network Performance for Image Translation.

A Generative Adversarial Network (GAN) can learn the relationship between two image domains and achieve unpaired image-to-image translation. One of the breakthroughs was Cycle-consistent Generative Adversarial Networks (CycleGAN), which is a popular method to transfer the content representations from the source domain to the target domain. Existing studies have gradually improved the performance of CycleGAN models by modifying the network structure or loss function of CycleGAN. However, these methods tend to suffer from training instability and the generators lack the ability to acquire the most discriminating features between the source and target domains, thus making the generated images of low fidelity and few texture details. To overcome these issues, this paper proposes a new method that combines Evolutionary Algorithms (EAs) and Attention Mechanisms to train GANs. Specifically, from an initial CycleGAN, binary vectors indicating the activation of the weights of the generators are progressively improved upon by means of an EA. At the end of this process, the best-performing configurations of generators can be retained for image generation. In addition, to address the issues of low fidelity and lack of texture details on generated images, we make use of the channel attention mechanism. The latter component allows the candidate generators to learn important features of real images and thus generate images with higher quality. The experiments demonstrate qualitatively and quantitatively that the proposed method, namely, Attention evolutionary GAN (AevoGAN) alleviates the training instability problems of CycleGAN training. In the test results, the proposed method can generate higher quality images and obtain better results than the CycleGAN training methods present in the literature, in terms of Inception Score (IS), Fréchet Inception Distance (FID) and Kernel Inception Distance (KID).

Translation of atherosclerotic disease features onto healthy carotid ultrasound images using domain-to-domain translation

ObjectiveIn this work, we evaluated a model for the translation of atherosclerotic disease features onto healthy carotid ultrasound images. MethodsAn un-paired domain-to-domain translation model – the cycle Generative Adversarial Network (cycleGAN) – was trained to translate between carotid ultrasound images of healthy arteries and images of pronounced disease. Translation performance was evaluated using the measurement of wall thickness in original and generated images. In addition, we explored disease translation in different tissue segments (subcutaneous tissue, muscle, lumen, far wall, and deep tissues), using structural similarity index measure (SSIM) maps. ResultsFeatures of pronounced disease were successfully translated to the healthy images (1.2 (0.33) mm vs 0.43 (0.07) mm, p < 0.001), while overall anatomy was retained as SSIM value was equal to 0.78 (0.02). Exploration of translated features showed that both arterial wall and subcutaneous tissues were modified in the translation, but that the subcutaneous tissue was subject to distortion of the anatomy in some cases. The image quality influenced the disease translation performance. ConclusionThe results show that the model can learn a mapping between healthy and diseased images while retaining the overall anatomical contents. This is the first study on atherosclerosis disease translation in medical images. SignificanceThe concept of translating disease onto existing healthy images may serve purposes such as education, cardiovascular risk communication in health conversations, or personalized modelling in precision medicine.

Deep learning-based middle cerebral artery blood flow abnormality detection using flow velocity waveform derived from transcranial Doppler ultrasound

Since the brain is unlike any other organ in that it cannot store energy and has a high metabolic demand, constant blood flow is essential for healthy brain function. The maximum flow velocity waveform that is produced by transcranial doppler echo ultrasonography has different qualities for a healthy subject and a critically ill patient with conditions such as intraparenchymal or subarachnoid hemorrhage, hydrocephalus, or traumatic brain injury. Depending on the degree of the injury, the symptoms of traumatic brain damage can present themselves right away or not until days or weeks later. To aid in the early and accurate detection of patients with severe brain conditions, a classification system is proposedthat can distinguish between healthy control and patient utilizing the maximum flow velocity waveform derived from Transcranial doppler ultrasound. In this research, we manually labelled the data to remove mediocre and corrupted signals and pre-processed low-quality signals into high-quality ones using a Cycle Generative Adversarial Network (CycleGAN). This study proposes a two-stream deep learning model, DopplerNet2+, based on a Self-organized Operational Neural Network (Self-ONN), which achieves an overall accuracy, precision, recall, sensitivity, f1 score, and specificity of 99.45%, 99.45%, 99.45%, and 99.37% for the classification issue. DopplerNet2+ has a better area under the curve (AUC) of 1.00 and a better Kolmogorov-Smirnov (KS) statistic of 0.996 at the 0.812 thresholds than 11 other Self-ONN models trained with different inputs. The results show that the proposed models can successfully carry out the targeted classification task.

Image Translation of Breast Ultrasound to Pseudo Anatomical Display by CycleGAN.

Ultrasound imaging is cost effective, radiation-free, portable, and implemented routinely in clinical procedures. Nonetheless, image quality is characterized by a granulated appearance, a poor SNR, and speckle noise. Specific for breast tumors, the margins are commonly blurred and indistinct. Thus, there is a need for improving ultrasound image quality. We hypothesize that this can be achieved by translation into a more realistic display which mimics a pseudo anatomical cut through the tissue, using a cycle generative adversarial network (CycleGAN). In order to train CycleGAN for this translation, two datasets were used, "Breast Ultrasound Images" (BUSI) and a set of optical images of poultry breast tissues. The generated pseudo anatomical images provide improved visual discrimination of the lesions through clearer border definition and pronounced contrast. In order to evaluate the preservation of the anatomical features, the lesions in both datasets were segmented and compared. This comparison yielded median dice scores of 0.91 and 0.70; median center errors of 0.58% and 3.27%; and median area errors of 0.40% and 4.34% for the benign and malignancies, respectively. In conclusion, generated pseudo anatomical images provide a more intuitive display, enhance tissue anatomy, and preserve tumor geometry; and can potentially improve diagnoses and clinical outcomes.

High-resolution restoration of solar images degraded by atmospheric turbulence effect using improved CycleGAN

Due to the atmospheric turbulence effect, the solar images received by solar telescopes are generally severely degraded. In order to restore and improve the solar images, we proposed an improved Cycle Generative Adversarial Network (CycleGAN). By incorporating the identity loss and the perceptual loss into the loss function of CycleGAN, the ability of the network to capture image pixels and high-frequency information is significantly improved. On the basis, the influence of the turbulence intensity and the image noise on the restoration effect of solar images is analyzed. Results show that the improved CycleGAN has the advantages of smaller error and higher clarity of the restored images. In addition, the proposed method does not require paired images as training datasets, providing a new processing method for optical system telescopes and other fields of image processing.

An Accurate Non-accelerometer-based PPG Motion Artifact Removal Technique using CycleGAN

A photoplethysmography (PPG) is an uncomplicated and inexpensive optical technique widely used in the healthcare domain to extract valuable health-related information, e.g., heart rate variability, blood pressure, and respiration rate. PPG signals can easily be collected continuously and remotely using portable wearable devices. However, these measuring devices are vulnerable to motion artifacts caused by daily life activities. The most common ways to eliminate motion artifacts use extra accelerometer sensors, which suffer from two limitations: (i) high power consumption, and (ii) the need to integrate an accelerometer sensor in a wearable device (which is not required in certain wearables). This paper proposes a low-power non-accelerometer-based PPG motion artifacts removal method outperforming the accuracy of the existing methods. We use Cycle Generative Adversarial Network to reconstruct clean PPG signals from noisy PPG signals. Our novel machine-learning-based technique achieves 9.5 times improvement in motion artifact removal compared to the state-of-the-art without using extra sensors such as an accelerometer, which leads to 45% improvement in energy efficiency.

A Deep Learning Approach to Visualize Aortic Aneurysm Morphology Without the Use of Intravenous Contrast Agents.

Intravenous contrast agents are routinely used in CT imaging to enable the visualization of intravascular pathology, such as with abdominal aortic aneurysms. However, the injection is contraindicated in patients with iodine allergy and is associated with renal complications. In this study, we investigate if the raw data acquired from a noncontrast CT image contains sufficient information to differentiate blood and other soft tissue components. A deep learning pipeline underpinned by generative adversarial networks was developed to simulate contrast enhanced CTA images using noncontrast CTs. Two generative models (cycle- and conditional) are trained with paired noncontrast and contrast enhanced CTs from seventy-five patients (total of 11,243 pairs of images) with abdominal aortic aneurysms in a 3-fold cross-validation approach with a training/testing split of 50:25 patients. Subsequently, models were evaluated on an independent validation cohort of 200 patients (total of 29,468 pairs of images). Both deep learning generative models are able to perform this image transformation task with the Cycle-generative adversarial network (GAN) model outperforming the Conditional-GAN model as measured by aneurysm lumen segmentation accuracy (Cycle-GAN: 86.1% ± 12.2% vs Con-GAN: 85.7% ± 10.4%) and thrombus spatial morphology classification accuracy (Cycle-GAN: 93.5% vs Con-GAN: 85.7%). This pipeline implements deep learning methods to generate CTAs from noncontrast images, without the need of contrast injection, that bear strong concordance to the ground truth and enable the assessment ofimportant clinical metrics. Our pipeline is poised to disrupt clinical pathways requiring intravenous contrast.

Cycle Generative Adversarial Network Based on Gradient Normalization for Infrared Image Generation

Image generation technology is currently one of the popular directions in computer vision research, especially regarding infrared imaging, bearing critical applications in the military field. Existing algorithms for generating infrared images from visible images are usually weak in perceiving the salient regions of images and cannot effectively highlight the ability to generate texture details in infrared images, resulting in less texture details and poorer generated image quality. In this study, a cycle generative adversarial network method based on gradient normalization was proposed to address the current problems of poor infrared image generation, lack of texture detail and unstable models. First, to address the problem of limited feature extraction capability of the UNet generator network that makes the generated IR images blurred and of low quality, the use of the residual network with better feature extraction capability in the generator was employed to make the generated infrared images highly defined. Secondly, in order to solve issues concerning severe lack of detailed information in the generated infrared images, channel attention and spatial attention mechanisms were introduced into the ResNet with the attention mechanism used to weight the generated infrared image features in order to enhance feature perception of the prominent regions of the image, helping to generate image details. Finally, to tackle the problem where the current training models of adversarial generator networks are insufficiently stable, which leads to easy collapse of the model, a gradient normalization module was introduced in the discriminator network to stabilize the model and render it less prone to collapse during the training process. The experimental results on several datasets showed that the proposed method obtained satisfactory data in terms of objective evaluation metrics. Compared with the cycle generative adversarial network method, the proposed method in this work exhibited significant improvement in data validity on multiple datasets.

Large-Field Microscopic Imaging Method Based on Cycle Generative Adversarial Networks

Large-Field Microscopic Imaging Method Based on Cycle Generative Adversarial Networks

Improving Brain Tumor Classification with Deep Learning Using燬ynthetic燚ata

Deep learning (DL) techniques, which do not need complex pre-processing and feature analysis, are used in many areas of medicine and achieve promising results. On the other hand, in medical studies, a limited dataset decreases the abstraction ability of the DL model. In this context, we aimed to produce synthetic brain images including three tumor types (glioma, meningioma, and pituitary), unlike traditional data augmentation methods, and classify them with DL. This study proposes a tumor classification model consisting of a Dense Convolutional Network (DenseNet121)-based DL model to prevent forgetting problems in deep networks and delay information flow between layers. By comparing models trained on two different datasets, we demonstrated the effect of synthetic images generated by Cycle Generative Adversarial Network (CycleGAN) on the generalization of DL. One model is trained only on the original dataset, while the other is trained on the combined dataset of synthetic and original images. Synthetic data generated by CycleGAN improved the best accuracy values for glioma, meningioma, and pituitary tumor classes from 0.9633, 0.9569, and 0.9904 to 0.9968, 0.9920, and 0.9952, respectively. The developed model using synthetic data obtained a higher accuracy value than the related studies in the literature. Additionally, except for pixel-level and affine transform data augmentation, synthetic data has been generated in the figshare brain dataset for the first time.

Generating Chest X-Ray Progression of Pneumonia Using Conditional Cycle Generative Adversarial Networks

Generating Chest X-Ray Progression of Pneumonia Using Conditional Cycle Generative Adversarial Networks