Cycle-consistent Adversarial Network Research Articles

Enhancing Semantic Consistency in Image-to-Image Translation with an Improved CycleGAN Framework

Abstract. This paper presents an enhanced Cycle-Consistent Adversarial Networks (CycleGAN) model aimed at preserving semantic consistency during image-to-image translation, with a focus on complex tasks such as autonomous driving and scientific simulations. The study's key contribution is the incorporation of a pre-trained semantic segmentation model to preserve important characteristics during translation, such as license plates, traffic signs, and pedestrian structures. By introducing a semantic consistency loss alongside the traditional cycle-consistency loss, the proposed approach ensures that key features are retained, even in challenging scenes. Extensive experiments conducted on the Cityscapes dataset demonstrate the effectiveness in maintaining both visual fidelity and semantic accuracy, significantly improving upon the traditional CycleGAN. This method proves particularly valuable in domains where precision is essential, such as cross-domain image generation for autonomous systems and medical imaging. Future research will focus on optimizing the model for real-time applications and exploring multi-domain frameworks to further enhance its performance in diverse environments. Overall, this study offers an efficient image style-transfer solution for preserving semantic integrity without sacrificing translation accuracy.

Facial expression morphing: enhancing visual fidelity and preserving facial details in CycleGAN-based expression synthesis

Recent advancements in facial expression synthesis using deep learning, particularly with Cycle-Consistent Adversarial Networks (CycleGAN), have led to impressive results. However, a critical challenge persists: the generated expressions often lack the sharpness and fine details of the original face, such as freckles, moles, or birthmarks. To address this issue, we introduce the Facial Expression Morphing (FEM) algorithm, a novel post-processing method designed to enhance the visual fidelity of CycleGAN-based outputs. The FEM method blends the input image with the generated expression, prioritizing the preservation of crucial facial details. We experimented with our method on the Radboud Faces Database (RafD) and evaluated employing the Fréchet Inception Distance (FID) standard benchmark for image-to-image translation and introducing a new metric, FSD (Facial Similarity Distance), to specifically measure the similarity between translated and real images. Our comprehensive analysis of CycleGAN, UNet Vision Transformer cycle-consistent GAN versions 1 (UVCGANv1) and 2 (UVCGANv2) reveals a substantial enhancement in image clarity and preservation of intricate details. The average FID score of 31.92 achieved by our models represents a remarkable 50% reduction compared to the previous state-of-the-art model’s score of 63.82, showcasing the significant advancements made in this domain. This substantial enhancement in image quality is further supported by our proposed FSD metric, which shows a closer resemblance between FEM-processed images and the original faces.

Semi-supervised Chinese poem-to-painting generation via cycle-consistent adversarial networks

Semi-supervised Chinese poem-to-painting generation via cycle-consistent adversarial networks

Integrating generative AI with ABCDE rule analysis for enhanced skin cancer diagnosis, dermatologist training and patient education.

The early detection and accurate monitoring of suspicious skin lesions are critical for effective dermatological diagnosis and treatment, particularly for reliable identification of the progression of nevi to melanoma. The traditional diagnostic framework, the ABCDE rule, provides a foundation for evaluating lesion characteristics by visual examination using dermoscopes. Simulations of skin lesion progression could improve the understanding of melanoma growth patterns. This study aims to enhance lesion analysis and understanding of lesion progression by providing a simulated potential progression of nevi into melanomas. The study generates a dataset of simulated lesion progressions, from nevi to simulated melanoma, based on a Cycle-Consistent Adversarial Network (Cycle-GAN) and frame interpolation. We apply an optical flow analysis to the generated dermoscopic image sequences, enabling the quantification of lesion transformation. In parallel, we evaluate changes in ABCDE rule metrics as example to assess the simulated evolution. We present the first simulation of nevi progressing into simulated melanoma counterparts, consisting of 152 detailed steps. The ABCDE rule metrics correlate with the simulation in a natural manner. For the seven samples studied, the asymmetry metric increased by an average of 19%, the border gradient metric increased by an average of 63%, the convexity metric decreased by an average of 3%, the diameter increased by an average of 2%, and the color dispersion metric increased by an average of 45%. The diagnostic value of the ABCDE rule is enhanced through the addition of insights based on optical flow. The outward expansion of lesions, as captured by optical flow vectors, correlates strongly with the expected increase in diameter, confirming the simulation's fidelity to known lesion growth patterns. The heatmap visualizations further illustrate the degree of change within lesions, offering an intuitive visual proxy for lesion evolution. The achieved simulations of potential lesion progressions could facilitate improved early detection and understanding of how lesions evolve. By combining the optical flow analysis with the established criteria of the ABCDE rule, this study presents a significant advancement in dermatoscopic diagnostics and patient education. Future research will focus on applying this integrated approach to real patient data, with the aim of enhancing the understanding of lesion progression and the personalization of dermatological care.

Adaptive assessment based on fractional CBCT images for cervical cancer.

Anatomical and other changes during radiotherapy will cause inaccuracy of dose distributions, therefore the expectation for online adaptive radiation therapy (ART) is high in effectively reducing uncertainties due to intra-variation. However, ART requires extensive time and effort. This study investigated an adaptive assessment workflow based on fractional cone-beam computed tomography (CBCT) images. Image registration, synthetic CT (sCT) generation, auto-segmentation, and dose calculation were implemented and integrated into ArcherQA Adaptive Check. The rigid registration was based on ITK open source. The deformable image registration (DIR) method was based on a 3D multistage registration network, and the sCT generation method was performed based on a 2D cycle-consistent adversarial network (CycleGAN). The auto-segmentation of organs at risk (OARs) on sCT images was finished by a deep learning-based auto-segmentation software, DeepViewer. The contours of targets were obtained by the structure-guided registration. Finally, the dose calculation was based on a GPU-based Monte Carlo (MC) dose code, ArcherQA. The dice similarity coefficient (DSCs) were over 0.86 for target volumes and over 0.79 for OARs. The gamma pass rate of ArcherQA versus Eclipse treatment planning system was more than 99% at the 2%/2mm criterion with a low-dose threshold of 10%. The time for the whole process was less than 3 min. The dosimetric results of ArcherQA Adaptive Check were consistent with the Ethos scheduled plan, which can effectively identify the fractions that need the implementation of the Ethos adaptive plan. This study integrated AI-based technologies and GPU-based MC technology to evaluate the dose distributions using fractional CBCT images, demonstrating remarkably high efficiency and precision to support future ART processes.

HEDehazeNet: Unpaired image dehazing via enhanced haze generation

Unpaired image dehazing models based on Cycle-Consistent Adversarial Networks (CycleGAN) typically consist of two cycle branches: dehazing-rehazing branch and hazing-dehazing branch. In these two branches, there is an asymmetry of information in the mutual transformation process between haze images and haze-free images. Previous models tended to focus more on the transformation process from haze images to haze-free images within the dehazing-rehazing branch, overlooking the provision of effective information for the formation of haze images in the hazing-dehazing branch. This oversight results in the production of haze patterns that are both monotonous and simplistic, ultimately impeding the overall performance and generalization capabilities of dehazing networks. In light of this, this paper proposes a novel model called HEDehazeNet (Dehazing Net based on Haze Generation Enhancement), which provides crucial information for the generation process of haze images through a dedicated haze generation enhancement module. This module is capable of producing three distinct modes of transmission maps - random transmission map, simulated transmission map, and mixed transmission maps combining both. Employing these transmission maps to generate hazing images with varying density and patterns provides the dehazing network with a more diverse and dynamically complex set of training samples, thereby enhancing its capacity to handle intricate scenes. Additionally, we made minor modifications to the U-Net, replacing residual blocks with multi-scale parallel convolutional blocks and channel self-attention, to further enhance the network's performance. Experiments were conducted on both synthetic and real-world datasets, substantiating the superiority of HEDehazeNet over the current state-of-the-art unpaired dehazing models.

Piston aero-engine fault cross-domain diagnosis based on unpaired generative transfer learning

As artificial intelligence (AI) and machine learning continue to advance, they have become key players in signal analysis and pattern recognition. However, challenges such as domain shift and a lack of labeled data hinder the application of machine learning to new, unfamiliar signals, particularly in the development of intelligent diagnostic methods for mechanical faults. In this study, we address these challenges by framing them as the extreme class imbalance problem. We introduce a novel generative transfer learning framework that leverages cycle-consistent adversarial networks with hard constraints (CycleGAN-HCs). This framework is designed to generate unpaired mechanical fault signals, which are then used to train classifiers that can operate across different domains. To enhance feature extraction capabilities for time series data, we developed a new generator model called U-Attention. Additionally, we refined the training loss function to better capture domain-specific and classification-specific features in mechanical vibration signals. The proposed method successfully facilitates feature transfer and data augmentation. Its effectiveness and reliability have been demonstrated through four cross-domain fault diagnosis tasks involving piston aero-engines. Comparative verification shows that the generative transfer learning diagnostic framework outperforms traditional methods, offering superior performance and better generalization in complex mechanical fault diagnosis scenarios.

Self-supervised Text Style Transfer using Cycle-Consistent Adversarial Networks

Self-supervised Text Style Transfer using Cycle-Consistent Adversarial Networks

Transferring Adult-like Phase Images for Robust Multi-view Isointense Infant Brain Segmentation.

Accurate tissue segmentation of infant brain in magnetic resonance (MR) images is crucial for charting early brain development and identifying biomarkers. Due to ongoing myelination and maturation, in the isointense phase (6-9 months of age), the gray and white matters of infant brain exhibit similar intensity levels in MR images, posing significant challenges for tissue segmentation. Meanwhile, in the adult-like phase around 12 months of age, the MR images show high tissue contrast and can be easily segmented. In this paper, we propose to effectively exploit adult-like phase images to achieve robustmulti-view isointense infant brain segmentation. Specifically, in one way, we transfer adult-like phase images to the isointense view, which have similar tissue contrast as the isointense phase images, and use the transferred images to train an isointense-view segmentation network. On the other way, we transfer isointense phase images to the adult-like view, which have enhanced tissue contrast, for training a segmentation network in the adult-like view. The segmentation networks of different views form a multi-path architecture that performs multi-view learning to further boost the segmentation performance. Since anatomy-preserving style transfer is key to the downstream segmentation task, we develop a Disentangled Cycle-consistent Adversarial Network (DCAN) with strong regularization terms to accurately transfer realistic tissue contrast between isointense and adult-like phase images while still maintaining their structural consistency. Experiments on both NDAR and iSeg-2019 datasets demonstrate a significant superior performance of our method over the state-of-the-art methods.

Sliced Wasserstein Distance-Guided Three-Dimensional Porous Media Reconstruction Based on Cycle-Consistent Adversarial Network and Few-Shot Learning

Sliced Wasserstein Distance-Guided Three-Dimensional Porous Media Reconstruction Based on Cycle-Consistent Adversarial Network and Few-Shot Learning

A Comparison of U-Net with Conditional Generative Adversarial Networks and Cycle-Consistent Adversarial Networks for real seismic data interpolation: Tupi Field

Deep learning models have been used to improve seismic trace interpolation in recent years. Encode-Decode models, such as U-Net, have been implemented to solve interpolation problems. The success of U-Net in seismic interpolation has inspired us to test the U-net also as an image generator for further Generative Adversarial Network (GAN) interpolation models. The objective of the present paper is to compare the performance of U-Net inside the GAN models for seismic interpolation: the U-Net alone and two GAN models, the conditional GAN (cGAN) and the Cycle Consistent GAN (CycleGAN), both using the U-Net as a generator inside their workflow. We test the methodologies for two scenarios: regular and irregular interpolation. All tests were performed in a real dataset from the Tupi Field which belongs to the Brazilian pre-salt region. A comparison of the statistical metrics shows that cGAN performs better than CycleGAN and the U-Net alone in most cases. The computational training time of the cGAN model, for all interpolation scenarios, is better than the CycleGAN. Finally, the cGAN training time is comparable to the training of the U-Net alone.

Virtual indigo carmine chromoendoscopy images: a novel modality for peroral cholangioscopy using artificial intelligence technology (with video)

Background and AimsAccurately diagnosing biliary strictures is crucial for surgical decisions, and although peroral cholangioscopy (POCS) aids in visual diagnosis, diagnosing malignancies or determining lesion margins via this route remains challenging. Indigo carmine is commonly used to evaluate lesions during gastrointestinal endoscopy. We aimed to establish the utility of virtual indigo carmine chromoendoscopy (VICI) converted from POCS images using artificial intelligence. MethodsThis single-center, retrospective study analyzed 40 patients with biliary strictures who underwent POCS using white light imaging (WLI) and narrow-band imaging (NBI). A “cycle-consistent adversarial network” (CycleGAN) was used to convert the WLI into VICI of POCS images. Three experienced endoscopists evaluated WLI, NBI, and VICI via POCS in all patients. The primary outcome was the visualization quality of surface structures, surface microvessels, and lesion margins. The secondary outcome was diagnostic accuracy. ResultsVICI showed superior visualization of the surface structures and lesion margins compared with WLI (P<0.001) and NBI (P<0.001). The diagnostic accuracies were 72.5%, 87.5%, and 90.0% in WLI alone, WLI and VICI simultaneously, and WLI and NBI simultaneously, respectively. WLI and VICI simultaneously tended to result in higher accuracy than WLI alone (P=0.083) and the results were not significantly different from WLI and NBI simultaneously (P=0.65). ConclusionsVICI in POCS proved valuable for visualizing surface structures and lesion margins and contributed to higher diagnostic accuracy comparable to NBI. In addition to NBI, VICI may be a novel supportive modality for POCS.

Thermal image visualization using multi-discriminator CycleGAN with unpaired thermal–visible image training set

Thermal imaging utilizes the differences in thermal radiation within a scene to capture images, effectively compensating for the limited imaging capabilities of visible-light cameras in low-light environments. In order to further enhance thermal imaging, we propose a novel approach that transforms the visualization of thermal images into an image conversion task. Our method, named TIVNet (Thermal Images Visualization Network), is based on a multi-discriminator cycle-consistent adversarial network (CycleGAN) and operates in the YUV color space to accurately colorize and represent thermal infrared images. The utilization of color and texture discriminators in our framework ensures that the converted images exhibit appropriate color styles and capture fine details, respectively. In the generator, we employed a Dense Residual Module (DRM) to facilitate the reuse of features and enhance the flow of information. By utilizing the DRM, we aim to enhance the overall performance and efficiency of the generator network. Additionally, we incorporated the Dual-channel Self-Attention Module(DCSA) to enhance the feature representation. By leveraging self-attention, the generator can effectively learn and exploit the inherent relationships between different regions, leading to improved feature representation and, consequently, more accurate colorization and representation of thermal infrared images. Notably, our proposed method leverages unsupervised learning, eliminating the need for a fully paired thermal infrared-visible image dataset. Through extensive experimentation on publicly available datasets as well as our own, we demonstrate that our approach outperforms state-of-the-art methods in thermal image visualization. Furthermore, the converted images produced by TIVNet exhibit a high level of realism, thereby enhancing the visual impact of thermal imaging. This improvement is particularly beneficial for human perception and subsequent target recognition tasks. Our findings suggest that the application of our method yields superior results both quantitatively and qualitatively, showcasing its potential for advancing thermal imaging capabilities.

The virtual staining method by quantitative phase imaging for label free lymphocytes based on self-supervised iteration cycle-consistent adversarial networks.

Quantitative phase imaging (QPI) provides 3D structural and morphological information for label free living cells. Unfortunately, this quantitative phase information cannot meet doctors' diagnostic requirements of the clinical "gold standard," which displays stained cells' pathological states based on 2D color features. To make QPI results satisfy the clinical "gold standard," the virtual staining method by QPI for label free lymphocytes based on self-supervised iteration Cycle-Consistent Adversarial Networks (CycleGANs) is proposed herein. The 3D phase information of QPI is, therefore, trained and transferred to a kind of 2D "virtual staining" image that is well in agreement with "gold standard" results. To solve the problem that unstained QPI and stained "gold standard" results cannot be obtained for the same label free living cell, the self-supervised iteration for the CycleGAN deep learning algorithm is designed to obtain a trained stained result as the ground truth for error evaluation. The structural similarity index of our virtual staining experimental results for 8756 lymphocytes is 0.86. Lymphocytes' area errors after converting to 2D virtual stained results from 3D phase information are less than 3.59%. The mean error of the nuclear to cytoplasmic ratio is 2.69%, and the color deviation from the "gold standard" is less than 6.67%.

Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images

The variation in histologic staining between different medical centers is one of the most profound challenges in the field of computer-aided diagnosis. The appearance disparity of pathological whole slide images causes algorithms to become less reliable, which in turn impedes the wide-spread applicability of downstream tasks like cancer diagnosis. Furthermore, different stainings lead to biases in the training which in case of domain shifts negatively affect the test performance. Therefore, in this paper we propose MultiStain-CycleGAN, a multi-domain approach to stain normalization based on CycleGAN. Our modifications to CycleGAN allow us to normalize images of different origins without retraining or using different models. We perform an extensive evaluation of our method using various metrics and compare it to commonly used methods that are multi-domain capable. First, we evaluate how well our method fools a domain classifier that tries to assign a medical center to an image. Then, we test our normalization on the tumor classification performance of a downstream classifier. Furthermore, we evaluate the image quality of the normalized images using the Structural similarity index and the ability to reduce the domain shift using the Fréchet inception distance. We show that our method proves to be multi-domain capable, provides a very high image quality among the compared methods, and can most reliably fool the domain classifier while keeping the tumor classifier performance high. By reducing the domain influence, biases in the data can be removed on the one hand and the origin of the whole slide image can be disguised on the other, thus enhancing patient data privacy.

Ancient Chinese painting style transfer based on CycleGAN

Style transfer aims to alter the visual aesthetic of images by giving them a different artistic style. With the rapid advancement of deep learning, style transfer tasks have made significant progress, introducing new perspectives and innovative potentials within the realm of image processing. This study seeks to explore style transfer methods based on Cycle-Consistent Adversarial Networks (CycleGAN), enabling contemporary landscape photographs to take on the form of ancient Chinese paintings. This endeavor opens up fresh possibilities for artistic creation, image editing and design applications. The research encompasses an exposition of the process involved in constructing the CycleGAN model, alongside presenting research findings. Furthermore, it delves into the discussion of crucial techniques employed during the model training process, specifically the utilization of cycle consistency loss in configuring the loss functions. Lastly, this study ventures into future research directions, including strategies for further enhancing the performance and expanding the application scope of this style transfer model.

Unsupervised image-to-image translation to reduce the annotation effort for instance segmentation of field vegetables

Due to the constantly increasing shortage of labour availability in the agricultural sector, automation is on the rise. This trend is particularly evident in field vegetable production, where the majority of costs are associated with human labour. To automate labour-intensive tasks, intelligent camera systems are required. Typical state of the art learning algorithms requires large amounts of labelled data for a reliable detection and segmentation. The effort for recording and labelling is increased by the diverse conditions on a field that need to be included in the datasets. In this paper, an unsupervised approach is presented, which unifies varying environmental conditions in the field, as well as an analysis to transfer this unpaired deep learning based unification to unknown exposure scenarios, different growth stages and other varieties. For this purpose, a Cycle-Consistent Adversarial Network (CycleGAN) is used which balance varying exposure and background situations to the training data variance without additional labelling effort. By adapting the exposure with a CycleGAN, the required labelling effort could be reduced by up to five times in relation to a heterogeneous dataset, as fewer variations need to be covered for a robust segmentation. The transfer of this approach to other growth stages, as well as the transfer to another cabbage cultivar shows the potential of additional data generalisation.

Extracting interpretable features for pathologists using weakly supervised learning to predict p16 expression in oropharyngeal cancer

One drawback of existing artificial intelligence (AI)-based histopathological prediction models is the lack of interpretability. The objective of this study is to extract p16-positive oropharyngeal squamous cell carcinoma (OPSCC) features in a form that can be interpreted by pathologists using AI model. We constructed a model for predicting p16 expression using a dataset of whole-slide images from 114 OPSCC biopsy cases. We used the clustering-constrained attention-based multiple-instance learning (CLAM) model, a weakly supervised learning approach. To improve performance, we incorporated tumor annotation into the model (Annot-CLAM) and achieved the mean area under the receiver operating characteristic curve of 0.905. Utilizing the image patches on which the model focused, we examined the features of model interest via histopathologic morphological analysis and cycle-consistent adversarial network (CycleGAN) image translation. The histopathologic morphological analysis evaluated the histopathological characteristics of image patches, revealing significant differences in the numbers of nuclei, the perimeters of the nuclei, and the intercellular bridges between p16-negative and p16-positive image patches. By using the CycleGAN-converted images, we confirmed that the sizes and densities of nuclei are significantly converted. This novel approach improves interpretability in histopathological morphology-based AI models and contributes to the advancement of clinically valuable histopathological morphological features.

RETRACTED ARTICLE: HR-CycleGAN: Face highlight reduction based on improved cycle-consistent adversarial networks

RETRACTED ARTICLE: HR-CycleGAN: Face highlight reduction based on improved cycle-consistent adversarial networks

COMIC: An unsupervised change detection method for heterogeneous remote sensing images based on copula mixtures and Cycle-Consistent Adversarial Networks

In this paper, we consider the problem of change detection (CD) with two heterogeneous remote sensing (RS) images. For this problem, an unsupervised change detection method has been proposed recently based on the image translation technique of Cycle-Consistent Adversarial Networks (CycleGANs), where one image is translated from its original modality to the modality of the other image so that the difference map can be obtained by performing arithmetical subtraction. However, the difference map derived from subtraction is susceptible to image translation errors, in which case the changed area and the unchanged area are less distinguishable. To overcome the above shortcoming, we propose a new unsupervised copula mixture and CycleGAN-based CD method (COMIC), which combines the advantages of copula mixtures on statistical modeling and the advantages of CycleGANs on data mining. In COMIC, the pre-event image is first translated from its original modality to the post-event image modality. After that, by constructing a copula mixture, the joint distribution of the features from the heterogeneous images can be learnt according to quantitive analysis of the dependence structure based on the translated image and the original pre-event image, which are of the same modality and contain totally the same objects. Then, we model the CD problem as a binary hypothesis testing problem and derive its test statistics based on the constructed copula mixture. Finally, the difference map can be obtained from the test statistics and the binary change map (BCM) is generated by K-means clustering. We perform experiments on real RS datasets, which demonstrate the superiority of COMIC over the state-of-the-art methods.