CycleGAN Model Research Articles

Film image processing and production based on high-performance calculation

In film and television production, efficient and precise image processing is vital for achieving realistic visual effects. Therefore, exploring and applying advanced image processing technologies has become an essential method for elevating the production quality of film and television projects. This work investigates the application of artificial intelligence (AI) technology in the processing and production of animated images in film and television scenarios. By comparing the performance of standard Generative Adversarial Network (GAN), DenseNet, and CycleGAN models under different noise conditions, it is found that CycleGAN performs the best in image denoising and detail restoration. Experimental results demonstrate that CycleGAN achieves a Peak Signal-to-noise Ratio (PSNR) of 30.1dB and a Structural Similarity Index Measure (SSIM) of 0.88 under Gaussian noise conditions. Moreover, CycleGAN achieves a PSNR of 29.5dB and an SSIM of 0.85 under salt-and-pepper noise conditions. It outperforms the other models in both conditions. Additionally, CycleGAN’s mean absolute error is significantly lower than that of the other models. This work demonstrates that CycleGAN can more effectively handle complex noise and generate high-quality images under unsupervised learning conditions. These findings provide new directions for future image processing research and offer important references for model selection in practical applications. This work not only offers new perspectives on the development of animation image processing technology but also establishes a theoretical foundation for applying advanced AI techniques in film and television production. Through comparative analysis of various deep learning models, this work highlights the superior performance of CycleGAN under complex noise conditions. This advancement not only drives progress in image processing technology but also provides effective solutions for efficient production and quality enhancement of future film and television works.

Visual detection of drilling robot position for rockburst prevention in mining processing by a new image dehazing method

The positioning of drill pipe based on visual detection is a crucial aspect in achieving unmanned operation of drilling robot for rockburst prevention. However, the images directly collected from mechanized mining face are always polluted. In order to eliminate the interference of dust and haze in the image, an image dehazing method based on improved CycleGAN model is proposed in this paper. Firstly, a pipe image dehazing dataset for the rockburst prevention drilling robot is collected and established. Moreover, a generator architecture with a multi-scale U-shaped network structure is designed to improve the quality and accuracy of image recovery. A new reconstruction block is designed and an SK fusion layer is introduced to improve the feature extraction capability of the model, and a MU-CycleGAN network structure is constructed. Finally, an experimental platform for drill pipe image dehazing of the drilling robot for rockburst prevention is set up in the intelligent mining equipment laboratory. Experimental results showed that the PSNR and SSIM of the image dehazing model achieved 27.04 and 0.946, and the success rate of drill pipe grabbing has increased by 13.75%. Experimental results reveal that the proposed framework achieves superior image enhancement performance than the comparison algorithms.

Fine-tuning GAN Models with Unpaired Aerial Images for RGB to NDVI Translation in Vegetation Index Estimation

Calculating the Normalized Difference Vegetation Index (NDVI) requires expensive multispectral cameras, posing challenges such as high costs and the need for technical expertise. This research aims to develop a method to transform RGB images obtained from Unmanned Aerial Vehicle (UAV) into NDVI images. Our proposed method leverages the CycleGAN model, an unsupervised image-to-image translation framework, to learn the intricate mapping between RGB values and NDVI. The model was trained on unpaired datasets of RGB and NDVI images, sourced from paddy fields located in Gresik and Yogyakarta, Indonesia. This process successfully encapsulated the complex correlation between these two modalities. Various training strategies were systematically investigated, including weight initialization schemes, fine-tuning procedures, and learning rate policies, to optimize the model's performance. The fine-tuned CycleGAN demonstrated superior performance in creating synthetic NDVI images from unpaired dataset, surpassing other methods in terms of fidelity, quality, and structural coherence. The results were impressive, with a Normalized Root Mean Square Error (NRMSE) of 0.327, a Peak Signal-to-Noise Ratio (PSNR) of 16.330, an Oriented FAST and Rotated BRIEF (ORB) score of 0.859, and a Structural Similarity Index (SSIM) of 0.757. The best performing CycleGAN model was then deployed on a low-spec microcomputer device, specifically the Raspberry Pi 4B with an average computation time of 21.0077 seconds. Raspberry Pi 4B was chosen for its lightweight, compact dimensions, and compatibility with efficient battery power connections, making it suitable for drone deployment.

Breaking the Haze: Enhanced CycleGAN Models with Self-Attention for Single-Image Dehazing

A common preprocessing step in vision-based applications is image dehazing, which has drawbacks such color distortion and detail loss. Conventional techniques frequently fall short of sufficiently retrieving sharp images from foggy situations. In order to improve single-image dehazing, this paper presents a unique method that combines Self-Attention Generative Adversarial Networks (SAGANs) with Cycle-Consistent Generative Adversarial Networks (CycleGAN). In order to capture long-range dependencies and enhance the perceptual quality of the dehazed images, our method includes self-attention processes from SAGANs and makes use of CycleGAN's strengths for efficient image-to-image translation without the need for paired training samples. We present a dual-framework design in which SAGANs improve textural and contextual details by focusing on relevant characteristics throughout the whole image space, while CycleGAN assures content fidelity through cycle consistency. We show through our studies that this integrated strategy achieves higher scores in common metrics like Peak Signal-to-Noise Ratio (PSNR) of 34.146 dB and Structural Similarity Index (SSIM) of 0.964, greatly outperforming both qualitative and quantitative assessments of previous methods. Together with establishing a new standard for single-image dehazing, this work shows how effective hybrid generative models can be when handling challenging image processing problems.

Cross-modality image translation of 3 Tesla Magnetic Resonance Imaging to 7 Tesla using Generative Adversarial Networks.

The rapid advancements in magnetic resonance imaging (MRI) technology have precipitated a new paradigm wherein cross-modality data translation across diverse imaging platforms, field strengths, and different sites is increasingly challenging. This issue is particularly accentuated when transitioning from 3 Tesla (3T) to 7 Tesla (7T) MRI systems. This study proposes a novel solution to these challenges using generative adversarial networks (GANs)-specifically, the CycleGAN architecture- to create synthetic 7T images from 3T data. Employing a dataset of 1112 and 490 unpaired 3T and 7T MR images, respectively, we trained a 2-dimensional (2D) CycleGAN model, evaluating its performance on a paired dataset of 22 participants scanned at 3T and 7T. Independent testing on 22 distinct participants affirmed the model's proficiency in accurately predicting various tissue types, encompassing cerebral spinal fluid, gray matter, and white matter. Our approach provides a reliable and efficient methodology for synthesizing 7T images, achieving a median Dice of 6.82%,7,63%, and 4.85% for Cerebral Spinal Fluid (CSF), Gray Matter (GM), and White Matter (WM), respectively, in the testing dataset, thereby significantly aiding in harmonizing heterogeneous datasets. Furthermore, it delineates the potential of GANs in amplifying the contrast-to-noise ratio (CNR) from 3T, potentially enhancing the diagnostic capability of the images. While acknowledging the risk of model overfitting, our research underscores a promising progression towards harnessing the benefits of 7T MR systems in research investigations while preserving compatibility with existent 3T MR data. This work was previously presented at the ISMRM 2021 conference (Diniz, Helmet, Santini, Aizenstein, & Ibrahim, 2021).

Unpaired data training enables super-resolution confocal microscopy from low-resolution acquisitions.

Supervised deep-learning models have enabled super-resolution imaging in several microscopic imaging modalities, increasing the spatial lateral bandwidth of the original input images beyond the diffraction limit. Despite their success, their practical application poses several challenges in terms of the amount of training data and its quality, requiring the experimental acquisition of large, paired databases to generate an accurate generalized model whose performance remains invariant to unseen data. Cycle-consistent generative adversarial networks (cycleGANs) are unsupervised models for image-to-image translation tasks that are trained on unpaired datasets. This paper introduces a cycleGAN framework specifically designed to increase the lateral resolution limit in confocal microscopy by training a cycleGAN model using low- and high-resolution unpaired confocal images of human glioblastoma cells. Training and testing performances of the cycleGAN model have been assessed by measuring specific metrics such as background standard deviation, peak-to-noise ratio, and a customized frequency content measure. Our cycleGAN model has been evaluated in terms of image fidelity and resolution improvement using a paired dataset, showing superior performance than other reported methods. This work highlights the efficacy and promise of cycleGAN models in tackling super-resolution microscopic imaging without paired training, paving the path for turning home-built low-resolution microscopic systems into low-cost super-resolution instruments by means of unsupervised deep learning.

Infrared Image Enhancement Method of Substation Equipment Based on Self-Attention Cycle Generative Adversarial Network (SA-CycleGAN)

During the acquisition of infrared images in substations, low-quality images with poor contrast, blurred details, and missing texture information frequently appear, which adversely affects subsequent advanced visual tasks. To address this issue, this paper proposes an infrared image enhancement algorithm for substation equipment based on a self-attention cycle generative adversarial network (SA-CycleGAN). The proposed algorithm incorporates a self-attention mechanism into the CycleGAN model’s transcoding network to improve the mapping ability of infrared image information, enhance image contrast, and reducing the number of model parameters. The addition of an efficient local attention mechanism (EAL) and a feature pyramid structure within the encoding network enhances the generator’s ability to extract features and texture information from small targets in infrared substation equipment images, effectively improving image details. In the discriminator part, the model’s performance is further enhanced by constructing a two-channel feature network. To accelerate the model’s convergence, the loss function of the original CycleGAN is optimized. Compared to several mainstream image enhancement algorithms, the proposed algorithm improves the quality of low-quality infrared images by an average of 10.91% in color degree, 18.89% in saturation, and 29.82% in feature similarity indices. Additionally, the number of parameters in the proposed algorithm is reduced by 37.89% compared to the original model. Finally, the effectiveness of the proposed method in improving recognition accuracy is validated by the Centernet target recognition algorithm.

Cross-modal Transfer Learning Based on an Improved CycleGAN Model for Accurate Kidney Segmentation in Ultrasound Images

ObjectiveDeep-learning algorithms have been widely applied in the field of automatic kidney ultrasound (US) image segmentation. However, obtaining a large number of accurate kidney labels clinically is very difficult and time-consuming. To solve this problem, we have proposed an efficient cross-modal transfer learning method to improve the performance of the segmentation network on a limited labeled kidney US dataset. MethodsWe aim to implement an improved image-to-image translation network called Seg-CycleGAN to generate accurate annotated kidney US data from labeled abdomen computed tomography images. The Seg-CycleGAN framework primarily consists of two structures: (i) a standard CycleGAN network to visually simulate kidney US from a publicly available labeled abdomen computed tomography dataset; (ii) and a segmentation network to ensure accurate kidney anatomical structures in US images. Based on the large number of simulated kidney US images and small number of real annotated kidney US images, we then aimed to employ a fine-tuning strategy to obtain better segmentation results. ResultsTo validate the effectiveness of the proposed method, we tested this method on both normal and abnormal kidney US images. The experimental results showed that the proposed method achieved a segmentation accuracy of 0.8548 in dice similarity coefficient on all testing datasets and 0.7622 on the abnormal testing dataset. ConclusionsCompared with existing data augmentation and transfer learning methods, the proposed method improved the accuracy and generalization of the kidney US image segmentation network on a limited number of training datasets. It therefore has the potential to significantly reduce annotation costs in clinical settings.

Bidirectional brain image translation using transfer learning from generic pre-trained models

Brain imaging plays a crucial role in the diagnosis and treatment of various neurological disorders, providing valuable insights into the structure and function of the brain. Techniques such as magnetic resonance imaging (MRI) and computed tomography (CT) enable non-invasive visualization of the brain, aiding in the understanding of brain anatomy, abnormalities, and functional connectivity. However, cost and radiation dose may limit the acquisition of specific image modalities, so medical image synthesis can be used to generate required medical images without actual addition. CycleGAN and other GANs are valuable tools for generating synthetic images across various fields. In the medical domain, where obtaining labeled medical images is labor-intensive and expensive, addressing data scarcity is a major challenge. Recent studies propose using transfer learning to overcome this issue. This involves adapting pre-trained CycleGAN models, initially trained on non-medical data, to generate realistic medical images. In this work, transfer learning was applied to the task of MR-CT image translation and vice versa using 18 pre-trained non-medical models, and the models were fine-tuned to have the best result. The models’ performance was evaluated using four widely used image quality metrics: Peak-signal-to-noise-ratio, Structural Similarity Index, Universal Quality Index, and Visual Information Fidelity. Quantitative evaluation and qualitative perceptual analysis by radiologists demonstrate the potential of transfer learning in medical imaging and the effectiveness of the generic pre-trained model. The results provide compelling evidence of the model’s exceptional performance, which can be attributed to the high quality and similarity of the training images to actual human brain images. These results underscore the significance of carefully selecting appropriate and representative training images to optimize performance in brain image analysis tasks.

IE-CycleGAN: improved cycle consistent adversarial network for unpaired PET image enhancement.

Technological advances in instruments have greatly promoted the development of positron emission tomography (PET) scanners. State-of-the-art PET scanners such as uEXPLORER can collect PET images of significantly higher quality. However, these scanners are not currently available in most local hospitals due to the high cost of manufacturing and maintenance. Our study aims to convert low-quality PET images acquired by common PET scanners into images of comparable quality to those obtained by state-of-the-art scanners without the need for paired low- and high-quality PET images. In this paper, we proposed an improved CycleGAN (IE-CycleGAN) model for unpaired PET image enhancement. The proposedmethod is based on CycleGAN, and the correlation coefficient loss and patient-specific prior loss were added to constrain the structure of the generated images. Furthermore, we defined a normalX-to-advanced training strategy to enhance the generalization ability of the network. The proposedmethod was validated on unpaired uEXPLORER datasets and Biograph Vision local hospital datasets. For the uEXPLORER dataset, the proposedmethod achieved betterresults than non-local mean filtering (NLM), block-matching and 3D filtering (BM3D), and deep image prior (DIP), which are comparable to Unet (supervised) and CycleGAN (supervised). For the Biograph Vision local hospital datasets, the proposedmethod achieved higher contrast-to-noise ratios(CNR) and tumor-to-background SUVmax ratios (TBR) than NLM, BM3D, and DIP. In addition, the proposedmethod showed higher contrast, SUVmax,and TBR than Unet (supervised) and CycleGAN (supervised) when applied to images from different scanners. The proposed unpaired PET image enhancementmethod outperforms NLM, BM3D, and DIP. Moreover, it performs better than the Unet(supervised) and CycleGAN(supervised) when implemented on local hospital datasets, which demonstrates its excellent generalization ability.

Synthetic temporal bone CT generation from UTE-MRI using a cycleGAN-based deep learning model: advancing beyond CT-MR imaging fusion.

The aim of this study is to develop a deep-learning model to create synthetic temporal bone computed tomography (CT) images from ultrashort echo-time magnetic resonance imaging (MRI) scans, thereby addressing the intrinsic limitations of MRI in localizing anatomic landmarks in temporal bone CT. This retrospective study included patients who underwent temporal MRI and temporal bone CT within one month between April 2020 and March 2023. These patients were randomly divided into training and validation datasets. A CycleGAN model for generating synthetic temporal bone CT images was developed using temporal bone CT and pointwise encoding-time reduction with radial acquisition (PETRA). To assess the model's performance, the pixel count in mastoid air cells was measured. Two neuroradiologists evaluated the successful generation rates of 11 anatomical landmarks. A total of 102 patients were included in this study (training dataset, n = 54, mean age 58 ± 14, 34 females (63%); validation dataset, n = 48, mean age 61 ± 13, 29 females (60%)). In the pixel count of mastoid air cells, no difference was observed between synthetic and real images (679 ± 342 vs 738 ± 342, p = 0.13). For the six major anatomical sites, the positive generation rates were 97-100%, whereas those of the five major anatomical structures ranged from 24% to 83%. We developed a model to generate synthetic temporal bone CT images using PETRA MRI. This model can provide information regarding the major anatomic sites of the temporal bone using MRI. The proposed algorithm addresses the primary limitations of MRI in localizing anatomic sites within the temporal bone. CT is preferred for imaging the temporal bone, but has limitations in differentiating pathology there. The model achieved a high success rate in generating synthetic images of six anatomic sites. This can overcome the limitations of MRI in visualizing key anatomic sites in the temporal skull.

Synchronizing Detection and Removal of Smoke in Endoscopic Images With Cyclic Consistency Adversarial Nets.

Smoke removal is an important and meaningful issue for endoscopic surgery, which can enhance the visual quality of endoscopic images. Because it is practically impossible to construct a large training dataset of pair-matched endoscopic images with/without smoke, the Generative Adversarial Nets (GANs) based models are usually used for endoscopic image desmoke. But they have difficulties in either locating the accurate smoke area, or recovering realistic internal organ or tissue details. In this paper, we propose a new approach, called Desmoke-CycleGAN, which combined detection, estimation, and removal of smoke together, to improve the CycleGAN model for endoscopic image smoke removal. In addition, both pixel-level and perceptual-level consistency loss have been incorporated in the proposed model, which helps the model to be more stable and efficient for recovering realistic details in endoscopic images. The experimental results have demonstrated that this method outperforms other state-of-the-art smoke removal approaches with unpaired real endoscopic images.

Automatic detection of the third molar and mandibular canal on panoramic radiographs based on deep learning

PurposeThis study aims to develop a deep learning framework for the automatic detection of the position relationship between the mandibular third molar (M3) and the mandibular canal (MC) on panoramic radiographs (PRs), to assist doctors in assessing and planning appropriate surgical interventions. MethodsDatasets D1 and D2 were obtained by collecting 253 PRs from a hospitals and 197 PRs from online platforms. The RPIFormer model proposed in this study was trained and validated on D1 to create a segmentation model. The CycleGAN model was trained and validated on both D1 and D2 to develop an image enhancement model. Ultimately, the segmentation and enhancement models were integrated with an object detection model to create a fully automated framework for M3 and MC detection in PRs. Experimental evaluation included calculating Dice coefficient, IoU, Recall, and Precision during the process. ResultsThe RPIFormer model proposed in this study achieved an average Dice coefficient of 92.56 % for segmenting M3 and MC, representing a 3.06 % improvement over the previous best study. The deep learning framework developed in this research enables automatic detection of M3 and MC in PRs without manual cropping, demonstrating superior detection accuracy and generalization capability. ConclusionThe framework developed in this study can be applied to PRs captured in different hospitals without the need for model fine-tuning. This feature is significant for aiding doctors in accurately assessing the spatial relationship between M3 and MC, thereby determining the optimal treatment plan to ensure patients' oral health and surgical safety.

A novel RCACycleGAN model is proposed for the high-precision reconstruction of sparse TFM images

The sparse total focusing method (TFM) has been shown to enhance the computational efficacy of ultrasound imaging but the image quality of ultrasound regrettably deteriorates with an increase in the sparsity rate of array elements. Deep learning has made remarkable advancements in image processing and cycle-consistent generative adversarial networks (CycleGANs) have been extensively employed to reconstruct diverse image categories. However, due to the incomplete extraction of image feature information by the generator and discriminator in a CycleGAN, high-quality sparse TFM images cannot be directly reconstructed using CycleGANs. There is also a risk of losing crucial feature information related to minor defects. As a result, this paper modifies the generator and discriminator in the CycleGAN to construct a new relativistic discriminator and coordinate attention CycleGAN (RCACycleGAN) model, which enables high-precision reconstruction of sparse TFM images. The addition of the coordinate attention module to the CycleGAN enhances the defective feature representation by fully considering the channel and spatial correlation between regions and using the fusion of spatially perceived feature maps in different directions. It solves the problem of easy loss of defective key feature information. The relativistic discriminator replaces the PatchGAN discriminator in the CycleGAN and evaluates the quality of both real and sparse TFM reconstructed images to ensure a relative image quality evaluation. This process solves the problem of unstable image quality of the sparse TFM reconstructed image. Experimental results demonstrate that RCACycleGAN can stably reconstruct sparse TFM images even in small sample dataset scenarios. The proposed network model reconstructs images with better accuracy, including in terms of structural similarity, defect roundness and area, and has a shorter training time than several existing network models.

Deep Learning Synthesis of White-Blood From Dark-Blood Late Gadolinium Enhancement Cardiac Magnetic Resonance.

Dark-blood late gadolinium enhancement (DB-LGE) cardiac magnetic resonance has been proposed as an alternative to standard white-blood LGE (WB-LGE) imaging protocols to enhance scar-to-blood contrast without compromising scar-to-myocardium contrast. In practice, both DB and WB contrasts may have clinical utility, but acquiring both has the drawback of additional acquisition time. The aim of this study was to develop and evaluate a deep learning method to generate synthetic WB-LGE images from DB-LGE, allowing the assessment of both contrasts without additional scan time. DB-LGE and WB-LGE data from 215 patients were used to train 2 types of unpaired image-to-image translation deep learning models, cycle-consistent generative adversarial network (CycleGAN) and contrastive unpaired translation, with 5 different loss function hyperparameter settings each. Initially, the best hyperparameter setting was determined for each model type based on the Fréchet inception distance and the visual assessment of expert readers. Then, the CycleGAN and contrastive unpaired translation models with the optimal hyperparameters were directly compared. Finally, with the best model chosen, the quantification of scar based on the synthetic WB-LGE images was compared with the truly acquired WB-LGE. The CycleGAN architecture for unpaired image-to-image translation was found to provide the most realistic synthetic WB-LGE images from DB-LGE images. The results showed that it was difficult for visual readers to distinguish if an image was true or synthetic (55% correctly classified). In addition, scar burden quantification with the synthetic data was highly correlated with the analysis of the truly acquired images. Bland-Altman analysis found a mean bias in percentage scar burden between the quantification of the real WB and synthetic white-blood images of 0.44% with limits of agreement from -10.85% to 11.74%. The mean image quality of the real WB images (3.53/5) was scored higher than the synthetic white-blood images (3.03), P = 0.009. This study proposed a CycleGAN model to generate synthetic WB-LGE from DB-LGE images to allow assessment of both image contrasts without additional scan time. This work represents a clinically focused assessment of synthetic medical images generated by artificial intelligence, a topic with significant potential for a multitude of applications. However, further evaluation is warranted before clinical adoption.

Data enhancement method based on cyclic generation adversarial network

This study introduces a novel data augmentation technique employing Cycle Generative Adversarial Networks (CycleGAN) to mitigate the challenges posed by the paucity of image datasets in deep learning domains. Through the adept training of a CycleGAN model, this method substantially enriches image datasets, thereby enhancing the efficiency of deep learning models in target detection tasks. Distinct from conventional approaches, our strategy incorporates advanced activation functions, relative discriminators, and residual connections, which collectively foster greater image diversity and mitigate mode collapse, all while maintaining a low computational overhead. Evaluations conducted on diverse datasets, including MNIST, Synthetic Aperture Radar (SAR), and medical blood cell images, demonstrate the method's superior augmentation capabilities compared to traditional Deep Convolutional GAN (DCGAN) techniques, underscoring its efficacy and potential utility in preprocessing for deep learning applications.

Virtual formalin-fixed and paraffin-embedded staining of fresh brain tissue via stimulated Raman CycleGAN model.

Intraoperative histology is essential for surgical guidance and decision-making. However, frozen-sectioned hematoxylin and eosin (H&E) staining suffers from degraded accuracy, whereas the gold-standard formalin-fixed and paraffin-embedded (FFPE) H&E is too lengthy for intraoperative use. Stimulated Raman scattering (SRS) microscopy has shown rapid histology of brain tissue with lipid/protein contrast but is challenging to yield images identical to nucleic acid-/protein-based FFPE stains interpretable to pathologists. Here, we report the development of a semi-supervised stimulated Raman CycleGAN model to convert fresh-tissue SRS images to H&E stains using unpaired training data. Within 3 minutes, stimulated Raman virtual histology (SRVH) results that matched perfectly with true H&E could be generated. A blind validation indicated that board-certified neuropathologists are able to differentiate histologic subtypes of human glioma on SRVH but hardly on conventional SRS images. SRVH may provide intraoperative diagnosis superior to frozen H&E in both speed and accuracy, extendable to other types of solid tumors.

Ancient Chinese painting style transfer based on CycleGAN

Style transfer aims to alter the visual aesthetic of images by giving them a different artistic style. With the rapid advancement of deep learning, style transfer tasks have made significant progress, introducing new perspectives and innovative potentials within the realm of image processing. This study seeks to explore style transfer methods based on Cycle-Consistent Adversarial Networks (CycleGAN), enabling contemporary landscape photographs to take on the form of ancient Chinese paintings. This endeavor opens up fresh possibilities for artistic creation, image editing and design applications. The research encompasses an exposition of the process involved in constructing the CycleGAN model, alongside presenting research findings. Furthermore, it delves into the discussion of crucial techniques employed during the model training process, specifically the utilization of cycle consistency loss in configuring the loss functions. Lastly, this study ventures into future research directions, including strategies for further enhancing the performance and expanding the application scope of this style transfer model.

Exploration of classical neural network architecture in cycleGAN framework with face photo-sketch synthesis

CycleGAN has been a benchmark in the style transfer field and various extensions with wide applications and excellent performance have been introduced in recent years, however, discussion about its architecture exploration which could enable us to further understand the concept of generative model is scarce. In this paper, several architectures referenced from classical convolutional neural networks are implemented into the generator and discriminator of the cycleGAN model, including AlexNet, DenseNet, GoogLeNet, and ResNet. Their feature extraction modes are imitated and modified into blocks to embed into the encoder part of the generator while the discriminator directly uses their model except it outputs a patch classification. In advance to mitigate the possible imbalance between generator and discriminator ability, a self-adjusting learning rate strategy based on the discriminator confidence is introduced. Multiple evaluation metrics are utilized to measure the performance of each model. Experimental results indicate an AlexNet-like architecture model could achieve a competitive performance than the baseline cycleGAN and present better fine details and high-frequency information.

CycleGAN generated pneumonia chest x-ray images: Evaluation with vision transformer

The use of generative models in image synthesis has become increasingly prevalent. Synthetic medical imaging data is of paramount importance, primarily because medical imaging data is scarce, costly, and encumbered by legal considerations pertaining to patient confidentiality. Synthetic medical images offer a potential answer to these issues. The predominant approaches primarily assess the quality of images and the degree of resemblance between these images and the original ones employed for their generation.The central idea of the work can be summarized in the question: Do the performance metrics of Frechet Inception Distance(FID) and Inception Score(IS) in the Cycle-consistent Generative Adversarial Networks (CycleGAN) model are adequate to determine how real a generated chest x-ray pneumonia image is? In this study, a CycleGAN model was employed to produce artificial images depicting 3 classes of chest x-ray pneumonia images: general(any type), bacterial, and viral pneumonia. The quality of the images were evaluated assessing and contrasting 3 criteria: performance metric of CycleGAN model, clinical assessment of respiratory experts and the results of classification of a visual transformer(ViT). The overall results showed that the evaluation metrics of the CycleGAN are insufficient to establish realism in generated medical images.