Cycle-consistent Adversarial Network Research Articles

The virtual staining method by quantitative phase imaging for label free lymphocytes based on self-supervised iteration cycle-consistent adversarial networks.

Quantitative phase imaging (QPI) provides 3D structural and morphological information for label free living cells. Unfortunately, this quantitative phase information cannot meet doctors' diagnostic requirements of the clinical "gold standard," which displays stained cells' pathological states based on 2D color features. To make QPI results satisfy the clinical "gold standard," the virtual staining method by QPI for label free lymphocytes based on self-supervised iteration Cycle-Consistent Adversarial Networks (CycleGANs) is proposed herein. The 3D phase information of QPI is, therefore, trained and transferred to a kind of 2D "virtual staining" image that is well in agreement with "gold standard" results. To solve the problem that unstained QPI and stained "gold standard" results cannot be obtained for the same label free living cell, the self-supervised iteration for the CycleGAN deep learning algorithm is designed to obtain a trained stained result as the ground truth for error evaluation. The structural similarity index of our virtual staining experimental results for 8756 lymphocytes is 0.86. Lymphocytes' area errors after converting to 2D virtual stained results from 3D phase information are less than 3.59%. The mean error of the nuclear to cytoplasmic ratio is 2.69%, and the color deviation from the "gold standard" is less than 6.67%.

Multi-domain stain normalization for digital pathology: A cycle-consistent adversarial network for whole slide images

The variation in histologic staining between different medical centers is one of the most profound challenges in the field of computer-aided diagnosis. The appearance disparity of pathological whole slide images causes algorithms to become less reliable, which in turn impedes the wide-spread applicability of downstream tasks like cancer diagnosis. Furthermore, different stainings lead to biases in the training which in case of domain shifts negatively affect the test performance. Therefore, in this paper we propose MultiStain-CycleGAN, a multi-domain approach to stain normalization based on CycleGAN. Our modifications to CycleGAN allow us to normalize images of different origins without retraining or using different models. We perform an extensive evaluation of our method using various metrics and compare it to commonly used methods that are multi-domain capable. First, we evaluate how well our method fools a domain classifier that tries to assign a medical center to an image. Then, we test our normalization on the tumor classification performance of a downstream classifier. Furthermore, we evaluate the image quality of the normalized images using the Structural similarity index and the ability to reduce the domain shift using the Fréchet inception distance. We show that our method proves to be multi-domain capable, provides a very high image quality among the compared methods, and can most reliably fool the domain classifier while keeping the tumor classifier performance high. By reducing the domain influence, biases in the data can be removed on the one hand and the origin of the whole slide image can be disguised on the other, thus enhancing patient data privacy.

Ancient Chinese painting style transfer based on CycleGAN

Style transfer aims to alter the visual aesthetic of images by giving them a different artistic style. With the rapid advancement of deep learning, style transfer tasks have made significant progress, introducing new perspectives and innovative potentials within the realm of image processing. This study seeks to explore style transfer methods based on Cycle-Consistent Adversarial Networks (CycleGAN), enabling contemporary landscape photographs to take on the form of ancient Chinese paintings. This endeavor opens up fresh possibilities for artistic creation, image editing and design applications. The research encompasses an exposition of the process involved in constructing the CycleGAN model, alongside presenting research findings. Furthermore, it delves into the discussion of crucial techniques employed during the model training process, specifically the utilization of cycle consistency loss in configuring the loss functions. Lastly, this study ventures into future research directions, including strategies for further enhancing the performance and expanding the application scope of this style transfer model.

Unsupervised image-to-image translation to reduce the annotation effort for instance segmentation of field vegetables

Due to the constantly increasing shortage of labour availability in the agricultural sector, automation is on the rise. This trend is particularly evident in field vegetable production, where the majority of costs are associated with human labour. To automate labour-intensive tasks, intelligent camera systems are required. Typical state of the art learning algorithms requires large amounts of labelled data for a reliable detection and segmentation. The effort for recording and labelling is increased by the diverse conditions on a field that need to be included in the datasets. In this paper, an unsupervised approach is presented, which unifies varying environmental conditions in the field, as well as an analysis to transfer this unpaired deep learning based unification to unknown exposure scenarios, different growth stages and other varieties. For this purpose, a Cycle-Consistent Adversarial Network (CycleGAN) is used which balance varying exposure and background situations to the training data variance without additional labelling effort. By adapting the exposure with a CycleGAN, the required labelling effort could be reduced by up to five times in relation to a heterogeneous dataset, as fewer variations need to be covered for a robust segmentation. The transfer of this approach to other growth stages, as well as the transfer to another cabbage cultivar shows the potential of additional data generalisation.

Extracting interpretable features for pathologists using weakly supervised learning to predict p16 expression in oropharyngeal cancer

One drawback of existing artificial intelligence (AI)-based histopathological prediction models is the lack of interpretability. The objective of this study is to extract p16-positive oropharyngeal squamous cell carcinoma (OPSCC) features in a form that can be interpreted by pathologists using AI model. We constructed a model for predicting p16 expression using a dataset of whole-slide images from 114 OPSCC biopsy cases. We used the clustering-constrained attention-based multiple-instance learning (CLAM) model, a weakly supervised learning approach. To improve performance, we incorporated tumor annotation into the model (Annot-CLAM) and achieved the mean area under the receiver operating characteristic curve of 0.905. Utilizing the image patches on which the model focused, we examined the features of model interest via histopathologic morphological analysis and cycle-consistent adversarial network (CycleGAN) image translation. The histopathologic morphological analysis evaluated the histopathological characteristics of image patches, revealing significant differences in the numbers of nuclei, the perimeters of the nuclei, and the intercellular bridges between p16-negative and p16-positive image patches. By using the CycleGAN-converted images, we confirmed that the sizes and densities of nuclei are significantly converted. This novel approach improves interpretability in histopathological morphology-based AI models and contributes to the advancement of clinically valuable histopathological morphological features.

HR-CycleGAN: Face highlight reduction based on improved cycle-consistent adversarial networks

HR-CycleGAN: Face highlight reduction based on improved cycle-consistent adversarial networks

COMIC: An unsupervised change detection method for heterogeneous remote sensing images based on copula mixtures and Cycle-Consistent Adversarial Networks

In this paper, we consider the problem of change detection (CD) with two heterogeneous remote sensing (RS) images. For this problem, an unsupervised change detection method has been proposed recently based on the image translation technique of Cycle-Consistent Adversarial Networks (CycleGANs), where one image is translated from its original modality to the modality of the other image so that the difference map can be obtained by performing arithmetical subtraction. However, the difference map derived from subtraction is susceptible to image translation errors, in which case the changed area and the unchanged area are less distinguishable. To overcome the above shortcoming, we propose a new unsupervised copula mixture and CycleGAN-based CD method (COMIC), which combines the advantages of copula mixtures on statistical modeling and the advantages of CycleGANs on data mining. In COMIC, the pre-event image is first translated from its original modality to the post-event image modality. After that, by constructing a copula mixture, the joint distribution of the features from the heterogeneous images can be learnt according to quantitive analysis of the dependence structure based on the translated image and the original pre-event image, which are of the same modality and contain totally the same objects. Then, we model the CD problem as a binary hypothesis testing problem and derive its test statistics based on the constructed copula mixture. Finally, the difference map can be obtained from the test statistics and the binary change map (BCM) is generated by K-means clustering. We perform experiments on real RS datasets, which demonstrate the superiority of COMIC over the state-of-the-art methods.

TDSCCNet: twin-depthwise separable convolution connect network for change detection with heterogeneous images

The task of change detection (CD) in optical and SAR images is an ever-evolving and demanding subject within the realm of remote sensing (RS). It holds great significance to identify the target areas by using complementary information between the two. Due to the distinct imaging mechanisms employed by optical and SAR sensors, effectively and accurately identifying changing regions can be challenging. To this end, a novel heterogeneous RS images CD network (Twin-Depthwise Separable Convolution Connect Network, TDSCCNet) is proposed in this paper. Image domain transformation is a front-end task, while the back-end employs a single-branch bilayer depthwise separable convolution-connected encoder-decoder to accomplish CD work. Specifically, first, the cycle-consistent adversarial network (CycleGAN) serves to integrate the optical and SAR visual domains, and a consistent feature expression is obtained. Second, the single-branch encoder structure of bilayer depthwise separable convolution is employed to realize change feature extraction. Finally, the multiscale connected decoder reconstructed by change map is utilized to reconstruct the original images and solve the local discontinuities and holes in the binary change map. Multiscale loss is designated for optimizing the global and local effects to alleviate the class imbalance problem. It was tested on four representative datasets from Gloucester, Shuguang Village, Italy and WV-3 datasets with an overall accuracy of 97.36%, 97.01%, 97.62%, and 98.01% respectively. By comparing the existing methods, experimental results confirmed the effectiveness of the proposed method.

CycleFusion: Automatic Annotation and Graph-to-Graph Transaction Based Cycle-Consistent Adversarial Network for Infrared and Visible Image Fusion

CycleFusion: Automatic Annotation and Graph-to-Graph Transaction Based Cycle-Consistent Adversarial Network for Infrared and Visible Image Fusion

A GPR Resolution Enhancement Method Based on Weakly Supervised Learning

Ground penetrating radar (GPR) is mainly used to detect underground structures in archaeology, military and other fields. In practice, the detection depth and resolution of GPR data are often difficult to balance. Therefore, it is vital to develop a method that can obtain both deeper distance and higher resolution. In this paper, a weakly supervised method for improving the resolution of GPR data based on the Cycle-Consistent Adversarial Network (Cycle-GAN) is proposed, which improves the quality of GPR data by learning the mapping relation between low-resolution and unpaired high-resolution data. The actual data is used to verify the validity and feasibility of the proposed method. Experimental results have shown that our method is able to recover detailed high-frequency components and the resolution is effectively improved.

A New Foreground-Perception Cycle-Consistent Adversarial Network for Surface Defect Detection With Limited High-Noise Samples

Surface defect detection (SDD) is critical in the smart manufacturing systems to ensure product quality. Nevertheless, the defective samples are always insufficient, and there exists high-noise backgrounds in the samples from the real-world SDD applications, which could significantly affect SDD models. To remedy the drawbacks, a new foreground-perception Cycle-consistent Adversarial Network (FCGAN) is proposed, and it can synthesize high-quality pseudo-defect images by recognizing the foreground of samples. Firstly, the attention networks are combined with the generator and discriminator to guide the training process. Secondly, the discriminative foreground is recognized and used to synthesize coordinated and realistic pseudo-defect images. Thirdly, the adversarial training for FCGAN is designed. The experiments are carried out on two well-known datasets and a real-world SDD for glass bottles. The results show that FCGAN can produce better pseudo-defect images than the state-of-the-art Generative Adversarial Networks, and it can promote the detection accuracy of SDD as well.

A three-dimensional deep learning model for inter-site harmonization of structural MR images of the brain: Extensive validation with a multicenter dataset

In multicenter MRI studies, pooling the imaging data can introduce site-related variabilities and can therefore bias the subsequent analyses. To harmonize the intensity distributions of brain MR images in a multicenter dataset, unsupervised deep learning methods can be employed. Here, we developed a model based on cycle-consistent adversarial networks for the harmonization of T1-weighted brain MR images. In contrast to previous works, it was designed to process three-dimensional whole-brain images in a stable manner while optimizing computation resources. Using six different MRI datasets for healthy adults (n=1525 in total) with different acquisition parameters, we tested the model in (i) three pairwise harmonizations with site effects of various sizes, (ii) an overall harmonization of the six datasets with different age distributions, and (iii) a traveling-subject dataset. Our results for intensity distributions, brain volumes, image quality metrics and radiomic features indicated that the MRI characteristics at the various sites had been effectively homogenized. Next, brain age prediction experiments and the observed correlation between the gray-matter volume and age showed that thanks to an appropriate training strategy and despite biological differences between the dataset populations, the model reinforced biological patterns. Furthermore, radiologic analyses of the harmonized images attested to the conservation of the radiologic information in the original images. The robustness of the harmonization model (as judged with various datasets and metrics) demonstrates its potential for application in retrospective multicenter studies.

Assessment of hyperspectral imaging and CycleGAN-simulated narrowband techniques to detect early esophageal cancer

The clinical signs and symptoms of esophageal cancer (EC) are often not discernible until the intermediate or advanced phases. The detection of EC in advanced stages significantly decreases the survival rate to below 20%. This study conducts a comparative analysis of the efficacy of several imaging techniques, including white light image (WLI), narrowband imaging (NBI), cycle-consistent adversarial network simulated narrowband image (CNBI), and hyperspectral imaging simulated narrowband image (HNBI), in the early detection of esophageal cancer (EC). In conjunction with Kaohsiung Armed Forces General Hospital, a dataset consisting of 1000 EC pictures was used, including 500 images captured using WLI and 500 images captured using NBI. The CycleGAN model was used to generate the CNBI dataset. Additionally, a novel method for HSI imaging was created with the objective of generating HNBI pictures. The evaluation of the efficacy of these four picture types in early detection of EC was conducted using three indicators: CIEDE2000, entropy, and the structural similarity index measure (SSIM). Results of the CIEDE2000, entropy, and SSIM analyses suggest that using CycleGAN to generate CNBI images and HSI model for creating HNBI images is superior in detecting early esophageal cancer compared to the use of conventional WLI and NBI techniques.

Cycle-generative adversarial network-based bone suppression imaging for highly accurate markerless motion tracking of lung tumors for cyberknife irradiation therapy.

Lung tumor tracking during stereotactic radiotherapy with the CyberKnife can misrecognize tumor location under conditions where similar patterns exist in the search area. This study aimed to develop a technique for bone signal suppression during kV-x-ray imaging. Paired CT images were created with or without bony structures using a 4D extended cardiac-torso phantom (XCAT phantom) in 56 cases. Subsequently, 3020 2D x-ray images were generated. Images with bone were input into cycle-consistent adversarial network (CycleGAN) and the bone suppressed images on the XCAT phantom (BSIphantom ) were created. They were then compared to images without bone using the structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR). Next, 1000 non-simulated treatment images from real cases were input into the training model, and bone-suppressed images of the patient (BSIpatient ) were created. Zero means normalized cross correlation (ZNCC) by template matching between each of the actual treatment images and BSIpatient were calculated. BSIphantom values were compared to their paired images without bone of the XCAT phantom test data; SSIM and PSNR were 0.90±0.06 and 24.54±4.48, respectively. It was visually confirmed that only bone was selectively suppressed without significantly affecting tumor visualization. The ZNCC values of the actual treatment images and BSIpatient were 0.763±0.136 and 0.773±0.143, respectively. The BSIpatient showed improved recognition accuracy over the actual treatment images. The proposed bone suppression imaging technique based on CycleGAN improves image recognition, making it possible to achieve highly accurate motion tracking irradiation.

Zero-Shot Generative AI for Rotating Machinery Fault Diagnosis: Synthesizing Highly Realistic Training Data via Cycle-Consistent Adversarial Networks

The Intelligent Fault Diagnosis of rotating machinery calls for a substantial amount of training data, posing challenges in acquiring such data for damaged industrial machinery. This paper presents a novel approach for generating synthetic data using a Generative Adversarial Network (GAN) with cycle consistency loss function known as cycleGAN. The proposed method aims to generate synthetic data that could effectively replace real experimental data. The generative model is trained to transform wavelet images of simulated vibrational signals into authentic data obtained from machinery with damaged bearings. The utilization of Maximum Mean Discrepancy (MMD) and Fréchet Inception Distance (FID) demonstrates a noteworthy resemblance between synthetic and real experimental data. Also, the generative model enables the synthesis of data that may have been entirely lacking from the experimental observation, indicating generative zero-shot learning capabilities. The efficacy of synthetic data in training diagnosis algorithms by means of Transfer Learning (TL) on Convolutional Neural Networks (CNNs) has been demonstrated to be comparable to that of real data. The study has been validated by means of the test rig for medium-sized industrial bearings accessible at the Politecnico di Torino.

LRFID-Net: A Local-Region-Based Fake-Iris Detection Network for Fake Iris Images Synthesized by a Generative Adversarial Network

Iris recognition is a biometric method using the pattern of the iris seated between the pupil and the sclera for recognizing people. It is widely applied in various fields owing to its high accuracy in recognition and high security. A spoof detection method for discriminating a spoof attack is essential in biometric recognition systems that include iris recognition. However, previous studies have mainly investigated spoofing attack detection methods based on printed or photographed images, video replaying, artificial eyes, and patterned contact lenses fabricated using iris images from information leakage. On the other hand, there have only been a few studies on spoof attack detection using iris images generated through a generative adversarial network (GAN), which is a method that has drawn considerable research interest with the recent development of deep learning, and the enhancement of spoof detection accuracy by the methods proposed in previous research is limited. To address this problem, the possibility of an attack on a conventional iris recognition system with spoofed iris images generated using cycle-consistent adversarial networks (CycleGAN), which was the motivation of this study, was investigated. In addition, a local region-based fake-iris detection network (LRFID-Net) was developed. It provides a novel method for discriminating fake iris images by segmenting the iris image into three regions based on the iris region. Experimental results using two open databases, the Warsaw LiveDet-Iris-2017 and the Notre Dame Contact Lens Detection LiveDet-Iris-2017 datasets, showed that the average classification error rate of spoof detection by the proposed method was 0.03% for the Warsaw dataset and 0.11% for the Notre Dame Contact Lens Detection dataset. The results confirmed that the proposed method outperformed the state-of-the-art methods.

A fatigue crack quantification model for metallic structure based on strain monitoring data

Obtaining the real-time fatigue crack length of a metallic structure is the prerequisite of the fatigue life monitoring and residual life estimation for an aircraft. This paper proposed a metallic structure's fatigue crack prediction model using strain monitoring data based on deep learning method. A cycle consistent adversarial network was developed to map the strain monitoring data from experimental measurement with those from finite element modeling. A crack size classification model and a crack length quantification model were proposed to classify the crack size range and identify the exact crack length, respectively. The proposed model was applied to predict the fatigue crack growth in centeral hole metallic plates subjected to random loading spectrum. The results showed that the prediction is effective and accurate.

CT synthesis from CBCT using a sequence-aware contrastive generative network.

Computerized tomography (CT) synthesis from cone-beam computerized tomography (CBCT) is a key step in adaptive radiotherapy. It uses a synthetic CT to calculate the dose to correct and adjust the radiotherapy plan in a timely manner. The cycle-consistent adversarial network (Cycle GAN) is commonly used in CT synthesis tasks but it has some defects: (a) the premise of the cycle consistency loss is that the conversion between domains is bijective, but the CBCT and CT conversion does not fully satisfy the bijective relationship, and (b) it does not take advantage of the complementary information between multiple sets of CBCTs for the same patient. To address these problems, we propose a novel framework named the sequence-aware contrastive generative network (SCGN) that introduces an attention sequence fusion module to improve the CBCT quality. In addition, it not only applies contrastive learning to the generative adversarial networks (GANs) to pay more attention to the anatomical structure of CBCT in feature extraction but also uses a new generator to improve the accuracy of the anatomical details. Experimental results on our datasets show that our method significantly outperforms the existing unsupervised CT synthesis methods.

More Efficient Auto-Segmentation Framework Using Patient-Specific Information for CBCT-Guided Adaptive Radiotherapy

Accurate and fast delineation of regions of interest (ROIs) is critical for online adaptive radiotherapy (ART). Due to the noise, artifacts and low contrast of soft tissue on cone-beam computed tomography (CBCT), the CBCT-guided ART (CBCTgART) needs an effective tool to assist the segmentation of ROIs. This study aims to develop an efficient auto-segmentation workflow with personalized modeling for CBCTgART based on deep learning (DL) model. Five hundred fifty-two patients with nasopharyngeal carcinoma were enrolled in this study. At the beginning, a cycle consistent adversarial network (CycleGAN) model was trained on 172 patients' CBCT/CT paired images to improve the image quality of CBCT to CT level. A generalized segmentation model was trained using the planning CT (pCT) and contour data from 530 patients. For personalized modeling, the generalized segmentation model was fine-tuned on the specific patient's pCT and contour to get the personalized model. When CBCT images were available, the trained CycleGAN model transformed the CBCT to synthetic pCT. Then the personalized auto-segmentation model generated the contour of ROIs on the synthetic pCT. We randomly selected 22 patients for model test. The proposed method (DL-personalized) was compared with other DL methods based on the same architecture of network: regular deep learning method (DL-regular), which was trained on the CBCT and corresponding contours, and generalized model in our framework (DL-generalized). So, 22 personalized, 1 generalized and 1 regular DL models were tested. The paired t-test was performed to test the significance for mean dice similarity coefficient (DSC), mean distance to agreement (MDA), and Hausdorff distance (HD) between the alternative and proposed methods. Two ROIs were included: the clinical target volume (CTV) and nasopharynx gross tumor volume (GTVnx). The proposed DL-personalized model achieved better results compared with others as shown in the table. The accuracy and robustness of our proposed framework was reliable. All of p values were under 0.01, which indicated the statistically significant difference. The proposed framework utilizing patient-specific information can improve the segmentation accuracy of ART. This method has potential to be integrated into the ART workflow to improve efficiency.

Raman spectrum model transfer method based on Cycle-GAN

The disparity in hardware quality among various models of Raman spectrometers gives rise to variations in the acquired Raman spectral data, even when the same substance is collected under identical external conditions. Conventionally, models constructed using data obtained from a particular instrument exhibit issues such as limited applicability or poor performance when deployed to different instruments. Currently, numerous model transfer algorithms grounded in chemometrics have been developed, all aiming to establish a mapping relationship capable of transforming spectral data from the source domain to the target domain. With the advancement of deep learning techniques, the utilization of deep learning enables the effective resolution of nonlinear mapping relationships between two spectral vectors. In the field of image translation, the Cycle-Consistent Adversarial Networks, Cycle-GAN, has already achieved mutual transformation between two distinct style images. However, due to images being multidimensional matrix data, unlike one-dimensional spectral data vectors, we have constructed a deep learning network based on Cycle-GAN for vector-to-vector transformation. This network allows the direct conversion of spectral data from the source domain to the target domain, without requiring parameter adjustments or other operations. Compared with traditional chemometric methods, our method is more intelligent and efficient. Finally, the cosine similarity between the source domain data and the transformed target domain data exceeds 99%.