Cycle Generative Adversarial Network Research Articles

Large-Field Microscopic Imaging Method Based on Cycle Generative Adversarial Networks

Large-Field Microscopic Imaging Method Based on Cycle Generative Adversarial Networks

Improving Brain Tumor Classification with Deep Learning Using燬ynthetic燚ata

Deep learning (DL) techniques, which do not need complex pre-processing and feature analysis, are used in many areas of medicine and achieve promising results. On the other hand, in medical studies, a limited dataset decreases the abstraction ability of the DL model. In this context, we aimed to produce synthetic brain images including three tumor types (glioma, meningioma, and pituitary), unlike traditional data augmentation methods, and classify them with DL. This study proposes a tumor classification model consisting of a Dense Convolutional Network (DenseNet121)-based DL model to prevent forgetting problems in deep networks and delay information flow between layers. By comparing models trained on two different datasets, we demonstrated the effect of synthetic images generated by Cycle Generative Adversarial Network (CycleGAN) on the generalization of DL. One model is trained only on the original dataset, while the other is trained on the combined dataset of synthetic and original images. Synthetic data generated by CycleGAN improved the best accuracy values for glioma, meningioma, and pituitary tumor classes from 0.9633, 0.9569, and 0.9904 to 0.9968, 0.9920, and 0.9952, respectively. The developed model using synthetic data obtained a higher accuracy value than the related studies in the literature. Additionally, except for pixel-level and affine transform data augmentation, synthetic data has been generated in the figshare brain dataset for the first time.

Generating Chest X-Ray Progression of Pneumonia Using Conditional Cycle Generative Adversarial Networks

Generating Chest X-Ray Progression of Pneumonia Using Conditional Cycle Generative Adversarial Networks

Intelligent synthesis of hyperspectral images from arbitrary web cameras in latent sparse space reconstruction

<abstract><p>Synthesizing hyperspectral images (HSI) from an ordinary camera has been accomplished recently. However, such computation models require detailed properties of the target camera, which can only be measured in a professional lab. This prerequisite prevents the synthesizing model from being installed on arbitrary cameras for end-users. This study offers a calibration-free method for transforming any camera into an HSI camera. Our solution requires no controllable light sources and spectrometers. Any consumer installing the program should produce high-quality HSI without the assistance of optical laboratories. Our approach facilitates a cycle-generative adversarial network (cycle-GAN) and sparse assimilation method to render the illumination-dependent spectral response function (SRF) of the underlying camera at the first part of the setup stage. The current illuminating function (CIF) must be identified for each image and decoupled from the underlying model. The HSI model is then integrated with the static SRF and dynamic CIF in the second part of the stage. The estimated SRFs and CIFs have been double-checked with the results by the standard laboratory method. The reconstructed HSIs have errors under 3% in the root mean square.</p></abstract>

The whole slide breast histopathology image detection based on a fused model and heatmaps

The analysis of whole slide breast histopathology image is one of the most effective techniques in the current breast cancer diagnosis. However, the size of whole slide image(WSI) is too large to be processed directly by machine learning or deep learning based classifier. To solve the problem of WSI-based computer aided diagnosis, a novel classification framework with three stages is proposed in this paper to locate and classify tumor region in the whole slide images. This framework contains three parts: patch-based classification, tumor region segmentation and location, and WSI-based classification. In the period of data processing, the Cycle-GAN(Cycle Generative Adversarial Network) model is firstly adopted to normalize the colors of the images. Then an improved DPN68(Dual Path Networks) based classifier and a Swin-Transformer based classifier are combined to improve the accuracy of patch-based classification. With the malignant confidence probability output by the fused model, the whole slide heatmap is generated to segment, locate and visually display the tumor regions. Statistical features are computed and selected from the heatmap based on medical diagnosis indicators, and then an SVM(Support Vector Machines) based classifier is applied to implement WSI-based classification to further confirm the tumor. Experimental results on the Camelyon16 dataset show that our proposed framework can effectively classify the whole slide breast histopathology image with high-precision.

An Online Rail Track Fastener Classification System Based on YOLO Models

In order to save manpower on rail track inspection, computer vision-based methodologies are developed. We propose utilizing the YOLOv4-Tiny neural network to identify track defects in real time. There are ten defects covering fasteners, rail surfaces, and sleepers from the upward and six defects about the rail waist from the sideward. The proposed real-time inspection system includes a high-performance notebook, two sports cameras, and three parallel processes. The hardware is mounted on a flat cart running at 30 km/h. The inspection results about the abnormal track components could be queried by defective type, time, and the rail hectometer stake. In the experiments, data augmentation by a Cycle Generative Adversarial Network (GAN) is used to increase the dataset. The number of images is 3800 on the upward and 967 on the sideward. Five object detection neural network models-YOLOv4, YOLOv4-Tiny, YOLOX-Tiny, SSD512, and SSD300-were tested. The YOLOv4-Tiny model with 150 FPS is selected as the recognition kernel, as it achieved 91.7%, 92%, and 91% for the mAP, precision, and recall of the defective track components from the upward, respectively. The mAP, precision, and recall of the defective track components from the sideward are 99.16%, 96%, and 94%, respectively.

Unsupervised domain adaptation using modified cycle generative adversarial network for aerial image classification

The unsupervised domain adaptation (UDA) of aerial image classification is of great significance in the traffic management and public safety applications. Due to the differences in the distribution of source and target domain datasets from different sensors and cities, and the acquisition difficulty and high annotation cost of target domain datasets, existing domain adaptive methods have difficulty obtaining high accuracy and stability. In this work, a modified cycle generative adversarial network (CycleGAN) (MGAN) based on the multiscale residual block (MSRB) and multipooling coordinate attention (MPCA) modules is proposed to generate fake images, which is the most critical stage of the UDA method. In the MGAN, the MSRB module can extract the multiscale information of the image, and the MPCA module can fuse the position and channel information in the image. These two modules better extract the features of edges, regions and semantic classes, which makes the generated fake images better retain the information of the source domain and possess the style of the target domain. Otherwise, the Deeplabv3+ model is used to obtain a pretrained and final segmentation model in the first and third stages. In the experiment, the proposed method is validated on the aerial images of the Potsdam and Vaihingen datasets in the International Society for Photogrammetry and Remote Sensing (ISPRS) two-dimensional semantic labeling benchmark and improves overall accuracy from 52% to 60% on the segmentation of the target domain, compared with the existing optimal method; it also achieves an accuracy of 77% for buildings. The results demonstrate that the proposed method can achieve a higher accuracy and stability.

Self-supervised deep partial adversarial network for micro-video multimodal classification

Micro-videos have gained popularity on various social media platforms because they provide a great medium for real-time storytelling. Although micro-videos can be naturally characterized by several modalities, for situations with uncertain missing modalities, a flexible multimodal representation learning framework that integrates complementary and consistent information has been difficult to develop. To better deal with the issue regarding incomplete modalities in multimodal micro-video classification, in this paper, we propose a self-supervised deep multimodal adversarial network (SDMAN) to learn comprehensive and robust micro-video representations. Specifically, we first consider a parallel multi-head attention (MHA) encoding module that simultaneously learns the representations of complete and incomplete modality groupings. We then present a multimodal self-supervised cycle generative adversarial network module, in which multiple generative adversarial networks are explored to transfer the information obtained from the complete modality grouping to the incomplete modality groupings. As a result, complementarity and consistency are mutually promoted among the modalities. Furthermore, experiments conducted on a large-scale micro-video dataset demonstrate that the SDMAN performs better than the state-of-the-art methods.

Deep learning-based dynamic PET parametric Ki image generation from lung static PET.

PET/CT is a first-line tool for the diagnosis of lung cancer. The accuracy of quantification may suffer from various factors throughout the acquisition process. The dynamic PET parametric Ki provides better quantification and improve specificity for cancer detection. However, parametric imaging is difficult to implement clinically due to the long acquisition time (~ 1 h). We propose a dynamic parametric imaging method based on conventional static PET using deep learning. Based on the imaging data of 203 participants, an improved cycle generative adversarial network incorporated with squeeze-and-excitation attention block was introduced to learn the potential mapping relationship between static PET and Ki parametric images. The image quality of the synthesized images was qualitatively and quantitatively evaluated by using several physical and clinical metrics. Statistical analysis of correlation and consistency was also performed on the synthetic images. Compared with those of other networks, the images synthesized by our proposed network exhibited superior performance in both qualitative and quantitative evaluation, statistical analysis, and clinical scoring. Our synthesized Ki images had significant correlation (Pearson correlation coefficient, 0.93), consistency, and excellent quantitative evaluation results with the Ki images obtained in standard dynamic PET practice. Our proposed deep learning method can be used to synthesize highly correlated and consistent dynamic parametric images obtained from static lung PET. • Compared with conventional static PET, dynamic PET parametric Ki imaging has been shown to provide better quantification and improved specificity for cancer detection. • The purpose of this work was to develop a dynamic parametric imaging method based on static PET images using deep learning. • Our proposed network can synthesize highly correlated and consistent dynamic parametric images, providing an additional quantitative diagnostic reference for clinicians.

Holographic reconstruction enhancement via unpaired image-to-image translation.

Digital holographic microscopy is an imaging process that encodes the 3D information of a sample into a single 2D hologram. The holographic reconstruction that decodes the hologram is conventionally based on the diffraction formula and involves various iterative steps in order to recover the lost phase information of the hologram. In the past few years, the deep-learning-based model has shown great potential to perform holographic reconstruction directly on a single hologram. However, preparing a large and high-quality dataset to train the models remains a challenge, especially when the holographic reconstruction images that serve as ground truth are difficult to obtain and can have a deteriorated quality due to various interferences of the imaging device. A cycle generative adversarial network is first trained with unpaired brightfield microscope images to restore the visual quality of the holographic reconstructions. The enhanced holographic reconstructions then serve as ground truth for the supervised learning of a U-Net that performs the holographic reconstruction on a single hologram. The proposed method was evaluated on plankton images and could also be applied to achieve super-resolution or colorization of the holographic reconstructions.

CX-DaGAN: Domain Adaptation for Pneumonia Diagnosis on a Small Chest X-Ray Dataset.

Recent advances in deep learning led to several algorithms for the accurate diagnosis of pneumonia from chest X-rays. However, these models require large training medical datasets, which are sparse, isolated, and generally private. Furthermore, these models in medical imaging are known to over-fit to a particular data domain source, i.e., these algorithms do not conserve the same accuracy when tested on a dataset from another medical center, mainly due to image distribution discrepancies. In this work, a domain adaptation and classification technique is proposed to overcome the over-fit challenges on a small dataset. This method uses a private-small dataset (target domain), a public-large labeled dataset from another medical center (source domain), and consists of three steps. First, it performs a data selection of the source domain's most representative images based on similarity constraints through principal component analysis subspaces. Second, the selected samples from the source domain are fit to the target distribution through an image to image translation based on a cycle-generative adversarial network. Finally, the target train dataset and the adapted images from the source dataset are used within a convolutional neural network to explore different settings to adjust the layers and perform the classification of the target test dataset. It is shown that fine-tuning a few specific layers together with the selected-adapted images increases the sorting accuracy while reducing the trainable parameters. The proposed approach achieved a notable increase in the target dataset's overall classification accuracy, reaching up to 97.78 % compared to 90.03 % by standard transfer learning.

Hiding Message Using a Cycle Generative Adversarial Network

Training an image steganography is an unsupervised problem, because it is impossible to obtain an ideal supervised steganographic image corresponding to the cover image and secret message. Inspired by the success of cycle generative adversarial networks in unsupervised tasks such as style transfer, this article proposes to use a cycle generative adversarial network to solve the problem of unsupervised image steganography. Specifically, this article jointly trains five networks, i.e., a steganographic network, an inverse steganographic network, a hidden message reconstruction network, and two discriminative networks, which together constitute a hidden message cycle generative adversarial network (HCGAN). Compared with the recent image steganography based on generative adversative network, HCGAN provides more accurate supervised information, which makes the training process of HCGAN converge faster and the performance of the trained image steganography network is better. In addition, this article introduces an image steganographic network based on residual learning and shows that residual learning can effectively improve the performance of steganography. Furthermore, to the best of our knowledge, we are the first to propose an inverse steganographic network for eliminating steganographic message from steganographic images, which can be used to avoid steganographic message being discovered or acquired by a third party. The experimental results show that compared with the steganography based on generative adversarial network, the proposed HCGAN has a higher correct decoding rate, better visual quality of steganographic image, and higher secrecy.

CycleADC-Net: A crack segmentation method based on multi-scale feature fusion

Crack detection is an important factor in structural safety assessment. But there are many challenges in detecting crack, especially for the ones with very thin shape, uneven intensity, or lying in a complex background. In addition, the crack image captured in poor lighting conditions makes it more difficult to be detected. Most of the existing crack detection methods are designed for well-light cracks, whose detection performance drops dramatically on low-light crack detection. To overcome these challenges, a novel encoder-decoder segmentation network, called CycleADC-Net is proposed in this work which opens a new idea to detect the crack images in low-light condition. A cycle generative adversarial network is employed to translate the low-light crack image to the bright domain, and its output is fed to the follow-up encoder-decoder segmentation network for final detection. The dual-channel feature extraction module and the attention mechanism are both introduced to extract semantic multi-scale image features, reduce information loss and suppress irrelevant background information when performing crack segmentation. The experimental results show that the proposed CycleADC-Net performs superior both on low-light crack or well-light crack databases over recent segmentation networks, suggesting it has good generalization ability and great potential in road inspection task under poor lighting environment.

A Multi-Centric Evaluation of AI-Driven Synthetic CT Generation Form Low Field Magnetic Resonance Imaging

<h3>Purpose/Objective(s)</h3> Magnetic Resonance is an essential modality in the context of radiotherapy primarily for providing additional information with respect to tumor functional information leading to better target delineation (brain, prostate, etc.) and secondary in the context of emergence of magnetic resonance guided radiotherapy. However, in both cases computed tomography acquisition remains necessary for dosimetric purposes hampering the workflow, introducing additional cost and dose simulation/optimization discrepancies (due to the registration issues between MR & CT) while being associated with additional patient toxicity. This study aims to investigate in a retrospective manner the relevance deep learning synthetic CT generation as an alternative for simulation and planning for pelvic tumors treated with low field (0.35T) MRgRT. <h3>Materials/Methods</h3> In the context of this study, a cycle generative adversarial neural network (GAN) deep learning architecture was trained to determine a bijective transformation between a low field (0.35T) MR and the associated computed tomography scan acquisition. The training set involved 350 pairs of weakly aligned data of pelvis cases. A retrospective cohort involving 20 test cases coming from eight different institutions (US: 2, EU: 5, AS: 1) involving different CT vendors was considered for testing. <h3>Results</h3> Reconstruction performance was assessed using the OARs used for treatment. The observed average scaled (between maximum and minimum HU values of the CT) difference between the reconstructed and the CT used for planning was 35.08 with significant discrepancies across organs. Femoral heads were the most reliably reconstructed (4.51 & 4.77) while rectum and sigmoid were the less precise ones (53.13 & 51.48). In terms of qualitative evaluation, the presence of fiducial markers heavily penalizes the reconstruction due to the implicit propagation of errors associated with the "convolution" nature of deep learning. The presence of air bubbles visible on the CT (sigmoid and rectum) is also an issue since these elements are marginally perceptible in the MR and therefore can hardly be predicted from generator. Detailed reconstruction results per organ are appended. <h3>Conclusion</h3> This retrospective multi-centric study is a first step toward assessing the potential of a fully low field MR-based treatment planning workflow that eliminates the need of CT acquisition. Integration of the synthetic CT in the simulation/optimization and assessment of the induced dosimetric errors are necessary steps to determine the clinical relevance of our preliminary findings.

Modern Crack Detection for Bridge Infrastructure Maintenance Using Machine Learning

Manual investigation of damages incurred to infrastructure is a challenging process, in that it is not only labour-intensive and expensive but also inefficient and error-prone. To automate the process, a method that is based on computer vision for automatically detecting cracks from 2D images is a viable option. Amongst the different methods of deep learning that are commonly used, the convolutional neural network (CNNs) is one that provides the opportunity for end-to-end mapping/learning of image features instead of using the manual suboptimal image feature extraction. Specifically, CNNs do not require human supervision and are more suitable to be used for indoor and outdoor applications requiring image feature extraction and are less influenced by internal and external noise. Additionally, the CNN’s are also computationally efficient since they are based on special convolution layers and pooling operations that enable the full execution of CNN frameworks on several hardware devices. Keeping this in mind, we propose a deep CNN framework that is based on 10 different convolution layers along with a cycle GAN (Generative Adversarial Network) for predicting the crack segmentation pixel by pixel in an end-to-end manner. The methods proposed here include the Deeply Supervised Nets (DSN) and Fully Convolutional Networks (FCN). The use of DSN enables integrated feature supervision for each stage of convolution. Furthermore, the model has been designed intricately for learning and aggregating multi-level and multiscale features while moving from the lower to higher convolutional layers through training. Hence, the architecture in use here is unique from the ones in practice which just use the final convolution layer. In addition, to further refine the predicted results, we have used a guided filter and CRFs (Conditional Random Fields) based methods. The verification step for the proposed framework was carried out with a set of 537 images. The deep hierarchical CNN framework of 10 convolutional layers and the Guided filtering achieved high-tech and advanced performance on the acquired dataset, showing higher F-score, Recall and Precision values of 0.870, 0.861, and 0.881 respectively, as compared to the traditional methods such as SegNet, Crack-BN, and Crack-GF.

MetaGanFi

Human has an unique gait and prior works show increasing potentials in using WiFi signals to capture the unique signature of individuals' gait. However, existing WiFi-based human identification (HI) systems have not been ready for real-world deployment due to various strong assumptions including identification of known users and sufficient training data captured in predefined domains such as fixed walking trajectory/orientation, WiFi layout (receivers locations) and multipath environment (deployment time and site). In this paper, we propose a WiFi-based HI system, MetaGanFi, which is able to accurately identify unseen individuals in uncontrolled domain with only one or few samples. To achieve this, the MetaGanFi proposes a domain unification model, CCG-GAN that utilizes a conditional cycle generative adversarial networks to filter out irrelevant perturbations incurred by interfering domains. Moreover, the MetaGanFi proposes a domain-agnostic meta learning model, DA-Meta that could quickly adapt from one/few data samples to accurately recognize unseen individuals. The comprehensive evaluation applied on a real-world dataset show that the MetaGanFi can identify unseen individuals with average accuracies of 87.25% and 93.50% for 1 and 5 available data samples (shot) cases, captured in varying trajectory and multipath environment, 86.84% and 91.25% for 1 and 5-shot cases in varying WiFi layout scenarios, while the overall inference process of domain unification and identification takes about 0.1 second per sample.

How much BiGAN and CycleGAN-learned hidden features are effective for COVID-19 detection from CT images? A comparative study

Bidirectional generative adversarial networks (BiGANs) and cycle generative adversarial networks (CycleGANs) are two emerging machine learning models that, up to now, have been used as generative models, i.e., to generate output data sampled from a target probability distribution. However, these models are also equipped with encoding modules, which, after weakly supervised training, could be, in principle, exploited for the extraction of hidden features from the input data. At the present time, how these extracted features could be effectively exploited for classification tasks is still an unexplored field. Hence, motivated by this consideration, in this paper, we develop and numerically test the performance of a novel inference engine that relies on the exploitation of BiGAN and CycleGAN-learned hidden features for the detection of COVID-19 disease from other lung diseases in computer tomography (CT) scans. In this respect, the main contributions of the paper are twofold. First, we develop a kernel density estimation (KDE)-based inference method, which, in the training phase, leverages the hidden features extracted by BiGANs and CycleGANs for estimating the (a priori unknown) probability density function (PDF) of the CT scans of COVID-19 patients and, then, in the inference phase, uses it as a target COVID-PDF for the detection of COVID diseases. As a second major contribution, we numerically evaluate and compare the classification accuracies of the implemented BiGAN and CycleGAN models against the ones of some state-of-the-art methods, which rely on the unsupervised training of convolutional autoencoders (CAEs) for attaining feature extraction. The performance comparisons are carried out by considering a spectrum of different training loss functions and distance metrics. The obtained classification accuracies of the proposed CycleGAN-based (resp., BiGAN-based) models outperform the corresponding ones of the considered benchmark CAE-based models of about 16% (resp., 14%).

Event recognition method based on dual-augmentation for a Φ-OTDR system with a few training samples.

Thanks to the development of machine learning and deep learning, data-driven pattern recognition based on neural network is a trend for Φ-OTDR system intrusion event recognition. The data-driven pattern recognition needs a large number of samples for training. However, in some scenarios, intrusion signals are difficult to collect, resulting in the lack of training samples. At the same time, labeling a large number of samples is also a very time-consuming work. This paper presents a few-shot learning classification method based on time series transfer and cycle generative adversarial network (CycleGAN) data augmentation for Φ-OTDR system. By expanding the rare samples based on time series transfer and CycleGAN, the number of samples in the dataset can finally meet the requirement of network training. The experimental result shows that even when the training set has two minor classes with only two samples, the average accuracy of the validation set with 5 classification tasks can still reach 90.84%, and the classification accuracy of minor classes can reach 79.28% with the proposed method.

Data/Model Jointly Driven High-Quality Case Generation for Power System Dynamic Stability Assessment

For data-driven dynamic stability assessment (DSA) in power systems, learning cases collected from actual historical records appear to be more reliable than those obtained from numerical simulations with an inevitable reality gap. However, due to the scarceness of transient events in practical systems, historical case sets generally encounter the small sample size and class-imbalance problems. To tackle these challenging issues, this article proposes a novel data/model jointly driven framework to generate high-quality cases for power system DSA applications. Model-driven numerical simulations are first utilized for rough case generation, based upon which case refinement is then intelligently carried out via cycle generative adversarial network (CycleGAN) learning. In this data-driven manner, the CycleGAN is able to produce refined cases highly resembling actual historical ones. A long short-term memory-based semisupervised learning scheme is further designed to reliably label all the refined cases. Numerical tests are comprehensively carried out on the realistic Guangdong Power Grid in South China. With only a small and skewed historical case set initially provided, the proposed framework is able to generate highly realistic cases to augment the set and mitigate the class-imbalance issue. These synthetic cases further help derive a more discerning DSA model, which contributes to enhanced reliability and adaptability of online DSA in practical power grids.

Unsupervised face frontalization using disentangled representation-learning CycleGAN

Face frontalization aims to normalize profile face images to frontal ones for pose-invariant face-related recognition tasks. Current works have achieved outstanding results in face frontalization by using deep learning techniques. However, paired training data is usually required to train a deep face frontalization model and it is always difficult and time-consuming to acquire this kind of data. To solve this problem, we propose an unsupervised face frontalization framework, named Disentangled Representation-learning Cycle Generative Adversarial Network (DRCycleGAN). The model can be trained with unpaired data through embedding face images onto two spaces, identity feature space and pose feature space, and jointly inputting the identity feature and the pose feature to the generators to implement a paired forward and backward mapping (i.e., face frontalization and its inverse process). To adapt to the face frontalization task, a semantic-level cycle consistency loss is proposed, which implements consistency constraint supervision by measuring high-level semantic feature differences. Extensive experiment results demonstrate that the proposed method can achieve promising face frontalization performance and improve the pose-invariant face recognition performance.