Cycle Generative Adversarial Network Research Articles

A Novel Weakly Supervised Remote Sensing Landslide Semantic Segmentation Method: Combining CAM and cycleGAN Algorithms

With the development of deep learning algorithms, more and more deep learning algorithms are being applied to remote sensing image classification, detection, and semantic segmentation. The landslide semantic segmentation of a remote sensing image based on deep learning mainly uses supervised learning, the accuracy of which depends on a large number of training data and high-quality data annotation. At this stage, high-quality data annotation often requires the investment of significant human effort. Therefore, the high cost of remote sensing landslide image data annotation greatly restricts the development of a landslide semantic segmentation algorithm. Aiming to resolve the problem of the high labeling cost of landslide semantic segmentation with a supervised learning method, we proposed a remote sensing landslide semantic segmentation with weakly supervised learning method combing class activation maps (CAMs) and cycle generative adversarial network (cycleGAN). In this method, we used the image level annotation data to replace pixel level annotation data as the training data. Firstly, the CAM method was used to determine the approximate position of the landslide area. Then, the cycleGAN method was used to generate the fake image without a landslide, and to make the difference with the real image to obtain the accurate segmentation of the landslide area. Finally, the pixel-level segmentation of the landslide area on remote sensing image was realized. We used mean intersection-over-union (mIOU) to evaluate the proposed method, and compared it with the method based on CAM, whose mIOU was 0.157, and we obtain better result with mIOU 0.237 on the same test dataset. Furthermore, we made a comparative experiment using the supervised learning method of a u-net network, and the mIOU result was 0.408. The experimental results show that it is feasible to realize landslide semantic segmentation in a remote sensing image by using weakly supervised learning. This method can greatly reduce the workload of data annotation.

Novel Blood Pressure Waveform Reconstruction from Photoplethysmography using Cycle Generative Adversarial Networks.

Continuous monitoring of blood pressure (BP) can help individuals manage their chronic diseases such as hypertension, requiring non-invasive measurement methods in free-living conditions. Recent approaches fuse Photoplethys-mograph (PPG) and electrocardiographic (ECG) signals using different machine and deep learning approaches to non-invasively estimate BP; however, they fail to reconstruct the complete signal, leading to less accurate models. In this paper, we propose a cycle generative adversarial network (CycleGAN) based approach to extract a BP signal known as ambulatory blood pressure (ABP) from a clean PPG signal. Our approach uses a cycle generative adversarial network that extends the GAN architecture for domain translation, and outperforms state-of-the-art approaches by up to 2× in BP estimation.

Modelling intra-muscular contraction dynamics using in silico to in vivo domain translation

BackgroundAdvances in sports medicine, rehabilitation applications and diagnostics of neuromuscular disorders are based on the analysis of skeletal muscle contractions. Recently, medical imaging techniques have transformed the study of muscle contractions, by allowing identification of individual motor units’ activity, within the whole studied muscle. However, appropriate image-based simulation models, which would assist the continued development of these new imaging methods are missing. This is mainly due to a lack of models that describe the complex interaction between tissues within a muscle and its surroundings, e.g., muscle fibres, fascia, vasculature, bone, skin, and subcutaneous fat. Herein, we propose a new approach to overcome this limitation.MethodsIn this work, we propose to use deep learning to model the authentic intra-muscular skeletal muscle contraction pattern using domain-to-domain translation between in silico (simulated) and in vivo (experimental) image sequences of skeletal muscle contraction dynamics. For this purpose, the 3D cycle generative adversarial network (cycleGAN) models were evaluated on several hyperparameter settings and modifications. The results show that there were large differences between the spatial features of in silico and in vivo data, and that a model could be trained to generate authentic spatio-temporal features similar to those obtained from in vivo experimental data. In addition, we used difference maps between input and output of the trained model generator to study the translated characteristics of in vivo data.ResultsThis work provides a model to generate authentic intra-muscular skeletal muscle contraction dynamics that could be used to gain further and much needed physiological and pathological insights and assess and overcome limitations within the newly developed research field of neuromuscular imaging.

Self-supervised learning for multi-center magnetic resonance imaging harmonization without traveling phantoms

Objective. With the progress of artificial intelligence (AI) in magnetic resonance imaging (MRI), large-scale multi-center MRI datasets have a great influence on diagnosis accuracy and model performance. However, multi-center images are highly variable due to the variety of scanners or scanning parameters in use, which has a negative effect on the generality of AI-based diagnosis models. To address this problem, we propose a self-supervised harmonization (SSH) method. Approach. Mapping the style of images between centers allows harmonization without traveling phantoms to be formalized as an unpaired image-to-image translation problem between two domains. The mapping is a two-stage transform, consisting of a modified cycle generative adversarial network (cycleGAN) for style transfer and a histogram matching module for structure fidelity. The proposed algorithm is demonstrated using female pelvic MRI images from two 3 T systems and compared with three state-of-the-art methods and one conventional method. In the absence of traveling phantoms, we evaluate harmonization from three perspectives: image fidelity, ability to remove inter-center differences, and influence on the downstream model. Main results. The improved image sharpness and structure fidelity are observed using the proposed harmonization pipeline. It largely decreases the number of features with a significant difference between two systems (from 64 to 45, lower than dualGAN: 57, cycleGAN: 59, ComBat: 64, and CLAHE: 54). In the downstream cervical cancer classification, it yields an area under the receiver operating characteristic curve of 0.894 (higher than dualGAN: 0.828, cycleGAN: 0.812, ComBat: 0.685, and CLAHE: 0.770). Significance. Our SSH method yields superior generality of downstream cervical cancer classification models by significantly decreasing the difference in radiomics features, and it achieves greater image fidelity.

A comprehensive analysis on single image-based deraining using generative adversarial network

Deraining is a process by which we can get a transparent image by removing raindrops from a rainy image. In the rainy time visibility of any scene decreases as vision property is affected by the rain. Recently generative adversarial network (GAN) is getting popular in the visual enhancement of hazy, dusty, and noisy images. It is essential to know the effectiveness of the diverse GAN algorithms in the natural rainy situations of different intensities. From this perspective, the present paper describes a comprehensive study on four single-image state-of-the-art GAN models, such as attentive GAN, cGAN, DHSGAN, and Cycle GAN for deraining. The experiment is done using the standard dataset consisting of real-world rainy images and the results are evaluated both objectively and subjectively. We have found somehow mixed results based on quantitative metrics and comparatively satisfactory results by the cGAN based on visual analysis.

Scope of generative adversarial networks (GANs) in image processing

Generative Adversarial Network is the topic of interest in today’s research in the field of image processing and computer vision. A basic GAN model was introduced by Ian Goodfellow et al. in 2014. After that advancement in the field of research in GAN models has been application specific. In computer vision and image to image translation GANs are playing very effective role either in the case of face detection and recognition or in image resolution enhancement and image augmentation. This paper represents a concise overview of various GAN models along with their features and applications. Pix2Pix and conditional GAN models work upon paired datasets while other models like cycle GAN, discover GAN, dual GAN, info GAN, deep convolutional GAN etc. work upon unpaired datasets. Various image datasets which are commonly used for training of generator and discriminator networks are also discussed in this paper. Since partial mode collapse is a common problem to occur during training process for all models, therefore various normalization techniques are also preferred during the training of generator and discriminator networks. As the advancements in GAN models are increasing at a very fast rate, soon these models will also be preferred in commercial applications.

An unsupervised heterogeneous change detection method based on image translation network and post-processing algorithm

ABSTRACT The change detection (CD) of heterogeneous remote sensing images is an important but challenging task. The difficulty is to obtain the change information by directly comparing the different statistical characteristics of the images acquired by different sensors. This paper proposes an unsupervised method for heterogeneous image CD based on an image domain transfer network. First, an attention mechanism is added to the Cycle-generative adversarial networks (Cycle-GANs) to obtain a more consistent feature expression by transferring bi-temporal heterogeneous images to the common domain. The Euclidean distance of the corresponding pixels is calculated in the common domain to form a difference map, and a threshold algorithm is applied to get a rough change map. Finally, the proposed adaptive Discrete Cosine Transform (DCT) algorithm reduces the noise introduced by false detection, and the final change map is obtained. The proposed method is verified on three real heterogeneous CD datasets and compared with the current state-of-the-art methods. The results show that the proposed method is accurate and robust for performing heterogeneous CD tasks.

Research on the Style of Art Works based on Deep Learning

In view of the unsatisfactory effect and major limitations of the style transfer of art works, this paper takes Chinese ink painting for the research subject. The obvious texture characteristics of Chinese ink painting are selected as the input of the Cycle Generative Adversarial Network (CycleGAN) model builder, and the relativistic evaluator is employed to improve the model loss function and the adversarial loss function. An improved art style transfer method of the CycleGAN model is proposed. The experiment shows that the improved CycleGAN model is efficient and feasible for style transfer. Compared with the traditional CycleGAN model, the proposed model performs better in GAN train and GAN test, with a higher average pass rate, which is an increase of nearly 10%. At the same time, with the increase of the number of iterations, the training time of the improved model is close to that of the original model, but the image of the improved model training is larger, which shows that it has more advantages.

Noise Interference Reduction in Vision Module of Intelligent Plant Cultivation Robot Using Better Cycle GAN

The vision recognition module is one of the very important modules in the normal operation of the intelligent plant cultivation robot. In the original vision module, by improving the original YOLOV3(The algorithm proposed in YOLOv3: An Incremental Improvement), a more excellent YOLOV3 was obtained to recognize plants under normal conditions. However, the improved YOLOV3 does not perform well in the visual recognition process when there is a lot of noise interference. To make the visual recognition module continue to work properly, in this paper, the better Cycle GAN (generative adversarial networks) model is proposed to deal with noise based on three common noises, and finally, through a lot of experiments, it is proved that the improvement has some significance for the vision recognition module, which extends the use of the vision recognition module and makes the intelligent plant cultivation robot avoid the interference of noise.

A Measurement Image Translation-Automatic Target Recognition Technique Based on CycleGAN with SAR Simulation DB

The proposed approach achieves the reliable accuracy of synthetic aperture radar-automatic target recognition (SAR-ATR) with a simulation database. The simulation images of targets-of-interest are generated from inverse SAR using high-frequency techniques. A measurement image translation-automatic target recognition (MIT-ATR) uses two deep learning networks. The unique feature of the MIT-ATR is that the measurement images are translated to the simulation-like images by cycle generative adversarial network (CycleGAN). CycleGAN does not need to have a dataset of paired images between the measurement and simulation images. The generated simulation-like images are used as the inputs of the Visual Geometry Group (VGG) network. The VGG network is trained on a simulation database with a softmax layer of multi-classes. Five classes, including a T-72 tank, are considered in the numerical experiments. The images of each class are simulated at all azimuth angles, but the elevation angles range from 6° to 30°. The accuracy of the proposed approach is 63% better than that of the traditional method with only the VGG network. The simulation database could definitely supplement the lack of measurement data. The accuracy of MIT-ATR is properly handled by CycleGAN and the VGG network.

Damage Detection for Conveyor Belt Surface Based on Conditional Cycle Generative Adversarial Network.

The belt conveyor is an essential piece of equipment in coal mining for coal transportation, and its stable operation is key to efficient production. Belt surface of the conveyor is vulnerable to foreign bodies which can be extremely destructive. In the past decades, much research and numerous approaches to inspect belt status have been proposed, and machine learning-based non-destructive testing (NDT) methods are becoming more and more popular. Deep learning (DL), as a branch of machine learning (ML), has been widely applied in data mining, natural language processing, pattern recognition, image processing, etc. Generative adversarial networks (GAN) are one of the deep learning methods based on generative models and have been proved to be of great potential. In this paper, a novel multi-classification conditional CycleGAN (MCC-CycleGAN) method is proposed to generate and discriminate surface images of damages of conveyor belt. A novel architecture of improved CycleGAN is designed to enhance the classification performance using a limited capacity images dataset. Experimental results show that the proposed deep learning network can generate realistic belt surface images with defects and efficiently classify different damaged images of the conveyor belt surface.

CycleGAN and SRGAN to Enrich the Dataset

When developments in the field of computer science are growing rapidly. For example, the development of image or video predictions for various fields has been widely applied to assist further processes. The field of computer vision has created many ideas about processing using deep learning algorithms. Sometimes the problem with using deep learning or machine learning is in the availability of the dataset or the unavailability of the dataset. Various methods are used to add to or enrich the dataset. One way is to add an image dataset by creating a synthetic image. One of the well-known algorithms is Generative Adversarial Networks as an algorithm for generating synthetic images. Currently, there are many variations of the GAN to around 500 variants. This research is to utilize the Cycle GAN architecture in order to enrich the dataset. By doing GAN as a synthetic image generator. This is very important in procuring image datasets, for training and testing models of Deep Learning algorithms such as Convolutional Neural Networks. In addition, the use of synthetic images produces a deep learning model to avoid overfitting. One of the causes of the overfitting problem is the lack of datasets. There are many ways to add image datasets, by cropping, continuously rotating 90 degrees, 180 degrees. The reason for using Cycle Generative Adversarial Networks is because this method is not as complicated as other GANs, but also not as simple. Cycle GAN synthetic images are processed with Super Resolution GAN, which aims to clarify image quality. So that it produces a different image and good image quality.

A cycle generative adversarial network for improving the quality of four-dimensional cone-beam computed tomography images

BackgroundFour-dimensional cone-beam computed tomography (4D-CBCT) can visualize moving tumors, thus adaptive radiation therapy (ART) could be improved if 4D-CBCT were used. However, 4D-CBCT images suffer from severe imaging artifacts. The aim of this study is to investigate the use of synthetic 4D-CBCT (sCT) images created by a cycle generative adversarial network (CycleGAN) for ART for lung cancer.MethodsUnpaired thoracic 4D-CBCT images and four-dimensional multislice computed tomography (4D-MSCT) images of 20 lung-cancer patients were used for training. High-quality sCT lung images generated by the CycleGAN model were tested on another 10 cases. The mean and mean absolute errors were calculated to assess changes in the computed tomography number. The structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) were used to compare the sCT and original 4D-CBCT images. Moreover, a volumetric modulation arc therapy plan with a dose of 48 Gy in four fractions was recalculated using the sCT images and compared with ideal dose distributions observed in 4D-MSCT images.ResultsThe generated sCT images had fewer artifacts, and lung tumor regions were clearly observed in the sCT images. The mean and mean absolute errors were near 0 Hounsfield units in all organ regions. The SSIM and PSNR results were significantly improved in the sCT images by approximately 51% and 18%, respectively. Moreover, the results of gamma analysis were significantly improved; the pass rate reached over 90% in the doses recalculated using the sCT images. Moreover, each organ dose index of the sCT images agreed well with those of the 4D-MSCT images and were within approximately 5%.ConclusionsThe proposed CycleGAN enhances the quality of 4D-CBCT images, making them comparable to 4D-MSCT images. Thus, clinical implementation of sCT-based ART for lung cancer is feasible.

A Real-Time Four-Dimensional Reconstruction Algorithm of Cine-Magnetic Resonance Imaging (Cine-MRI) Using Deep Learning.

PurposeThe purpose of this study is to propose algorithms and methods for achieving high accuracy in tracking and interception irradiation technology for tumors that move by respiration using MR-linac (MRIdian®, ViewRay Inc.) and to use deep learning to predict the movement of moving tumors in real time during radiation therapy and reconstruct cine magnetic resonance imaging (cine-MRI) into four-dimensional (4D) movies.MethodsIn this study, we propose a reconstruction algorithm using 4DCT for treatment planning taken before irradiation as training data in consideration of the actual treatment flow. In the algorithm, two neural networks made before treatment are used to reconstruct 4D movies that predict tumor movement in real time during treatment. Cycle GAN (generative adversarial network) was used to convert MR images to CT images, and long short-term memory was used to convert cine-MRI to 4D movies and predict tumor movement.ResultsWe succeeded in predicting the time including the imaging time of the MR images, the lag until irradiation, and the calculation time in the algorithm. In addition, the conversion and prediction results at each phase of reconstruction were generally good so that they could be clinically applied.ConclusionsThe reconstruction algorithm proposed in this study enables high-precision radiotherapy while predicting the volume information of the tumor and the actual tumor position, which could not be obtained during radiotherapy.

Synthetic CT generation from cone-beam CT using deep-learning for breast adaptive radiotherapy

We investigated the feasibility of the generation of synthetic CT (sCT) from CBCT images with deep learning and the dose evaluation for CBCT-guided breast cancer adaptive radiotherapy.A total of sixty-eight patients receiving radiotherapy after breast-conserving surgery were retrospectively included in this study. We compared the performance of three deep-learning methods in generating sCT from CBCT, including U-Net, Cycle generative adversarial network (CycleGAN) and pix2pix. The original treatment plan was transferred to sCT keeping the same parameters. The dosimetric evaluation was performed by a quick dose recalculation on sCT based on gamma analysis.The U-Net model obtained the lowest mean absolute error (MAE) within the body, clinical target volume (CTV) and organs at risk (OARs), with 62.53 ± 9.14 HU within the body, 35.99 ± 6.32 HU within tumor bed, and 30.15 ± 6.36 HU within CTV. In terms of dose comparison, the gamma pass rates under 3%/3 mm and 2%/2 mm criteria were 91.40 ± 3.52% and 85.95 ± 4.75% for the U-Net model, whereas 89.50 ± 3.46% and 83.65 ± 4.00% for the pix2pix model, and 89.84 ± 3.47% and 83.69 ± 4.28% for the CycleGAN model, respectively.The sCT images generated by the U-Net model can provide higher image similarity and dosimetric accuracy than those generated by the pix2pix and CycleGAN models. The approach could be used to realize accurate dose calculation for breast cancer adaptive radiotherapy based on CBCT.

Brain Tumor Detection and Classification Using Cycle Generative Adversarial Networks.

Brain cancer ranks tenth on the list of leading causes of death in both men and women. Biopsy is one of the most used methods for diagnosing cancer. However, the biopsy process is quite dangerous and take a long time to reach a decision. Furthermore, as the tumor size is rising quickly, non-invasive, automatic diagnostic equipment is required which can automatically detect the tumor and its stage precisely in a few seconds. In recent years, techniques based on Machine Learning and Deep Learning (DL) for detecting and classifying cancers has gained remarkable success in recent years. This paper suggested an ensemble method for detecting and classifying brain tumor and its stages using brain Magnetic Resonance Imaging (MRI). A modified InceptionResNetV2 pre-trained model is used for tumor detection from MRI image. After tumor detection, a combination of InceptionResNetV2 and Random Forest Tree (RFT) is used to determine the cancer stage, which includes glioma, meningioma, and pituitary cancer. The size of the dataset is small, so C-GAN (Cyclic Generative Adversarial Networks) is used to increase the dataset size. The experiment results demonstrate that the suggested tumor detection and tumor classification models achieve the accuracy of 99% and 98%, respectively.

Speech emotion recognition using data augmentation method by cycle-generative adversarial networks

One of the obstacles in developing speech emotion recognition (SER) systems is the data scarcity problem, i.e., the lack of labeled data for training these systems. Data augmentation is an effective method for increasing the amount of training data. In this paper, we propose a cycle-generative adversarial network (cycle-GAN) for data augmentation in the SER systems. For each of the five emotions considered, an adversarial network is designed to generate data that have a similar distribution to the main data in that class but have a different distribution to those of other classes. These networks are trained in an adversarial way to produce feature vectors similar to those in the training set, which are then added to the original training sets. Instead of using the common cross-entropy loss to train cycle-GANs, we use the Wasserstein divergence to mitigate the gradient vanishing problem and to generate high-quality samples. The proposed network has been applied to SER using the EMO-DB dataset. The quality of the generated data is evaluated using two classifiers based on support vector machine and deep neural network. The results showed that the recognition accuracy in unweighted average recall was about 83.33%, which is better than the baseline methods compared.

CT2US: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data

Accurate segmentation of kidney in ultrasound images is a vital procedure in clinical diagnosis and interventional operation. In recent years, deep learning technology has demonstrated promising prospects in medical image analysis. However, due to the inherent problems of ultrasound images, data with annotations are scarce and arduous to acquire, hampering the application of data-hungry deep learning methods. In this paper, we propose cross-modal transfer learning from computerized tomography (CT) to ultrasound (US) by leveraging annotated data in the CT modality. In particular, we adopt cycle generative adversarial network (CycleGAN) to synthesize US images from CT data and construct a transition dataset to mitigate the immense domain discrepancy between US and CT. Mainstream convolutional neural networks such as U-Net, U-Res, PSPNet, and DeepLab v3+ are pretrained on the transition dataset and then transferred to real US images. We first trained CNN models on a data set composed of 50 ultrasound images and validated them on a validation set composed of 30 ultrasound images. In addition, we selected 82 ultrasound images from another hospital to construct a cross-site data set to verify the generalization performance of the models. The experimental results show that with our proposed transfer learning strategy, the segmentation accuracy in dice similarity coefficient (DSC) reaches 0.853 for U-Net, 0.850 for U-Res, 0.826 for PSPNet and 0.827 for DeepLab v3+ on the cross-site test set. Compared with training from scratch, the accuracy improvement was 0.127, 0.097, 0.105 and 0.036 respectively. Our transfer learning strategy effectively improves the accuracy and generalization ability of ultrasound image segmentation model with limited training data.

Civil Infrastructure Damage and Corrosion Detection: An Application of Machine Learning

Automatic detection of corrosion and associated damages to civil infrastructures such as bridges, buildings, and roads, from aerial images captured by an Unmanned Aerial Vehicle (UAV), helps one to overcome the challenges and shortcomings (objectivity and reliability) associated with the manual inspection methods. Deep learning methods have been widely reported in the literature for civil infrastructure corrosion detection. Among them, convolutional neural networks (CNNs) display promising applicability for the automatic detection of image features less affected by image noises. Therefore, in the current study, we propose a modified version of deep hierarchical CNN architecture, based on 16 convolution layers and cycle generative adversarial network (CycleGAN), to predict pixel-wise segmentation in an end-to-end manner using the images of Bolte Bridge and sky rail areas in Victoria (Melbourne). The convolutedly designed model network proposed in the study is based on learning and aggregation of multi-scale and multilevel features while moving from the low convolutional layers to the high-level layers, thus reducing the consistency loss in images due to the inclusion of CycleGAN. The standard approaches only use the last convolutional layer, but our proposed architecture differs from these approaches and uses multiple layers. Moreover, we have used guided filtering and Conditional Random Fields (CRFs) methods to refine the prediction results. Additionally, the effectiveness of the proposed architecture was assessed using benchmarking data of 600 images of civil infrastructure. Overall, the results show that the deep hierarchical CNN architecture based on 16 convolution layers produced advanced performances when evaluated for different methods, including the baseline, PSPNet, DeepLab, and SegNet. Overall, the extended method displayed the Global Accuracy (GA); Class Average Accuracy (CAC); mean Intersection Of the Union (IOU); Precision (P); Recall (R); and F-score values of 0.989, 0.931, 0.878, 0.849, 0.818 and 0.833, respectively.

Label2label: training a neural network to selectively restore cellular structures in fluorescence microscopy.

ABSTRACTImmunofluorescence microscopy is routinely used to visualise the spatial distribution of proteins that dictates their cellular function. However, unspecific antibody binding often results in high cytosolic background signals, decreasing the image contrast of a target structure. Recently, convolutional neural networks (CNNs) were successfully employed for image restoration in immunofluorescence microscopy, but current methods cannot correct for those background signals. We report a new method that trains a CNN to reduce unspecific signals in immunofluorescence images; we name this method label2label (L2L). In L2L, a CNN is trained with image pairs of two non-identical labels that target the same cellular structure. We show that after L2L training a network predicts images with significantly increased contrast of a target structure, which is further improved after implementing a multiscale structural similarity loss function. Here, our results suggest that sample differences in the training data decrease hallucination effects that are observed with other methods. We further assess the performance of a cycle generative adversarial network, and show that a CNN can be trained to separate structures in superposed immunofluorescence images of two targets.