CycleGAN Model Research Articles

Self-Supervised Learning-Based High-Resolution Ultrasound Imaging for Prostate Brachytherapy

Ultrasound (US) imaging has been widely used in image-guided needle/seed placement in prostate brachytherapy. For 3D US images with large slice thickness, high frequency information in the slice direction is missing and cannot be resolved through interpolation. As an ill-posed problem, most high-resolution generation methods rely on the presence of external/training atlases to learn the transform from low resolution to high resolution images. The anatomical variations among individuals bring unavoidable uncertainties in this atlas-based learning mechanism. In this study, we propose a self-supervised learning method, which does not use any external high-resolution atlas images, yet can resolve high resolution images from the 3D images with a large slice thickness.In 3D US imaging, in-plane spatial resolution is generally much higher than through-plane resolution. Considering the natural isotropy of US imaging on biological tissue, for a certain patient, the mapping from low-resolution to high-resolution images can be established by learning the mapping from down-sampled in-plane images (low-resolution) to original in-plane US images (high-resolution), which enables the generation of high-resolution through-plane images. Two independent cycle consistent generative adversarial networks (CycleGANs) are trained with paired original in-plane US images and two sets of in-plane down-sampled US images. Finally, high-resolution 3D US images are reconstructed by combining the generated 2D images obtained via feeding through-plane images into the two CycleGAN models. A validation study was conducted to quantitatively assess the proposed method using 3D high in-/through-plane resolution breast US images of 50 breast cancer patients. The feasibility of the proposed method for obtaining high-resolution US images for prostate brachytherapy was tested using 3D US images with 2-mm slice thickness from 45 prostate cancer patients.In the breast US images validation test using a three-times spatial resolution enhancement, the proposed method achieved the mean absolute error value of 0.90 (range 0.57-1.35), the peak signal-to-noise ratio value of 37.88 (range 35.84-40.48) dB, and the visual information fidelity value of 0.69 (range 0.68-0.72), which significantly outperforms bicubic interpolation. For the prostate cases, the proposed method outperformed the bicubic interpolation for the spatial resolution enhancement factors of 5 and 10.A novel deep learning-based algorithm for reconstructing high-resolution 3D US images from sparely acquired 2D images was developed and validated. Significant improvement on through-plane resolution has been achieved by using the acquired 2D images without any external atlas images. Its self-supervision capability could accelerate high-resolution US imaging for prostate brachytherapy.

Cast suppression in radiographs by generative adversarial networks.

Injured extremities commonly need to be immobilized by casts to allow proper healing. We propose a method to suppress cast superimpositions in pediatric wrist radiographs based on the cycle generative adversarial network (CycleGAN) model. We retrospectively reviewed unpaired pediatric wrist radiographs (n = 9672) and sampled them into 2 equal groups, with and without cast. The test subset consisted of 718 radiographs with cast. We evaluated different quadratic input sizes (256, 512, and 1024 pixels) for U-Net and ResNet-based CycleGAN architectures in cast suppression, quantitatively and qualitatively. The mean age was 11 ± 3 years in images containing cast (n = 4836), and 11 ± 4 years in castless samples (n = 4836). A total of 5956 X-rays had been done in males and 3716 in females. A U-Net 512 CycleGAN performed best (P ≤ .001). CycleGAN models successfully suppressed casts in pediatric wrist radiographs, allowing the development of a related software tool for radiology image viewers.

3D Isotropic Super-resolution Prostate MRI Using Generative Adversarial Networks and Unpaired Multiplane Slices.

We developed a deep learning-based super-resolution model for prostate MRI. 2D T2-weighted turbo spin echo (T2w-TSE) images are the core anatomical sequences in a multiparametric MRI (mpMRI) protocol. These images have coarse through-plane resolution, are non-isotropic, and have long acquisition times (approximately 10-15min). The model we developed aims to preserve high-frequency details that are normally lost after 3D reconstruction. We propose a novel framework for generating isotropic volumes using generative adversarial networks (GAN) from anisotropic 2D T2w-TSE and single-shot fast spin echo (ssFSE) images. The CycleGAN model used in this study allows the unpaired dataset mapping to reconstruct super-resolution (SR) volumes. Fivefold cross-validation was performed. The improvements from patch-to-volume reconstruction (PVR) to SR are 80.17%, 63.77%, and 186% for perceptual index (PI), RMSE, and SSIM, respectively; the improvements from slice-to-volume reconstruction (SVR) to SR are 72.41%, 17.44%, and 7.5% for PI, RMSE, and SSIM, respectively. Five ssFSE cases were used to test for generalizability; the perceptual quality of SR images surpasses the in-plane ssFSE images by 37.5%, with 3.26% improvement in SSIM and a higher RMSE by 7.92%. SR images were quantitatively assessed with radiologist Likert scores. Our isotropic SR volumes are able to reproduce high-frequency detail, maintaining comparable image quality to in-plane TSE images in all planes without sacrificing perceptual accuracy. The SR reconstruction networks were also successfully applied to the ssFSE images, demonstrating that high-quality isotropic volume achieved from ultra-fast acquisition is feasible.

Depth-based branching level estimation for bronchoscopic navigation

Bronchoscopists rely on navigation systems during bronchoscopy to reduce the risk of getting lost in the complex bronchial tree-like structure and the homogeneous bronchus lumens. We propose a patient-specific branching level estimation method for bronchoscopic navigation because it is vital to identify the branches being examined in the bronchus tree during examination. We estimate the branching level by integrating the changes in the number of bronchial orifices and the camera motions among the frames. We extract the bronchial orifice regions from a depth image, which is generated using a cycle generative adversarial network (CycleGAN) from real bronchoscopic images. We calculate the number of orifice regions using the vertical and horizontal projection profiles of the depth images and obtain the camera-moving direction using the feature point-based camera motion estimation. The changes in the number of bronchial orifices are combined with the camera-moving direction to estimate the branching level. We used three in vivo and one phantom case to train the CycleGAN model and four in vivo cases to validate the proposed method. We manually created the ground truth of the branching level. The experimental results showed that the proposed method can estimate the branching level with an average accuracy of 87.6%. The processing time per frame was about 61 ms. Experimental results show that it is feasible to estimate the branching level using the number of bronchial orifices and camera-motion estimation from real bronchoscopic images.

Building damage detection based on multi-source adversarial domain adaptation

Building damage assessment plays an essential role during post-disaster rescue operations. Given that labeled samples are difficult to timely obtain after a disaster, transfer learning attracts increasing attention. However, different sensors employed cause considerable discrepancies not only between historical and current scenes but also among historical scenes, which could exert an effect on transfer performance. Therefore, a multi-source adversarial domain adaptation (MADA) method is proposed in this paper to fulfill the task of post-disaster building assessment. This method consists of two phases. First, imageries of several historical scenes are transformed into the same style of the current scene through the CycleGAN model with a classifier, ensuring class invariance, to be fused to make an adapted source domain. Second, feature alignment between adapted source and target domains is executed based on adversarial discriminative domain adaptation. The MADA method enhances the transformed image quality, fully utilizes relevant information in historical scenes, solves inter-scene interference problems among historical images, and improves the transfer efficiency from historical to the current disaster scene. Two experiments are conducted with Hurricane Sandy, Irma, and Maria datasets as multi-source and target domains to validate MADA’s effectiveness. Results show that the classification performance is better than other methods.

RECONSTRUCTED TEETH IMAGE FROM BRACES WITH GAN

Recent advances in deep learning models have shown promising potential in object removal, which is the task of replacing undesired objects with appropriate pixel values using known context. Object removal-based deep learning can commonly be solved by modeling it as the Img2Img (image to image) translation or Inpainting. Instead of dealing with a large context, this paper aims at a specific application of object removal, that is, erasing braces trace out of an image having teeth with braces (called braces2teeth problem). We solved the problem by three methods corresponding to different datasets. Firstly, we use the CycleGAN model to deal with the problem that paired training data is not available. In the second case, we try to create pseudo-paired data to train the Pix2Pix model. In the last case, we utilize GraphCut combining generative inpainting model to build a user-interactive tool that can improve the result in case the user is not satisfied with previous results. To our best knowledge, this study is one of the first attempts to take the braces2teeth problem into account by using deep learning techniques and it can be applied in various fields, from health care to entertainment.

Weakly supervised anomaly segmentation in retinal OCT images using an adversarial learning approach.

Lesion detection is a critical component of disease diagnosis, but the manual segmentation of lesions in medical images is time-consuming and experience-demanding. These issues have recently been addressed through deep learning models. However, most of the existing algorithms were developed using supervised training, which requires time-intensive manual labeling and prevents the model from detecting unaware lesions. As such, this study proposes a weakly supervised learning network based on CycleGAN for lesions segmentation in full-width optical coherence tomography (OCT) images. The model was trained to reconstruct underlying normal anatomic structures from abnormal input images, then the lesions can be detected by calculating the difference between the input and output images. A customized network architecture and a multi-scale similarity perceptual reconstruction loss were used to extend the CycleGAN model to transfer between objects exhibiting shape deformations. The proposed technique was validated using an open-source retinal OCT image dataset. Image-level anomaly detection and pixel-level lesion detection results were assessed using area-under-curve (AUC) and the Dice similarity coefficient, producing results of 96.94% and 0.8239, respectively, higher than all comparative methods. The average test time required to generate a single full-width image was 0.039 s, which is shorter than that reported in recent studies. These results indicate that our model can accurately detect and segment retinopathy lesions in real-time, without the need for supervised labeling. And we hope this method will be helpful to accelerate the clinical diagnosis process and reduce the misdiagnosis rate.

Indoor camera pose estimation via style‐transfer 3D models

AbstractMany vision‐based indoor localization methods require tedious and comprehensive pre‐mapping of built environments. This research proposes a mapping‐free approach to estimating indoor camera poses based on a 3D style‐transferred building information model (BIM) and photogrammetry technique. To address the cross‐domain gap between virtual 3D models and real‐life photographs, a CycleGAN model was developed to transform BIM renderings into photorealistic images. A photogrammetry‐based algorithm was developed to estimate camera pose using the visual and spatial information extracted from the style‐transferred BIM. The experiments demonstrated the efficacy of CycleGAN in bridging the cross‐domain gap, which significantly improved performance in terms of image retrieval and feature correspondence detection. With the 3D coordinates retrieved from BIM, the proposed method can achieve near real‐time camera pose estimation with an accuracy of 1.38 m and 10.1° in indoor environments.

Facilitating MR-Guided Adaptive Proton Therapy in Children Using Deep Learning-Based Synthetic CT.

PurposeTo determine whether self-attention cycle-generative adversarial networks (cycle-GANs), a novel deep-learning method, can generate accurate synthetic computed tomography (sCT) to facilitate adaptive proton therapy in children with brain tumors.Materials and MethodsBoth CT and T1-weighted magnetic resonance imaging (MRI) of 125 children (ages 1-20 years) with brain tumors were included in the training dataset. A model introducing a self-attention mechanism into the conventional cycle-GAN was created to enhance tissue interfaces and reduce noise. The test dataset consisted of 7 patients (ages 2-14 years) who underwent adaptive planning because of changes in anatomy discovered on MRI during proton therapy. The MRI during proton therapy-based sCT was compared with replanning CT (ground truth).ResultsThe Hounsfield unit-mean absolute error was significantly reduced with self-attention cycle-GAN, as compared with conventional cycle-GAN (65.3 ± 13.9 versus 88.9 ± 19.3, P < .01). The average 3-dimensional gamma passing rates (2%/2 mm criteria) for the original plan on the anatomy of the day and for the adapted plan were high (97.6% ± 1.2% and 98.9 ± 0.9%, respectively) when using sCT generated by self-attention cycle-GAN. The mean absolute differences in clinical target volume (CTV) receiving 95% of the prescription dose and 80% distal falloff along the beam axis were 1.1% ± 0.8% and 1.1 ± 0.9 mm, respectively. Areas of greatest dose difference were distal to the CTV and corresponded to shifts in distal falloff. Plan adaptation was appropriately triggered in all test patients when using sCT.ConclusionThe novel cycle-GAN model with self-attention outperforms conventional cycle-GAN for children with brain tumors. Encouraging dosimetric results suggest that sCT generation can be used to identify patients who would benefit from adaptive replanning.

Self-supervised learning for accelerated 3D high-resolution ultrasound imaging.

Ultrasound (US) imaging has been widely used in diagnosis, image-guided intervention, and therapy, where high-quality three-dimensional (3D) images are highly desired from sparsely acquired two-dimensional (2D) images. This study aims to develop a deep learning-based algorithm to reconstruct high-resolution (HR) 3D US images only reliant on the acquired sparsely distributed 2D images. We propose a self-supervised learning framework using cycle-consistent generative adversarial network (cycleGAN), where two independent cycleGAN models are trained with paired original US images and two sets of low-resolution (LR) US images, respectively. The two sets of LR US images are obtained through down-sampling the original US images along the two axes, respectively. In US imaging, in-plane spatial resolution is generally much higher than through-plane resolution. By learning the mapping from down-sampled in-plane LR images to original HR US images, cycleGAN can generate through-plane HR images from original sparely distributed 2D images. Finally, HR 3D US images are reconstructed by combining the generated 2D images from the two cycleGAN models. The proposed method was assessed on two different datasets. One is automatic breast ultrasound (ABUS) images from 70 breast cancer patients, the other is collected from 45 prostate cancer patients. By applying a spatial resolution enhancement factor of 3 to the breast cases, our proposed method achieved the mean absolute error (MAE) value of 0.90±0.15, the peak signal-to-noise ratio (PSNR) value of 37.88±0.88dB, and the visual information fidelity (VIF) value of 0.69±0.01, which significantly outperforms bicubic interpolation. Similar performances have been achieved using the enhancement factor of 5 in these breast cases and using the enhancement factors of 5 and 10 in the prostate cases. We have proposed and investigated a new deep learning-based algorithm for reconstructing HR 3D US images from sparely acquired 2D images. Significant improvement on through-plane resolution has been achieved by only using the acquired 2D images without any external atlas images. Its self-supervision capability could accelerate HR US imaging.

Deep Learning–Based CT-to-CBCT Deformable Image Registration for Autosegmentation in Head and Neck Adaptive Radiation Therapy

The purpose of this study is to develop a deep learning–based method that can automatically generate segmentations on conebeam computed tomography (CBCT) for head and neck online adaptive radiation therapy (ART), where expert-drawn contours in planning CT (pCT) images serve as prior knowledge. Because of the many artifacts and truncations that characterize CBCT, we propose to utilize a learning-based deformable image registration method and contour propagation to get updated contours on CBCT. Our method takes CBCT and pCT as inputs, and it outputs a deformation vector field and synthetic CT (sCT) simultaneously by jointly training a CycleGAN model and 5-cascaded Voxelmorph model. The CycleGAN generates the sCT from CBCT, while the 5-cascaded Voxelmorph warps the pCT to the sCT’s anatomy. We compared the segmentation results to Elastix, Voxelmorph and 5-cascaded Voxelmorph models on 18 structures including target and organs-at-risk. Our proposed method achieved an average Dice similarity coefficient of 0.83 ± 0.09 and an average 95% Hausdorff distance of 2.01 ± 1.81 mm. Our method showed better accuracy than Voxelmorph and 5-cascaded Voxelmorph and comparable accuracy to Elastix, but with much higher efficiency. The proposed method can rapidly and simultaneously generate sCT with correct CT numbers and propagate contours from pCT to CBCT for online ART replanning.

Artifact-Free Single Image Defogging

In this paper, we present a novel defogging technique, named CurL-Defog, with the aim of minimizing the insertion of artifacts while maintaining good contrast restoration and visibility enhancement. Many learning-based defogging approaches rely on paired data, where fog is artificially added to clear images; this usually provides good results on mildly fogged images but is not effective for difficult cases. On the other hand, the models trained with real data can produce visually impressive results, but unwanted artifacts are often present. We propose a curriculum learning strategy and an enhanced CycleGAN model to reduce the number of produced artifacts, where both synthetic and real data are used in the training procedure. We also introduce a new metric, called HArD (Hazy Artifact Detector), to numerically quantify the number of artifacts in the defogged images, thus avoiding the tedious and subjective manual inspection of the results. HArD is then combined with other defogging indicators to produce a solid metric that is not deceived by the presence of artifacts. The proposed approach compares favorably with state-of-the-art techniques on both real and synthetic datasets.

Improving generalization in MR-to-CT synthesis in radiotherapy by using an augmented cycle generative adversarial network with unpaired data.

MR-to-CT synthesis is one of the first steps in the establishment of an MRI-only workflow in radiotherapy. Current MR-to-CT synthesis methods in deep learning use unpaired MR and CT training images with a cycle generative adversarial network (CycleGAN) to minimize the effect of misalignment between paired images. However, this approach critically assumes that the underlying interdomain mapping is approximately deterministic and one-to-one. In the current study, we use an Augmented CycleGAN (AugCGAN) model to create a robust model that can be applied to different scanners and sequences using unpaired data. This study included T2-weighted MR and CT pelvic images of 38 patients in treatment position from five different centers. The AugCGAN was trained on 2D transverse slices of 19 patients from three different sites. The network was then used to generate synthetic CT (sCT) images of 19 patients from the two other sites. Mean absolute errors (MAEs) for each patient were evaluated between real and synthetic CT images. Original treatment plans of nine patients were recalculated using sCT images to assess the dose distribution in terms of voxel-wise dose difference, gamma, and dose-volume histogram analysis. The mean MAEs were Hounsfield units ( ) and 65.8HU for the first and second test sites, respectively. The maximum dose difference to the target was with a gamma pass rate using the 3%, 3mm criteria above 99%. The average time required to generate a complete sCT image for a patient on our GPU was 8.5s. This study suggests that our unpaired approach achieves good performance in generalization with respect to sCT image generation.

Speckle Noise Reduction for OCT Images Based on Image Style Transfer and Conditional GAN.

Raw optical coherence tomography (OCT) images typically are of low quality because speckle noise blurs retinal structures, severely compromising visual quality and degrading performances of subsequent image analysis tasks. In our previous study(Ma et al., 2018), we have developed a Conditional Generative Adversarial Network (cGAN) for speckle noise removal in OCT images collected by several commercial OCT scanners, which we collectively refer to as scanner T. In this paper, we improve the cGAN model and apply it to our in-house OCT scanner (scanner B) for speckle noise suppression. The proposed model consists of two steps: 1) We train a Cycle-Consistent GAN (CycleGAN) to learn style transfer between two OCT image datasets collected by different scanners. The purpose of the CycleGAN is to leverage the ground truth dataset created in our previous study. 2) We train a mini-cGAN model based on the PatchGAN mechanism with the ground truth dataset to suppress speckle noise in OCT images. After training, we first apply the CycleGAN model to convert raw images collected by scanner B to match the style of the images from scanner T, and subsequently use the mini-cGAN model to suppress speckle noise in the style transferred images. We evaluate the proposed method on a dataset collected by scanner B. Experimental results show that the improved model outperforms our previous method and other state-of-the-art models in speckle noise removal, retinal structure preservation and contrast enhancement.

Herniated Lumbar Disc Generation and Classification Using Cycle Generative Adversarial Networks on Axial View MRI

A frequent cause of lower back pain presenting with leg pain is a herniated lumbar intervertebral disc. A herniation or a herniated lumbar disc is a change of position of disc material (nucleus pulpous or annulus fibrosis). Usually, the lower back pain goes away within days or weeks. Regular treatment techniques for lower back pain include medication, exercises, relaxation methods and surgery. Back pain and back problems regularly occur in the lumbar region. The spinal canal is made up of vertebrae; each one protects the spinal nerves. Intervertebral discs and facet joints connect the vertebrae above and below. Groups of muscles and ligaments hold the vertebrae and the discs together. Muscles support the spine and the body weight, and they allow us to move. Pressure can result in excessive wear and tear of the other structures. For example, a common problem in the lower back is disc herniation. In this case, pressure on an intervertebral disc makes its center, the nucleus pulposus, protrude backwards and push against the spinal nerves, leading to lower back pain. Detection and classification are the two most important tasks in computer aided diagnosing systems. Detection of a herniated lumbar disc from magnetic resonance imaging (MRI) is a very difficult task for radiologist. The extraction of herniated discs has been achieved by different approaches such as active contours, region growing, watershed techniques, thresholding and deep learning. In this study, to detect intervertebral disc from axial MRIs we develop a method using generative adversarial networks (GANs), especially the CycleGAN model, to automatically generate and detect intervertebral disc and to classify the type of the herniated lumbar disc such as foraminal or median. We propose to explore the importance of axial view MRI to determine the herniation type. Accurately, GANs and other generative networks have created several ways to tackle different problems well known and challenging of medical image analysis, such as segmentation, reconstruction, data simulation, medical image de-noising, and classification. Moreover, their efficiency to synthesize images and data at unprecedented levels of realism also gives hope that the chronic scarcity of labeled data in the medical field can be resolved with the help of these generative models. In our case, having a database that contains several images is a very difficult task. In this paper, we put forward a new approach based on GANs, in order to solve the problem of lumbar intervertebral disc images reduction. This method is based especially on CycleGAN. Consequently, the essential objective of our work is to generate and automatically classify the herniation type as foraminal or median using GANs. Our computer aided diagnosis (CAD) system achieved a 97.2% accuracy on our dataset. This result represents a very high-performance results by providing the state of the art and our work utilizing the GANs technique. Our CAD is very effective and efficient for classifying herniations of lumbar intervertebral discs. Therefore, the contribution of this study appears in: firstly, the use of the CycleGAN model based on convolutional layers to detect and classify the herniation type (median or foraminal) in lumbar intervertebral discs, secondly, the use of axial view MRI in order to classify the type of the herniated intervertebral disc. The main objective of this paper is to help radiologists automatically recognize and classify herniated lumbar discs.

LighterGAN: An Illumination Enhancement Method for Urban UAV Imagery

In unmanned aerial vehicle based urban observation and monitoring, the performance of computer vision algorithms is inevitably limited by the low illumination and light pollution caused degradation, therefore, the application image enhancement is a considerable prerequisite for the performance of subsequent image processing algorithms. Therefore, we proposed a deep learning and generative adversarial network based model for UAV low illumination image enhancement, named LighterGAN. The design of LighterGAN refers to the CycleGAN model with two improvements—attention mechanism and semantic consistency loss—having been proposed to the original structure. Additionally, an unpaired dataset that was captured by urban UAV aerial photography has been used to train this unsupervised learning model. Furthermore, in order to explore the advantages of the improvements, both the performance in the illumination enhancement task and the generalization ability improvement of LighterGAN were proven in the comparative experiments combining subjective and objective evaluations. In the experiments with five cutting edge image enhancement algorithms, in the test set, LighterGAN achieved the best results in both visual perception and PIQE (perception based image quality evaluator, a MATLAB build-in function, the lower the score, the higher the image quality) score of enhanced images, scores were 4.91 and 11.75 respectively, better than EnlightenGAN the state-of-the-art. In the enhancement of low illumination sub-dataset Y (containing 2000 images), LighterGAN also achieved the lowest PIQE score of 12.37, 2.85 points lower than second place. Moreover, compared with the CycleGAN, the improvement of generalization ability was also demonstrated. In the test set generated images, LighterGAN was 6.66 percent higher than CycleGAN in subjective authenticity assessment and 3.84 lower in PIQE score, meanwhile, in the whole dataset generated images, the PIQE score of LighterGAN is 11.67, 4.86 lower than CycleGAN.

Synthesizing CT images from MR images with deep learning: model generalization for different datasets through transfer learning

Background and purpose. Replacing CT imaging with MR imaging for MR-only radiotherapy has sparked the interest of many scientists and is being increasingly adopted in radiation oncology. Although many studies have focused on generating CT images from MR images, only models on data with the same dataset were tested. Therefore, how well the trained model will work for data from different hospitals and MR protocols is still unknown. In this study, we addressed the model generalization problem for the MR-to-CT conversion task. Materials and methods. Brain T2 MR and corresponding CT images were collected from SZSPH (source domain dataset), brain T1-FLAIR, T1-POST MR, and corresponding CT images were collected from The University of Texas Southwestern (UTSW) (target domain dataset). To investigate the model’s generalizability ability, four potential solutions were proposed: source model, target model, combined model, and adapted model. All models were trained using the CycleGAN network. The source model was trained with a source domain dataset from scratch and tested with a target domain dataset. The target model was trained with a target domain dataset and tested with a target domain dataset. The combined model was trained with both source domain and target domain datasets, and tested with the target domain dataset. The adapted model used a transfer learning strategy to train a CycleGAN model with a source domain dataset and retrain the pre-trained model with a target domain dataset. MAE, RMSE, PSNR, and SSIM were used to quantitatively evaluate model performance on a target domain dataset. Results. The adapted model achieved best quantitative results of 74.56 ± 8.61, 193.18 ± 17.98, 28.30 ± 0.83, and 0.84 ± 0.01 for MAE, RMSE, PSNR, and SSIM using the T1-FLAIR dataset and 74.89 ± 15.64, 195.73 ± 31.29, 27.72 ± 1.43, and 0.83 ± 0.04 for MAE, RMSE, PSNR, and SSIM using the T1-POST dataset. The source model had the poorest performance. Conclusions. This work indicates high generalization ability to generate synthetic CT images from small training datasets of MR images using pre-trained CycleGAN. The quantitative results of the test data, including different scanning protocols and different acquisition centers, indicated the proof of this concept.

Unpaired medical image colorization using generative adversarial network

We consider medical image transformation problems where a grayscale image is transformed into a color image. The colorized medical image should have the same features as the input image because extra synthesized features can increase the possibility of diagnostic errors. In this paper, to secure colorized medical images and improve the quality of synthesized images, as well as to leverage unpaired training image data, a colorization network is proposed based on the cycle generative adversarial network (CycleGAN) model, combining a perceptual loss function and a total variation (TV) loss function. Visual comparisons and experimental indicators from the NRMSE, PSNR, and SSIM metrics are used to evaluate the performance of the proposed method. The experimental results show that GAN-based style conversion can be applied to colorization of medical images. As well, the introduction of perceptual loss and TV loss can improve the quality of images produced as a result of colorization better than the result generated by only using the CycleGAN model.

Auto-Annotated Deep Segmentation for Surface Defect Detection

This article presents a deep learning scheme for automatic defect detection in material surfaces. The success of deep learning model training is generally determined by the number of representative training samples and the quality of the annotation. It is extremely tedious and tiresome to annotate defects pixel-by-pixel in an image to train a semantic network model for defect segmentation. In this study, we propose a two-stage deep learning scheme to tackle the pixel-wise defect detection in textured surfaces without manual annotation. The first stage of the deep learning scheme uses two cycle-consistent adversarial network (CycleGAN) models to automatically synthesize and annotate defect pixels in an image. The synthesized defect images and their corresponding annotated results from the CycleGAN models are then used as the input-output pairs for training the U-Net semantic network. The proposed scheme requires only a few real defect samples for the training and completely requires no manual annotation work. It is practical and computationally very efficient for the implementation in manufacturing. Experimental results show that the proposed deep learning scheme can be applied for defect detection in a variety of textured and patterned surfaces, and results in high detection accuracy.

Remote Sensing Data Augmentation Through Adversarial Training

The lack of remote sensing images and poor quality limit the performance improvement of follow-up research such as remote sensing interpretation. In this paper, a Generative Adversarial Network(GAN) is proposed for data augmentation of remote sensing images abstracted from Jiangxi and Anhui Provinces in China, i.e., Deeply-supervised GAN(D-sGAN). D-sGAN can generate high-quality images that are rich in changes, greatly shorten the generation time, and provide data support for applications such as semantic interpretation of remote sensing images. At First, to modulate the layer activations, a down-sampling scheme is designed based on the segmentation map. Then, the architecture of the generator is Unet++ with the proposed down-sampling module. Next, the generator of this net is deeply supervised by the discriminator using deep Convolutional Neural Network(CNN). This paper further proved that the proposed down-sampling module and the dense connection characteristics of UNet++ are significantly beneficial to the retention of semantic information of remote sensing images. Numerical results demonstrated that the images generated by D-sGAN could be used to improve accuracy of the segmentation network, with the faster generation speed compared to the CoGAN, SimGAN and CycleGAN models. Furthermore, the remote sensing data generated by the model helped the interpretation network to increase the accuracy by 9\%, meeting actual generation requirements.