CycleGAN Model Research Articles

CycleStyleGAN-Based Knowledge Transfer for a Machining Digital Twin.

Digitalisation of manufacturing is a crucial component of the Industry 4.0 transformation. The digital twin is an important tool for enabling real-time digital access to precise information about physical systems and for supporting process optimisation via the translation of the associated big data into actionable insights. Although a variety of frameworks and conceptual models addressing the requirements and advantages of digital twins has been suggested in the academic literature, their implementation has received less attention. The work presented in this paper aims to make a proposition that considers the novel challenges introduced for data analysis in the presence of heterogeneous and dynamic cyber-physical systems in Industry 4.0. The proposed approach defines a digital twin simulation tool that captures the dynamics of a machining vibration signal from a source model and adapts them to a given target environment. This constitutes a flexible approach to knowledge extraction from the existing manufacturing simulation models, as information from both physics-based and data-driven solutions can be elicited this way. Therefore, an opportunity to reuse the costly established systems is made available to the manufacturing businesses, and the paper presents a process optimisation framework for such use case. The proposed approach is implemented as a domain adaptation algorithm based on the generative adversarial network model. The novel CycleStyleGAN architecture extends the CycleGAN model with a style-based signal encoding. The implemented model is validated in an experimental scenario that aims to replicate a real-world manufacturing knowledge transfer problem. The experiment shows that the transferred information enables the reduction of the required target domain data by one order of magnitude.

Cycle-Consistent Adversarial Networks for Smoke Detection and Removal in Endoscopic Images.

During endoscopic surgery, smoke removal is important and meaningful for increasing the visual quality of endoscopic images. However, unlike natural image dehaze, it is practical impossible to build a large paired endoscopic image training dataset with/without smoke. Therefore, in this paper, we propose a new approach, called Desmoke-CycleGAN, which combined detection and removal of smoke together, to improve the CycleGAN model for endoscopic image smoke removal. The detector can provide information about smoke locations and densities, which helps the GAN model to be more stable and efficient for smoke removal. Although some imperfections still exist, the experimental results have demonstrated that this method outperforms other state-of-the-art smoke removal approaches with unpaired real endoscopic images.Clinical Relevance- This can help improve the visibility in endoscopic surgery and to get smoke-free endoscopic images with better quality.

MRI Knee Domain Translation for Unsupervised Segmentation By CycleGAN (data from Osteoarthritis initiative (OAI)).

Accurate quantification of bone and cartilage features is the key to efficient management of knee osteoarthritis (OA). Bone and cartilage tissues can be accurately segmented from magnetic resonance imaging (MRI) data using supervised Deep Learning (DL) methods. DL training is commonly conducted using large datasets with expert-labeled annotations. DL models perform better if distributions of testing data (target domains) are close to those of training data (source domains). However, in practice, data distributions of images from different MRI scanners and sequences are different and DL models need to re-trained on each dataset separately. We propose a domain adaptation (DA) framework using the CycleGAN model for MRI translation that would aid in unsupervised MRI data segmentation. We have validated our pipeline on five scans from the Osteoarthritis Initiative (OAI) dataset. Using this pipeline, we translated TSE Fat Suppressed MRI sequences to pseudo-DESS images. An improved MaskRCNN (IMaskRCNN) instance segmentation network trained on DESS was used to segment cartilage and femoral head regions in TSE Fat Suppressed sequences. Segmentations of the I-MaskRCNN correlated well with approximated manual segmentation obtained from nearest DESS slices (DICE = 0.76) without the need for retraining. We anticipate this technique will aid in automatic unsupervised assessment of knee MRI using commonly acquired MRI sequences and save experts' time that would otherwise be required for manual segmentation.Clinical relevance- This technique paves the way to automatically convert one MRI sequence to its equivalent as if acquired by a different protocol or different magnet, facilitating robust, hardware-independent automated analysis. For example, routine clinically acquired knee MRI could be converted to high-resolution high-contrast images suitable for automated detection of cartilage defects.

Cycle-Generative Adversarial Network-Based Image Correction of 4-Dimensional Cone-Beam Computed Tomography for Lung Cancer Adaptive Radiation Therapy

To achieve high-precision stereotactic body radiotherapy, visualization of moving tumor/target estimation is performed using 4-dimensional cone-beam computed tomography (4D-CBCT) before each treatment fraction. Currently, respiration-correlated 4D-CBCT is achieved with subgroups of cone-beam projections into different respiratory phases. However, insufficient projections inside each phase bin cause severe imaging artifacts. Convolutional neural networks are widely used to correct CBCT specific artifacts. Previous research has shown that mean absolute error in Hounsfield units improved up to 93% over the original CBCT image. However, slight voxel-wise misalignment between the training images may lead to blurred synthesis, and hence, due to respiratory motion, it is impossible to perform 4D-CBCT imaging without organ misalignment with the 4D-computed tomography (4D-CT). Cycle-generative adversarial network (cycle-GAN) framework enforces an inverse transformation and achieves high accurate consistency even in mapping nonlinear domains. The aim of this study was to improve 4D-CBCT image quality using cycle-GAN and evaluate its use in adaptive radiation therapy (ART) planning for lung cancer.Unpaired 4D-CT and 4D-CBCT taken from publicly available datasets in 5 patients who underwent lung cancer treatment were used for training using cycle-GAN model. Then, Synthesis of 4D-CBCT (Syn-CBCT) images with improved quality conferred by the cycle-GAN model was tested in another 5 cases in the lung region. To assess pixel value deviation and image similarity, mean absolute errors (MAE) and structural similarity index measure (SSIM) were calculated for Syn-CBCT and paired 4D-CT images and compared with those of original 4D-CBCT images. Moreover, volumetric modulation arc therapy plans with a dose of 48 Gy in 4 fractions were recalculated based on Syn-CBCT images, and 3-dimensional (3D)-gamma (3%/3 mm) analysis was conducted to compare the former with ideal dose distributions observed in 4D-CT images.4D-CBCT imaging quality was improved and lung tumor regions were clearly observed after noise and artifact removal by applying the trained cycle-GAN model. MAEs were improved by 21% and SSIMs by 46% for whole images. Hence, pixel value and structural visualization in Syn-CBCT images could achieve near-ideal quality of 4D-CT images. Large structures were generally consistent with those in paired 4D-CT images, whereas fine structures (pulmonary vessels and fine bones) were not completely duplicated. However, a mean 3D gamma pass rate of > 95% was achieved in all cases.The proposed cycle-GAN method enhances 4D-CBCT image quality, making it comparable to that of 4D-CT images. Thus, clinical implementation of Syn-CBCT-based ART is feasible.

Self-Supervised Learning-Based High-Resolution Ultrasound Imaging for Prostate Brachytherapy

Ultrasound (US) imaging has been widely used in image-guided needle/seed placement in prostate brachytherapy. For 3D US images with large slice thickness, high frequency information in the slice direction is missing and cannot be resolved through interpolation. As an ill-posed problem, most high-resolution generation methods rely on the presence of external/training atlases to learn the transform from low resolution to high resolution images. The anatomical variations among individuals bring unavoidable uncertainties in this atlas-based learning mechanism. In this study, we propose a self-supervised learning method, which does not use any external high-resolution atlas images, yet can resolve high resolution images from the 3D images with a large slice thickness.In 3D US imaging, in-plane spatial resolution is generally much higher than through-plane resolution. Considering the natural isotropy of US imaging on biological tissue, for a certain patient, the mapping from low-resolution to high-resolution images can be established by learning the mapping from down-sampled in-plane images (low-resolution) to original in-plane US images (high-resolution), which enables the generation of high-resolution through-plane images. Two independent cycle consistent generative adversarial networks (CycleGANs) are trained with paired original in-plane US images and two sets of in-plane down-sampled US images. Finally, high-resolution 3D US images are reconstructed by combining the generated 2D images obtained via feeding through-plane images into the two CycleGAN models. A validation study was conducted to quantitatively assess the proposed method using 3D high in-/through-plane resolution breast US images of 50 breast cancer patients. The feasibility of the proposed method for obtaining high-resolution US images for prostate brachytherapy was tested using 3D US images with 2-mm slice thickness from 45 prostate cancer patients.In the breast US images validation test using a three-times spatial resolution enhancement, the proposed method achieved the mean absolute error value of 0.90 (range 0.57-1.35), the peak signal-to-noise ratio value of 37.88 (range 35.84-40.48) dB, and the visual information fidelity value of 0.69 (range 0.68-0.72), which significantly outperforms bicubic interpolation. For the prostate cases, the proposed method outperformed the bicubic interpolation for the spatial resolution enhancement factors of 5 and 10.A novel deep learning-based algorithm for reconstructing high-resolution 3D US images from sparely acquired 2D images was developed and validated. Significant improvement on through-plane resolution has been achieved by using the acquired 2D images without any external atlas images. Its self-supervision capability could accelerate high-resolution US imaging for prostate brachytherapy.

Cast suppression in radiographs by generative adversarial networks.

Injured extremities commonly need to be immobilized by casts to allow proper healing. We propose a method to suppress cast superimpositions in pediatric wrist radiographs based on the cycle generative adversarial network (CycleGAN) model. We retrospectively reviewed unpaired pediatric wrist radiographs (n = 9672) and sampled them into 2 equal groups, with and without cast. The test subset consisted of 718 radiographs with cast. We evaluated different quadratic input sizes (256, 512, and 1024 pixels) for U-Net and ResNet-based CycleGAN architectures in cast suppression, quantitatively and qualitatively. The mean age was 11 ± 3 years in images containing cast (n = 4836), and 11 ± 4 years in castless samples (n = 4836). A total of 5956 X-rays had been done in males and 3716 in females. A U-Net 512 CycleGAN performed best (P ≤ .001). CycleGAN models successfully suppressed casts in pediatric wrist radiographs, allowing the development of a related software tool for radiology image viewers.

3D Isotropic Super-resolution Prostate MRI Using Generative Adversarial Networks and Unpaired Multiplane Slices.

We developed a deep learning-based super-resolution model for prostate MRI. 2D T2-weighted turbo spin echo (T2w-TSE) images are the core anatomical sequences in a multiparametric MRI (mpMRI) protocol. These images have coarse through-plane resolution, are non-isotropic, and have long acquisition times (approximately 10-15min). The model we developed aims to preserve high-frequency details that are normally lost after 3D reconstruction. We propose a novel framework for generating isotropic volumes using generative adversarial networks (GAN) from anisotropic 2D T2w-TSE and single-shot fast spin echo (ssFSE) images. The CycleGAN model used in this study allows the unpaired dataset mapping to reconstruct super-resolution (SR) volumes. Fivefold cross-validation was performed. The improvements from patch-to-volume reconstruction (PVR) to SR are 80.17%, 63.77%, and 186% for perceptual index (PI), RMSE, and SSIM, respectively; the improvements from slice-to-volume reconstruction (SVR) to SR are 72.41%, 17.44%, and 7.5% for PI, RMSE, and SSIM, respectively. Five ssFSE cases were used to test for generalizability; the perceptual quality of SR images surpasses the in-plane ssFSE images by 37.5%, with 3.26% improvement in SSIM and a higher RMSE by 7.92%. SR images were quantitatively assessed with radiologist Likert scores. Our isotropic SR volumes are able to reproduce high-frequency detail, maintaining comparable image quality to in-plane TSE images in all planes without sacrificing perceptual accuracy. The SR reconstruction networks were also successfully applied to the ssFSE images, demonstrating that high-quality isotropic volume achieved from ultra-fast acquisition is feasible.

Depth-based branching level estimation for bronchoscopic navigation

Bronchoscopists rely on navigation systems during bronchoscopy to reduce the risk of getting lost in the complex bronchial tree-like structure and the homogeneous bronchus lumens. We propose a patient-specific branching level estimation method for bronchoscopic navigation because it is vital to identify the branches being examined in the bronchus tree during examination. We estimate the branching level by integrating the changes in the number of bronchial orifices and the camera motions among the frames. We extract the bronchial orifice regions from a depth image, which is generated using a cycle generative adversarial network (CycleGAN) from real bronchoscopic images. We calculate the number of orifice regions using the vertical and horizontal projection profiles of the depth images and obtain the camera-moving direction using the feature point-based camera motion estimation. The changes in the number of bronchial orifices are combined with the camera-moving direction to estimate the branching level. We used three in vivo and one phantom case to train the CycleGAN model and four in vivo cases to validate the proposed method. We manually created the ground truth of the branching level. The experimental results showed that the proposed method can estimate the branching level with an average accuracy of 87.6%. The processing time per frame was about 61 ms. Experimental results show that it is feasible to estimate the branching level using the number of bronchial orifices and camera-motion estimation from real bronchoscopic images.

Building damage detection based on multi-source adversarial domain adaptation

Building damage assessment plays an essential role during post-disaster rescue operations. Given that labeled samples are difficult to timely obtain after a disaster, transfer learning attracts increasing attention. However, different sensors employed cause considerable discrepancies not only between historical and current scenes but also among historical scenes, which could exert an effect on transfer performance. Therefore, a multi-source adversarial domain adaptation (MADA) method is proposed in this paper to fulfill the task of post-disaster building assessment. This method consists of two phases. First, imageries of several historical scenes are transformed into the same style of the current scene through the CycleGAN model with a classifier, ensuring class invariance, to be fused to make an adapted source domain. Second, feature alignment between adapted source and target domains is executed based on adversarial discriminative domain adaptation. The MADA method enhances the transformed image quality, fully utilizes relevant information in historical scenes, solves inter-scene interference problems among historical images, and improves the transfer efficiency from historical to the current disaster scene. Two experiments are conducted with Hurricane Sandy, Irma, and Maria datasets as multi-source and target domains to validate MADA’s effectiveness. Results show that the classification performance is better than other methods.

RECONSTRUCTED TEETH IMAGE FROM BRACES WITH GAN

Recent advances in deep learning models have shown promising potential in object removal, which is the task of replacing undesired objects with appropriate pixel values using known context. Object removal-based deep learning can commonly be solved by modeling it as the Img2Img (image to image) translation or Inpainting. Instead of dealing with a large context, this paper aims at a specific application of object removal, that is, erasing braces trace out of an image having teeth with braces (called braces2teeth problem). We solved the problem by three methods corresponding to different datasets. Firstly, we use the CycleGAN model to deal with the problem that paired training data is not available. In the second case, we try to create pseudo-paired data to train the Pix2Pix model. In the last case, we utilize GraphCut combining generative inpainting model to build a user-interactive tool that can improve the result in case the user is not satisfied with previous results. To our best knowledge, this study is one of the first attempts to take the braces2teeth problem into account by using deep learning techniques and it can be applied in various fields, from health care to entertainment.

Weakly supervised anomaly segmentation in retinal OCT images using an adversarial learning approach.

Lesion detection is a critical component of disease diagnosis, but the manual segmentation of lesions in medical images is time-consuming and experience-demanding. These issues have recently been addressed through deep learning models. However, most of the existing algorithms were developed using supervised training, which requires time-intensive manual labeling and prevents the model from detecting unaware lesions. As such, this study proposes a weakly supervised learning network based on CycleGAN for lesions segmentation in full-width optical coherence tomography (OCT) images. The model was trained to reconstruct underlying normal anatomic structures from abnormal input images, then the lesions can be detected by calculating the difference between the input and output images. A customized network architecture and a multi-scale similarity perceptual reconstruction loss were used to extend the CycleGAN model to transfer between objects exhibiting shape deformations. The proposed technique was validated using an open-source retinal OCT image dataset. Image-level anomaly detection and pixel-level lesion detection results were assessed using area-under-curve (AUC) and the Dice similarity coefficient, producing results of 96.94% and 0.8239, respectively, higher than all comparative methods. The average test time required to generate a single full-width image was 0.039 s, which is shorter than that reported in recent studies. These results indicate that our model can accurately detect and segment retinopathy lesions in real-time, without the need for supervised labeling. And we hope this method will be helpful to accelerate the clinical diagnosis process and reduce the misdiagnosis rate.

Indoor camera pose estimation via style‐transfer 3D models

AbstractMany vision‐based indoor localization methods require tedious and comprehensive pre‐mapping of built environments. This research proposes a mapping‐free approach to estimating indoor camera poses based on a 3D style‐transferred building information model (BIM) and photogrammetry technique. To address the cross‐domain gap between virtual 3D models and real‐life photographs, a CycleGAN model was developed to transform BIM renderings into photorealistic images. A photogrammetry‐based algorithm was developed to estimate camera pose using the visual and spatial information extracted from the style‐transferred BIM. The experiments demonstrated the efficacy of CycleGAN in bridging the cross‐domain gap, which significantly improved performance in terms of image retrieval and feature correspondence detection. With the 3D coordinates retrieved from BIM, the proposed method can achieve near real‐time camera pose estimation with an accuracy of 1.38 m and 10.1° in indoor environments.

Facilitating MR-Guided Adaptive Proton Therapy in Children Using Deep Learning-Based Synthetic CT.

PurposeTo determine whether self-attention cycle-generative adversarial networks (cycle-GANs), a novel deep-learning method, can generate accurate synthetic computed tomography (sCT) to facilitate adaptive proton therapy in children with brain tumors.Materials and MethodsBoth CT and T1-weighted magnetic resonance imaging (MRI) of 125 children (ages 1-20 years) with brain tumors were included in the training dataset. A model introducing a self-attention mechanism into the conventional cycle-GAN was created to enhance tissue interfaces and reduce noise. The test dataset consisted of 7 patients (ages 2-14 years) who underwent adaptive planning because of changes in anatomy discovered on MRI during proton therapy. The MRI during proton therapy-based sCT was compared with replanning CT (ground truth).ResultsThe Hounsfield unit-mean absolute error was significantly reduced with self-attention cycle-GAN, as compared with conventional cycle-GAN (65.3 ± 13.9 versus 88.9 ± 19.3, P < .01). The average 3-dimensional gamma passing rates (2%/2 mm criteria) for the original plan on the anatomy of the day and for the adapted plan were high (97.6% ± 1.2% and 98.9 ± 0.9%, respectively) when using sCT generated by self-attention cycle-GAN. The mean absolute differences in clinical target volume (CTV) receiving 95% of the prescription dose and 80% distal falloff along the beam axis were 1.1% ± 0.8% and 1.1 ± 0.9 mm, respectively. Areas of greatest dose difference were distal to the CTV and corresponded to shifts in distal falloff. Plan adaptation was appropriately triggered in all test patients when using sCT.ConclusionThe novel cycle-GAN model with self-attention outperforms conventional cycle-GAN for children with brain tumors. Encouraging dosimetric results suggest that sCT generation can be used to identify patients who would benefit from adaptive replanning.

Self-supervised learning for accelerated 3D high-resolution ultrasound imaging.

Ultrasound (US) imaging has been widely used in diagnosis, image-guided intervention, and therapy, where high-quality three-dimensional (3D) images are highly desired from sparsely acquired two-dimensional (2D) images. This study aims to develop a deep learning-based algorithm to reconstruct high-resolution (HR) 3D US images only reliant on the acquired sparsely distributed 2D images. We propose a self-supervised learning framework using cycle-consistent generative adversarial network (cycleGAN), where two independent cycleGAN models are trained with paired original US images and two sets of low-resolution (LR) US images, respectively. The two sets of LR US images are obtained through down-sampling the original US images along the two axes, respectively. In US imaging, in-plane spatial resolution is generally much higher than through-plane resolution. By learning the mapping from down-sampled in-plane LR images to original HR US images, cycleGAN can generate through-plane HR images from original sparely distributed 2D images. Finally, HR 3D US images are reconstructed by combining the generated 2D images from the two cycleGAN models. The proposed method was assessed on two different datasets. One is automatic breast ultrasound (ABUS) images from 70 breast cancer patients, the other is collected from 45 prostate cancer patients. By applying a spatial resolution enhancement factor of 3 to the breast cases, our proposed method achieved the mean absolute error (MAE) value of 0.90±0.15, the peak signal-to-noise ratio (PSNR) value of 37.88±0.88dB, and the visual information fidelity (VIF) value of 0.69±0.01, which significantly outperforms bicubic interpolation. Similar performances have been achieved using the enhancement factor of 5 in these breast cases and using the enhancement factors of 5 and 10 in the prostate cases. We have proposed and investigated a new deep learning-based algorithm for reconstructing HR 3D US images from sparely acquired 2D images. Significant improvement on through-plane resolution has been achieved by only using the acquired 2D images without any external atlas images. Its self-supervision capability could accelerate HR US imaging.

Deep Learning–Based CT-to-CBCT Deformable Image Registration for Autosegmentation in Head and Neck Adaptive Radiation Therapy

The purpose of this study is to develop a deep learning–based method that can automatically generate segmentations on conebeam computed tomography (CBCT) for head and neck online adaptive radiation therapy (ART), where expert-drawn contours in planning CT (pCT) images serve as prior knowledge. Because of the many artifacts and truncations that characterize CBCT, we propose to utilize a learning-based deformable image registration method and contour propagation to get updated contours on CBCT. Our method takes CBCT and pCT as inputs, and it outputs a deformation vector field and synthetic CT (sCT) simultaneously by jointly training a CycleGAN model and 5-cascaded Voxelmorph model. The CycleGAN generates the sCT from CBCT, while the 5-cascaded Voxelmorph warps the pCT to the sCT’s anatomy. We compared the segmentation results to Elastix, Voxelmorph and 5-cascaded Voxelmorph models on 18 structures including target and organs-at-risk. Our proposed method achieved an average Dice similarity coefficient of 0.83 ± 0.09 and an average 95% Hausdorff distance of 2.01 ± 1.81 mm. Our method showed better accuracy than Voxelmorph and 5-cascaded Voxelmorph and comparable accuracy to Elastix, but with much higher efficiency. The proposed method can rapidly and simultaneously generate sCT with correct CT numbers and propagate contours from pCT to CBCT for online ART replanning.

Artifact-Free Single Image Defogging

In this paper, we present a novel defogging technique, named CurL-Defog, with the aim of minimizing the insertion of artifacts while maintaining good contrast restoration and visibility enhancement. Many learning-based defogging approaches rely on paired data, where fog is artificially added to clear images; this usually provides good results on mildly fogged images but is not effective for difficult cases. On the other hand, the models trained with real data can produce visually impressive results, but unwanted artifacts are often present. We propose a curriculum learning strategy and an enhanced CycleGAN model to reduce the number of produced artifacts, where both synthetic and real data are used in the training procedure. We also introduce a new metric, called HArD (Hazy Artifact Detector), to numerically quantify the number of artifacts in the defogged images, thus avoiding the tedious and subjective manual inspection of the results. HArD is then combined with other defogging indicators to produce a solid metric that is not deceived by the presence of artifacts. The proposed approach compares favorably with state-of-the-art techniques on both real and synthetic datasets.

Improving generalization in MR-to-CT synthesis in radiotherapy by using an augmented cycle generative adversarial network with unpaired data.

MR-to-CT synthesis is one of the first steps in the establishment of an MRI-only workflow in radiotherapy. Current MR-to-CT synthesis methods in deep learning use unpaired MR and CT training images with a cycle generative adversarial network (CycleGAN) to minimize the effect of misalignment between paired images. However, this approach critically assumes that the underlying interdomain mapping is approximately deterministic and one-to-one. In the current study, we use an Augmented CycleGAN (AugCGAN) model to create a robust model that can be applied to different scanners and sequences using unpaired data. This study included T2-weighted MR and CT pelvic images of 38 patients in treatment position from five different centers. The AugCGAN was trained on 2D transverse slices of 19 patients from three different sites. The network was then used to generate synthetic CT (sCT) images of 19 patients from the two other sites. Mean absolute errors (MAEs) for each patient were evaluated between real and synthetic CT images. Original treatment plans of nine patients were recalculated using sCT images to assess the dose distribution in terms of voxel-wise dose difference, gamma, and dose-volume histogram analysis. The mean MAEs were Hounsfield units ( ) and 65.8HU for the first and second test sites, respectively. The maximum dose difference to the target was with a gamma pass rate using the 3%, 3mm criteria above 99%. The average time required to generate a complete sCT image for a patient on our GPU was 8.5s. This study suggests that our unpaired approach achieves good performance in generalization with respect to sCT image generation.

Speckle Noise Reduction for OCT Images Based on Image Style Transfer and Conditional GAN.

Raw optical coherence tomography (OCT) images typically are of low quality because speckle noise blurs retinal structures, severely compromising visual quality and degrading performances of subsequent image analysis tasks. In our previous study(Ma et al., 2018), we have developed a Conditional Generative Adversarial Network (cGAN) for speckle noise removal in OCT images collected by several commercial OCT scanners, which we collectively refer to as scanner T. In this paper, we improve the cGAN model and apply it to our in-house OCT scanner (scanner B) for speckle noise suppression. The proposed model consists of two steps: 1) We train a Cycle-Consistent GAN (CycleGAN) to learn style transfer between two OCT image datasets collected by different scanners. The purpose of the CycleGAN is to leverage the ground truth dataset created in our previous study. 2) We train a mini-cGAN model based on the PatchGAN mechanism with the ground truth dataset to suppress speckle noise in OCT images. After training, we first apply the CycleGAN model to convert raw images collected by scanner B to match the style of the images from scanner T, and subsequently use the mini-cGAN model to suppress speckle noise in the style transferred images. We evaluate the proposed method on a dataset collected by scanner B. Experimental results show that the improved model outperforms our previous method and other state-of-the-art models in speckle noise removal, retinal structure preservation and contrast enhancement.

Herniated Lumbar Disc Generation and Classification Using Cycle Generative Adversarial Networks on Axial View MRI

A frequent cause of lower back pain presenting with leg pain is a herniated lumbar intervertebral disc. A herniation or a herniated lumbar disc is a change of position of disc material (nucleus pulpous or annulus fibrosis). Usually, the lower back pain goes away within days or weeks. Regular treatment techniques for lower back pain include medication, exercises, relaxation methods and surgery. Back pain and back problems regularly occur in the lumbar region. The spinal canal is made up of vertebrae; each one protects the spinal nerves. Intervertebral discs and facet joints connect the vertebrae above and below. Groups of muscles and ligaments hold the vertebrae and the discs together. Muscles support the spine and the body weight, and they allow us to move. Pressure can result in excessive wear and tear of the other structures. For example, a common problem in the lower back is disc herniation. In this case, pressure on an intervertebral disc makes its center, the nucleus pulposus, protrude backwards and push against the spinal nerves, leading to lower back pain. Detection and classification are the two most important tasks in computer aided diagnosing systems. Detection of a herniated lumbar disc from magnetic resonance imaging (MRI) is a very difficult task for radiologist. The extraction of herniated discs has been achieved by different approaches such as active contours, region growing, watershed techniques, thresholding and deep learning. In this study, to detect intervertebral disc from axial MRIs we develop a method using generative adversarial networks (GANs), especially the CycleGAN model, to automatically generate and detect intervertebral disc and to classify the type of the herniated lumbar disc such as foraminal or median. We propose to explore the importance of axial view MRI to determine the herniation type. Accurately, GANs and other generative networks have created several ways to tackle different problems well known and challenging of medical image analysis, such as segmentation, reconstruction, data simulation, medical image de-noising, and classification. Moreover, their efficiency to synthesize images and data at unprecedented levels of realism also gives hope that the chronic scarcity of labeled data in the medical field can be resolved with the help of these generative models. In our case, having a database that contains several images is a very difficult task. In this paper, we put forward a new approach based on GANs, in order to solve the problem of lumbar intervertebral disc images reduction. This method is based especially on CycleGAN. Consequently, the essential objective of our work is to generate and automatically classify the herniation type as foraminal or median using GANs. Our computer aided diagnosis (CAD) system achieved a 97.2% accuracy on our dataset. This result represents a very high-performance results by providing the state of the art and our work utilizing the GANs technique. Our CAD is very effective and efficient for classifying herniations of lumbar intervertebral discs. Therefore, the contribution of this study appears in: firstly, the use of the CycleGAN model based on convolutional layers to detect and classify the herniation type (median or foraminal) in lumbar intervertebral discs, secondly, the use of axial view MRI in order to classify the type of the herniated intervertebral disc. The main objective of this paper is to help radiologists automatically recognize and classify herniated lumbar discs.

LighterGAN: An Illumination Enhancement Method for Urban UAV Imagery

In unmanned aerial vehicle based urban observation and monitoring, the performance of computer vision algorithms is inevitably limited by the low illumination and light pollution caused degradation, therefore, the application image enhancement is a considerable prerequisite for the performance of subsequent image processing algorithms. Therefore, we proposed a deep learning and generative adversarial network based model for UAV low illumination image enhancement, named LighterGAN. The design of LighterGAN refers to the CycleGAN model with two improvements—attention mechanism and semantic consistency loss—having been proposed to the original structure. Additionally, an unpaired dataset that was captured by urban UAV aerial photography has been used to train this unsupervised learning model. Furthermore, in order to explore the advantages of the improvements, both the performance in the illumination enhancement task and the generalization ability improvement of LighterGAN were proven in the comparative experiments combining subjective and objective evaluations. In the experiments with five cutting edge image enhancement algorithms, in the test set, LighterGAN achieved the best results in both visual perception and PIQE (perception based image quality evaluator, a MATLAB build-in function, the lower the score, the higher the image quality) score of enhanced images, scores were 4.91 and 11.75 respectively, better than EnlightenGAN the state-of-the-art. In the enhancement of low illumination sub-dataset Y (containing 2000 images), LighterGAN also achieved the lowest PIQE score of 12.37, 2.85 points lower than second place. Moreover, compared with the CycleGAN, the improvement of generalization ability was also demonstrated. In the test set generated images, LighterGAN was 6.66 percent higher than CycleGAN in subjective authenticity assessment and 3.84 lower in PIQE score, meanwhile, in the whole dataset generated images, the PIQE score of LighterGAN is 11.67, 4.86 lower than CycleGAN.