Cycle-consistent Generative Adversarial Network Research Articles

Comparative Study of Traditional and Deep-Learning Denoising Approaches for Image-Based Petrophysical Characterization of Porous Media

Digital rock physics has seen significant advances owing to improvements in micro-computed tomography (MCT) imaging techniques and computing power. These advances allow for the visualization and accurate characterization of multiphase transport in porous media. Despite such advancements, image processing and particularly the task of denoising MCT images remains less explored. As such, selection of proper denoising method is a challenging optimization exercise of balancing the tradeoffs between minimizing noise and preserving original features. Despite its importance, there are no comparative studies in the geoscience domain that assess the performance of different denoising approaches, and their effect on image-based rock and fluid property estimates. Further, the application of machine learning and deep learning-based (DL) denoising models remains under-explored. In this research, we evaluate the performance of six commonly used denoising filters and compare them to five DL-based denoising protocols, namely, noise-to-clean (N2C), residual dense network (RDN), and cycle consistent generative adversarial network (CCGAN)—which require a clean reference (ground truth), as well as noise-to-noise (N2N) and noise-to-void (N2V)—which do not require a clean reference. We also propose hybrid or semi-supervised DL denoising models which only require a fraction of clean reference images. Using these models, we investigate the optimal number of high-exposure reference images that balances data acquisition cost and accurate petrophysical characterization. The performance of each denoising approach is evaluated using two sets of metrics: (1) standard denoising evaluation metrics, including peak signal-to-noise ratio (PSNR) and contrast-to-noise ratio (CNR), and (2) the resulting image-based petrophysical properties such as porosity, saturation, pore size distribution, phase connectivity, and specific surface area (SSA). Petrophysical estimates show that most traditional filters perform well when estimating bulk properties but show large errors for pore-scale properties like phase connectivity. Meanwhile, DL-based models give mixed outcomes, where supervised methods like N2C show the best performance, and an unsupervised model like N2V shows the worst performance. N2N75, which is a newly proposed semi-supervised variation of the N2N model, where 75% of the clean reference data is used for training, shows very promising outcomes for both traditional denoising performance metrics and petrophysical properties including both bulk and pore-scale measures. Lastly, N2C is found to be the most computationally efficient, while CCGAN is found to be the least, among the DL-based models considered in this study. Overall, this investigation shows that application of sophisticated supervised and semi-supervised DL-based denoising models can significantly reduce petrophysical characterization errors introduced during the denoising step. Furthermore, with the advancement of semi-supervised DL-based models, requirement of clean reference or ground truth images for training can be reduced and deployment of fast X-ray scanning can be made possible.

A two-stage deep generative adversarial quality enhancement network for real-world 3D CT images

High-quality (HQ) three-dimensional (3D) images are the premise of analyzing the properties of porous media such as rocks. X-ray computed tomography (CT) is one of the most widely used imaging tools to capture the 3D images of rock samples. Nevertheless, the quality (e.g., resolution, sharpness, and the signal-to-noise ratio) of the collected rock CT images may not meet the needs of practical applications in some cases due to the limitations of imaging systems, leading to inaccurate results of property analysis. In this paper, aiming at improving the quality of rock CT images as well as the accuracy of property analysis, we develop a two-stage deep generative adversarial quality enhancement network for real-world 3D CT images, namely the CTQENet. More specifically, the proposed CTQENet consists of a two-dimensional (2D) reconstruction module (2DRM) and a 3D fusion module (3DFM), which enhance the quality of 3D CT images from the perspective of 2D slices and 3D volumes, respectively. In order to remove artifacts and enhance the resolution of real-world CT images, the 2DRM takes the cycle-consistent generative adversarial network as the backbone to learn the mapping from low-quality (LQ) 2D slices to HQ ones without one-to-one paired training data. Then, the 3D CT volumes stacked by the reconstructed HQ slices along the x/y/z-axis are adaptively fused in the generative adversarial network-based 3DFM, to achieve more reliable 3D morphological structures. Qualitative and quantitative comparisons show the effectiveness of the proposed CTQENet for real-world 3D CT images of rock samples. In particular, the reconstructed HQ 3D CT images by CTQENet show similar morphological characteristics and statistical properties with HQ targets. This study makes it possible to obtain higher quality 3D CT images that partly exceed the limitations of CT imaging systems for better visual experience and more accurate property analysis.

Self-Supervised Pansharpening Based on a Cycle-Consistent Generative Adversarial Network

In the field of remote sensing image pansharpening, deep learning-based methods have shown impressive performances recently. However, most deep learning-based pansharpening methods are based on supervised learning, which requires a large number of training images. In addition, obtaining large amounts of images with a high spatial and spectral resolution for training may be difficult in practice. In this letter, a novel self-supervised learning method based on a cycle-consistent generative adversarial network (CycleGAN) is proposed for remote sensing image pansharpening, without requiring large volumes of data for training. The framework contains two generators and two discriminators, and applies a residual neural network to the first generator. The panchromatic (PAN) image and multispectral (MS) image are input into the first generator to obtain the fused image, and then the fused image is input into the second generator to obtain a PAN image, which should be consistent with the input PAN image. The experimental results show that the proposed method performs better than the state-of-the-art unsupervised pansharpening method, and also achieves a competitive performance when compared with a supervised method.

Selection of Augmented Data for Overcoming the Imbalance Problem in Facies Classification

Facies classification refers to the classification of rock types and pore fluids using information obtained from well log data and core samples. A range of elastic properties provide the main input for classification models. The elastic properties are closely related to water saturation, porosity, and shale volume. In addition, if impedance inversion is performed, the same elastic properties can be obtained from the surface seismic area, thus linking well log and surface seismic data. Machine learning (ML)-based facies classification has the advantage of minimizing the subjectivity associated with human interpretations and maximizing the time efficiency. However, due to the insufficiency of well log data, class imbalance and absolute data shortages can easily arise. Therefore, in this study, we used a cycle-consistent generative adversarial network (CycleGAN) to augment the synthetic data simulating well log data. In addition, we determined which classes of data required augmentation when using CycleGAN and proposed criteria for selecting the augmented data to be used for class-balanced training. The developed algorithm was verified using the Vincent oil field data. The classification results were improved, and more physically valid predictions were achieved in the surface seismic survey area. The data augmentation scheme developed in this study will be useful for facies classification in environments where well log data are very limited.

An Instance and Feature-Based Hybrid Transfer Model for Fault Diagnosis of Rotating Machinery With Different Speeds

Recently, transfer learning approaches using pre-trained models to learn new tasks enable many real-world applications. However, when detecting rotating machinery faults under different conditions like rotating speeds, classical instance-based transfer (IBT) algorithms represented by generative adversarial networks (GANs) require the source and target domains to have similar feature spaces. Meanwhile, the other feature-based transfer (FBT) algorithms often face feature extraction difficulty and domain over-adaptation. To overcome these problems, a novel hybrid transfer model that combines both ideas of IBT and FBT algorithms is proposed in this paper. This hybrid model firstly introduces a cycle-consistent generative adversarial network (CycleGAN) to generate labeled pseudo samples. By adding an operating-condition discriminator and a fault-consistency discriminator to the CycleGAN, generated samples are easier to transfer operating conditions and retain the domain-invariant fault information. Secondly, generated and target samples are sent to a deep domain confusion (DDC) based network to adapt feature domain spaces and further predict the labels of target samples. Compared with the traditional IBT or FBT model, the proposed hybrid transfer model can combine both advantages of these two competitive models, broadening the transferable domain and improving the task accuracy. In addition, a pseudo-label based transition strategy for labeling intermediate domain samples is also proposed to help the transfer model perform better when unlabeled intermediate domain samples are available. Lab experiments verify that the proposed hybrid model and transition strategy can significantly improve the fault diagnosis accuracy of rotating machinery with different speeds.

Through-the-Wall Micro-Doppler De-Wiring Technique via Cycle-Consistent Adversarial Network

The radar penetrating technique has aroused a keen interest in the research community, due to its superior abilities for through-the-wall indoor human motion monitoring. Micro-Doppler signatures in this situation play a significant role in recognition and classification for human activities. However, the live wire buried in the wall introduces additive clutters to the spectrograms. Such degraded spectrograms drastically affect the performance of behind-the-wall human activity detection. In this paper, an ultra-wideband (UWB) radar system is utilized in the through-the-wall scenario to get the feature enhanced micro-Doppler signature called range-max time-frequency representation (R-max TFR). Then, a recently introduced Cycle-Consistent Generative Adversarial Network (Cycle GAN) is employed to realize the end-to-end de-wiring task. Cycle GAN can learn the mapping between spectrograms with and without the live wire effect. To minimize the wiring clutters, a loss function called identity loss is introduced in this work. Finally, the proposed de-wiring approach is evaluated through classification. The results show that the proposed Cycle GAN architecture outperforms other state-of-art de-wiring methods.

Feasibility of Synthetic Computed Tomography Images Generated from Magnetic Resonance Imaging Scans Using Various Deep Learning Methods in the Planning of Radiation Therapy for Prostate Cancer.

Simple SummaryMRI-only simulation in radiation therapy (RT) planning has received attention because the CT scan can be omitted. For MRI-only simulation, synthetic CT (sCT) is necessary for the dose calculation. Various methodologies have been suggested for the generation of sCT and, recently, methods using the deep learning approaches are actively investigated. GAN and cycle-consistent GAN (CycGAN) have been mainly tested, however, very limited studies compared the qualities of sCTs generated from these methods or suggested other models for sCT generation. We have compared GAN, CycGAN, and, reference-guided GAN (RgGAN), a new model of deep learning method. We found that the performance in the HU conservation for soft tissue was poorest for GAN. All methods could generate sCTs feasible for VMAT planning with the trend that sCT generated from the RgGAN showed best performance in dosimetric conservation D98% and D95% than sCTs from other methodologies.We aimed to evaluate and compare the qualities of synthetic computed tomography (sCT) generated by various deep-learning methods in volumetric modulated arc therapy (VMAT) planning for prostate cancer. Simulation computed tomography (CT) and T2-weighted simulation magnetic resonance image from 113 patients were used in the sCT generation by three deep-learning approaches: generative adversarial network (GAN), cycle-consistent GAN (CycGAN), and reference-guided CycGAN (RgGAN), a new model which performed further adjustment of sCTs generated by CycGAN with available paired images. VMAT plans on the original simulation CT images were recalculated on the sCTs and the dosimetric differences were evaluated. For soft tissue, a significant difference in the mean Hounsfield unites (HUs) was observed between the original CT images and only sCTs from GAN (p = 0.03). The mean relative dose differences for planning target volumes or organs at risk were within 2% among the sCTs from the three deep-learning approaches. The differences in dosimetric parameters for D98% and D95% from original CT were lowest in sCT from RgGAN. In conclusion, HU conservation for soft tissue was poorest for GAN. There was the trend that sCT generated from the RgGAN showed best performance in dosimetric conservation D98% and D95% than sCTs from other methodologies.

Connected-UNets: a deep learning architecture for breast mass segmentation

Breast cancer analysis implies that radiologists inspect mammograms to detect suspicious breast lesions and identify mass tumors. Artificial intelligence techniques offer automatic systems for breast mass segmentation to assist radiologists in their diagnosis. With the rapid development of deep learning and its application to medical imaging challenges, UNet and its variations is one of the state-of-the-art models for medical image segmentation that showed promising performance on mammography. In this paper, we propose an architecture, called Connected-UNets, which connects two UNets using additional modified skip connections. We integrate Atrous Spatial Pyramid Pooling (ASPP) in the two standard UNets to emphasize the contextual information within the encoder–decoder network architecture. We also apply the proposed architecture on the Attention UNet (AUNet) and the Residual UNet (ResUNet). We evaluated the proposed architectures on two publically available datasets, the Curated Breast Imaging Subset of Digital Database for Screening Mammography (CBIS-DDSM) and INbreast, and additionally on a private dataset. Experiments were also conducted using additional synthetic data using the cycle-consistent Generative Adversarial Network (CycleGAN) model between two unpaired datasets to augment and enhance the images. Qualitative and quantitative results show that the proposed architecture can achieve better automatic mass segmentation with a high Dice score of 89.52%, 95.28%, and 95.88% and Intersection over Union (IoU) score of 80.02%, 91.03%, and 92.27%, respectively, on CBIS-DDSM, INbreast, and the private dataset.

Defect attention template generation cycleGAN for weakly supervised surface defect segmentation

Surface defect segmentation is very important for the quality inspection of industrial production and is an important pattern recognition problem. Although deep learning (DL) has achieved remarkable results in surface defect segmentation, most of these results have been obtained by using massive images with pixel-level annotations, which are difficult to obtain at industrial sites. This paper proposes a weakly supervised defect segmentation method based on the dynamic templates generated by an improved cycle-consistent generative adversarial network (CycleGAN) trained by image-level annotations. To generate better templates for defects with weak signals, we propose a defect attention module by applying the defect residual for the discriminator to strengthen the elimination of defect regions and suppress changes in the background. A defect cycle-consistent loss is designed by adding structural similarity (SSIM) to the original L1 loss to include the grayscale and structural features; the proposed loss can better model the inner structure of defects. After obtaining the defect-free template, a defect segmentation map can easily be obtained through a simple image comparison and threshold segmentation. Experiments show that the proposed method is both efficient and effective, significantly outperforms other weakly supervised methods, and achieves performance that is comparable or even superior to that of supervised methods on three industrial datasets (intersection over union (IoU) on the DAGM 2007, KSD and CCSD datasets of 78.28%, 59.43%,and 68.83%, respectively). The proposed method can also be employed as a semiautomatic annotation tool combined with active learning.

Synthetic CT Generation From Multi-Sequence MR Images for Head and Neck MRI-Only Radiotherapy via Cycle-Consistent Generative Adversarial Network

Synthetic CT (sCT) generation with MR image plays an essential role in MR-only radiotherapy, which can be used for electron density estimation and dose calculation in treatment planning. Currently, the reliability of deep learning-based methods for sCT generation depends on two aspects: the MR-CT registration errors, which can eliminate by a cycle-consistent generative adversarial network (CycleGAN) with unpaired training data, and the different anatomy information and contract divergence of various tissue in different sequences MRI which will significantly affect the nonlinear mapping of MR-CT. The purpose of this study is to investigate the effect of different sequence MRI by evaluating the image quality of corresponded sCTs.Multi-sequence MR images (T1, T2, T1C) and plan CT images of 151 patients with nasopharyngeal carcinoma were acquired on the same day in the radiation treatment position. 126 cases were randomly selected as the training set with the same preprocessing. Three unsupervised CycleGAN-based models were respectively trained with different sequence MRI with Architecture, which consists of 2 ResUnet-based generators and 2 Patch-GAN-based discriminators. The remaining 25 CT-MR cases with non-rigid registration were selected as the independent paired testing set. The mean error (ME) and mean absolute error (MAE) between plan CT and sCT of each model were used to measure the CT value estimation accuracy, which was calculated within air (Hounsfield Unit, HU < 100), soft tissue (-100 < HU < 150), bone (HU > 150) and whole body, respectively. Structural similarity index (SSIM) was applied to evaluate the similarity between error (plan CT and sCT.The MEs calculated between plan CT and T1, T2, T1C-based sCT images, were (-0.62 ± 3.06) vs. (-1.81 ± 3.13) vs. (0.13 ± 2.56) within air, (154.92 ± 87.93) vs. (173.27 ± 97.62) vs. (262.78 ± 91.74) within soft tissue, (1363.99 ± 294.94) vs. (337.21 ± 50.4) vs. (661.63 ± 168.43) within bone, (22.12 ± 29.47) vs. (17.05 ± 10.51) vs. (35.28 ± 20.75) within the whole body. The MAEs correspondingly were (4.49 ± 3.13) vs. (4.6 ± 3.47) vs. (4.6 ± 3.47) within air, (186.41 ± 61.75) vs. (197.33 ± 58.62) vs. (293.56 ± 72.72) within soft tissue, (186.41 ± 61.75) vs. (197.33 ± 58.62) vs. (293.56 ± 72.72) within bone, (31.28 ± 28.1) vs. (27.64 ± 10.41) vs. (42.6 ± 23.82) within the whole body. The quantitative analysis between T1, T2, T1C-sCTs indicates that the T1C based generated model achieves higher accuracy than the other models in nasopharyngeal carcinoma.The T1C MRI can provide sufficient anatomy information and tissue contract in synthetic CT prediction of head and neck region, which offers a substantial potential for density estimation and dose calculation in radiotherapy. The generation method and evaluation will investigate synthetic CT prediction with more complex tissue in various body parts.

Generating Synthesized Computed Tomography (CT) from Magnetic Resonance Imaging Using Cycle-Consistent Generative Adversarial Network for Brain Tumor Radiation Therapy

<h3>Purpose/Objective(s)</h3> Brain tumor is the most common malignant tumor of the head and neck in China, postoperative radiotherapy is one of the main methods to improve the survival rate of patients. Magnetic resonance imaging (MRI) has the advantage of its soft tissue contrast, for the sake of getting the tumor boundary accurately, it is necessary to combine CT/MR image fusing in gross tumor volume (GTV) delineation. In order to avoid the uncertainty caused by multimodal images registration and reduce unnecessary radiation dose, replacing CT with MRI has become one of the research hotspots in the field of radiotherapy. The aim of this study was to use unregistered MRI and CT images from brain tumor patients to generate pseudo-CT images based on a cycle-consistent generative adversarial network (CycleGAN) framework. <h3>Materials/Methods</h3> T1, T2-weighted MRI and CT-simulation images of the whole brain were collected from 31 brain tumor patients. In this work, we have used a CycleGAN framework to generate pseudo-CT images from MRI, this model is capable of image-to-image translation using unpaired MRI and CT images in an unsupervised learning method. Due to the influence of head frame in CT-simulation images and the different scanning range, imaging resolution and contrast of each patient, preprocessing of MRI and CT images, such as clipping, background removal, resampling and normalization, was required firstly. Secondly, in order to compensate for the insufficient to separate all major tissue types by the single MR sequence, each patient's T1/T2-weighted MR image pair was served as the input of the CycleGAN framework after registering and fusing. Finally, a simple cross-validation study, randomly selecting 70% samples as the training set and the remaining images as the testing set, was performed to compare the quality of synthetic CT and real CT image. <h3>Results</h3> The CycleGAN method produced the overall average MAE below 0.24 ± 0.02 and the 0.79 ± 0.03 SSIM value for the testing set images, the heterogeneity between the synthetic CT image and the actual CT image was acceptable. <h3>Conclusion</h3> We successfully realized the image-to-image translation using unregistered MRI and CT images based on the CycleGAN framework. The evaluation of pseudo-CT image showed the feasibility and accuracy of this method, which can effectively reduce the error caused by multi-mode image registration in gross tumor volume (GTV) delineation for brain tumor patients.

ESA‐CycleGAN: Edge feature and self‐attention based cycle‐consistent generative adversarial network for style transfer

Nowadays, style transfer is used in a wide range of commercial applications, such as image beautification, film rendering etc. However, many existing methods of style transfer suffer from loss of details and poor overall visual effect. To address these problems, an edge feature and self-attention based cycle-consistent generative adversarial network (ESA-CycleGAN) is proposed. The model architecture consists of a generator, a discriminator, and an edge feature extraction network. Both the generator and the discriminator contain a self-attention module to capture global features of the image. The edge feature extraction network extracts the edge of the original image and feeds it into the network together with the original image, thereby allowing better processing of details. Besides, a perceptual loss term is added to optimize the network, resulting in better perceptual results. ESA-CycleGAN is applied on four datasets, respectively. The experimental results show that the authors’ computed final IS and FID values have good results compared to the results of several other existing models, indicating the superiority of the model in style transfer, which can better preserve the details of the original images with better image quality.

A Smart Surveillance System for Uncooperative Gait Recognition Using Cycle Consistent Generative Adversarial Networks (CCGANs).

Surveillance remains an important research area, and it has many applications. Smart surveillance requires a high level of accuracy even when persons are uncooperative. Gait Recognition is the study of recognizing people by the way they walk even when they are unwilling to cooperate. It is another form of a behavioral biometric system in which unique attributes of an individual's gait are analyzed to determine their identity. On the other hand, one of the big limitations of the gait recognition system is uncooperative environments in which both gallery and probe sets are made under different and unknown walking conditions. In order to tackle this problem, we propose a deep learning-based method that is trained on individuals with the normal walking condition, and to deal with an uncooperative environment and recognize the individual with any dynamic walking conditions, a cycle consistent generative adversarial network is used. This method translates a GEI disturbed from different covariate factors to a normal GEI. It works like unsupervised learning, and during its training, a GEI disrupts from different covariate factors of each individual and acts as a source domain while the normal walking conditions of individuals are our target domain to which translation is required. The cycle consistent GANs automatically find an individual pair with the help of the Cycle Loss function and generate the required GEI, which is tested by the CNN model to predict the person ID. The proposed system is evaluated over a publicly available data set named CASIA-B, and it achieved excellent results. Moreover, this system can be implemented in sensitive areas, like banks, seminar halls (events), airports, embassies, shopping malls, police stations, military areas, and other public service areas for security purposes.

Synthetic CT-aided multiorgan segmentation for CBCT-guided adaptive pancreatic radiotherapy.

The delineation of organs at risk (OARs) is fundamental to cone-beam CT (CBCT)-based adaptive radiotherapy treatment planning, but is time consuming, labor intensive, and subject to interoperator variability. We investigated a deep learning-based rapid multiorgan delineation method for use in CBCT-guided adaptive pancreatic radiotherapy. To improve the accuracy of OAR delineation, two innovative solutions have been proposed in this study. First, instead of directly segmenting organs on CBCT images, a pretrained cycle-consistent generative adversarial network (cycleGAN) was applied to generating synthetic CT images given CBCT images. Second, an advanced deep learning model called mask-scoring regional convolutional neural network (MS R-CNN) was applied on those synthetic CT to detect the positions and shapes of multiple organs simultaneously for final segmentation. The OAR contours delineated by the proposed method were validated and compared with expert-drawn contours for geometric agreement using the Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), mean surface distance (MSD), and residual mean square distance (RMS). Across eight abdominal OARs including duodenum, large bowel, small bowel, left and right kidneys, liver, spinal cord, and stomach, the geometric comparisons between automated and expert contours are as follows: 0.92 (0.89-0.97) mean DSC, 2.90mm (1.63-4.19mm) mean HD95, 0.89mm (0.61-1.36mm) mean MSD, and 1.43mm (0.90-2.10mm) mean RMS. Compared to the competing methods, our proposed method had significant improvements (p<0.05) in all the metrics for all the eight organs. Once the model was trained, the contours of eight OARs can be obtained on the order of seconds. We demonstrated the feasibility of a synthetic CT-aided deep learning framework for automated delineation of multiple OARs on CBCT. The proposed method could be implemented in the setting of pancreatic adaptive radiotherapy to rapidly contour OARs with high accuracy.

On the application of generative adversarial networks for nonlinear modal analysis

Linear modal analysis is a useful and effective tool for the design and analysis of structures. However, a comprehensive basis for nonlinear modal analysis remains to be developed. In the current work, a machine learning scheme is proposed with a view to performing nonlinear modal analysis. The scheme is focussed on defining a one-to-one mapping from a latent ‘modal’ space to the natural coordinate space, whilst also imposing orthogonality of the mode shapes. The mapping is achieved via the use of the recently-developed cycle-consistent generative adversarial network (cycle-GAN) and an assembly of neural networks targeted on maintaining the desired orthogonality. The method is tested on simulated data from structures with cubic nonlinearities and different numbers of degrees of freedom, and also on data from an experimental three-degree-of-freedom set-up with a column-bumper nonlinearity. The results reveal the method’s efficiency in separating the ‘modes’. The method also provides a nonlinear superposition function, which in most cases has very good accuracy.

Remaining useful life prediction with insufficient degradation data based on deep learning approach

Remaining useful life (RUL) prediction plays a crucial role in decision-making in conditionbased maintenance for preventing catastrophic field failure. For degradation-failed products, the data of performance deterioration process are the key for lifetime estimation. Deep learning has been proved to have excellent performance in RUL prediction given that the degradation data are sufficiently large. However, in some applications, the degradation data are insufficient, under which how to improve the prediction accuracy is yet a challenging problem. To tackle such a challenge, we propose a novel deep learning-based RUL prediction framework by amplifying the degradation dataset. Specifically, we leverage the cycle-consistent generative adversarial network to generate the synthetic data, based on which the original degradation dataset is amplified so that the data characteristics hidden in the sample space could be captured. Moreover, the sliding time window strategy and deep bidirectional long short-term memory network are employed to complete the RUL prediction framework. We show the effectiveness of the proposed method by running it on the turbine engine data set from the National Aeronautics and Space Administration. The comparative experiments show that our method outperforms a case without the use of the synthetically generated data.

An unsupervised domain adaptation approach for change detection and its application to deforestation mapping in tropical biomes

Changes in environmental conditions, geographical variability and different sensor properties typically make it almost impossible to employ previously trained classifiers for new data without a significant drop in classification accuracy. Domain adaptation (DA) techniques been proven useful to alleviate that problem. In particular, appearance adaptation techniques may be used to adapt images from a specific dataset in such a way that the generated images have a style that is similar to the images from another dataset. Such techniques are, however, prone to creating artifacts that hinder proper classification of the adapted images. In this work we propose an unsupervised DA approach for change detection tasks, which is based on a particular appearance adaptation method: the Cycle-Consistent Generative Adversarial Network (CycleGAN). Specifically, we extend that method by introducing additional constraints in the training phase of the model components, which make it preserve the semantic structure and class transitions in the adapted images. We evaluate the proposed approach on a deforestation detection application, considering different sites in the Amazon rain-forest and in the Brazilian Cerrado (savanna) using Landsat-8 images. In the experiments, each site corresponds to a domain, and the accuracy of a classifier trained with images and references from one (source) domain is measured in the classification of another (target) domain. The results show that the proposed approach is successful in producing artifact-free adapted images, which can be satisfactory classified by the pre-trained source classifiers. On average, the accuracies achieved in the classification of the adapted images outperformed the baselines (when no adaptation was made) by 7.1% in terms of mean average precision, and 9.1% in terms of F1-Score. To the best of our knowledge, the proposed method is the first unsupervised domain adaptation approach devised for change detection.

Enhancing robustness of zero resource children's speech recognition system through bispectrum based front-end acoustic features

Automatic recognition of children's speech for low resource languages is extremely challenging. This is due to the fact that speech data from child speakers for system training is unavailable in the case of such languages i.e., zero-resourced in terms of children's speech. Consequently, we are forced to decode children's speech on systems trained using a limited amount of adults' data. However, the acoustic mismatch between adults' and children's speech, such as pitch and speaking-rate differences, leads to highly degraded recognition rates. At the same time, since ASR systems embedded in smart devices are being used anywhere, effect of surrounding noise further degrades the recognition rates. Motivated by this fact, a noise robust front-end acoustic feature extraction approach exploiting bispectrum analysis is proposed in this paper. The proposed features are noise robust due to the inherent immunity of bispectrum towards additive noise. An added advantage of bispectrum analysis is reduced pitch sensitivity. This, in turn, helps reduce the aforementioned pitch-induced acoustic mismatch. Further to that, in order to deal with the unavailability of speech data from child speakers, we have explored the role of voice conversion through cycle-consistent generative adversarial network (CGAN) to modify the acoustic attributes of adults' data. Voice conversion via CGAN renders the adults' speech perceptually similar to children's speech as noted during the listening test. On augmenting the voice converted adults' speech into training, the ill-effects of the acoustic mismatch between adults' and children's speech is reduced to a large extent. Consequently, the recognition performance is significantly improved. The experimental evaluations presented in this paper demonstrate that the use of proposed features as well as CGAN-based voice conversion are highly suited for zero-resource children's speech recognition task under noisy conditions.

Automated delineation of head and neck organs at risk using synthetic MRI-aided mask scoring regional convolutional neural network.

Auto-segmentation algorithms offer a potential solution to eliminate the labor-intensive, time-consuming, and observer-dependent manual delineation of organs-at-risk (OARs) in radiotherapy treatment planning. This study aimed to develop a deep learning-based automated OAR delineation method to tackle the current challenges remaining in achieving reliable expert performance with the state-of-the-art auto-delineation algorithms. The accuracy of OAR delineation is expected to be improved by utilizing the complementary contrasts provided by computed tomography (CT) (bony-structure contrast) and magnetic resonance imaging (MRI) (soft-tissue contrast). Given CT images, synthetic MR images were firstly generated by a pre-trained cycle-consistent generative adversarial network. The features of CT and synthetic MRI were then extracted and combined for the final delineation of organs using mask scoring regional convolutional neural network. Both in-house and public datasets containing CT scans from head-and-neck (HN) cancer patients were adopted to quantitatively evaluate the performance of the proposed method against current state-of-the-art algorithms in metrics including Dice similarity coefficient (DSC), 95th percentile Hausdorff distance (HD95), mean surface distance (MSD), and residual mean square distance (RMS). Across all of 18 OARs in our in-house dataset, the proposed method achieved an average DSC, HD95, MSD, and RMS of 0.77 (0.58-0.90), 2.90mm (1.32-7.63mm), 0.89mm (0.42-1.85mm), and 1.44mm (0.71-3.15mm), respectively, outperforming the current state-of-the-art algorithms by 6%, 16%, 25%, and 36%, respectively. On public datasets, for all nine OARs, an average DSC of 0.86 (0.73-0.97) were achieved, 6% better than the competing methods. We demonstrated the feasibility of a synthetic MRI-aided deep learning framework for automated delineation of OARs in HN radiotherapy treatment planning. The proposed method could be adopted into routine HN cancer radiotherapy treatment planning to rapidly contour OARs with high accuracy.

Normalization of HE-stained histological images using cycle consistent generative adversarial networks

BackgroundHistological images show strong variance (e.g. illumination, color, staining quality) due to differences in image acquisition, tissue processing, staining, etc. This can impede downstream image analysis such as staining intensity evaluation or classification. Methods to reduce these variances are called image normalization techniques.MethodsIn this paper, we investigate the potential of CycleGAN (cycle consistent Generative Adversarial Network) for color normalization in hematoxylin-eosin stained histological images using daily clinical data with consideration of the variability of internal staining protocol variations. The network consists of a generator network GB that learns to map an image X from a source domain A to a target domain B, i.e. GB:XA→XB. In addition, a discriminator network DB is trained to distinguish whether an image from domain B is real or generated. The same process is applied to another generator-discriminator pair (GA,DA), for the inverse mapping GA:XB→XA. Cycle consistency ensures that a generated image is close to its original when being mapped backwards (GA(GB(XA))≈XA and vice versa). We validate the CycleGAN approach on a breast cancer challenge and a follicular thyroid carcinoma data set for various stain variations. We evaluate the quality of the generated images compared to the original images using similarity measures. In addition, we apply stain normalization on pathological lymph node data from our institute and test the gain from normalization on a ResNet classifier pre-trained on the Camelyon16 data set.ResultsQualitative results of the images generated by our network are compared to original color distributions. Our evaluation indicates that by mapping images to a target domain, the similarity training images from that domain improves up to 96%. We also achieve a high cycle consistency for the generator networks by obtaining similarity indices greater than 0.9. When applying the CycleGAN normalization to HE-stain images from our institute the kappa-value of the ResNet-model that is only trained on Camelyon16 data is increased more than 50%.ConclusionsCycleGANs have proven to efficiently normalize HE-stained images. The approach compensates for deviations resulting from image acquisition (e.g. different scanning devices) as well as from tissue staining (e.g. different staining protocols), and thus overcomes the staining variations in images from various institutions.The code is publicly available at https://github.com/m4ln/stainTransfer_CycleGAN_pytorch. The data set supporting the solutions is available at https://doi.org/10.11588/data/8LKEZF.