Fusion of Brain PET and MRI Images Using Tissue-Aware Conditional Generative Adversarial Network With Joint Loss

Jiayin Kang,Wu Lu,Wenjuan Zhang

doi:10.1109/access.2019.2963741

Jiayin Kang, Wu Lu + Show 1 more

Open Access

https://doi.org/10.1109/access.2019.2963741

Copy DOI

Journal: IEEE Access	Publication Date: Jan 1, 2020
Citations: 17	License type: CC BY 4.0

Affiliation: Shanghai Ocean University

Abstract

Positron emission tomography (PET) has rich pseudo color information that reflects the functional characteristics of tissue, but lacks structural information and its spatial resolution is low. Magnetic resonance imaging (MRI) has high spatial resolution as well as strong structural information of soft tissue, but lacks color information that shows the functional characteristics of tissue. For the purpose of integrating the color information of PET with the anatomical structures of MRI to help doctors diagnose diseases better, a method for fusing brain PET and MRI images using tissue-aware conditional generative adversarial network (TA-cGAN) is proposed. Specifically, the process of fusing brain PET and MRI images is treated as an adversarial machine between retaining the color information of PET and preserving the anatomical information of MRI. More specifically, the fusion of PET and MRI images can be regarded as a min-max optimization problem with respect to the generator and the discriminator, where the generator attempts to minimize the objective function via generating a fused image mainly contains the color information of PET, whereas the discriminator tries to maximize the objective function through urging the fused image to include more structural information of MRI. Both the generator and the discriminator in TA-cGAN are conditioned on the tissue label map generated from MRI image, and are trained alternatively with joint loss. Extensive experiments demonstrate that the proposed method enhances the anatomical details of the fused image while effectively preserving the color information from the PET. In addition, compared with other state-of-the-art methods, the proposed method achieves better fusion effects both in subjectively visual perception and in objectively quantitative assessment.

Highlights

Positron emission tomography (PET), a nuclear medicine imaging technology, provides a color image with functional information that reflects the metabolism of different tissues
For the purpose of validating the performance of our proposed method, the following five state-of-the-art methods are used to compare with our method: the IHS combined with retina-inspired models (IHS-Retina) method [5], the non-subsampled shearlet transform (NSST) method [10], the low-rank sparse dictionaries learning (LSDL) method [11], the nonparametric density model (NDM) method [14], and the convolutional neural networks (CNNs) method [22]
We adopt the following four commonly used metrics to evaluate the performances of different methods: the entropy (EN) [37], the average gradient (AG) [38], the spectral discrepancy (SD) [5], and the QAB/F [39]

Summary

Introduction

Positron emission tomography (PET), a nuclear medicine imaging technology, provides a color image with functional information that reflects the metabolism of different tissues. PET image has a low spatial resolution and lacks. The associate editor coordinating the review of this manuscript and approving it for publication was Bohui Wang. Structural information of tissues [1]. Magnetic resonance imaging (MRI), another non-invasive imaging tool, presents strong soft tissue structure information with higher spatial resolution. MRI image lacks color information that reflects the metabolic function of specific tissues [2], [3]. Effectively integrating PET with MRI via image fusion can provide more meaningfully complementary information.

Objectives

Methods

Results

Conclusion