DiCyc: GAN-based deformation invariant cross-domain information fusion for medical image synthesis

Chengjia Wang,Guang Yang,Giorgos Papanastasiou,Sotirios A Tsaftaris,David E Newby,Calum Gray,Gillian Macnaught,Tom J Macgillivray

doi:10.1016/j.inffus.2020.10.015

Chengjia Wang, Guang Yang + Show 6 more

Open Access

https://doi.org/10.1016/j.inffus.2020.10.015

Copy DOI

Abstract

Cycle-consistent generative adversarial network (CycleGAN) has been widely used for cross-domain medical image synthesis tasks particularly due to its ability to deal with unpaired data. However, most CycleGAN-based synthesis methods cannot achieve good alignment between the synthesized images and data from the source domain, even with additional image alignment losses. This is because the CycleGAN generator network can encode the relative deformations and noises associated to different domains. This can be detrimental for the downstream applications that rely on the synthesized images, such as generating pseudo-CT for PET-MR attenuation correction. In this paper, we present a deformation invariant cycle-consistency model that can filter out these domain-specific deformation. The deformation is globally parameterized by thin-plate-spline (TPS), and locally learned by modified deformable convolutional layers. Robustness to domain-specific deformations has been evaluated through experiments on multi-sequence brain MR data and multi-modality abdominal CT and MR data. Experiment results demonstrated that our method can achieve better alignment between the source and target data while maintaining superior image quality of signal compared to several state-of-the-art CycleGAN-based methods.

Highlights

Multi-modal medical imaging, i.e. acquiring images of the same organ or structure using different imaging techniques that are based on different physical phenomena, is increasingly used towards improving clinical decision-making
We introduce a global transformation model and modified layers of the deformable convolutional network (DCN) into the CycleGAN image generator and propose to the use of a novel image alignment loss based on normalized mutual information (NMI)
We introduced the DiCyc cross-domain medical image synthesis model which addresses the issue of and is resilient to domain-specific deformations

Summary

Introduction

Multi-modal medical imaging, i.e. acquiring images of the same organ or structure using different imaging techniques (or modalities) that are based on different physical phenomena, is increasingly used towards improving clinical decision-making. Collecting data from the same patient using different imaging techniques is often impractical, due to, limited access to different imaging devices, additional time needed for multiple scanning sessions, and the associated cost This makes cross-domain medical image synthesis a technology that is gaining popularity. Cross-domain image synthesis has been used to impute incomplete information in standard statistical analysis [1,2], to predict and simulate developments of missing information [3], or to improve intermediate steps of analysis such as registration [4], information fusion [5,6,7], segmentation [8,9,10], atlas construction [11,12] and disease classification [13,14] These methods map between MRI, computed tomography (CT), positron emission tomography (PET) and ultrasound imaging from one domain to another. Using MRI to achieve attenuation correction of PET data can be a disadvantage as, unlike CT, the MR signal is not physically related to Information Fusion 67 (2021) 147–160

Objectives

Methods

Results

Conclusion