Multi-modal Perceptual Adversarial Learning for Longitudinal Prediction of Infant MR Images

Abstract

Longitudinal magnetic resonance imaging (MRI) is essential in neuroimaging studies of early brain development. However, incomplete data is an inevitable problem in longitudinal studies because of participant attrition and scan failure. Data imputation is a possible way to address such missing data. Here, we propose a novel 3D multi-modal perceptual adversarial network (MPGAN) to predict a missing MR image from an existing longitudinal image of the same subject. To the best of our knowledge, this is the first application of deep generative methods for longitudinal image prediction of structural MRI in the first year of life, where brain volume and image intensities are changing dramatically. In order to produce sharper and more realistic images, we incorporate the perceptual loss into the adversarial training process. To leverage complementary information contained in the multi-modality data, MPGAN predicts T1w and T2w images jointly in the prediction process. We evaluated MPGAN versus six alternative approaches based on visual as well as quantitative assessment. The results indicate that our MPGAN predicts missing MR images in an accurate and visually realistic fashion, and shows better performance than the alternative methods.