Conditional GAN with an Attention-Based Generator and a 3D Discriminator for 3D Medical Image Generation

Abstract

Conditional Generative Adversarial Networks (cGANs) are a set of methods able to synthesize images that match a given condition. However, existing models designed for natural images are impractical to generate high-quality 3D medical images due to enormous computation. To address this issue, most cGAN models used in the medical field process either 2D slices or small 3D crops and join them together in subsequent steps to reconstruct the full-size 3D image. However, these approaches often cause spatial inconsistencies in adjacent slices or crops, and the changes specified by the target condition may not consider the 3D image as a whole. To address these problems, we propose a novel cGAN that can synthesize high-quality 3D MR images at different stages of the Alzheimer’s disease (AD). First, our method generates a sequence of 2D slices using an attention-based 2D generator with a disease condition for efficient transformations depending on brain regions. Then, consistency in 3D space is enforced by the use of a set of 2D and 3D discriminators. Moreover, we propose an adaptive identity loss based on the attention scores to properly transform features relevant to the target condition. Our experiments show that the proposed method can generate smooth and realistic 3D images at different stages of AD, and the image change with respect to the condition is better than the images generated by existing GAN-based methods.