End-to-End 3D Liver CT Image Synthesis from Vasculature Using a Multi-Task Conditional Generative Adversarial Network

Abstract

Acquiring relevant, high-quality, and heterogeneous medical images is essential in various types of automated analysis, used for a variety of downstream data augmentation tasks. However, a large number of real image samples are expensive to obtain, especially for 3D medical images. Therefore, there is an urgent need to synthesize realistic 3D medical images. However, the existing generator models have poor stability and lack the guidance of prior medical knowledge. To this end, we propose a multi-task (i.e., segmentation task and generation task) 3D generative adversarial network (GAN) for the synthesis of 3D liver CT images (3DMT-GAN). To the best of our knowledge, this is the first application for a 3D liver CT image synthesis task. Specifically, we utilize a mask of vascular segmentation as the input because it contains structural information about a variety of rich anatomical structures. We use the semantic mask of the liver as prior medical knowledge to guide the 3D CT image generation, reducing the calculation of a large number of backgrounds, thus making the model more focused on the generation of the region of the liver. In addition, we introduce a stable multiple gradient descent algorithm (MGDA) reconstruction method into our model to balance the weights of the multi-task framework. Experiments were conducted on a real dataset, and the experimental results show that the segmentation task achieves a Dice similarity coefficient (DSC) of 0.87, while the synthesis task outperforms existing state-of-the-art methods. This study demonstrates the feasibility of using vascular images to synthesize images of the liver.