3D Medical Image Synthesis by Factorised Representation and Deformable Model Learning

Abstract

In this paper we propose a model for controllable synthesis of 3D (volumetric) medical image data. The model is comprised of three components which are learnt simultaneously from unlabelled data through self-supervision: (i) a multi-tissue anatomical model, (ii) a probability distribution over deformations of this anatomical model, and, (iii) a probability distribution over ‘renderings’ of the anatomical model (where a rendering defines the relationship between anatomy and resulting pixel intensities). After training, synthetic data can be generated by sampling the deformation and rendering distributions. To encourage meaningful correspondence in the learnt anatomical model the renderer is kept simple during training, however once trained the (deformed) anatomical model provides dense multi-class segmentation masks for all training volumes, which can be used directly for state-of-the-art conditional image synthesis. This factored model based approach to data synthesis has a number of advantages: Firstly, it allows for coherent synthesis of realistic 3D data, as it is only necessary to learn low dimensional generative models (over deformations and renderings) rather than over the high dimensional 3D images themselves. Secondly, as a by-product of the anatomical model we implicitly learn a dense correspondence between all training volumes, which can be used for registration, or one-shot segmentation (through label transfer). Lastly, the factored representation allows for modality transfer (rendering one image in the modality of another), and meaningful interpolation between volumes. We demonstrate the proposed approach on cardiac MR, and multi-modal abdominal MR/CT datasets.