Joint Clinical Data and CT Image Based Prognosis: A Case Study on Postoperative Pulmonary Venous Obstruction Prediction

Abstract

Very often doctors diagnose diseases and prescribe treatments through cross-referencing patients’ clinical data as well as radiology reports. On the other hand, while a few existing machine learning frameworks for diagnosis, treatment planning, and prognosis have used both clinical data and medical images, they all have prior knowledge about what information should be extracted from medical images. However, this is not the case for many diseases. For example, cardiac anatomical structure and tissue shapes are essential for pulmonary venous obstruction (PVO) prediction after correction of total anomalous pulmonary venous connection (TAPVC), but the exact graphical features in the computed tomography (CT) images that should be measured remain unclear. In this paper, we propose to use convolutional neural network to automatically obtain features from CT images and combine them with clinical data in an end-to-end trainable manner. We further collect a dataset consisting of 132 TAPVC patients for evaluation, and find that jointly using clinical data and CT images to predict postoperative PVO outperforms the method based on either clinical data or CT images alone. Our dataset is released to the community to promote further research.