Tensor based multi-modality fusion for prediction of postoperative early recurrence of single hepatocellular carcinoma

Abstract

Clinical decision-making in oncology involves multi-modal data such as radiology and clinical factors. In recent years, several computer-aided multi-modal decision systems have been developed to predict the recurrence of hepatocellular carcinoma (HCC) after hepatectomy, but these models simply concatenate features naively at the feature level, which would create redundancy and hinder model performance. Particularly, we found that integrating deep multi-modal models based on tensor fusion, which can better handle relevant information between different modalities and reduce model redundancy to obtain better results. This finding suggests the presence of independent, complementary prognostic information between radiology and clinical modalities. Hence we propose a multi-modal learning network that uses a tensor-based fusion approach with clinical parameters, radiomics features and corresponding complex interrelationships between pathological data to predict postoperative early recurrence of single hepatocellular carcinoma. In order to maximize the information gathered from each modality, we introduced a multi-modal fusion loss function based on orthogonal loss to assist multi-modal training. With IRB’s approval, we collected 176 cases with radiomics (MRI) and pathological features diagnosed by experienced clinicians, and established an approach based on tensor fusion that takes pathological findings and postoperative early recurrence as ground truth. Training with 140 cases and tested with 36 cases, the proposed network achieved the AUC of 0.883, which showed great potential in predicting postoperative early HCC recurrence. The ablation experimental results also showed that the auxiliary loss function had a statistically significant improvement in model performance (p<0.01).