2D and 3D Convolutional Neural Network Fusion for Predicting the Histological Grade of Hepatocellular Carcinoma

Abstract

Preoperative Knowledge of the histological grade of hepatocellular carcinoma (HCC) is significant for patient management and prognosis in clinical practice. Recent studies reported that 3D Convolutional Neural Network (CNN) outperformed 2D CNN for lesion characterization. Since 2D and 3D deep feature derived from CNN embed different spatial information of neoplasm, we hypothesize that the performance of lesion characterization might be improved if taking full advantage of both 2D and 3D characterization. In this work, we propose a 2D and 3D CNN fusion architecture to integrate both 2D and 3D spatial information of neoplasm for predicting the histological grade of HCC. Specifically, correlated and individual component analysis (CICA) is performed to fuse the 2D deep features in three orthogonal views and the 3D deep feature in volumetric images of HCC. Experimental results of 46 clinical patients with HCCs demonstrate several encouraging features of the proposed 2D and 3D deep feature fusion framework as follows: (1) Fusion of 2D and 3D deep feature using CICA outperforms 2D or 3D deep feature for predicting the histological grade of HCC. (2) Fusion of 2D deep features derived from three orthogonal views using CICA yields better results than those of 3D deep feature. (3) CICA is better than the conventional concatenation and the correlation learning model for deep feature fusion.