Deep Learning-Driven Real-Time Liver Tumor Localization via Optical Surface Imaging and Biomechanical Modeling

Abstract

<h3>Purpose/Objective(s)</h3> To track liver tumors in real time via a deep learning-based framework, by exploring patient-specific correlations between patients' body surface maps and internal anatomical motion. <h3>Materials/Methods</h3> Real-time liver tumor localization was achieved via deformable registration-based motion estimation, in a two-step framework: (a) liver boundary motion estimation via deep features learnt from optical body surface imaging; and (b) intra-liver motion propagation via deep learning-based biomechanical modeling from liver boundary motion. In step (a), a patient-specific, fully-connected convolutional neural network (SurfCNN) predicts the motion of nodes residing on the boundary of a patient-specific reference liver mesh, by extracting saliency features from curvature entropy maps derived out of real-time optical body surface images. Subsequently, step (b) uses the liver boundary motion solved in (a) to infer intra-liver tumor motion, using a U-Net-style model (UNet-Bio) inspired by biomechanical modeling via finite element analysis (FEA). The cascaded framework was evaluated using a dataset of 8 liver cancer patients from our institute. Each patient had a 10-phase, contrast-enhanced 4D-CT set with liver and liver tumor contoured. To generate sufficient motion variations to train and test the patient-specific SurfCNN model, we augmented the dataset of each patient by simulating varying real-time motion patterns and magnitudes. The augmentation was achieved via a principal component analysis-based statistical model, expanding the nine inter-phase deformation-vector-fields (DVFs) of each 4D-CT into 1,728 different motion states including both rigid and deformable motion. Optical surface imaging was simulated from each motion state via extracted body surface maps, while corresponding liver boundary motion was derived via the augmented DVFs to supervise the training of SurfCNN. UNet-Bio, on the other hand, could be trained as a population-based model using supervisions from intra-liver DVFs solved by FEA-based biomechanical modeling. The liver tumor localization accuracy was assessed through Dice similarity coefficient (DSC), Hausdorff distance (HD), and center-of-mass-error (COME), by comparing the augmentation DVF-propagated ‘ground-truth' liver tumor volumes against ones deformed by the cascaded framework. <h3>Results</h3> Tested using 576 unseen scenarios for each patient case, the cascaded SurfCNN and UNet-Bio scheme can localize liver tumors to 0.791(mean) ± 0.141 (s.d.) in DSC, 3.6 ± 1.9 mm in HD, and 2.3 ± 1.7 mm in COME. In comparison, the prior reference image without deformable registration yielded 0.534 ± 0.277 in DSC, 8.3 ± 6.1 mm in HD and 6.8 ± 5.8 mm in COME. The whole model inference time was less than 100 milliseconds, satisfying the temporal constraints of real-time imaging. <h3>Conclusion</h3> The deep learning-based framework allows accurate 3D liver tumor tracking in real-time via non-ionizing, marker-less, and high frame-rate optical surface imaging.