Real-Time 3D Liver Tumor Localization via Combined Optical Surface Imaging and an X-Ray Projection from Arbitrary Imaging Angles

Abstract

A deep learning (DL)-based, deformable registration-driven liver tumor localization technique was developed for onboard deformable motion tracking. The technique (Surf-X360-Bio) uses in-room optical surface imaging and an x-ray projection at an arbitrary scan angle to solve volumetric liver and liver tumor motion in real-time. Surf-X360-Bio solves the volumetric motion of the liver and localizes the liver tumor, through deforming liver and liver tumor meshes segmented on prior 4D-CTs/CBCTs. It uses real-time onboard information from an optical surface image and a simultaneously-acquired x-ray projection (from an arbitrary scan angle). Surf-X360-Bio localizes tumors via two steps: liver boundary motion estimation and intra-liver motion derivation. Surf-X360-Bio first estimates liver boundary motion by a patient-specific surface imaging model (Surf), utilizing the correlation between the external body surface motion and the internal liver boundary motion. As the correlation can be imperfect, the residual motion estimation errors were corrected by a patient-specific, angle-agnostic x-ray imaging model (X360). X360 deformed the liver boundary to match motion-related imaging features encoded in an arbitrarily-angled x-ray projection, using the Surf output for initialization. X360 adopted a geometry-aware learning module to extract and adapt to angle-varying features of arbitrarily-angled x-ray images, by calculating the projection system matrix of each x-ray image on-the-fly during model training and inference. After the liver boundary motion estimation by Surf and X360, intra-liver deformation was solved by a biomechanical model (Bio) to propagate the liver boundary motion inside to localize the tumors. The DL-based Bio model used domain knowledge of tissue biomechanics and finite element analysis (FEA) to solve intra-liver motion, with a much faster speed than conventional FEA methods to meet the real-time constraint. Surf-X360-Bio was evaluated using a dataset of 34 liver patients. Liver tumor localization accuracy was evaluated by the Dice similarity coefficient (DSC), the 95-percentile Hausdorff distance (HD95), and the center-of-mass error (COME). Using 3,306 motion scenarios spanning the 360 degree x-ray scan angles for each testing patient, Surf-X360-Bio localized the liver tumors to 0.81 (mean) ± 0.16 (s.d.) in DSC, 2.5 ± 1.7 mm in HD95, and 2.1 ± 1.8 mm in COME. In comparison, the prior reference image without deformable registration-driven localization yielded 0.42 ± 0.29 in DSC, 8.1 ± 5.2 mm in HD95, and 8.5 ± 5.2 mm in COME. Via Surf-X360-Bio, the overall inference time was below 230 ms for each case. Combining optical surface imaging and x-ray imaging, Surf-X360-Bio achieved fast (<250 ms inference time), accurate (mean error < 2.1 mm), and robust liver tumor localizations at arbitrary x-ray scan angles for real-time, marker-less 3D deformable motion tracking.