A Markerless 3D Tracking Framework for Continuum Surgical Tools Using a Surgical Tool Partial Pose Estimation Network Based on Domain Randomization

Abstract

3D tracking of single‐port continuum surgical tools is an essential step toward their closed‐loop control in robot‐assisted‐laparoscopy, since single‐port tools possess multiple degrees‐of‐freedom (DoFs) without distal joint sensors and hence have lower motion precision compared to rigid straight‐stemmed tools used in multi‐port robotic laparoscopy. This work proposes a novel markerless 3D tracking framework for continuum surgical tools using a proposed surgical tool partial pose estimation network (STPPE‐Net) based on U‐Net and ResNet. The STPPE‐Net estimates the segmentation and a 5‐DoF pose of the tool end‐effector. This network is entirely trained by a synthetic data generator based on domain randomization (DR) and requires zero manual annotation. The 5‐DoF pose estimation from the STPPE‐Net is combined with the surgical tool axial rotation from the robot control system. Then, the entire pose is further refined via a region‐based optimization that maximizes the overlap between the tool end‐effector segmentation from the STPPE‐Net and its projection onto the image plane of the endoscopic camera. The segmentation accuracy and 6‐DoF pose estimation precision of the proposed framework are validated on the images captured from an endoscopic single‐port system. The experimental results show the effectiveness and robustness of the proposed tracking framework for continuum surgical tools.