Automatic Tracking of Surgical Instruments with a Continuum Laparoscope Using Data‐Driven Control in Robotic Surgery

Abstract

In existing surgery process, surgeons need to manually adjust the laparoscopes to provide a better field of view (FOV) during operation, which may distract surgeons and slow down the surgery process. Herein, a data‐driven control method that uses a continuum laparoscope to adjust the FOV by tracking the surgical instruments is presented. A Koopman‐based system identification method is first applied to linearize the nonlinear system. Shifted Chebyshev polynomials are used to construct observation functions that transfer low‐dimension observations to high‐dimension ones. The Koopman operator is approximated using a finite‐dimensional estimation method. An optimal controller is further developed according to the trained linear dynamic model. Furthermore, a learning‐based pose estimation framework is designed to detect keypoints on surgical instruments and provides visual feedback for the control system. Compared with other detection methods, the proposed scheme achieves a higher detection precision and provides more optional keypoints for tracking. Simulation and experiments validate the feasibility of the proposed control method. Experimental results show that the proposed method can automatically adjust the field of continuum laparoscope by tracking surgical instruments in real time and satisfy the clinical requirements.