Endoscope FOV Autonomous Tracking Method for Robot-Assisted Surgery Considering Pose Control, Hand–Eye Coordination, and Image Definition

Abstract

In laparoscopic surgeries, intelligent assisted endoscope-holding robots (AEHRs) with traditional servo control have limited performance due to the lack of endoscopic attitude control, hand-eye coordination, and image definition. This paper proposes an endoscope field-of-view (FOV) autonomous tracking method for robot-assisted surgery (RAS) considering pose (i.e., position and attitude) control, hand-eye coordination and image definition, it can realize that the AEHR system can safely, stably and quickly track the movement of surgical instruments. Firstly, the general framework of the intelligent tracking system with robot-assisted endoscope-holding is constructed according to the clinical needs and shortcomings of traditional research. Secondly, the kinematic model and remote center of motion (RCM) constraint equations of AEHRs are established. Then, surgical instrument tips are segmented and localized using the deep learning method (DLM), and further, a surgical FOV intelligent adjustment control model with multiple indexes is developed. Finally, an experimental system based on the hybrid of AEHRs and speech interaction is built. The simulation and experimental results demonstrate that the proposed method can achieve fast, accurate, and effective instrument tips tracking (ITT), reduce hand-eye incoordination, and improve the safety and stability of FOV intelligent adjustment.