Learning Laparoscope Actions via Video Features for Proactive Robotic Field-of-View Control

Abstract

Smart laparoscope motion control for adjusting surgical field-of-view is an increasingly hot topic in robot-assisted surgery. Previous off-the-shelf methods have been conducted in reactive ways which heavily rely on human input signals, e.g., gaze or voice, thus cannot avoid cognitive burdens to surgeons. In this paper, we introduce a novel proactive framework that learns the motion strategy from clinical surgical videos to achieve autonomous laparoscope control in surgery. In specific, we first propose a robust estimation method to acquire laparoscope motion in dynamic surgical scenes, which provides the basis for later supervised learning process. To imitate operating manners in different surgical domains, we design consistent dynamic image-level motion features that consist of semantic segmentation and dense optical flow. A sequence-to-sequence recurrent network is proposed to learn the connection between previous and future motion features at multiple temporal scales within a time-dependent sliding window. Then a laparoscope action head is built to convert the motion features into three-dimensional actions, hence establishing an end-to-end laparoscope motion imitation network. We have extensively validated the proposed proactive field-of-view control framework on both our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ex-vivo</i> phantom-based robotic platform, as well as <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in-vivo</i> videos collected from real clinical videos. Our promising results demonstrate the feasibility to directly learn laparoscope behavioral actions from collected video data of experienced surgical assistants.