Sensorless Real-Time Force Estimation in Microsurgery Robots Using a Time Series Convolutional Neural Network

Jiuyun Xia,Kazuo Kiguchi

doi:10.1109/access.2021.3124304

Jiuyun Xia, Kazuo Kiguchi

Open Access

https://doi.org/10.1109/access.2021.3124304

Copy DOI

Abstract

Robotic-assisted microsurgeries provide several benefits to both patients and surgeons. Nevertheless, there are still some limitations and challenges associated with their outcome, one of which is a lack of force feedback. Without force information, the risk of delicate tissue damage from the excessive force applied by surgeons would be increased. Since it is difficult to install force sensors on microsurgical tools, a novel approach for estimating a force vector from the deformation of the surgical tool is proposed in this paper. In the proposed approach, a surgical instrument that deforms according to the magnitude of the tool-to-tissue force is designed, and a time series convolution neural network is used to make the nonlinear relationship between the visual information of the deformation of the surgical tool and the applied forces in such a way that the tool-to-tissue force can be estimated according to the deformation of the surgical instrument in a real-time manner. The experimental results prove that the applied force can be successfully estimated with high accuracy in three dimensions using the proposed method.

Highlights

Microsurgery is an operation for suturing vessels or nerves under a microscope and is regarded as one of the most technically demanding surgical disciplines because it requires precise motion to manipulate delicate tissue in a small and constrained workspace [1][2]
Force feedback is a major feature that can improve the microsurgical performance since it enables surgeons to control the interaction forces [6], and it helps in the proper execution of surgical procedures
Without force information, it is difficult for surgeons to feel how much force is applied to delicate tissues, and excessive force might result in irreversible damage [7]-[9]

Summary

INTRODUCTION

Microsurgery is an operation for suturing vessels or nerves under a microscope and is regarded as one of the most technically demanding surgical disciplines because it requires precise motion to manipulate delicate tissue in a small and constrained workspace [1][2]. Deep neural networks are applied together with time series visual information from a deformable surgical tool to estimate the applied force. This setup was designed to acquire a dataset of image information of the proposed deformable surgical tool and the actual applied interaction forces. The video is converted into an image frame by frame, and a resize method is used to extract a region of interest with a size of 256 × 256 pixels from 1920 × 1080 pixels From this environment, a dataset with 5400 images and the corresponding force information labels was established, and this dataset was used to train and evaluate our proposed time series CNN model network. Deformation of surgical that result from applying a force at different time instants the proposed model by evaluating the difference between the estimated forces and the real forces

RESULTS AND DISCUSSION

COMPARISON RESULTS

CONCLUSIONS