Real-Time 3D Tracking of Laparoscopy Training Instruments for Assessment and Feedback.

Benjamin Gautier,Benjie Tang,Ghulam Nabi,Ghulam Nabi,Mustafa Suphi Erden,Harun Tugal

doi:10.3389/frobt.2021.751741

Benjamin Gautier, Benjie Tang + Show 4 more

Open Access

https://doi.org/10.3389/frobt.2021.751741

Copy DOI

Abstract

Assessment of minimally invasive surgical skills is a non-trivial task, usually requiring the presence and time of expert observers, including subjectivity and requiring special and expensive equipment and software. Although there are virtual simulators that provide self-assessment features, they are limited as the trainee loses the immediate feedback from realistic physical interaction. The physical training boxes, on the other hand, preserve the immediate physical feedback, but lack the automated self-assessment facilities. This study develops an algorithm for real-time tracking of laparoscopy instruments in the video cues of a standard physical laparoscopy training box with a single fisheye camera. The developed visual tracking algorithm recovers the 3D positions of the laparoscopic instrument tips, to which simple colored tapes (markers) are attached. With such system, the extracted instrument trajectories can be digitally processed, and automated self-assessment feedback can be provided. In this way, both the physical interaction feedback would be preserved and the need for the observance of an expert would be overcome. Real-time instrument tracking with a suitable assessment criterion would constitute a significant step towards provision of real-time (immediate) feedback to correct trainee actions and show them how the action should be performed. This study is a step towards achieving this with a low cost, automated, and widely applicable laparoscopy training and assessment system using a standard physical training box equipped with a fisheye camera.

Highlights

Laparoscopy is a minimal invasive surgery performed in the abdominal cavity with the most important advantage of fast recovery of patients, compared to conventional open surgery procedures
This study develops an algorithm for real-time tracking of laparoscopy instruments in the video cues of a standard physical laparoscopy training box with a single fisheye camera
In this paper a real time 3D instrument trajectory tracking is developed for single camera laparoscopy training boxes

Summary

INTRODUCTION

Laparoscopy is a minimal invasive surgery performed in the abdominal cavity with the most important advantage of fast recovery of patients, compared to conventional open surgery procedures. In order to speed up the detection process, we are directly using the raw (non-flattened) fisheye camera output, the shape retrieved before application of the convex hull does not have straight lines This would result in having a large set of candidate points for corner selection after the Hough transform. For the corner selection we use the knowledge on the contour pose in the image and the properties of the trapezoidal shape when the cylinders (markers) are viewed from top (Figure 5) where the detected line segments C and B in Figure 6A must remain parallel This method allows to find the best candidates for the corner points from the list output in the previous steps. The four corners of both instruments estimated by the Kalman Filter are flattened as in the previous sub-section

Overall Procedure of Detection of Marker Corners

CONCLUSION

Findings

DATA AVAILABILITY STATEMENT