Feature Tracking and Segmentation in Real Time via Deep Learning in Vitreoretinal Surgery: A Platform for Artificial Intelligence-Mediated Surgical Guidance

Abstract

This study investigated whether a deep-learning neural network can detect and segment surgical instrumentation and relevant tissue boundaries and landmarks within the retina using imaging acquired from a surgical microscope in real time, with the goal of providing image-guided vitreoretinal (VR) microsurgery. Retrospective analysis via a prospective, single-center study. One hundred and one patients undergoing VR surgery, inclusive of core vitrectomy, membrane peeling, and endolaser application, in a university-based ophthalmology department between July 1, 2020, and September 1,2021. A dataset composed of 606 surgical image frames was annotated by 3 VR surgeons. Annotation consisted of identifying the location and area of the following features, when present in-frame: vitrector-, forceps-, and endolaser tooltips, optic disc, fovea, retinal tears, retinal detachment, fibrovascular proliferation, endolaser spots, area where endolaser was applied, and macular hole. An instance segmentation fully convolutional neural network (YOLACT++) was adapted and trained, and fivefold cross-validation was employed to generate metrics for accuracy. Area under the precision-recall curve (AUPR) for the detection of elements tracked and segmented in the final test dataset; the frames per second (FPS) for the assessment of suitability for real-time performance of the model. The platform detected and classified the vitrector tooltip with a mean AUPR of 0.972 ± 0.009. The segmentation of target tissues, such as the optic disc, fovea, and macular hole reached mean AUPR values of 0.928 ± 0.013, 0.844 ± 0.039, and 0.916 ± 0.021, respectively. The postprocessed image was rendered at a full high-definition resolution of 1920× 1080 pixels at 38.77 ± 1.52 FPS when attached to a surgical visualization system, reaching up to 87.44 ± 3.8 FPS. Neural Networks can localize, classify, and segment tissues and instruments during VR procedures in real time. We propose a framework for developing surgical guidance and assessment platform that may guide surgical decision-making and help in formulating tools for systematic analyses of VR surgery. Potential applications include collision avoidance to prevent unintended instrument-tissue interactions and the extraction of spatial localization and movement of surgical instruments for surgical data science research. Proprietary or commercial disclosure may be found after the references.