Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform

Giovanni Badiali,Vincenzo Ferrari,Sara Condino,Fabrizio Cutolo,Marina Carbone,Renzo D’Amato,Laura Cercenelli

doi:10.3390/info13020081

Abstract

In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.

Highlights

Many authors report proof of concept experience in the surgical room with commercially available devices [8,9,10,11,12,13,14], underlining the enthusiasm of the surgical world in exploring the potentialities of Augmented reality (AR) head-mounted displays (HMDs) and appraising the limitation of using a device not engineered for the surgical tasks: (1) perceptual issues related to the rendering appropriate focus cues [15,16,17], (2) the absence of a framework dedicated to the medical applications [18], (3) the need for a design that takes into consideration the surgeon posture to avoid neck overload [19,20] and the need of maintaining the sterility of his hands, and (4) the need for a millimetric virtual to real registration accuracy in many interventions
The objective of the paper is to describe the full architecture of a wearable surgical navigator implementing a new AR paradigm developed in the framework of the Horizon2020 VOSTARS project (G.A. 731974) and the challenges faced to entering the surgical room in compliance with the EU regulations for non-CE medical devices in clinical trials
AR HMDs offer the enhancement experience of the natural view with the superimposition of computer-generated three-dimensional (3D) objects implementing either an optical see-through (OST) or a video see-through (VST) paradigm [22]

Summary

Introduction

Augmented reality (AR) head-mounted displays (HMD) allow the real-time visualization in front of the user’s eyes of additional virtual information within the real environment for an enhanced and user interactive experience in different fields of application [1,2].The use of commercial AR HMDs in the medical domain, and in particular in surgery, is investigated by many authors [3,4,5,6,7].In the case of open surgery, the translation of the state-of-the-art surgical navigators based on the virtual reality (VR) paradigm into a wearable system is the natural evolution of such devices, potentially providing surgeons with an improved and more effective navigated experience with a positive impact on the clinical outcome. Many authors report proof of concept experience in the surgical room with commercially available devices [8,9,10,11,12,13,14], underlining the enthusiasm of the surgical world in exploring the potentialities of AR HMDs and appraising the limitation of using a device not engineered for the surgical tasks: (1) perceptual issues related to the rendering appropriate focus cues [15,16,17], (2) the absence of a framework dedicated to the medical applications [18], (3) the need for a design that takes into consideration the surgeon posture to avoid neck overload [19,20] and the need of maintaining the sterility of his hands, and (4) the need for a millimetric virtual to real registration accuracy in many interventions

Objectives

Methods

Results