Extending medical interfaces towards virtual reality and augmented reality

Abstract

Introduction: The growing interest of Augmented Reality (AR) together with the renaissance of Virtual Reality (VR) opened new possibilities to redesign how professionals interact with medical images. Several medical specialties already rely on 2D and 3D image data for diagnosis, surgical planning, surgical navigation, medical education and patient-clinician communication. However, the vast majority of conventional medical interfaces and interaction techniques continue unchanged, while the most innovative solutions have not yet untapped the full potential of VR and AR because extending conventional workstations to accommodate VR and AR interaction paradigms is not free of challenges. Notably, VR and AR-based workstations, besides having to render complex anatomical data in interactive frame rates, must also promote proper anatomical insight, boost visual memory through seamless visual collaboration between professionals, unrestrained interaction from being seated at a desk (e.g., using mouse and keyboard) to adopt non-stationary postures and freely walk within a work space, and must support a fluid exchange of image data and 3D models as this foments interesting discussions to solve clinical cases. Moreover, VR and AR-based workstations should likewise be designed according to good human-computer interaction principles since it is well known that medical professionals can be resistant to changes in their workflow. To meet these challenges, we present several case studies that serve as proofs of concept represented as VR or AR prototypes, which were tested by professionals of different medical specialities. Case studies: We have identified specific areas where VR and AR paradigms can make a difference on how healthcare professionals visualize, manipulate and read 2D and 3D medical image data. The most obvious is medical education, where we are currently exploring implant planning using mobile AR and a tablet to train dentistry students. Another specific area is radiodiagnostics, where we evaluated the positive impact of VR, namely, on how radiologists analyze images when immersed inside a virtual reading room [1], and on how radiologists may perform immersive virtual colonography. Also in surgical navigation, AR head-mounted displays carry the promise to assist surgeons performing laparoscopic surgery, in particular, how optical see-through head-mounted displays can assist the surgeon’s eye-hand coordination and to clarify the location of anatomical landmarks displayed on the video stream. Finally, we explore how AR, using multiple projection surfaces, can assist users to perform rehabilitation exercises by themselves under the offline supervision of a physiotherapist [2].