A workflow to generate patient-specific three-dimensional augmented reality models from medical imaging data and example applications in urologic oncology

Andrew B. Rosenkrantz,James S. Wysock,Hersh Chandarana,Daniel K. Sodickson,Samir S. Taneja,Nicole Wake,William C. Huang

doi:10.1186/s41205-021-00125-5

Abstract

Augmented reality (AR) and virtual reality (VR) are burgeoning technologies that have the potential to greatly enhance patient care. Visualizing patient-specific three-dimensional (3D) imaging data in these enhanced virtual environments may improve surgeons’ understanding of anatomy and surgical pathology, thereby allowing for improved surgical planning, superior intra-operative guidance, and ultimately improved patient care. It is important that radiologists are familiar with these technologies, especially since the number of institutions utilizing VR and AR is increasing. This article gives an overview of AR and VR and describes the workflow required to create anatomical 3D models for use in AR using the Microsoft HoloLens device. Case examples in urologic oncology (prostate cancer and renal cancer) are provided which depict how AR has been used to guide surgery at our institution.

Highlights

In modern surgery, pre-operative imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), and ultrasonography are used for both presurgical planning and intra-operative guidance
These imaging techniques allow surgeons to visualize the surgical anatomy, they are limited since interpretation of the three-dimensional (3D) anatomy must be made on a two-dimensional (2D) screen
While the methods of image visualization used in these laboratories allow for improved understanding of anatomy as compared to viewing images in 2D, they are still limited by the fact that 3D data is being visualized on a 2D screen

Summary

Introduction

Pre-operative imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), and ultrasonography are used for both presurgical planning and intra-operative guidance. It is believed that these methods of advanced image visualization can improve understanding of anatomy and surgical pathology, thereby enhancing surgical planning, providing improved intra-operative guidance, and improving patient care. Workflow to create patient-specific AR models AR devices do not currently accept Digital Imaging and Communications in Medicine (DICOM) images as an input.

Results

Conclusion