Augmented Holographic HPB Surgical Navigation Using Extended Reality: XR (VR/AR/MR)

Abstract

Surgical navigation have become essential for surgeons to accurately and safely perform HPB operations. HPB surgery requires a high degree of spatial awareness and orientation because of complex anatomies and procedures. The traditional navigation for HPB surgery was displayed only for 2D observation on a flat screen by surgeons, the image-based navigation interface is separated from the operating area, and the surgeon needs to switch the field of vision between the screen and the patient´s lesion area. We developed extended reality (XR) navigation that combines virtual reality (VR), augmented reality (AR), and mixed reality (MR), and holographic technology for HPB surgery to provide more spatial and intuitive information to surgeons. From patient´s individual CT data, organs and abnormal lesions were extracted into individually colored 3D polygons, and represented into real space with a transparent holographic wearable glasses built-in position sensors (HoloLens and MagicLeap) using our original XR application. In Results, each organ was floated in the actual surgical space These holographic organ models were able to share and move freely in all directions by gesture interface, and complex procedures could be confirmed with pointing by all surgeons. The ability to spatial awareness for understanding the extent of resection, blood vessel processing, and lymph node dissection were improved during surgery. Our XR navigation system has high accuracy and stability for registration. Our patient-specific XR surgical navigation is highly effective and reduce surgical time, blood loss, and adverse event. This system must have value for future HPB surgeons