Spatiotemporal characteristics of medical extended-reality systems

Abstract

Emerging uses for extended-reality (XR) head-mounted displays (HMDs) within medical environments include visualizations of medical data across various imaging modalities including radiography, computed tomography, ultrasound, and magnetic resonance images. Rendering medical data in XR environments requires real-time updates to account for user movement within the environment. Unlike stationary 2D medical displays, XR HMDs also require real-time stereoscopic rendering capabilities with high performance graphics processing units. Furthermore, performance depends on the status of added systems including tracking sensor technology, user's input data, and in the case of augmented reality (AR), spatial mapping and image registration. These temporal considerations have implications for the interpretation of medical data. However, methods for the evaluation of their effects on image quality are not yet well defined. The definition of these effects in the context of medical XR devices is at best inconsistent if not completely lacking. In this work, we compare the effects and causes for three classes of XR spatiotemporal characteristics affecting medical image quality: temporal artifacts, luminance artifacts, and spatial mapping artifacts. We describe the XR system components starting from user movement recognized by inertial measurement unit and camera sensors and ending with user perception of the display through the optics of the HMD. We summarize our findings and highlight device performance areas contributing to the different effects.