8.2: Anatomy Education Method using Autostereoscopic 3D Image Overlay and Mid‐Air Augmented Reality Interaction

Abstract

The augmented reality (AR) medical education systems offer unique advantages for an interactive and hands‐on integration with the student in surgical skill training. However, there are still major differences in visualization and interaction between real surgical environments and AR‐based medical education systems. In this paper, we propose an anatomy education method using an autostereoscopic 3D image overlay and mid‐air augmented reality interaction with soft tissue deformation. To achieve good depth perception and intuitive interaction, we introduce a model‐based occlusion handling for accurate occlusion processing in real time and a position‐based dynamics (PBD) simulation method for virtual soft tissue deformation in real time. The experimental results showed that our approach could provide 3D visualization and interaction of virtual soft tissues and handle the mutual occlusion automatically in real time. Our proposed medical education system will provide users with a visualization and interaction experience close to real surgical environments.