Using Multimodal Interaction in a Virtual Reality Thoracic Diagnostic Scenario

Abstract

Background: Stethoscopes and mannequins are basic tools for medical practitioners to diagnose various problems. Restricted by space, resources, and time, these tools and common video training are sometimes inaccessible. These obstacles may be solved by immersive virtual reality (VR) that improves interaction and blurs the line between virtual and reality. Therefore, this study combines virtual and real interaction technologies to investigate the impact of multi-sensory interaction on learning outcomes in chest auscultation.Methods: This study integrated VIVE Tracker technology to provide synchronized tactile stimulation in VR. University students in medical-related disciplines participated in the study. Interviews were conducted to understand how students perceived the effectiveness and usefulness of the thoracic auscultation VR.Conclusion: Most students expressed that tactile stimulation gave realistic sensations and effectively reinforced the immersive effect. Simultaneously, it increased their interest and motivation in learning. The integration of synchronized tactile stimulation is a novel approach that expands the realm of interactivity beyond handheld controller manipulation and creates a platform to simulate medical-related scenarios to learn complex knowledgeand skills.