Physical object interaction in first responder mixed reality training

Abstract

Virtual Reality (VR) systems can improve the training of first responders and soldiers in multi-domain operations in a number of ways. Realistic simulation of physical objects, however, is challenging. The huge variety of equipment pieces and other objects specialists in first responder units - and in particular CBRN-troops - interact with further increases the effort. In this paper, we present a novel and flexible Mixed Reality (MR) training system for first responders that enables the integration of physical objects by using Augmented Virtuality (AV) and binary tracking. A Head Mounted Display (HMD) immerses the user in VR, while augmenting the visualization with 3D imagery of real objects, captured by an RGB-D sensor. In addition, a RFID-reader at the user's hand detects the presence or absence (binary response) of certain equipment items. Our proposed MR system fuses this information with data of an inertial motion capture suit to an approximate global object pose and distributes it. Our solution provides a wide range of options for physical object interaction and collaboration in a multi-user MR environment. In addition, we demonstrate the training capabilities of our proposed system with a multi-user training scenario, simulating a CBRN crisis. Results from our technical and quantitative user evaluation with 13 experts in CBRN response from the Austrian Armed Forces (National Defense Academy and Competence Center NBC Defense) indicate strong applicability and user acceptance. Over 80% of the participants found it easy or very easy to interact with physical objects and liked the multi-user training much or very much.