Combining immersion and interaction in XR training with 360-degree video and 3D virtual objects

Abstract

Research shows that XR experiences activate portions of the brain that facilitate learning. Case studies in industry bear this out, showing enormous benefits of XR when used to train employees, including reduced time to learn and complete a task, increased ability to perform the task correctly the first time, and better knowledge retention than with traditional training methods. The emerging technologies that fall under the umbrella term XR—virtual reality, augmented reality, mixed reality, and 360-degree video—can facilitate or enhance learning through different affordances. The degree of immersion and the ability to exercise agency through interaction also enhances learning. 360-degree video is highly immersive. But it is not inherently interactive. It is theorized that by inserting interactive 3D objects into a 360-degree video environment, both immersion and interactivity can be achieved, with the result of enhanced learning. The solution presented here also has the advantage of being more easily produced and cheaper than virtual reality. The objectives of this preliminary study were to create a simple and relatively inexpensive prototype virtual environment very close to reality and test the technical effectiveness of learning and situational awareness using 360-degree videos together with 3D objects. An iterative human-centered design (HCD) process that integrates airplane pilots and training personnel as end users and participants in the cocreation of an aviation training system; actively involves users; provides a clear understanding of tasks; facilitates good interaction between user and technology; and includes well-designed interactions was used. A few scripts were created in the prototype to improve the user experience (UX) when interacting with the user interface (UI). The prototype app was presented to flight instructors and flight pilots using a Samsung S7 smartphone with a Samsung Gear VR by Oculus headset. The prototype also allows the experience to be viewed on a mobile device without a VR headset. The experience is based on a 360-degree video taken within a Cessna 150, with a view of the flight panel. 3D objects were inserted and were made interactive so the user can control them. A survey that asked for suggestions to improve the next version of this prototype was given to the test subjects. Preliminary findings suggest system user acceptance, improved pilot confidence before the actual flight, decreased errors, and better situational awareness. In conclusion, the combination of 360-degree video and 3D virtual objects can help train pilots better, more safely, in less time, and with more confidence in the use of the aircraft checklist. The technology provides an alternative to costly and time-consuming VR solutions for training those without access to aircraft, and permits learning with the use of their smartphone.