Tangible digital twin with shared visualization for collaborative air traffic management operations

Abstract

In recent years, digital twin technology has become increasingly popular and been widely applied across different industries including air traffic management (ATM). While previous studies on digital twin in ATM focused on factors such as system architecture, synthetic data generation and transfer, and real-time simulation, there is currently a lack of research on visualization and interaction design of digital twin from human factors’ perspective. Moreover, recent advances in virtual and mixed reality (VR/MR) technology give potential for designing interactive digital twin in 3D space that can assist human operators in performing their duties. Therefore, this research aims to explore visualization and interaction design of an MR-based digital twin prototype that can assist air traffic controllers (ATCOs) in carrying out ground control tasks. Specifically, MR headsets (Microsoft HoloLens 2) are leveraged to project out-of-tower view of airport traffic onto a 3D printed airport model at a 1:1 scale. The spatially aligned tangible system enables ATCOs to perform typical ATM operations by directly touching the 3D printed airport model. In addition, collaborative features are implemented into the system, which enable multiple ATCOs to have a shared view of the airport traffic and collaboratively perform ATM operations when wearing the MR headsets. Finally, a user study was conducted with the recruitment of ten licensed ATCOs to investigate usability of the system and ATCOs’ workload, situational awareness and trust when using the system. The feedback from the ATCOs in the user study suggested that: (1) The visual and interactive design were acceptable; (2) The ATCOs indicated varied workload under different scenarios; (3) The ATCOs developed a high level of shared situational awareness with shared view of the system; and (4) The ATCOs were open and confident to use the system in its’ mature form. The findings show that the MR-based tangible airport digital tower system has high potential in collaborative ATM operations by improving user performance and user experience.