Multimodal Multi-User Mixed Reality Human–Robot Interface for Remote Operations in Hazardous Environments

Abstract

In hazardous environments, where conditions present risks for humans, the maintenance and interventions are often done with teleoperated remote systems or mobile robotic manipulators to avoid human exposure to dangers. The increasing need for safe and efficient teleoperation requires advanced environmental awareness and collision avoidance. The up-to-date screen-based 2D or 3D interfaces do not fully allow the operator to immerse in the controlled scenario. This problem can be addressed with the emerging Mixed Reality (MR) technologies with Head-Mounted Devices (HMDs) that offer stereoscopic immersion and interaction with virtual objects. Such human-robot interfaces have not yet been demonstrated in telerobotic interventions in particle physics accelerators. Moreover, robotic operations often require a few experts to collaborate, which increases the system complexity and requires sharing a multi-user Augmented Reality (AR) workspace. The multi-user telerobotics with shared control in the AR has not yet been approached in the industrial state-of-the-art. In this work, the developed MR human-robot interface using the AR HMD is presented. The interface adapts to the constrained wireless networks in particle accelerator facilities and provides reliable high-precision interaction and specialized visualization. The multimodal operation uses hands, eyes and user motion tracking, and voice recognition for control, as well as offers video, 3D point cloud and audio feedback from the robot. Multiple experts can collaborate in the AR workspace locally or remotely, share the robot’s control and monitor robotic teleoperation. Ten (10) operators tested the interface in intervention scenarios in the European Organization for Nuclear Research (CERN) with complete network characterization and measurements to conclude if operational requirements were met and if the network architecture could support single and multi-user communication load. The interface system has proved to be operationally ready at the Technical Readiness Level (TRL) 8 - and was validated through successful tests and demonstration in single and multi-user missions. Some areas of system limitations and further work were identified, such as optimising the network architecture for multi-user scenarios or high-level interface actions applying automatic interaction strategies with the robot depending on network conditions.