Using Mixed Reality for Control and Monitoring of Robot Model Based on Robot Operating System 2

Abstract

This article presents the design and implementation of an innovative human–machine interface (HMI) in mixed reality for a robot model operating within Robot Operating System 2 (ROS 2). The interface is specifically developed for compatibility with Microsoft HoloLens 2 hardware and leverages the Unity game engine alongside the Mixed Reality Toolkit (MRTK) to create an immersive mixed reality application. The project uses the Turtlebot 3 Burger model robot, simulated within the Gazebo virtual environment, as a representative mechatronic system for demonstration purposes. Communication between the mixed reality application and ROS 2 is facilitated through a publish–subscribe mechanism, utilizing ROS TCP Connector for message serialization between nodes. This interface not only enhances the user experience by allowing for the real-time monitoring and control of the robotic system but also aligns with the principles of Industry 5.0, emphasizing human-centric and inclusive technological advancements. The practical outcomes of this research include a fully functional mixed reality application that integrates seamlessly with ROS 2, showcasing the potential of mixed reality technologies in advancing the field of industrial automation and human–machine interaction.