Cloud-based multimodal human-robot interaction simulator utilizing ROS and unity frameworks

Abstract

A new software architecture for a simulator of human-robot interaction (HRI) in virtual reality (VR) environments is proposed. Since collecting and storing a massive amount of data concerning multimodal interaction experiences is an important task concerning research on HRI, a cloud-based VR platform, named “SIGVerse,” which reduces costs of developing real robots and interaction experiments in the real world, is proposed. The reusability of virtual robot software is restricted in a real environment due to difference between VR and real robot architectures; therefore, a new architecture utilizing the Unity and ROS frameworks is proposed. The proposed architecture provides functionalities for constructing scalable 3D environments, embodied and social interaction via the Internet, compatible robot software, high-fidelity sensor feedback, and recording/playback of interaction. To demonstrate the feasibility of the proposed architecture, the performance of SIGVerse in terms of simulating of multimodal information in actual interaction applications was evaluated, and the latency between avatars synchronized via cloud computing was measured. Additionally, interactive behaviors of robots and avatars in a VR environment and a real environment were experimentally compared, and the comparison results confirm that the VR behaviors of robots and avatars were almost the same as the behavior of a robot in a real environment.