Mixed reality-integrated 3D/2D vision mapping for intuitive teleoperation of mobile manipulator

Abstract

• This work evaluates the impact of the integrated 3D/2D vision and motion mapping schemes on teleoperation efficiency and user experience in a mixed reality (MR) environment. • A significant enhancement in task performance is achieved with the point cloud within an MR environment (MR-3DPC) vision and motion mapping scheme, while minimizing training effort and cognitive workload. • The MR-integrated 3D/2D vision and motion mapping schemes of teleoperation reduced overall task completion times by 34% and 17%. • It is found using point cloud within an MR environment (MR-3DPC) outperforms displaying stereoscopic vision (MR-3DSV) in MR and can enhance users’ stereoscopic perception and task performance. Depth cues are crucial to increase user perception and spatial awareness of the remote environment when remotely guiding complex robotic systems. A mixed reality (MR) integrated 3D/2D vision and motion mapping framework for immersive and intuitive telemanipulation of a complex mobile manipulator is presented. The proposed 3D immersive telerobotic schemes provide the users with depth perception through the merging of multiple 3D/2D views of the remote environment via MR subspace. The mobile manipulator platform consists of a 6°-of-freedom (DOF) industrial manipulator, 3D-printed parallel gripper, and mobile base, which can be controlled by non-skilled operators who are physically separated from the robot working space through a velocity-based imitative motion mapping approach. This work evaluates the impact of depth perception and immersion provided by integrated 3D/2D vision and motion mapping schemes on teleoperation efficiency and user experience in an MR environment. In particular, the MR enhanced systems maintain spatial awareness and perceptual salience of the remote scene in 3D, facilitating intuitive mixed reality human-robot interaction (MR-HRI). This study compared two MR-integrated 3D/2D vision and motion mapping schemes against a typical 2D Baseline visual display method through pick-and-place, assembly, and dexterous manufacturing tasks. The MR-integrated 3D/2D vision and motion mapping schemes of teleoperation reduced overall task completion times by 34% and 17%, compared to the MR-2D Baseline, while minimizing training effort and cognitive workload.