Real-time multitask multihuman–robot interaction based on context awareness

Abstract

AbstractThis study presents a novel context awareness multihuman–robot interaction (MHRI) system that allows multiple operators to interact with a robot. In the system, a monocular multihuman 3D pose estimator is first developed with the convolutional neural network. The estimator first regresses a set of 2D joints representations of body parts and then restores the 3D joints positions based on these 2D representations. Further, the 3D joints are assigned to the corresponding individual with a priority–redundancy association algorithm. The whole 3D pose of each person is reconstructed in real time, even in crowded scenes containing both self-occlusion of the body and inter-person occlusion. Then, the identities of multiple persons are recognized with action context and 3D skeleton tracking to improve interactive efficiency. For context-awareness multitask interaction, the robot control strategy is designed based on target goal generation and correction. The generated goal is taken as a reference to the model predictive controller (MPC) to generate motion trajectory. Different interactive requirements are adapted by adjusting the weight parameters of the energy function of the MPC controller. Multihuman–robot interactive experiments, including dynamic obstacle avoidance (human–robot safety) and cooperative handling, demonstrate the feasibility and effectiveness of the MHRI, and the safety and collaborative efficiency of the system are evaluated with HRI metrics.