Image Processing-Based Supervised Learning to Predict Robot Intention for Multimodal Interactions Between a Virtual Human and a Social Robot

Abstract

A humanoid robot and a virtual human with human-like appearance were developed, and various similar intelligence, functions, autonomy and interaction modalities were developed for them that were deemed necessary for their real-world collaboration. The virtual human was integrated with the robot through a common platform based on some control algorithms so that they were capable of cooperating with each other in a real-world task, which was the task of assisting each other in searching for a hidden object in a homely environment. One important activity for such dynamic collaboration was the virtual human’s ability to predict the robot’s intention so that the virtual human could plan to collaborate with the robot, and vice versa. To address this problem, three representative robot behaviors such as hand shaking, hand pointing and hand waving were analyzed using supervised learning-based algorithms associated with image processing to help the virtual human learn robot behaviors so that the learned behaviors could help the virtual human predict the robot’s intention during their intended social interactions. In the training phase, the robot showed each of the three behaviors separately for 500 times. Images of each type of behaviors were taken separately, and the properties of the images were extracted following the image processing method. Then, each robot behavior extracted through image processing was labeled properly. Then, the supervised learning algorithm was developed that classified three different robot behaviors into three different classes. In the testing phase, the robot showed each of its three behaviors to the virtual human randomly for 100 times, the images were taken in real-time, and sent to the image processing-based machine learning algorithm. It was monitored whether the virtual human was able to recognize the robot behavior properly. The results showed that the virtual human was able to recognize robot behaviors and thus predict the robot’s intention with above 95% accuracy level.