Learning sign language from a social robot peer by playing an interactive game

Abstract

This work presents a humanoid robot assisted interactive game for Sign Language (SL) tutoring. The game is specially designed for children with communication impairments. The children play the game with a modified Robovie R3 humanoid robot which is able to express a set of chosen words in SL using hand movements, body and face gestures. The robotic platform is specially modified with LEDs in face, additional DOFs in wrist and 5 independent fingers at hands for robust SL generation. The robot is able to communicate with the children by recognizing colored flashcards through a RGB-D camera based system and generating a selected subset of signs, including motivating facial gestures, in return. The presented game is composed of three stages: 1) Introduction of the selected signs to a group of children, 2) Each child's one-to-one play with the robot using flashcards 3) Paper-based test to evaluate the recognition of the signed demonstrated by the robot. Current video presents Screenshots from a preliminary study with a group of hearing-impaired children (7–14 years), where children had almost 100% score in recognizing the robot's signs from a subset of 10 words.