Towards Improving Social Communication Skills With Multimodal Sensory Information

Abstract

How to improve social communication skills for children, especially those with social communication difficulties such as attention deficit/hyperactivity disorder, has long been a challenge faced by researchers and therapists. Recent research indicates that computer-assisted approaches may be effective in addressing this issue. This study aimed to understand children's behaviors and then provide appropriate support to improve their social communication skills. We have established an intelligent system, inside which a child can freely play interactive social skills games with virtual characters. The virtual characters can adjust their own behaviors by adapting to the child's cognitive state (e.g., focus of attention) and affective state (e.g., happiness or surprise). The child's behavior is identified in real-time by recognition of multimodal sensory information, which includes head pose and eye gaze estimation, gesture detection, and affective state detection supported by a series of algorithms proposed in this study. Furthermore, this intelligent system has been enabled in a nonintrusive manner using a novel approach of multicamera surveillance to provide the child with natural interaction with the system. Experimental results show the system can estimate a user's attention and affective states with correctness rates of 93% and 91.3%, respectively. The results obtained suggest that the methods have strong potential as alternative methods for sensing human behavior and providing appropriate support.