AR Computer-Assisted Learning for Children with ASD based on Hand Gesture and Voice Interaction

Abstract

For children with autism spectrum disorder (ASD), innovative computer-assisted solutions can provide useful teaching, learning, and support tools. Speech, video, narrative, and a mix of 2D/3D images of virtual objects shown on screens and mobile phones can help to improve the learning experience. Augmented Reality (AR) presents a valuable technology for achieving certain educational objectives. This study presents a gesture and voice-based learning framework for children with ASD that offers interactivity, engagement, and visual support during school/cognitive therapy sessions. The proposed prototype was designed to increase children's engagement and focus during the session and achieve a high level of social interaction and inclusion. It was implemented in accordance with the requirements of specialists from regional rehabilitation centres for children with ASD and structured activities and AR-based games were defined. A multi-baseline feasibility pilot study was carried out with 18 children, aged 2–12 years, diagnosed with ASD and with learning difficulties (concentration and language development problems, social interaction and inclusion), and 3 therapists/parents. Preliminary qualitative and quantitative results show significantly higher engagement and concentration time using the proposed AR-computer system compared to a non-computer condition. In addition to improved receptive vocabulary and social interaction for the children, the prototype reduces teacher workload while engaging the children.