Automatic assessment of cognitive and emotional states in virtual reality‐based flexibility training for four adolescents with autism

Abstract

AbstractTracking students’ learning states to provide tailored learner support is a critical element of an adaptive learning system. This study explores how an automatic assessment is capable of tracking learners’ cognitive and emotional states during virtual reality (VR)‐based representational‐flexibility training. This VR‐based training program aims to promote the flexibility of adolescents with autism spectrum disorder (ASD) in interpreting, selecting and creating multimodal representations during STEM‐related design problem solving. For the automatic assessment, we used both natural language processing (NLP) and machine‐learning techniques to develop a multi‐label classification model. We then trained the model with the data from a total of audio‐ and video‐recorded 66 training sessions of four adolescents with ASD. To validate the model, we implemented both k‐fold cross‐validations and the manual evaluations by expert reviewers. The study finding suggests the feasibility of implementing the NLP and machine‐learning driven automatic assessment to track and assess the cognitive and emotional states of individuals with ASD during VR‐based flexibility training. The study finding also denotes the importance and viability of providing adaptive supports to maintain learners’ cognitive and affective engagement in a highly interactive digital learning environment.