Image and Video-Based Autism Spectrum Disorder Detection via Deep Learning

Abstract

People with Autism Spectrum Disorder (ASD) show atypical attention to social stimuli and aberrant gaze when viewing images of the physical world. However, it is unknown how they perceive the world from a first-person perspective. In this study, we used machine learning to classify photos taken in three different categories (people, indoors, and outdoors) as either having been taken by individuals with ASD or by peers without ASD. Our classifier effectively discriminated photos from all three categories but was particularly successful at classifying photos of people with >80% accuracy. Importantly, the visualization of our model revealed critical features that led to successful discrimination and showed that our model adopted a strategy similar to that of ASD experts. Furthermore, for the first time, we showed that photos were taken by individuals with ASD contained less salient objects, especially in the central visual field. Notably, our model outperformed the classification of these photos by ASD experts. Together, we demonstrate an effective and novel method that is capable of discerning photos taken by individuals with ASD and revealing aberrant visual attention in ASD from a unique first-person perspective. Our method may in turn provide an objective measure for evaluations of individuals with ASD. People with ASD also show atypical behavior when they are doing the same action with peers without ASD. However, it is challenging to efficiently extract this feature from spatial and temporal information. In this study, we applied Graph Convolutional Network (GCN) to the 2D skeleton sequence to classify video recording the same action (brush teeth and wash face) as either from individuals with ASD or by peers without ASD. Furthermore, we adopted an adaptive graph mechanism that allows the model to learn a kernel flexibly and exclusively for each layer, which means the model can learn more useful and robust features. Our classifier can effectively reach80% accuracy. Our method may play an important role in the evaluations of individuals with ASD.