6.2 Evaluation of an Artificial Intelligence–Based Medical Device Designed to Aid in the Diagnosis of Autism Spectrum Disorder in the Primary Care Setting

Abstract

The lack of diagnostic tools for autism spectrum disorder (ASD) in primary care settings and long waiting lists for specialist assessment contribute to an average delay of 3 years between first parental concern and diagnosis. This study examined the performance of an artificial intelligence–based device intended to aid primary care practitioners (PCPs) in the diagnosis of ASD. This was a prospective multisite pivotal study conducted in 6 states using a double-blind active comparator design with 425 completed subjects (36% female) ages 18 to 72 months with concern for developmental delay. Previous research developed, tuned, and tested a device that uses a gradient-boosted decision-tree machine learning algorithm that analyzes 64 behavioral features from 3 distinct inputs: 1) caregiver questionnaire; 2) 2 to 4 minutes of home videos analyzed by trained video analysts; and 3) PCP questionnaire. Device results were compared to diagnosis by independent agreement of specialist clinicians based on clinical assessment, including a modified Child Autism Rating Scale (CARS-2) and DSM-5 criteria. Specialists were child and adolescent psychiatrists, child psychologists, pediatric neurologists, and developmental behavioral pediatricians experienced in diagnosing ASD. Comparison of device results to specialist diagnosis found the following for subjects with determinate device results: positive predictive value (PPV) = 80.8% (95% CI, 70.3-88.8); negative predictive value (NPV) = 98.3% (90.6-100); sensitivity = 98.4% (91.6-100); and specificity = 78.9% (67.6-87.7). There was no evidence that device performance significantly varied when the PCP used the device remotely compared to in-person. Using this device, PCPs could efficiently, accurately, and equitably diagnose a subset of children aged 18 to 72 months old, thereby streamlining specialist referrals and facilitating earlier ASD diagnosis and interventions. The results further provide preliminary evidence that PCP evaluation of the child can be done via telemedicine or in-person with no degradation in device performance.