A Preliminary Evaluation of the Diagnostic Performance of a Smartphone-Based Machine Learning-Assisted System for Evaluation of Clinical Activity Score in Digital Images of Thyroid-Associated Orbitopathy.

Abstract

Background: We previously developed a machine learning (ML)-assisted system for predicting the clinical activity score (CAS) in thyroid-associated orbitopathy (TAO) using digital facial images taken by a digital single-lens reflex camera in a studio setting. In this study, we aimed to apply this system to smartphones and detect active TAO (CAS ≥3) using facial images captured by smartphone cameras. We evaluated the performance of our system on various smartphone models and compared it with the performance of ophthalmologists with varying clinical experience. Methods: We applied the preexisting ML architecture to classify photos taken with smartphones (Galaxy S21 Ultra, iPhone 12 pro, iPhone 11, iPhone SE 2020, Galaxy M20, and Galaxy A21S). The performance was evaluated with smartphone-captured images from 100 patients with TAO. Three ophthalmology residents, three general ophthalmologists with <5 years of clinical experience, and three oculoplastic specialists independently interpreted the same set of images taken under a studio environment and compared their results with those generated by the smartphone-based ML-assisted system. Reference CAS was determined by a consensus of three oculoplastic specialists. Results: Active TAO (CAS ≥3) was identified in 28 patients. Smartphone model used in capturing facial images influenced active TAO detection performance (F1 score 0.59-0.72). The smartphone-based system showed 74.5% sensitivity, 84.8% specificity, and F1 score 0.70 on top three smartphones. On images from all six smartphones, average sensitivity, specificity, and F1 score were 71.4%, 81.6%, and 0.66, respectively. Ophthalmology residents' values were 69.1%, 55.1%, and 0.46. General ophthalmologists' values were 61.9%, 79.6%, and 0.55. Oculoplastic specialists' values were 73.8%, 90.7%, and 0.75. This smartphone-based ML-assisted system predicted CAS within 1 point of reference CAS in 90.7% using facial images from smartphones. Conclusions: Our smartphone-based ML-assisted system shows reasonable accuracy in detecting active TAO, comparable with oculoplastic specialists and outperforming residents and general ophthalmologists. It may enable reliable self-monitoring for disease activity, but confirmatory research is needed for clinical application.