Accuracy of Integrated Artificial Intelligence Grading Using Handheld Retinal Imaging in a Community Diabetic Eye Screening Program

Abstract

To evaluate mydriatic handheld retinal imaging performance assessed by point-of-care (POC) artificial intelligence (AI) as compared with retinal image graders at a centralized reading center (RC) in identifying diabetic retinopathy (DR) and diabetic macular edema (DME). Prospective, comparative study. Five thousand five hundred eighty-five eyes from 2793 adult patients with diabetes. Point-of-care AI assessment of disc and macular handheld retinal images was compared with RC evaluation of validated 5-field handheld retinal images (disc, macula, superior, inferior, and temporal) in identifying referable DR (refDR; defined as moderate nonproliferative DR [NPDR], or worse, or any level of DME) and vision-threatening DR (vtDR; defined as severe NPDR or worse, or any level of center-involving DME [ciDME]). Reading center evaluation of the 5-field images followed the international DR/DME classification. Sensitivity (SN) and specificity (SP) for ungradable images, refDR, and vtDR were calculated. Agreement for DR and DME; SN and SP for refDR, vtDR, and ungradable images. Diabetic retinopathy severity by RC evaluation: no DR, 67.3%; mild NPDR, 9.7%; moderate NPDR, 8.6%; severe NPDR, 4.8%; proliferative DR, 3.8%; and ungradable, 5.8%. Diabetic macular edema severity by RC evaluation was as follows: no DME (80.4%), non-ciDME (7.7%), ciDME (4.4%), and ungradable (7.5%). Referable DR was present in 25.3% and vtDR was present in 17.5% of eyes. Images were ungradable for DR or DME in 7.5% by RC evaluation and 15.4% by AI. There was substantial agreement between AI and RC for refDR (κ= 0.66) and moderate agreement for vtDR (κ= 0.54). The SN/SP of AI grading compared with RC evaluation was 0.86/0.86 for refDR and 0.92/0.80 for vtDR. This study demonstrates that POC AI following a defined handheld retinal imaging protocol at the time of imaging has SN and SP for refDR that meets the current United States Food and Drug Administration thresholds of 85% and 82.5%, but not for vtDR. Integrating AI at the POC could substantially reduce centralized RC burden and speed information delivery to the patient, allowing more prompt eye care referral. Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.