Objective or PurposeTo compare the performance of three phenotyping methods in identifying diabetic retinopathy (DR) and related clinical conditions. DesignThree phenotyping methods were used to identify clinical conditions including unspecified DR, non-proliferative diabetic retinopathy (NPDR) (mild, moderate, severe), consolidated NPDR (unspecified DR or any NPDR), proliferative diabetic retinopathy (PDR), diabetic macular edema (DME), vitreous hemorrhage (VH), retinal detachment (RD) (tractional RD or combined tractional and rhegmatogenous RD), and neovascular glaucoma (NVG). The first method used only International Classification of Diseases, 10th Revision (ICD-10) diagnosis codes (ICD-10 Lookup System). The next two methods used a Bidirectional Encoder Representations from Transformers (BERT) with a dense Multilayer Perceptron (MLP) output layer natural language processing (NLP) framework. The NLP framework was applied either to free-text of provider notes (Text-Only NLP System) or both free-text and ICD-10 diagnosis codes (Text-and-ICD NLP System). Subjects, Participants, and/or ControlsAdults ≥18 years with diabetes mellitus seen at the Wilmer Eye Institute. Methods, Intervention or TestingWe compared the performance of the three phenotyping methods in identifying the diabetic retinopathy related conditions with gold standard chart review. We also compared the estimated disease prevalence using each method. Main Outcome MeasuresPerformance of each method was reported as the macro F1 score. The agreement between the methods was calculated using the kappa statistic. Prevalence estimates were also calculated for each method. ResultsA total of 91,097 patients and 692,486 office visits were included in the study. Compared to the gold standard, the Text-and-ICD NLP System had the highest F1 score for most clinical conditions (range 0.39 to 0.64). The agreement between the ICD-10 Lookup System and Text-Only NLP System varied (kappa of 0.21 to 0.81). The prevalence of diabetic retinopathy and related conditions ranged from 1.1% for NVG to 17.9% for DME (using the Text-and-ICD NLP System). ConclusionsThe prevalence of diabetic retinopathy and related conditions varied significantly depending on the methodology of identifying cases. The best performing phenotyping method was the Text-and-ICD NLP System that used information in both diagnosis codes as well as free-text notes.