Inter-observer Reliability and Agreement Study on Early Diagnosis of Diabetic Retinopathy and Diabetic Macular Edema Risk

Abstract

The degree of inter-observer agreement on early diagnosis of diabetic retinopathy (DR) and diabetic macular edema (DME) risk has been assessed in this paper. Three sets of DR and DME risk ratings on 529 diabetic patients were independently built by ophthalmologists of the Andalusian (Spain) Health Service through observation of two macula-centered retinographies from these patients (one image per eye, 1058 images). DR was graded on a 0–3 scale from DR-unrelated to severe DR, while DME risk was graded on a 0–2 scale from no risk to moderate-severe risk. Inter-rater reliability (IRR) assessment was performed by the intra-class correlation (ICC) and two kappa-like statistical variants —Light’s kappa and Fleiss’ kappa. ICC-computed IRR showed excellent agreement between our three coders: values were 0.844 (95 % CI, 0.822–0.865) and 0.833 (95 % CI, 0.805–0.853) for DR and DME ratings, respectively. Kappa index-quantified assessment resulted in substantial agreement, as both kappa indexes rendered values around 0.60 for DR and 0.75 for DME ratings. All computed IRR metrics proved high inter-observer agreement and consistency among DR degree and DME risk diagnoses. Reliable diagnosis provided by human experts supports the generation of reference standards that can be used in the development of automatic DR diagnosis systems.