Statistical framework for validation without ground truth of choroidal thickness changes detection

Tiziano Ronchetti,Christoph Meier,Selim Orgül,Alina T Giger,Peter M Maloca,Chi-Ho To,Christoph Jud,Philippe C Cattin,Rachel Ka Man Chun,Quan Liu,Hendrik P N Scholl,Boris Považay,Sanjoy Bhattacharya

doi:10.1371/journal.pone.0218776

Abstract

Monitoring subtle choroidal thickness changes in the human eye delivers insight into the pathogenesis of various ocular diseases such as myopia and helps planning their treatment. However, a thorough evaluation of detection-performance is challenging as a ground truth for comparison is not available. Alternatively, an artificial ground truth can be generated by averaging the manual expert segmentations. This makes the ground truth very sensitive to ambiguities due to different interpretations by the experts. In order to circumvent this limitation, we present a novel validation approach that operates independently from a ground truth and is uniquely based on the common agreement between algorithm and experts. Utilizing an appropriate index, we compare the joint agreement of several raters with the algorithm and validate it against manual expert segmentation. To illustrate this, we conduct an observational study and evaluate the results obtained using our previously published registration-based method. In addition, we present an adapted state-of-the-art evaluation method, where a paired t-test is carried out after leaving out the results of one expert at the time. Automated and manual detection were performed on a dataset of 90 OCT 3D-volume stack pairs of healthy subjects between 8 and 18 years of age from Asian urban regions with a high prevalence of myopia.

Highlights

The choroid is a vascular structure located at the posterior part of the uveal tract in the eye, between the relative rigid sclera and the more flexible, light-sensitive retina
The choroidal thickness is defined as the vertical distance between the Bruch’s Membrane (BM) and the Choroid-Sclera Interface (CSI), see Fig 1(b)
This is due to the low contrast in the image acquisition in this area of the choroid which makes a clear identification of the CSI difficult

Summary

Introduction

The choroid is a vascular structure located at the posterior part of the uveal tract in the eye, between the relative rigid sclera and the more flexible, light-sensitive retina. Longitudinal studies using OCT imaging offer a unique possibility to gain insight into the dynamics of anatomical changes in the retina and choroid. This leads to an understanding of the mechanisms regulating such processes. Due to the lack of alternative high precision in-vivo measurement methodologies, signals are typically compared to their histological equivalent This proves to be notoriously difficult, subjective, and unreliable in view of the large amount of data points and the weak signals that are frequently hard to interpret for the human observer [14]. Automated detection of noisy and speckled OCT-images at the low-signal-end of the depth scan already has a long history and usually focuses on segmentation by delineating borders that are associated with large scale changes of the refractive index, or by determining tissue texture appearance

Objectives

Methods

Results

Conclusion