Analysis of a pilot study collecting pathologist annotations for validating machine learning algorithms

Abstract

<strong>Purpose:</strong> We describe registration accuracy studies of a custom hardware-software system called eeDAP that registers fields of view (FOVs) of a glass slide on a microscope to the digital presentations of regions of interest (ROIs) of whole slide images (WSI) of the same glass slide. In this manuscript, we describe the results of adding new hardware and use the results to size a larger pathologist data collection study. <br/> <strong>Methods:</strong> We create a registration accuracy task by identifying a visually distinct target. This target will be the center of a WSI ROI and is expected to appear in the center of the microscope FOV. We examined the registration accuracy of 60 ROIs from six slides, alternating registration methods and slide order within each study. We measure the distance between the target and the FOV center (registration error) using an eye piece reticle ruler as the stage moves from target to target. We summarize each error as a success (≤ 5.0 µm) or failure (&gt; 5.0 µm). We completed a multi-reader multicase (MRMC) analysis of the registration successes and failures to estimate the variance components due to the readers and the cases. <br/> <strong> Results:</strong> When using eeDAP in-focus, accuracy was within 5 µm in more than 97% of the FOVs. <br/> <strong>Conclusions:</strong> The eeDAP registration methods were robust to new hardware, and the MRMC analysis has provided variance components for sizing future registration accuracy studies to account for the variability from readers and cases.