Clustered nuclei splitting based on recurrent distance transform in digital pathology images

Abstract

The accuracy of the applied technique for automated nuclei segmentation is critical in obtaining high-quality and efficient diagnostic results. Unfortunately, multiple objects in histopathological images are connected (clustered) and frequently counted as one. In this study, we present a new method for cluster splitting based on distance transform binarized with the recurrently increased threshold value and modified watershed algorithm. The proposed method treats clusters separately, splitting them into smaller sub-clusters and conclusively into separate objects, based solely on the shape feature, making it independent of the pixel intensity. The efficiency of these algorithms is validated based on the labeled set of images from two datasets: BBBC004v1 and breast cancer tissue microarrays. Results of initial nuclei detection were significantly improved by applying the proposed algorithms. Our approach outperformed the state-of-the-art techniques based on recall, precision, F1-score, and Jaccard index. The proposed method achieves very low amount of under-segmented, as well as over-segmented objects. In summary, we provide novel and efficient method for dividing the clustered nuclei in digital images of histopathological slides.