From Astronomy to Histology: Adapting the FellWalker Algorithm to Deep Nuclear Instance Segmentation

Abstract

AbstractAccurate cell nuclei segmentation is necessary for subsequent histopathology image analysis, including tumour classification, grading and prognosis. Manually identifying cell nuclei is both difficult and time-consuming, with cell nuclei exhibiting dramatic differences in morphology and staining characteristics. Recently, significant advancements in automatic cell nuclei segmentation have been achieved using deep learning, with methods particularly successful in identifying cell nuclei from background tissue. However, delineating individual cell nuclei remains challenging, with often unclear boundaries between neighbouring nuclei. In this paper, we incorporate the FellWalker algorithm, originally developed for analysing molecular clouds, into a deep learning-based pipeline to perform instance cell nuclei segmentation. We evaluate our proposed method on the Lizard dataset, the largest publicly available nuclear segmentation dataset in digital pathology, and compare it against popular methods such as U-Net with Watershed and Mask R-CNN. Our proposed method consistently outperforms the other methods across dataset sizes, achieving an object Dice of 0.7876, F1 score of 0.8245 and Aggregated Jaccard Index of 0.6526. The flexible nature of our pipeline incorporating the FellWalker algorithm has the potential for broader application in biomedical image instance segmentation tasks.KeywordsDigital pathologyDeep learningImage processingInstance segmentation