DeepRNNetSeg: Deep Residual Neural Network for Nuclei Segmentation on Breast Cancer Histopathological Images

Abstract

Nuclear segmentation in the histopathological images is a very important and prerequisite step in computer aided breast cancer grading and diagnosis systems. Nuclei segmentation is very challenging in the complex histopathological images due to uneven color distribution, cell overlapping, variability in size, shape and texture. In this paper, we have developed a deep residual neural network (DeepRNNetSeg) model for automatic nuclei segmentation on the breast cancer histopathological images. DeepRNNetSeg learns high-level of discriminative features of the nuclei from the pixel intensities and produces probability maps. Annotated image mask is applied to the image in order to obtain the image patches, which are then fed to DeepRNNetSeg, which classify each image patches as nuclei or non-nuclei. We evaluate our proposed model on publicly available 143 H&E stain images of estrogen receptor positive (ER+) breast cancer. DeepRNNetSeg model has achieved an improved mean F1-score of 0.8513 and a mean accuracy of 86.87%.