Multi-task generative adversarial learning for nuclei segmentation with dual attention and recurrent convolution

Abstract

Pathological image is the gold standard for diagnosis and evaluation of cancer. Nuclei segmentation is the basis for quantitative analysis of the pathological image. Although the current deep learning-based nuclei segmentation methods generally perform better than the traditional ones, They are still plagued by over-segmentation and under-segmentation, especially when the nuclei are adherent and overlapping with each other. Therefore, how to effectively distinguish different nuclei has always been a challenging task. In this paper, we proposed a novel segmentation method for nuclei via integrating improved U-Net and generative adversarial learning. By introducing spatial and channel mapping table (SC-MT) attention mechanism, the issues about nuclei over-segmentation and under-segmentation have been alleviated and recurrent convolution units will contribute to the continuity of nuclei contours topology. Extensive experimental results on multiple nuclei segmentation datasets show that the proposed method can effectively distinguish the adherent and overlapping nuclei with robust performance. The code will be available at: https://github.com/antifen/Nuclei-Segmentation.