A scale and region-enhanced decoding network for nuclei classification in histology image

Abstract

Accurate classification of nuclei in histology images is essential for clinical diagnosis, prognosis, and therapeutic response prediction of cancer. However, this is still a challenging task due to (1) nuclei exhibiting a high level of heterogeneity within different types and (2) large intra-class variability including complex morphology and large variations of scale. To solve these problems, we propose a novel scale and region-enhanced decoding network based on the traditional U-shape structure for nuclei classification. We employ a nuclei detection head as region enhancement module in the decoding branch, which can enhance the nuclear regional information by locating the approximate bounding regions and provide more distinguish information for producing better feature maps of subsequent classification. Then, we propose a scale-aware feature fusion module, which fuses stage-wise feature maps generated from the decoder branch, to effectively learn multi-scale features. Finally, we utilize a scale attention module to calibrate the features and adapt to the most suitable scale in the hybrid multi-scale feature maps. In comparison with several state-of-the-art methods on two publicly available colonic cancer nuclei classification datasets, namely ConSep and Lizard, the proposed method obtains the highest accuracy of 0.860 and 0.927, respectively. It also achieves the highest accuracy of 0.838 on the PanNuke dataset collected from different tissues at different magnitudes. The independent validation on two subsets of the Lizard dataset indicates the proposed method obtains the highest accuracy. In conclusion, the proposed method can greatly improve classification performance, particularly for challenging nuclei with complex contexts and large-scale variations.