Multi-layer boosting sparse convolutional model for generalized nuclear segmentation from histopathology images

Abstract

It is a challenging problem to achieve generalized nuclear segmentation in digital histopathology images. Existing techniques, using either handcrafted features in learning-based models or traditional image analysis-based approaches, do not effectively tackle the challenging cases, such as crowded nuclei, chromatin-sparse, and heavy background clutter. In contrast, deep networks have achieved state-of-the-art performance in modeling various nuclear appearances. However, their success is limited due to the size of the considered networks. We solve these problems by reformulating nuclear segmentation in terms of a cascade 2-class classification problem and propose a multi-layer boosting sparse convolutional (ML-BSC) model. In the proposed ML-BSC model, discriminative probabilistic binary decision trees (PBDTs) are designed as weak learners in each layer to cope with challenging cases. A sparsity-constrained cascade structure enables the ML-BSC model to improve representation learning. Comparing to the existing techniques, our method can accurately separate individual nuclei in complex histopathology images, and it is more robust against chromatin-sparse and heavy background clutter. An evaluation carried out using three disparate datasets demonstrates the superiority of our method over the state-of-the-art supervised approaches in terms of segmentation accuracy.