Optimizing the cervix cytological examination based on deep learning and dynamic shape modeling

Abstract

The task of segmenting nuclei and cytoplasm in Papanicolau smear images is one of the most challenging tasks in automated cervix cytological analysis owing to the high degree of overlapping, the multiform shape of the cells and their complex structures resulting from inconsistent staining, poor contrast, and the presence of inflammatory cells. This article presents a robust variational segmentation framework based on superpixelwise convolutional neutral network and a learned shape prior enabling an accurate analysis of overlapping cervical mass. The cellular components of Pap image are first classified by automatic feature learning and classification model. Then, a learning shape prior model is employed to delineate the actual contour of each individual cytoplasm inside the overlapping mass. The shape prior is dynamically modeled during the segmentation process as a weighted linear combination of shape templates from an over-complete shape dictionary under sparsity constraints. We provide quantitative and qualitative assessment of the proposed method using two databases of 153 cervical cytology images, with 870 cells in total, synthesised by accumulating real isolated cervical cells to generate overlapping cellular masses with a varying number of cells and degree of overlap. The experimental results have demonstrated that our methodology can successfully segment nuclei and cytoplasm from highly overlapping mass. Our segmentation is also competitive when compared to the state-of-the-art methods.