Nuclei instance segmentation using a transformer-based graph convolutional network and contextual information augmentation

Abstract

Nucleus instance segmentation is an important task in medical image analysis involving cell-level pathological analysis and is of great significance for many biomedical applications, such as disease diagnosis and drug screening. However, the high-density and tight-contact between cells is a common feature of most cell images, which poses a great technical challenge for nuclei instance segmentation. The latest research focuses on CNN-based methods for nuclei instance segmentation, which typically rely on bounding box regression and non-maximum suppression to locate nuclei. However, this frequently results in poor local bounding boxes for nuclei that are adhered or clustered together. In response to the challenges of high-density and tight-contact in cellular images, we propose a novel end-to-end nuclei instance segmentation model. Specifically, we first employ the Swin Transformer as the backbone network of our model, which captures global multi-scale information by combining the global modelling capability of transformers and the local modelling capability of convolutional neural networks (CNNs). Additionally, we integrate a graph convolutional feature fusion module (GCFM), that combines deep and shallow features to learn an affinity matrix. The module also adopts graph convolution to guide the network in learning the object-level local information. Finally, we design a hybrid dilated convolution module (HDC) and insert it into the backbone network to enhance the contextual information over a large range. These components assist the network in extracting rich features. The experimental results demonstrate that our algorithm outperforms several state-of-the-art models on the DSB2018 and LIVECell datasets.