Multi-view orientational attention network combining point-based affinity for polyp segmentation

Abstract

Most existing deep learning-based polyp segmentation methods neglect two important aspects of polyps: the geometric orientation information of polyps and the point information of the entire colonoscopy area. In this paper, we introduce a multi-view orientational attention network (MVOA-Net), which incorporates orientation and point awareness to effectively address the issue of intra-class inconsistency resulting from variations in polyp shape, size, and position, as well as the inter-class indistinction caused by the high similarity between polyp lesions and surrounding tissues. To achieve robust orientation awareness, we propose a novel geometric orientation transformer encoder (GOTE) based on horizontal and vertical views. Moreover, To simultaneously capture the global context information of GOTE and emphasize the important local information of the convolution-based attention encoder (CBAE), a global and local cross attention fusion module (CAFM) is also proposed to simultaneously model the long-range dependencies of polyps and pay sufficient attention to the local boundaries of polyps. Additionally, a efficient atrous spatial pyramid pooling (E-ASPP) module is proposed to enhance the semantic representation of high-level features. Finally, a point-based affinity module (PBAM) and a multi-scale fusion module (MSFM) are proposed to distinguish the disguise of polyps, further alleviating inter-class indistinction. The ablation study results demonstrate the effectiveness of each component. Quantitative and qualitative experimental results show that MVOA-Net achieves the best segmentation accuracy across domain polyp datasets and has obvious advantages in segmenting multiple polyp objects.