A spatial squeeze and multimodal feature fusion attention network for multiple tumor segmentation from PET–CT Volumes

Abstract

Tumor segmentation is a key step in computer-aided diagnosis. The PET–CT co-segmentation method combines the high sensitivity of PET images and the anatomical information of CT images. For whole-body multiple tumors, such as soft tissue sarcoma, lymphoma, etc., due to the different lesion location and size, it is necessary to segment the tumor area according to the whole body anatomical information. How to effectively leverage whole-body contextual information and the fusion of multimodal information is the key to the problem. To address this issue, we propose a spatial squeeze and multimodal feature fusion attention network for whole-body multiple tumors segmentation based on PET–CT volumes. Our proposed method consists of two parts, a Coronal-Spatial Squeeze Attention Extraction Network (CSAE-Net) and a Precise PET–CT Fusion Attention Segmentation Network (PFAS-Net), respectively. In CSAE-Net, we squeeze a 3D PET–CT volume along the coronal plane into m 2D images, and obtain 3D Coronal Spatial Squeeze Attention Volume based on these 2D images. In PFAS-Net, the input is a 2D axial PET–CT slice, and the previously obtained coronal spatial squeeze attention map is used to guide the segmentation. Moreover, a Multimodal Fusion Attention (MFA) module is proposed to fuse the metabolic information of PET and the anatomical information of CT. We perform experiments on PET–CT datasets of two whole-body multiple tumors, Soft Tissue Sarcoma (STS) and Lymphoma. The results show that our proposed method improved Dice values by 8.03% in STS and 1.74% in Lymphoma. Also the visualization results show that our proposed method is able to suppress high-uptake regions of normal tissues.