PMFN-SSL: Self-supervised learning-based progressive multimodal fusion network for cancer diagnosis and prognosis

Abstract

The integration of digital pathology images and genetic data is a developing field in cancer research, presenting potential opportunities for predicting survival and classifying grades through multiple source data. However, obtaining comprehensive annotations proves challenging in practical medical settings, and the extraction of features from high-resolution pathology images is hindered by inter-domain disparities. Current data fusion methods ignore the spatio-temporal incongruity among multimodal data. To address the above challenges, we propose a novel self-supervised transformer-based pathology feature extraction strategy, and construct an interpretable Progressive Multimodal Fusion Network (PMFN-SSL) for cancer diagnosis and prognosis. Our contributions are mainly divided into three aspects. Firstly, we propose a joint patch sampling strategy based on the information entropy and HSV components of an image, which reduces the demand for sample annotations and avoid image quality degradation caused by manual contamination. Secondly, a self-supervised transformer-based feature extraction module for pathology images is proposed and innovatively leverages partially weakly supervised labeling to align the extracted features with downstream medical tasks. Further, we improve the existing multimodal feature fusion model with an progressive fusion strategy to reduce the inconsistency between multimodal data due to differences in collection of temporal and spatial. Abundant ablation and comparison experiments demonstrate that the proposed data preprocessing method and multimodal fusion paradigm strengthen the quality of feature extraction and improve the prediction based on real cancer grading and prognosis. Code and trained models are made available at: https://github.com/Mercuriiio/PMFN-SSL.