Abstract
Multimodality medical imaging has played a significant role in lesion diagnosis and characterization. However, there are remaining challenges in the procedure of multimodality feature fusion based lesion characterization. First, large inter-modality variations make it difficult to harness the complementary information between modalities for better characterization. Subsequently, large intra-class and small inter-class variations due to the heterogeneity of neoplasm makes the classification more challenging. Finally, the relative importance of modalities for the characterization has not been thoroughly investigated, easily resulting in non-optimal fusion performance. In this study, we propose an attention guided discriminative and adaptive fusion (AGDAF) method based on deep learning architecture to address above three problems. Specifically, we first design a novel cross-modal intra- and inter-attention module to focus on learning both the intra-modality relations and inter-modality relations. Then, we introduce a discriminative feature learning loss to reduce the distance of features in the same class and increase the distance of features in different classes of neoplasm in single modalities. Finally, we design an adaptive weighting strategy to increase the contribution of modalities with relatively lower loss values and reduce the impact of modalities with large loss values for the final loss function. Experimental results of grading clinical hepatocellular carcinoma demonstrate that the proposed method significantly outperforms the previously reported multimodality feature fusion methods. In addition, ablation study also demonstrates the effectiveness of the proposed cross-modal intra- and inter-attention module, discriminative module, and adaptive weight adjustment module for multimodality feature fusion in lesion characterization.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.