Biomedical Text Similarity Evaluation Using Attention Mechanism and Siamese Neural Network

Zhengguang Li,Heng Chen,Huayue Chen

doi:10.1109/access.2021.3099021

Zhengguang Li, Heng Chen + Show 1 more

Open Access

https://doi.org/10.1109/access.2021.3099021

Copy DOI

Abstract

It is a crucial component to estimate the similarity of biomedical sentence pair. Siamese neural network (SNN) can achieve better performance for non-biomedical corpora. However, SNN alone cannot obtain satisfactory biomedical text similarity evaluation results due to syntactic complexity and long sentences. In this paper, a cross self-attention (CSA) is proposed to design a new attention mechanism, namely self2self-attention(S2SA). Then the S2SA is introduced into SNN to construct a novel self2self-attentive siamese neural network, namely S2SA-SNN. In the S2SA-SNN, self-attention is used to learn the different weights of words and complex syntactic features in a single sentence. The means of the CSA are used to learn inherent interactive semantic information between sentences, and it employs self-attention instead of global attention to perform cross attention between sentences. Finally, three biomedical benchmark datasets of Pearson Correlation of 0.66 and 0.72/0.66 on DBMI and CDD-ful/-ref are used to test and prove the effectiveness of the S2SA-SNN. The experiment results show that the S2SA-SNN can achieve better performances with pre-trained word embedding and obtain better generalization ability than other compared methods.

Highlights

More and more medical texts have been accumulated with an amount of biomedical information growing
Siamese neural networks (SNN): our baseline like MaLSTM, but double bi-directional long short-term memory (biLSTM) in each branch network are employed in the model
A cross self-attention is proposed, which is integrated with self-attention for designing a novel hybrid attention mechanism, namely self2self-attention mechanism

Summary

Introduction

More and more medical texts have been accumulated with an amount of biomedical information growing. Some researchers utilized biomedical resources [3] or corpus [4] to improve the evaluation similarity performance, the generalization of these methods is poor due to the limitation of resources and corpus. Some researchers utilized word embedding [7], sentence embedding [8][9] and shared sentence encoder [10] to obtain sentence semantic representation and estimate the similarity. The attention mechanisms[12] are integrated with SNN to focus on crucial words. These words have an important impact on the sentence semantic representation. These neural networks with attention mechanisms achieve good results, but they ignored the importance of interactive information between sentences. The interaction contributed to enhance the semantic information of two sentences, and promise the semantic similarity estimation performance

Methods

Results

Conclusion