Deep scaled dot-product attention based domain adaptation model for biomedical question answering

Abstract

Biomedical text mining is becoming increasingly important as the number of biomedical documents grow rapidly. Deep learning has boosted the development of biomedical text mining models. However, as deep learning models require a large amount of training data, a hierarchical attention based transfer learning model is proposed in this paper for the question answering task in biomedical field which lacks of sufficient training data. We adopt BERT (Bidirectional Encoder Representation Transformers), which has the ability to learn from large-scale unsupervised data, to enrich the semantic representation in our model. Especially, the scaled dot-product attention mechanism captures the question interaction clues for passage encoding. The domain adaptation technique of fine-tuning is used to reinforce the performance, which penalizes the deviations from the source model's parameters and remembers the knowledge of source domain. We evaluate the system performance on the open data set of BioASQ-Task B. The results show that our system achieves the state-of-the-art performance without any handcrafted features and outperforms the best solution for factoid questions in 2016 and 2017 BioASQ-Task B.