Named Entity Recognition Using Transfer Learning with the Fusion of Pre-trained SciBERT Language Model and Bi-directional Long Short Term Memory

Abstract

The foundation of the study of medical texts and one of the primary information extraction sub-activities is biomedical named entity recognition (BioNER) which brings out important named structures such as diseases, gene, protein, species etc. from biomedical text. Although the BioNER task is fairly comparable to regular NER, it is more difficult to identify biological named entities because of their unique nomenclatural features. In this regard, we present an architecture for neural network using transfer learning with the fusion of pre-trained SciBERT language model and bidirectional long short term memory (BiLSTM). First, the paradigm uses SciBERT, which performs better on biomedical tasks than BERT base, and a Bi-LSTM network in order to gather more comprehensive context data. Ultimately a conditional random field (CRF) layer combined with Bi-LSTM is used to model the dependency between each state and the entire input sequences. At the end, we have evaluated our system on a benchmark dataset named NCBI-Disease corpus. The suggested model has a competitive effectiveness for the NCBI-Disease possessing a 91.00% f1-score, 89.00% precision, 93.00% recall, and 98.00% accuracy. Therefore, with the usage of pre-trained SciBERT model and no feature engineering, our methodology can improve the neural network’s capacity to extract significant information.