Applying Transformer Models for Disease Named Entity Recognition

Abstract

Disease Named Entity Recognition (DNER) is an emerging field of research that identifies disease mentions in health-related texts. Downstream tasks of DNER includes exploring the cause of a given disease, analysis of relationships among diseases, prevention mechanisms and specialized treatment procedures. Various models have been proposed for DNER, including dictionary-based approaches, rule-based systems, machine learning models with handcrafted features and deep learning models. Though deep learning models perform better for DNER, they highly depend on large annotated datasets, which is often not feasible. Multi-task approaches that combine manually annotated datasets with similar domains boost the performance of DNER; however, it is often difficult to find the combination of datasets that give a performance gain. This study addresses such drawbacks by investigating the applicability of transfer learning via Bidirectional Encoder Representations from Transformers (BERT) and proposes a model using BERT and Conditional Random Fields (CRF), which performs better than similar state-of-the-art models based on general deep learning and multi-task learning methods.