Named Entity Recognition Model Research Articles

Online biomedical named entities recognition by data and knowledge-driven model

Named entity recognition (NER) is an important task for the natural language processing of biomedical text. Currently, most NER studies standardized biomedical text, but NER for unstandardized biomedical text draws less attention from researchers. Named entities in online biomedical text exist with errors and polymorphisms, which negatively impact NER models’ performance and impede support from knowledge representation methods. In this paper, we propose a neural network method that can effectively recognize entities in unstandardized online medical/health text. We introduce a new pre-training scheme that uses large-scale online question-answering pairs to enhance transformers’ model capacity on online biomedical text. Moreover, we supply models with knowledge representations from a knowledge base called multi-channel knowledge labels, and this method overcomes the restriction from languages, like Chinese, that require word segmentation tools to represent knowledge. Our model outperforms other baseline methods significantly in experiments on a dataset for Chinese online medical entity recognition and achieves state-of-the-art results.

Automatic de-identification of French electronic health records: a cost-effective approach exploiting distant supervision and deep learning models

BackgroundElectronic health records (EHRs) contain valuable information for clinical research; however, the sensitive nature of healthcare data presents security and confidentiality challenges. De-identification is therefore essential to protect personal data in EHRs and comply with government regulations. Named entity recognition (NER) methods have been proposed to remove personal identifiers, with deep learning-based models achieving better performance. However, manual annotation of training data is time-consuming and expensive. The aim of this study was to develop an automatic de-identification pipeline for all kinds of clinical documents based on a distant supervised method to significantly reduce the cost of manual annotations and to facilitate the transfer of the de-identification pipeline to other clinical centers.MethodsWe proposed an automated annotation process for French clinical de-identification, exploiting data from the eHOP clinical data warehouse (CDW) of the CHU de Rennes and national knowledge bases, as well as other features. In addition, this paper proposes an assisted data annotation solution using the Prodigy annotation tool. This approach aims to reduce the cost required to create a reference corpus for the evaluation of state-of-the-art NER models. Finally, we evaluated and compared the effectiveness of different NER methods.ResultsA French de-identification dataset was developed in this work, based on EHRs provided by the eHOP CDW at Rennes University Hospital, France. The dataset was rich in terms of personal information, and the distribution of entities was quite similar in the training and test datasets. We evaluated a Bi-LSTM + CRF sequence labeling architecture, combined with Flair + FastText word embeddings, on a test set of manually annotated clinical reports. The model outperformed the other tested models with a significant F1 score of 96,96%, demonstrating the effectiveness of our automatic approach for deidentifying sensitive information.ConclusionsThis study provides an automatic de-identification pipeline for clinical notes, which can facilitate the reuse of EHRs for secondary purposes such as clinical research. Our study highlights the importance of using advanced NLP techniques for effective de-identification, as well as the need for innovative solutions such as distant supervision to overcome the challenge of limited annotated data in the medical domain.

ATBBC: Named entity recognition in emergency domains based on joint BERT-BILSTM-CRF adversarial training

With the rapid growth of Internet penetration, identifying emergency information from network news has become increasingly significant for emergency monitoring and early warning. Although deep learning models have been commonly used in Chinese Named Entity Recognition (NER), they require a significant amount of well-labeled training data, which is difficult to obtain for emergencies. In this paper, we propose an NER model that combines bidirectional encoder representations from Transformers (BERT), bidirectional long-short-term memory (BILSTM), and conditional random field (CRF) based on adversarial training (ATBBC) to address this issue. Firstly, we constructed an emergency dataset (ED) based on the classification and coding specifications of the national emergency platform system. Secondly, we utilized the BERT pre-training model with adversarial training to extract text features. Finally, BILSTM and CRF were used to predict the probability distribution of entity labels and decode the probability distribution into corresponding entity labels.Experiments on the ED show that our model achieves an F1-score of 85.39% on the test dataset, which proves the effectiveness of our model.

Building knowledge graphs from technical documents using named entity recognition and edge weight updating neural network with triplet loss for entity normalization

Attempts to express information from various documents in graph form are rapidly increasing. The speed and volume in which these documents are being generated call for an automated process, based on machine learning techniques, for cost-effective and timely analysis. Past studies responded to such needs by building knowledge graphs or technology trees from the bibliographic information of documents, or by relying on text mining techniques in order to extract keywords and/or phrases. While these approaches provide an intuitive glance into the technological hotspots or the key features of the select field, there still is room for improvement, especially in terms of recognizing the same entities appearing in different forms so as to interconnect closely related technological concepts properly. In this paper, we propose to build a patent knowledge network using the United States Patent and Trademark Office (USPTO) patent filings for the semiconductor device sector by fine-tuning Huggingface’s named entity recognition (NER) model with our novel edge weight updating neural network. For the named entity normalization, we employ edge weight updating neural network with positive and negative candidates that are chosen by substring matching techniques. Experiment results show that our proposed approach performs very competitively against the conventional keyword extraction models frequently employed in patent analysis, especially for the named entity normalization (NEN) and document retrieval tasks. By grouping entities with named entity normalization model, the resulting knowledge graph achieves higher scores in retrieval tasks. We also show that our model is robust to the out-of-vocabulary problem by employing the fine-tuned BERT NER model.

A Study of Entity Relationship Extraction Algorithms Based on Symmetric Interaction between Data, Models, and Inference Algorithms

The purpose of this paper is to address the extraction of entities and relationships from unstructured Chinese text, with a particular emphasis on the challenges of Named Entity Recognition (NER) and Relation Extraction (RE). This will be achieved by integrating external lexical information and utilizing the abundant semantic information available in Chinese. We utilize a pipeline model that is applied separately to NER and RE by introducing an innovative NER model that integrates Chinese pinyin, characters, and words to enhance recognition capabilities. Simultaneously, we incorporate information such as entity distance, sentence length, and part-of-speech to improve the performance of relation extraction. We also delve into the interactions among data, models, and inference algorithms to improve learning efficiency in addressing this challenge. In comparison to existing methods, our model has achieved significant results.

Extracting Symptoms of Agitation in Dementia from Free-Text Nursing Notes Using Advanced Natural Language Processing.

Nursing staff record observations about older people under their care in free-text nursing notes. These notes contain older people's care needs, disease symptoms, frequency of symptom occurrence, nursing actions, etc. Therefore, it is vital to develop a technique to uncover important data from these notes. This study developed and evaluated a deep learning and transfer learning-based named entity recognition (NER) model for extracting symptoms of agitation in dementia from the nursing notes. We employed a Clinical BioBERT model for word embedding. Then we applied bidirectional long-short-term memory (BiLSTM) and conditional random field (CRF) models for NER on nursing notes from Australian residential aged care facilities. The proposed NER model achieves satisfactory performance in extracting symptoms of agitation in dementia with a 75% F1 score and 78% accuracy. We will further develop machine learning models to recommend the optimal nursing actions to manage agitation.

A template augmented distant supervision framework for Chinese named entity recognition

Distant supervision has been proven to be an efficient way of generating labeled instances for Named Entity Recognition (NER). However, it suffers from dictionary biases and ambiguous entities, resulting in noisy and incomplete labels. To overcome this drawback, this paper proposes a template augmented distant supervision framework, which generates high-quality labeled training data with minimal human effort. Specifically, we use distant supervision to extract sentences that contain entities and apply a pre-trained language model to encode these sentences. The encoded sentences are clustered and then for each cluster, three sentences are sampled out to form a seed template pool. The seed templates are calibrated and decomposed to decouple the connection between different parts. Finally, the seed templates and entity dictionary are combined with pre-trained language model to generate semantically coherent and precisely labeled training data. Experimental results on the EC and NEWS datasets and a practical electronic after-sale Q&A dataset with multiple pre-trained language models demonstrate that the proposed framework is able to improve the F1 score of the distantly supervised NER models by 7.9%–12.9%.

An Approach to a Linked Corpus Creation for a Literary Heritage Based on the Extraction of Entities from Texts

Working with the literary heritage of writers requires the studying of a large amount of materials. Finding them can take a considerable amount of time even when using search engines. The solution to this problem is to create a linked corpus of literary heritage. Texts in such a corpus will be united by common entities, which will make it possible to select texts not only by the occurrence of certain phrases in a query but also by common entities. To solve this problem, we propose the use of a Named Entity Recognition model trained on examples from a corpus of texts and a database structure for storing connections between texts. We propose to automate the process of creating a dataset for training a BERT-based NER model. Due to the specifics of the subject area, methods, techniques, and strategies are proposed to increase the accuracy of the model trained with a small set of examples. As a result, we created a dataset and a model trained on it which showed high accuracy in recognizing entities in the text (the average F1-score for all entity types is 0.8952). The database structure provides for the storage of unique entities and their relationships with texts and a selection of texts based on the entities. The method was tested for a corpus of texts from the literary heritage of Alexander Sergeevich Pushkin, which is also a difficult task due to the specifics of the Russian language.

Research on Chinese Nested Entity Recognition Based on IDCNNLR and GlobalPointer

The task of named entity recognition (NER) is to identify entities in the text and predict their categories. In real-life scenarios, the context of the text is often complex, and there may exist nested entities within an entity. This kind of entity is called a nested entity, and the task of recognizing entities with nested structures is referred to as nested named entity recognition. Most existing NER models can only handle flat entities, and there has been limited research progress in Chinese nested named entity recognition, resulting in relatively few models in this direction. General NER models have limited semantic extraction capabilities and cannot capture deep semantic information between nested entities in the text. To address these issues, this paper proposes a model that uses the GlobalPointer module to identify nested entities in the text and constructs the IDCNNLR semantic extraction module to extract deep semantic information. Furthermore, multiple-head self-attention mechanisms are incorporated into the model at multiple positions to achieve data denoising, enhancing the quality of semantic features. The proposed model considers each possible entity boundary through the GlobalPointer module, and the IDCNNLR semantic extraction module and multi-position attention mechanism are introduced to enhance the model’s semantic extraction capability. Experimental results demonstrate that the proposed model achieves F1 scores of 69.617% and 79.285% on the CMeEE Chinese nested entity recognition dataset and CLUENER2020 Chinese fine-grained entity recognition dataset, respectively. The model exhibits improvement compared to baseline models, and each innovation point shows effective performance enhancement in ablative experiments.

Named entity recognition in aerospace based on multi-feature fusion transformer

In recent years, along with the rapid development in the domain of artificial intelligence and aerospace, aerospace combined with artificial intelligence is the future trend. As an important basic tool for Natural Language Processing, Named Entity Recognition technology can help obtain key relevant knowledge from a large number of aerospace data. In this paper, we produced an aerospace domain entity recognition dataset containing 30 k sentences in Chinese and developed a named entity recognition model that is Multi-Feature Fusion Transformer (MFT), which combines features such as words and radicals to enhance the semantic information of the sentences. In our model, the double Feed-forward Neural Network is exploited as well to ensure MFT better performance. We use our aerospace dataset to train MFT. The experimental results show that MFT has great entity recognition performance, and the F1 score on aerospace dataset is 86.10%.

Utilising crowdsourcing and text mining to enhance information extraction from social media: A case study in handling COVID-19 supply requests in Thailand

Social media platforms are critical for disaster communication and relief efforts. Rapid and precise social media post analysis is required for effective disaster response. This article presents a comprehensive study of a framework that combines crowdsourcing and text mining techniques to enhance data extraction from social media. The research focuses on a particular case study of COVID-19 pandemic medical supply request, which shows several key findings. First, the incorporation of domain-specific data during the training of named entity recognition (NER) models is essential for accurately identifying and retrieving important entities, such as the names of medical supplies and hospitals. Second, the implementation of a hybrid system leads to improvement in the extraction of information from social media posts. Finally, the involvement of crowdsourcing is found to be significant in the validation, verification, and filtering of disorganised information within the hybrid system. Our performance analysis demonstrates that the use of hybrid models has the potential to significantly improve the extraction of supply names (by up to 37%) and hospital names (by up to 66%), especially in the absence of a comprehensive vocabulary or specially trained NER models. During the COVID-19 supply shortage in Thailand, volunteers utilised hybrid models to expedite the identification of the necessary information. Experiment results demonstrated significant improvement in the accuracy of extracted data, the ability to acquire relevant information in real-time, the capacity to handle a substantial number of posts and the practical benefit of the proposed framework.

Text mining for contexts and relationships in cancer genomics literature.

Scientific advances build on the findings of existing research. The 2001 publication of the human genome has led to the production of huge volumes of literature exploring the context-specific functions and interactions of genes. Technology is needed to perform large-scale text mining of research papers to extract the reported actions of genes in specific experimental contexts and cell states, such as cancer, thereby facilitating the design of new therapeutic strategies. We present a new corpus and Text Mining methodology that can accurately identify and extract the most important details of cancer genomics experiments from biomedical texts. We build a Named Entity Recognition model that accurately extracts relevant experiment details from PubMed abstract text, and a second model that identifies the relationships between them. This system outperforms earlier models and enables the analysis of gene function in diverse and dynamically evolving experimental contexts. Code and data are available here: https://github.com/cambridgeltl/functional-genomics-ie.

Context-Aware Attentive Multilevel Feature Fusion for Named Entity Recognition.

In the era of information explosion, named entity recognition (NER) has attracted widespread attention in the field of natural language processing, as it is fundamental to information extraction. Recently, methods of NER based on representation learning, e.g., character embedding and word embedding, have demonstrated promising recognition results. However, existing models only consider partial features derived from words or characters while failing to integrate semantic and syntactic information, e.g., capitalization, inter-word relations, keywords, and lexical phrases, from multilevel perspectives. Intuitively, multilevel features can be helpful when recognizing named entities from complex sentences. In this study, we propose a novel attentive multilevel feature fusion (AMFF) model for NER, which captures the multilevel features in the current context from various perspectives. It consists of four components to, respectively, capture the local character-level (CL), global character-level (CG), local word-level (WL), and global word-level (WG) features in the current context. In addition, we further define document-level features crafted from other sentences to enhance the representation learning of the current context. To this end, we introduce a novel context-aware attentive multilevel feature fusion (CAMFF) model based on AMFF, to fully leverage document-level features from all the previous inputs. The obtained multilevel features are then fused and fed into a bidirectional long short-term memory (BiLSTM)-conditional random field (CRF) network for the final sequence labeling. Extensive experiments on four benchmark datasets demonstrate that our proposed AMFF and CAMFF models outperform a set of state-of-the-art baseline methods and the features learned from multiple levels are complementary.

Automatic Detection of Distant Metastasis Mentions in Radiology Reports in Spanish.

A critical task in oncology is extracting information related to cancer metastasis from electronic health records. Metastasis-related information is crucial for planning treatment, evaluating patient prognoses, and cancer research. However, the unstructured way in which findings of distant metastasis are often written in radiology reports makes it difficult to extract information automatically. The main aim of this study was to extract distant metastasis findings from free-text imaging and nuclear medicine reports to classify the patient status according to the presence or absence of distant metastasis. We created a distant metastasis annotated corpus using positron emission tomography-computed tomography and computed tomography reports of patients with prostate, colorectal, and breast cancers. Entities were labeled M1 or M0 according to affirmative or negative metastasis descriptions. We used a named entity recognition model on the basis of a bidirectional long short-term memory model and conditional random fields to identify entities. Mentions were subsequently used to classify whole reports into M1 or M0. The model detected distant metastasis mentions with a weighted average F1 score performance of 0.84. Whole reports were classified with an F1 score of 0.92 for M0 documents and 0.90 for M1 documents. These results show the usefulness of the model in detecting distant metastasis findings in three different types of cancer and the consequent classification of reports. The relevance of this study is to generate structured distant metastasis information from free-text imaging reports in Spanish. In addition, the manually annotated corpus, annotation guidelines, and code are freely released to the research community.

Natural Language Processing Applied to Forensics Information Extraction With Transformers and Graph Visualization

Digital forensics analysis is a slow process mainly due to the large amount and variety of data. Some forensic tools help categorize files by type and allow automatization of tasks, like named entity recognition (NER). NER is a key component in many natural language processing (NLP) applications, such as relation extraction (RE) and information retrieval. The introduction of neural networks and transformer architectures in the last few years made it possible to develop more accurate models in different languages. This work proposes a reproducible setup to build a forensic pipeline for information extraction using NLP of texts. Our results show that it is possible to develop both NER and RE models in any language and tune its hyper-parameters to achieve state-of-art performance and build comprehensive knowledge graphs, decreasing the amount of time required for human supervision and review. We also find that solving this task in phases can further improve the performance, not only for digital investigation applications, but also for general-purpose information extraction and analysis.

Active Learning with Pre-trained Language Models for Named Entity Recognition in Requirements Engineering

The ever-increasing number of textual requirements and the handling of these consumes significant time in product design. Therefore, automating requirements analysis with natural language processing is critical to increase efficiency. Pre-trained language models have proven useful for requirements analysis. However, to deploy pre-trained language models, a fully labeled dataset must first be created to finetune language models towards the requirements domain. The creation of a fully labeled dataset, especially for named entity recognition (NER) applications, poses a major challenge as it entails high manual labeling effort by requirements engineers. Furthermore, in an ever-changing requirements environment, the manual labeling process must be repeated to avoid model drift, adding additional workload. Recent advances of pre-trained language models combined with active learning can significantly reduce the manual labeling effort. Thus, we present a novel approach for leveraging active learning with pre-trained language models for NER. We demonstrate the efficacy of this combination in an experiment, where we are able to reduce the manual requirements labeling effort by 74% while even improving model performance. We argue that by reducing the manual effort, active learning can shorten the ramp-up time of language model deployment for requirements engineering and enable industrial adoption.

Partial Annotation Learning for Biomedical Entity Recognition.

Named Entity Recognition (NER) is a key task to support biomedical research. In Biomedical Named Entity Recognition (BioNER), obtaining high-quality expert annotated data is laborious and expensive, leading to the development of automatic approaches such as distant supervision. However, manually and automatically generated data often suffer from the unlabeled entity problem, whereby many entity annotations are missing, degrading the performance of full annotation NER models. To conquer this issue, we undertake a systematic exploration of the efficacy of partial annotation learning methods for BioNER, which encompasses a comprehensive evaluation conducted across a spectrum of distinct simulated scenarios of missing entity annotations. Furthermore, we propose a TS-PubMedBERT-Partial-CRF partial annotation learning model. We standardize a compilation of 16 BioNER corpora, encompassing a range of five distinct entity types, to establish a gold standard. And we compare against the state-of-the-art partial annotation model EER-PubMedBERT, the widely acknowledged partial annotation model BiLSTM-Partial-CRF model, and the state-of-the-art full annotation learning BioNER model PubMedBERT tagger. Results show that partial annotation learning-based methods can effectively learn from biomedical corpora with missing entity annotations. Our proposed model outperforms alternatives and, specifically, the PubMedBERT tagger by 38% in F1-score under high missing entity rates. Moreover, the recall of entity mentions in our model demonstrates a competitive alignment with the upper threshold observed on the fully annotated dataset. We have published our data, source code and training records at https://github.com/possible1402/partial\_annotation\_learning.

RoBGP: A Chinese Nested Biomedical Named Entity Recognition Model Based on RoBERTa and Global Pointer

RoBGP: A Chinese Nested Biomedical Named Entity Recognition Model Based on RoBERTa and Global Pointer

Big Data Knowledge Graph of Charging Safety Influencing Factors and Database Construction Method of Safety Features

Abstract In this paper, we utilize big data to screen relevant data on charging safety influencing factors and perform data cleaning to constitute a charging safety influencing factors dataset. BERT is selected as the baseline model for the named entity recognition task, together with the CRF model, to exclude irrelevant features, resulting in an effective model for entity recognition in line with the knowledge graph. Introducing a security database, a graph attention network model that simultaneously obtains the structural features and textual description features of the security knowledge graph is proposed to improve the performance of knowledge graph relationship extraction. The dataset of high-frequency charging security composition, as well as the random dataset, are used as experimental samples, respectively, to compare and analyze the performance of the BERT-CRF named entity recognition model in terms of each index. The link prediction evaluation task is evaluated using the structure- and text-based graph attention network model, and experimental analysis is carried out using three benchmark models. From the overall results of the test, it can be seen that the BERT-CRF model learns 90% of the lexicon’s knowledge and passes the model test by keeping each evaluation metric in the range of 0.9 to 1.0 under the large data volume experimental environment. The proposed graph attention network model, which uses structure and text, has a better link prediction performance than other models and performs better in the FB15K-237 dataset.

Supporting the working life exposome: Annotating occupational exposure for enhanced literature search.

An individual's likelihood of developing non-communicable diseases is often influenced by the types, intensities and duration of exposures at work. Job exposure matrices provide exposure estimates associated with different occupations. However, due to their time-consuming expert curation process, job exposure matrices currently cover only a subset of possible workplace exposures and may not be regularly updated. Scientific literature articles describing exposure studies provide important supporting evidence for developing and updating job exposure matrices, since they report on exposures in a variety of occupational scenarios. However, the constant growth of scientific literature is increasing the challenges of efficiently identifying relevant articles and important content within them. Natural language processing methods emulate the human process of reading and understanding texts, but in a fraction of the time. Such methods can increase the efficiency of both finding relevant documents and pinpointing specific information within them, which could streamline the process of developing and updating job exposure matrices. Named entity recognition is a fundamental natural language processing method for language understanding, which automatically identifies mentions of domain-specific concepts (named entities) in documents, e.g., exposures, occupations and job tasks. State-of-the-art machine learning models typically use evidence from an annotated corpus, i.e., a set of documents in which named entities are manually marked up (annotated) by experts, to learn how to detect named entities automatically in new documents. We have developed a novel annotated corpus of scientific articles to support machine learning based named entity recognition relevant to occupational substance exposures. Through incremental refinements to the annotation process, we demonstrate that expert annotators can attain high levels of agreement, and that the corpus can be used to train high-performance named entity recognition models. The corpus thus constitutes an important foundation for the wider development of natural language processing tools to support the study of occupational exposures.