Performance Of Named Entity Recognition Research Articles

RT: a Retrieving and Chain-of-Thought framework for few-shot medical named entity recognition.

This article aims to enhance the performance of larger language models (LLMs) on the few-shot biomedical named entity recognition (NER) task by developing a simple and effective method called Retrieving and Chain-of-Thought (RT) framework and to evaluate the improvement after applying RT framework. Given the remarkable advancements in retrieval-based language model and Chain-of-Thought across various natural language processing tasks, we propose a pioneering RT framework designed to amalgamate both approaches. The RT approach encompasses dedicated modules for information retrieval and Chain-of-Thought processes. In the retrieval module, RT discerns pertinent examples from demonstrations during instructional tuning for each input sentence. Subsequently, the Chain-of-Thought module employs a systematic reasoning process to identify entities. We conducted a comprehensive comparative analysis of our RT framework against 16 other models for few-shot NER tasks on BC5CDR and NCBI corpora. Additionally, we explored the impacts of negative samples, output formats, and missing data on performance. Our proposed RT framework outperforms other LMs for few-shot NER tasks with micro-F1 scores of 93.50 and 91.76 on BC5CDR and NCBI corpora, respectively. We found that using both positive and negative samples, Chain-of-Thought (vs Tree-of-Thought) performed better. Additionally, utilization of a partially annotated dataset has a marginal effect of the model performance. This is the first investigation to combine a retrieval-based LLM and Chain-of-Thought methodology to enhance the performance in biomedical few-shot NER. The retrieval-based LLM aids in retrieving the most relevant examples of the input sentence, offering crucial knowledge to predict the entity in the sentence. We also conducted a meticulous examination of our methodology, incorporating an ablation study. The RT framework with LLM has demonstrated state-of-the-art performance on few-shot NER tasks.

Named entity recognition model based on Multi‐BiLSTM and competition mechanism

AbstractA named entity recognition (NER) model based on multiple bidirectional long short‐term memory networks (Multi‐BiLSTM) and competition mechanism (CM) is proposed. This model includes three parts: word vectorization module, learning module and application module. In the word vectorization module, the word vectors containing semantic features are generated based on BERT (Bidirectional Encoder Representations from Transformers). In the learning module, the multiple small‐scale BiLSTM are used to extract the contextual features of the word vectors synchronously. In the application module, the competition mechanism is adopted to select the relatively better models obtained in the learning module for recognizing entities. The results show the proposed model can reduce the parameters in the learning process, and the NER performance of the proposed model is better than that of the BERT‐BiLSTM‐CRF model. For example, the F1 can be increased by 1.02% and 0.39% for CoNLL2003 and Chinese resume datasets respectively.

ADAPTIVE DOMAIN-SPECIFIC NAMED ENTITY RECOGNITION METHOD WITH LIMITED DATA

The ever-evolving volume of digital information requires the development of innovative search strategies aimed at obtaining the necessary data efficiently and economically feasible. The urgency of the problem is emphasized by the growing complexity of information landscapes and the need for fast data extraction methodologies. In the field of natural language processing, named entity recognition (NER) is an essential task for extracting useful information from unstructured text input for further classification into predefined categories. Nevertheless, conventional methods frequently encounter difficulties when confronted with a limited amount of labeled data, posing challenges in real-world scenarios where obtaining substantial annotated datasets is problematic or costly. In order to address the problem of domain-specific NER with limited data, this work investigates NER techniques that can overcome these constraints by continuously learning from newly collected information on pre-trained models. Several techniques are also used for making the greatest use of the limited labeled data, such as using active learning, exploiting unlabeled data, and integrating domain knowledge. Using domain-specific datasets with different levels of annotation scarcity, the fine-tuning process of pre-trained models, such as transformer-based models (TRF) and Toc2Vec (token-to-vector) models is investigated. The results show that, in general, expanding the volume of training data enhances most models' performance for NER, particularly for models with sufficient learning ability. Depending on the model architecture and the complexity of the entity label being learned, the effect of more data on the model's performance can change. After increasing the training data by 20%, the LT2V model shows the most balanced growth in accuracy overall by 11% recognizing 73% of entities and processing speed. Meanwhile, with consistent processing speed and the greatest F1-score, the Transformer-based model (TRF) shows promise for effective learning with less data, achieving 74% successful prediction and a 7% increase in performance after expanding the training data to 81%. Our results pave the way for the creation of more resilient and efficient NER systems suited to specialized domains and further the field of domain-specific NER with sparse data. We also shed light on the relative merits of various NER models and training strategies, and offer perspectives for future research.

Named entity recognition using transfer learning and small human‐ and meta‐pseudo‐labeled datasets

AbstractWe introduce a high‐performance named entity recognition (NER) model for written and spoken language. To overcome challenges related to labeled data scarcity and domain shifts, we use transfer learning to leverage our previously developed KorBERT as the base model. We also adopt a meta‐pseudo‐label method using a teacher/student framework with labeled and unlabeled data. Our model presents two modifications. First, the student model is updated with an average loss from both human‐ and pseudo‐labeled data. Second, the influence of noisy pseudo‐labeled data is mitigated by considering feedback scores and updating the teacher model only when below a threshold (0.0005). We achieve the target NER performance in the spoken language domain and improve that in the written language domain by proposing a straightforward rollback method that reverts to the best model based on scarce human‐labeled data. Further improvement is achieved by adjusting the label vector weights in the named entity dictionary.

Transformer-based Named Entity Recognition for Clinical Cancer Drug Toxicity by Positive-unlabeled Learning and KL Regularizers

Background: With increasing rates of polypharmacy, the vigilant surveillance of clinical drug toxicity has emerged as an important concern. Named Entity Recognition (NER) stands as an indispensable undertaking, essential for the extraction of valuable insights regarding drug safety from the biomedical literature. In recent years, significant advancements have been achieved in the deep learning models on NER tasks. Nonetheless, the effectiveness of these NER techniques relies on the availability of substantial volumes of annotated data, which is labor-intensive and inefficient. background: With increasing rates of polypharmacy, clinical drug toxicity has been closely monitored. Named Entity Recognition (NER) is a vital task for extracting valuable drug safety information from biomedical literature. Recently, many deep learning models in biomedical domain have made great progress for NER, especially pre-trained language models. However, these NER methods require large amounts of high-quality manually annotated data with named entities, which is labor intensive and inefficient. Methods: This study introduces a novel approach that diverges from the conventional reliance on manually annotated data. It employs a transformer-based technique known as Positive-Unlabeled Learning (PULearning), which incorporates adaptive learning and is applied to the clinical cancer drug toxicity corpus. To improve the precision of prediction, we employ relative position embeddings within the transformer encoder. Additionally, we formulate a composite loss function that integrates two Kullback-Leibler (KL) regularizers to align with PULearning assumptions. The outcomes demonstrate that our approach attains the targeted performance for NER tasks, solely relying on unlabeled data and named entity dictionaries. objective: To improve the performance of prediction Conclusion: Our model achieves an overall NER performance with an F1 of 0.819. Specifically, it attains F1 of 0.841, 0.801 and 0.815 for DRUG, CANCER, and TOXI entities, respectively. A comprehensive analysis of the results validates the effectiveness of our approach in comparison to existing PULearning methods on biomedical NER tasks. Additionally, a visualization of the associations among three identified entities is provided, offering a valuable reference for querying their interrelationships. method: In this work, instead of relying on the manually labeled data, a transformer-based Positive-Unlabeled Learning (PULearning) is proposed with adaptive learning and applied on the clinical cancer drug toxicity corpus. To improve the precision of prediction, relative position embeddings are used in transformer encoder. And then, a mixed loss is designed with two Kullback-Leibler (KL) regularizers for PULearning assumptions. Through adaptive sampling, our approach meets the expected performance for NER task only using unlabeled data and named entity dictionaries. result: The overall NER performance of our model obtains 0.819 of F1-score, while 0.841, 0.801 and 0.815 of F1-score on DRUG, CANCER and TOXI, respectively. other: None

Ontology Attention Layer for Medical Named Entity Recognition

Named entity recognition (NER) is particularly challenging for medical texts due to the high domain specificity, abundance of technical terms, and sparsity of data in this field. In this work, we propose a novel attention layer, called the “ontology attention layer”, that enhances the NER performance of a language model for clinical text by utilizing an ontology consisting of conceptual classes related to the target entity set. The proposed layer computes the relevance between each input token and the classes in the ontology and then fuses the encoded token vectors and the class vectors to enhance the token vectors by explicit superior knowledge. In our experiments, we apply the proposed layer to various language models for an NER task based on a Chinese clinical dataset to evaluate the performance of the layer. We also investigate the influence of the granularity of the classes utilized in the ontology attention layer. The experimental results show that the proposed ontology attention layer improved F1 scores by 0.4% to 0.5%. The results suggest that the proposed method is an effective approach to improving the NER performance of existing language models for clinical datasets.

Class-Imbalanced-Aware Distantly Supervised Named Entity Recognition.

Distantly supervised named entity recognition (NER), which automatically learns NER models without manually labeling data, has gained much attention recently. In distantly supervised NER, positive unlabeled (PU) learning methods have achieved notable success. However, existing PU learning-based NER methods are unable to automatically handle the class imbalance and further depend on the estimation of the unknown class prior; thus, the class imbalance and imperfect estimation of the class prior degenerate the NER performance. To address these issues, this article proposes a novel PU learning method for distantly supervised NER. The proposed method can automatically handle the class imbalance and does not need to engage in class prior estimation, which enables the proposed methods to achieve the state-of-the-art performance. Extensive experiments support our theoretical analysis and validate the superiority of our method.

Named Entity Recognition of an Oversampled and Preprocessed Manufacturing Data Corpus

In recent manufacturing industry, improving the manufacturing process is of paramount importance. One area that holds great potential for enhancement is the application and manipulation of maintenance data. By effectively leveraging this data, manufacturers can optimize maintenance schedules, leading to increased efficiency, reduced costs, and minimized downtime. However, the challenge lies in handling vast amounts of maintenance data that often come in various formats, making it difficult to extract valuable insights. Without proper analysis, this unprocessed data can result in unforeseen issues, costly disruptions, and extended downtime periods. To overcome this obstacle, modern manufacturing companies are turning to advanced technologies such as language modelling, text classification, machine translation, and Named Entity Recognition (NER). To the best of our knowledge, no investigation has been conducted to assess the impact of text preprocessing on NER performance. Improving the initial stage of NER, such as text preprocessing, can enhance NER performance which leads to the training model’s efficiency performance. In this study, Hidden Markov Model (HMM) is employed to improve NER performance by utilizing oversampling and text preprocessing techniques. The study is performed without IOB labelling and consider seven specific entities and the preprocessing text tasks include tokenization, lemmatization, erase punctuation, stop words removal, and elimination of long and short words. As a result, HMM for NER with oversampling and preprocessed text outperformed the one without any of both by 20.10% and 27.59%, respectively, due to consideration of significant classes and words among the entity classes in preprocessed factory reports. This finding highlights the importance of text preprocessing method selection in NER and its capability to optimize maintenance schedule and reduce downtime.

A System for Interviewing and Collecting Statements Based on Intent Classification and Named Entity Recognition Using Augmentation

In cases of child sexual abuse, interviewing and obtaining trustworthy statements from victims and witnesses is essential because their statements are the only evidence. It is crucial to ascertain objectively the credibility of the victim’s statements, which may vary based on the nature of the questions posed by the forensic interviewer. Therefore, interview skills that eliminate subjective opinions require a high level of training for forensic interviewers. To reduce high-risk subjective interviews, objectively analyzing statements is essential. Understanding the victim’s intent and named entity recognition (NER) in the statements is necessary to give the victim open-ended questions and memory recall. Therefore, the system provides an intent classification and NER method that follows the National Institute of Child Health and Human Development Investigative Interview Protocol, which outlines the collection of objective statements. Large language models such as BERT and KoBERT, along with data augmentation techniques, were proposed using a restricted training dataset of limited size to achieve effective intent classification and NER performance. Additionally, a system that can collect objective statements with the proposed model was developed and it was confirmed that it could assist statement analysts. The verification results showed that the model achieved average F1-scores of 95.5% and 97.8% for intent classification and NER, respectively, which improved the results of the limited data by 3.4% and 3.7%, respectively.

Two evaluations on Ontology-style relation annotations

In this paper, we propose an Ontology-Style Relation (OSR) annotation approach. In conventional Relation Extraction (RE) datasets, relations are annotated as a link between two entity mentions. In contrast, in our OSR annotation, a relation is annotated as a relation mention (i.e., not a link but a node) and rdfs:domain and rdfs:range links are annotated from the relation mention to its argument entity mentions. This approach has the following benefits: (1) the relation annotations can be easily converted to Resource Description Framework (RDF) triples to populate an Ontology, (2) some part of conventional RE tasks can be tackled as Named Entity Recognition (NER) tasks, and the relation classes are limited to several RDF properties, and (3) OSR annotations can be used for clear documentations of Ontology contents. We conducted two kinds of evaluation to investigate effects of OSR annotation. We converted (1) an in-house corpus of Japanese Rules of the Road (RoR) in conventional annotations into the OSR annotations and built a novel OSR-RoR corpus and (2) SemEval-2010 Task 8 dataset into the OSR annotations (called OSR-SemEval corpus). We compared the NER and RE performance using neural NER/RE tool DyGIE++ on the conventional and OSR annotations. The experimental results show that the OSR annotations make the RE task easier while introducing slight complexity into the NER task.

EPT: Data Augmentation with Embedded Prompt Tuning for Low-Resource Named Entity Recognition

Data augmentation methods are often used to address data scarcity in natural language processing (NLP). However, token-label misalignment, which refers to situations where tokens are matched with incorrect entity labels in the augmented sentences, hinders the data augmentation methods from achieving high scores in token-level tasks like named entity recognition (NER). In this paper, we propose embedded prompt tuning (EPT) as a novel data augmentation approach to low-resource NER. To address the problem of token-label misalignment, we implicitly embed NER labels as prompt into the hidden layer of pre-trained language model, and therefore entity tokens masked can be predicted by the finetuned EPT. Hence, EPT can generate high-quality and high-diverse data with various entities, which improves performance of NER. As datasets of cross-domain NER are available, we also explore NER domain adaption with EPT. The experimental results show that EPT achieves substantial improvement over the baseline methods on low-resource NER tasks.

The named entity recognition of vessel power equipment fault using the multi-details embedding model

The fault diagnosis of vessel power equipment is established by the manual work with low efficiency. The knowledge graph(KG) usually is applied to extract the experience and operation logic of controllers into knowledge, which can enrich the means of fault judgment and recovery decision. As an important part of KG building, the performance of named entity recognition (NER) is critical to the following tasks. Due to the challenges of information insufficiency and polysemous words in the entities of vessel power equipment fault, this study adopts the fusion model of Bidirectional Encoder Representations from Transformers (BERT), revised Convolutional neural network (CNN), bidirectional long short-term memory (BiLSTM), and conditional random field (CRF). Firstly, the adjusted BERT and revised CNN are respectively adopted to acquire the multiple embeddings including semantic information and contextual glyph features. Secondly, the local context features are effectively extracted by adopting the channel-wised fusion structures. Finally, BiLSTM and CRF are respectively adopted to obtain the semantic information of the long sequences and the prediction sequence labels. The experimental results show that the performance of NER by the proposed model outperforms other mainstream models. Furthermore, this work provides the foundation of the tasks of intelligent diagnosis and NER in other fields.

Deep Neural Network with Embedding Fusion for Chinese Named Entity Recognition

Chinese Named Entity Recognition (NER) is an essential task in natural language processing, and its performance directly impacts the downstream tasks. The main challenges in Chinese NER are the high dependence of named entities on context and the lack of word boundary information. Therefore, how to integrate relevant knowledge into the corresponding entity has become the primary task for Chinese NER. Both the lattice LSTM model and the WC-LSTM model did not make excellent use of contextual information. Additionally, the lattice LSTM model had a complex structure and did not exploit the word information well. To address the preceding problems, we propose a Chinese NER method based on the deep neural network with multiple ways of embedding fusion. First, we use a convolutional neural network to combine the contextual information of the input sequence and apply a self-attention mechanism to integrate lexicon knowledge, compensating for the lack of word boundaries. The word feature, context feature, bigram feature, and bigram context feature are obtained for each character. Second, four different features are used to fuse information at the embedding layer. As a result, four different word embeddings are obtained through cascading. Last, the fused feature information is input to the encoding and decoding layer. Experiments on three datasets show that our model can effectively improve the performance of Chinese NER.

Chemical identification and indexing in full-text articles: an overview of the NLM-Chem track at BioCreative VII.

The BioCreative National Library of Medicine (NLM)-Chem track calls for a community effort to fine-tune automated recognition of chemical names in the biomedical literature. Chemicals are one of the most searched biomedical entities in PubMed, and-as highlighted during the coronavirus disease 2019 pandemic-their identification may significantly advance research in multiple biomedical subfields. While previous community challenges focused on identifying chemical names mentioned in titles and abstracts, the full text contains valuable additional detail. We, therefore, organized the BioCreative NLM-Chem track as a community effort to address automated chemical entity recognition in full-text articles. The track consisted of two tasks: (i) chemical identification and (ii) chemical indexing. The chemical identification task required predicting all chemicals mentioned in recently published full-text articles, both span [i.e.named entity recognition (NER)] and normalization (i.e.entity linking), using Medical Subject Headings (MeSH). The chemical indexing task required identifying which chemicals reflect topics for each article and should therefore appear in the listing of MeSH terms for the document in the MEDLINE article indexing. This manuscript summarizes the BioCreative NLM-Chem track and post-challenge experiments. We received a total of 85 submissions from 17 teams worldwide. The highest performance achieved for the chemical identification task was 0.8672 F-score (0.8759 precision and 0.8587 recall) for strict NER performance and 0.8136 F-score (0.8621 precision and 0.7702 recall) for strict normalization performance. The highest performance achieved for the chemical indexing task was 0.6073 F-score (0.7417 precision and 0.5141 recall). This community challenge demonstrated that (i) the current substantial achievements in deep learning technologies can be utilized to improve automated prediction accuracy further and (ii) the chemical indexing task is substantially more challenging. We look forward to further developing biomedical text-mining methods to respond to the rapid growth of biomedical literature. The NLM-Chem track dataset and other challenge materials are publicly available at https://ftp.ncbi.nlm.nih.gov/pub/lu/BC7-NLM-Chem-track/. Database URL https://ftp.ncbi.nlm.nih.gov/pub/lu/BC7-NLM-Chem-track/.

Multi-task Joint Learning to Enhance Named Entity Recognition

Named Entity Recognition (NER) models have achieved good performance in recent years but also have some shortcomings. Existing models regard NER as a sequence labeling task for label prediction, without considering the impact of different stages in the entity recognition process on the final result. NER can be viewed as two separate subtasks: boundary detection task and type prediction task. The two subtasks can transmit information and cooperate in the process of entity recognition, so the synergy of the two subtasks is beneficial to NER. In this paper, we propose a method to split the NER task into multiple subtasks and use the information of each subtask to enhance the NER task. According to the characteristics of different subtasks, we use different feature extraction method models to extract structural information useful for this task effectively. Using the information extracted from each subtask or the final results of the subtasks, the performance of NER was enhanced through the gating network. We conducted extensive experiments on the CONLL2003 dataset, and the experimental results show that our proposed multi-task joint learning enhances the effectiveness of the named entity recognition model.

Cross-Lingual Named Entity Recognition for Heterogenous Languages

Previous works on cross-lingual Named Entity Recognition (NER) have achieved great success. However, few of them consider the effect of language families between the source and target languages. In this study, we find that the cross-lingual NER performance of a target language would decrease when its source language is changed from the same (homogenous) into a different (heterogenous) language family with that target language. To improve the NER performance in this situation, we propose a novel cross-lingual NER framework based on self-distillation mechanism and Bilateral-Branch Network (SD-BBN). SD-BBN learns source-language NER knowledge from supervised datasets and obtains target-language knowledge from weakly supervised datasets. These two kinds of knowledge are then fused based on self-distillation mechanism for better identifying entities in the target language. We evaluate SD-BBN on 9 language datasets from 4 different language families. Results show that SD-BBN tends to outperform baseline methods. Remarkably, when the target and source languages are heterogenous, SD-BBN can achieve a greater boost. Our results might suggest that obtaining language-specific knowledge from the target language is essential for improving cross-lingual NER when the source and target languages are heterogenous. This finding could provide a novel insight into further research.

Generation of training data for named entity recognition of artworks

As machine learning techniques are being increasingly employed for text processing tasks, the need for training data has become a major bottleneck for their application. Manual generation of large scale training datasets tailored to each task is a time consuming and expensive process, which necessitates their automated generation. In this work, we turn our attention towards creation of training datasets for named entity recognition (NER) in the context of the cultural heritage domain. NER plays an important role in many natural language processing systems. Most NER systems are typically limited to a few common named entity types, such as person, location, and organization. However, for cultural heritage resources, such as digitized art archives, the recognition of fine-grained entity types such as titles of artworks is of high importance. Current state of the art tools are unable to adequately identify artwork titles due to unavailability of relevant training datasets. We analyse the particular difficulties presented by this domain and motivate the need for quality annotations to train machine learning models for identification of artwork titles. We present a framework with heuristic based approach to create high-quality training data by leveraging existing cultural heritage resources from knowledge bases such as Wikidata. Experimental evaluation shows significant improvement over the baseline for NER performance for artwork titles when models are trained on the dataset generated using our framework.

Named entity recognition of building construction defect information from text with linguistic noise

Neither traditional rule-based named entity recognition (NER) nor the latest language models perform well in information extraction from noisy text—the text that contains linguistic errors, slang, loanwords, and jargon. Building defect complaints filed by residents via online systems is a representative example of such noisy text. This paper proposes an NER method for automatically extracting defect information from noisy text using a defect thesaurus and transfer learning. The thesaurus built herein included 1097 defect named entities in 23 categories. The NER performance was tested using 69,750 defect complaints through transfer learning of three representative pre-trained language models: Multilingual Bidirectional Encoder Representations from Transformers (BERT), Korean BERT (KoBERT), and Korean Efficiently Learning an Encoder that Classifies Token Replacements Accurately (KoELECTRA). The proposed method achieved an average F1 score of 91.0% using KoBERT. This NER performance was higher than that of the open benchmark NER performance for clean text (86.1%).

An End-To-End NER Model with Explicit Boundary and Type Information

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP). The purpose is to locate and classify named entities with specific meanings, including the person and location. However, there are two problems with the existing methods: Firstly, the model cannot significantly distinguish boundaries and types of entities, especially if there are many types of entities. Secondly, the labels do not provide information on the differences in the boundaries and types of model entities. Therefore, the Explicit Boundary and Type Information (EBTI) models are used to address both issues to improve NER performance. Specifically, the boundary detection module uses for pure boundary recognition, and the type module uses to determine entity type information and then combine them to improve the performance of NER. Experiments are conducted on the widely used dataset CoNLL2003 to demonstrate the advantages of our method, with results comparable to the current state-of-the-art (SOTA).

Multi-attention deep neural network fusing character and word embedding for clinical and biomedical concept extraction

Clinical and biomedical concept extraction is critical in medical analysis using clinical and biomedical documents from professional literature, EHRs and PHRs. Named entity recognition (NER) accurately marks essential information in the literature based on the characteristics of the target entity, providing a method for extracting clinical and biomedical concepts. The performance of NER is heavily embedding-dependent, so recent studies have proposed the method of generating word embedding from character-level information, which can strengthen the representation ability for word embedding.In this paper, we present a novel neural network model including an attention mechanism network and a convolutional neural network (CNN) to further improve character-level embedding. First, an attention mechanism is applied simultaneously to the local and global character embedding. Then, a CNN with multi-size filters is used to extract more information from the character level, which can capture more meaningful features from words with various lengths. In addition, a cross-attention method is used to leverage the interaction between word embedding and character embedding to generate the final word representation. Finally, we modified Mogrifier LSTM to make it suitable for NER tasks and integrated it into our model. Experimental results show that our method is effective and that the model performs better than the baseline models. We also apply our methods proposed in this paper to the transformer-based model and obtain a 90.36 F1-score on NCBI-Disease.