SAG: A Session-Augmented Heterogeneous Graph-based model for Named Entity Recognition in query

Drug named entity recognition (NER) is a critical step for complex biomedical NLP tasks such as the extraction of pharmacogenomic, pharmacodynamic and pharmacokinetic parameters. Large quantities of high quality training data are almost always a prerequisite for employing supervised machine-learning techniques to achieve high classification performance. However, the human labour needed to produce and maintain such resources is a significant limitation. In this study, we improve the performance of drug NER without relying exclusively on manual annotations. We perform drug NER using either a small gold-standard corpus (120 abstracts) or no corpus at all. In our approach, we develop a voting system to combine a number of heterogeneous models, based on dictionary knowledge, gold-standard corpora and silver annotations, to enhance performance. To improve recall, we employed genetic programming to evolve 11 regular-expression patterns that capture common drug suffixes and used them as an extra means for recognition. Our approach uses a dictionary of drug names, i.e. DrugBank, a small manually annotated corpus, i.e. the pharmacokinetic corpus, and a part of the UKPMC database, as raw biomedical text. Gold-standard and silver annotated data are used to train maximum entropy and multinomial logistic regression classifiers. Aggregating drug NER methods, based on gold-standard annotations, dictionary knowledge and patterns, improved the performance on models trained on gold-standard annotations, only, achieving a maximum F-score of 95%. In addition, combining models trained on silver annotations, dictionary knowledge and patterns are shown to achieve comparable performance to models trained exclusively on gold-standard data. The main reason appears to be the morphological similarities shared among drug names. We conclude that gold-standard data are not a hard requirement for drug NER. Combining heterogeneous models build on dictionary knowledge can achieve similar or comparable classification performance with that of the best performing model trained on gold-standard annotations.

In the field of technology policy, a large number of technology policies are released every day, and scientific researchers need to always pay attention to a great number of technology policy information on different websites, and it is arduous to find crucial policy information from them. Using named entity recognition technology to convert a great number of unstructured text information in technology policy fields into structured information can help scientific researchers obtain crucial policy information. Compared with named entity recognition in the general field, the main challenge of entity recognition in the professional field is that there is less data in the professional field with annotations. In order to reduce the resource overhead of annotated data, a semi-supervised learning method for named entity recognition is produced. The advantage of the semi-supervised learning training model is that it can use the text data with label information and the text data without label information to train the recognition model, and improve the generalization ability of the named entity recognition model. This paper innovatively proposes a dynamic adversarial training method DAT (Dynamic Adversarial Training) that dynamically adjusts the loss weights of supervised data and unsupervised data, and applies it to semi-supervised entity recognition tasks, and proposes the DAT-Bert-CRF model. Effectively solve the problem of semi-supervised entity recognition. The result of our experiment show that compared with other semi-supervised entity recognition methods, the performance of our model in this paper is better.

We consider the problem of entity alignment in knowledge graphs. Previous works mainly focus on two aspects: One is to improve the TransE-based models which mostly only consider triple-level structural information i.e. relation triples or to make use of graph convolutional networks holding the assumption that equivalent entities are usually neighbored by some other equivalent entities. The other is to incorporate external features, such as attributes types, attribute values, entity names and descriptions to enhance the original relational model. However, the long-term structural dependencies between entities have not been exploited well enough and sometimes external resources are incomplete and unavailable. These will impair the accuracy and robustness of combinational models that use relations and other types of information, especially when iteration is performed. To better explore structural information between entities, we novelly propose a Self-Attentive heterogeneous sequence learning model for Entity Alignment (SAEA) that allows us to capture long-term structural dependencies within entities. Furthermore, considering low-degree entities and relations appear much less in sequences prodeced by traditional random walk methods, we design a degree-aware random walk to generate heterogeneous sequential data for self-attentive learning. To evaluate our proposed model, we conduct extensive experiments on real-world datasets. The experimental results show that our method outperforms various state-of-the-art entity alignment models using relation triples only.

Document-level event extraction aims to recognize event information from a whole piece of article. Existing methods are not effective due to two challenges of this task: a) the target event arguments are scattered across sentences; b) the correlation among events in a document is non-trivial to model. In this paper, we propose Heterogeneous Graph-based Interaction Model with a Tracker (GIT) to solve the aforementioned two challenges. For the first challenge, GIT constructs a heterogeneous graph interaction network to capture global interactions among different sentences and entity mentions. For the second, GIT introduces a Tracker module to track the extracted events and hence capture the interdependency among the events. Experiments on a large-scale dataset (Zheng et al, 2019) show GIT outperforms the previous methods by 2.8 F1. Further analysis reveals \modelname is effective in extracting multiple correlated events and event arguments that scatter across the document.

Named Entity Recognition (NER) models have achieved good performance in recent years but also have some shortcomings. Existing models regard NER as a sequence labeling task for label prediction, without considering the impact of different stages in the entity recognition process on the final result. NER can be viewed as two separate subtasks: boundary detection task and type prediction task. The two subtasks can transmit information and cooperate in the process of entity recognition, so the synergy of the two subtasks is beneficial to NER. In this paper, we propose a method to split the NER task into multiple subtasks and use the information of each subtask to enhance the NER task. According to the characteristics of different subtasks, we use different feature extraction method models to extract structural information useful for this task effectively. Using the information extracted from each subtask or the final results of the subtasks, the performance of NER was enhanced through the gating network. We conducted extensive experiments on the CONLL2003 dataset, and the experimental results show that our proposed multi-task joint learning enhances the effectiveness of the named entity recognition model.

Automatically detecting mentions of pharmaceutical drugs and chemical substances is key for the subsequent extraction of relations of chemicals with other biomedical entities such as genes, proteins, diseases, adverse reactions or symptoms. The identification of drug mentions is also a prior step for complex event types such as drug dosage recognition, duration of medical treatments or drug repurposing. Formally, this task is known as named entity recognition (NER), meaning automatically identifying mentions of predefined entities of interest in running text. In the domain of medical texts, for chemical entity recognition (CER), techniques based on hand-crafted rules and graph-based models can provide adequate performance. In the recent years, the field of natural language processing has mainly pivoted to deep learning and state-of-the-art results for most tasks involving natural language are usually obtained with artificial neural networks. Competitive resources for drug name recognition in English medical texts are already available and heavily used, while for other languages such as Spanish these tools, although clearly needed were missing. In this work, we adapt an existing neural NER system, NeuroNER, to the particular domain of Spanish clinical case texts, and extend the neural network to be able to take into account additional features apart from the plain text. NeuroNER can be considered a competitive baseline system for Spanish drug and CER promoted by the Spanish national plan for the advancement of language technologies (Plan TL).

Entity recognition, also known as named entity recognition (NER), is a fundamental task in natural language processing (NLP) that involves identifying and categorizing entities within text. These entities, such as names of people, organizations, locations, dates, and numerical values, provide structured information from unstructured text data. NER models, ranging from rule-based to machine learning-based approaches, decode linguistic patterns and contextual information to extract entities effectively. This article explores the roles of entities, tokens, and NER models in NLP, detailing their significance in various applications like information retrieval and border security. It delves into the practices of implementing NER in legal document analysis, travel history analysis, and document verification, showcasing its transformative impact in streamlining processes and enhancing security measures. Despite challenges such as ambiguity and data scarcity, ongoing research and emerging trends in multilingual NER and ethical considerations promise to drive innovation in the field. By addressing these challenges and embracing new developments, entity recognition is poised to continue advancing NLP capabilities and powering diverse real-world applications.

Purpose To solve the problems of annotation noise, ambiguity recognition and nested entity recognition in the field of Chinese furniture, this paper aims to design a new recognition model ALE-BiLSTM-CRF. Design/methodology/approach This paper addresses the relative independence of text characters in the Chinese furniture domain named entity recognition (NER) task. It also considers the limited information provided by these text characters in this task. Therefore, a model named ALE-BiLSTM-CRF for Chinese furniture domain NER is proposed. First, the ERNIE pre-trained model is used to transform text into a dynamic vector that integrates contextual information. And adversarial learning is combined to generate adversarial samples to enhance the robustness of the model. Next, the BiLSTM module captures the temporal information of the context, and the multi-head attention mechanism integrates long-distance semantic features into the character vectors. Finally, a CRF layer is used to learn the constraints between labels, enabling the model to generate more reasonable and semantically consistent label sequences. This paper conducts comparative experiments with mainstream models on the Weibo data set, achieving an F1 score of 75.52%, demonstrating its generality and robustness. Additionally, comparative and ablation experiments are conducted on a self-constructed furniture data set in the Chinese furniture field, achieving an F1 score of 89.62%, verifying the model’s superiority and effectiveness. Findings This paper conducts comparative experiments with mainstream models on the Weibo data set, achieving an F1 score of 75.52%, demonstrating its generality and robustness. Additionally, comparative and ablation experiments are conducted on a self-constructed furniture data set in the Chinese furniture field, achieving an F1 score of 89.62%, verifying the model’s superiority and effectiveness. Research limitations/implications This paper demonstrates its universality and generalization by conducting comparative experiments with mainstream models on the Weibo data set. It also conducts comparative experiments with representative pre-trained models on the furniture data set and conducts ablation experiments on the model itself, further demonstrating the superiority and effectiveness of the model. Practical implications In the furniture domain, NER aims to use various methods, including rule templates, machine learning and deep learning techniques, to extract structured information related to furniture from unstructured text. These pieces of information may include the name, material, brand, style and function of the furniture. By extracting and identifying these named entities, this paper can provide more accurate data support for furniture design, manufacturing and marketing, thereby promoting further development and innovation in the furniture industry. Social implications In the furniture field, NER faces some special challenges, which are different from entity recognition in general fields. Furniture terminology is often highly specialized and complex in structure. At the same time, there may be a large number of nested entities in the text of the furniture field. For example, the furniture name “sofa bed” contains two entities “sofa” and “bed.” Current sequence labeling methods often find it difficult to recognize such nested entity structures simultaneously. Additionally, because furniture terminology and descriptions may change with trends and design styles, the model also needs to have a certain degree of adaptability and update capabilities. These reasons make it more difficult to extract information in the furniture field, and NER in the furniture field faces huge challenges. Originality/value This paper conducts comparative experiments with mainstream models on the Weibo data set, achieving an F1 score of 75.52%, demonstrating its generality and robustness. Additionally, comparative and ablation experiments are conducted on a self-constructed furniture data set in the Chinese furniture field, achieving an F1 score of 89.62%, verifying the model’s superiority and effectiveness.

1 - Unified neural architecture for drug, disease, and clinical entity recognition

BackgroundThe unstructured data of Chinese cancer electronic health records (EHRs) contains valuable medical expertise. Accurate medical entity recognition is crucial for building a medical-assisted decision system. Named entity recognition (NER) in cancer EHRs typically uses general models designed for English medical records. There is a lack of specialized handling for cancer-specific records and limited application to Chinese medical records.ObjectiveThis study aims to propose a specific NER model to enhance the recognition of medical entities in Chinese cancer EHRs.MethodsDesensitized inpatient EHRs related to breast cancer were collected from a leading hospital in Beijing. Building upon the MC Bidirectional Encoder Representations from Transformers (BERT) foundation, the study further incorporated a Chinese cancer corpus for pretraining, resulting in the construction of the ChCancerBERT pretrained model. In conjunction with dilated-gated convolutional neural networks, bidirectional long short-term memory, multihead attention mechanism, and a conditional random field, this model forms a multimodel, multilevel integrated NER approach.ResultsThis approach effectively extracts medical entity features related to symptoms, signs, tests, treatments, and time in Chinese breast cancer EHRs. The entity recognition performance of the proposed model surpasses that of the baseline model and other models compared in the experiment. The F1-score reached 86.93%, precision reached 87.24%, and recall reached 86.61%. The model introduced in this study demonstrates exceptional performance on the CCKS2019 dataset, attaining a precision rate of 87.26%, a recall rate of 87.27%, and an impressive F1-score of 87.26%, surpassing that of existing models.ConclusionsThe experiments demonstrate that the approach proposed in this study exhibits excellent performance in NER within breast cancer EHRs. This advancement will further contribute to clinical decision support for cancer treatment and research. In addition, the study reveals that incorporating domain-specific corpora in clinical NER tasks can further enhance the performance of BERT models in specialized domains.

IntroductionThe labor market is rapidly evolving, leading to a mismatch between existing Knowledge, Skills, and Abilities (KSAs) and future occupational requirements. Reports from organizations like the World Economic Forum and the OECD emphasize the need for dynamic skill identification. This paper introduces a novel system for constructing a dynamic taxonomy using Natural Language Processing (NLP) techniques, specifically Named Entity Recognition (NER) and Relation Extraction (RE), to identify and predict future skills. By leveraging machine learning models, this taxonomy aims to bridge the gap between current skills and future demands, contributing to educational and professional development.MethodsTo achieve this, an NLP-based architecture was developed using a combination of text preprocessing, NER, and RE models. The NER model identifies and categorizes KSAs and occupations from a corpus of labor market reports, while the RE model establishes the relationships between these entities. A custom pipeline was used for PDF text extraction, tokenization, and lemmatization to standardize the data. The models were trained and evaluated using over 1,700 annotated documents, with the training process optimized for both entity recognition and relationship prediction accuracy.ResultsThe NER and RE models demonstrated promising performance. The NER model achieved a best micro-averaged F1-score of 65.38% in identifying occupations, skills, and knowledge entities. The RE model subsequently achieved a best micro-F1 score of 82.2% for accurately classifying semantic relationships between these entities at epoch 1,009. The taxonomy generated from these models effectively identified emerging skills and occupations, offering insights into future workforce requirements. Visualizations of the taxonomy were created using various graph structures, demonstrating its applicability across multiple sectors. The results indicate that this system can dynamically update and adapt to changes in skill demand over time.DiscussionThe dynamic taxonomy model not only provides real-time updates on current competencies but also predicts emerging skill trends, offering a valuable tool for workforce planning. The high recall rates in NER suggest strong entity recognition capabilities, though precision improvements are needed to reduce false positives. Limitations include the need for a larger corpus and sector-specific models. Future work will focus on expanding the corpus, improving model accuracy, and incorporating expert feedback to further refine the taxonomy.

Named Entity Recognition (NER) is a crucial task in Natural Language Processing (NLP) that identifies entities such as person names, locations, and organizations within the text. While many NER studies have concentrated on the English language, there is a significant need for further research on Indonesian NER. Indonesia presents unique challenges due to its structural complexities, polysemy, and ambiguities. Conventional machine learning and deep learning techniques have been widely applied in NER; however, more detailed exploration into integrating these methods for performance improvement is needed. This study introduces a novel hybrid model, TWBiL, which combines Transformer mechanisms, Word2Vec embeddings, Bidirectional Long Short-Term Memory (Bi-LSTM), and Attention mechanisms to enhance NER performance on Indonesian text. TWBiL harnesses the strengths of each component to generate superior word vector representations, extract intricate sentence features, and disambiguate entities contextually. Our experimental results demonstrate the effectiveness of the proposed hybrid model, revealing a significant improvement in NER performance. Specifically, TWBiL achieves an F1-Score of 85.11 on an Indonesian online news dataset, outperforming the traditional Bi-LSTM model, which achieved a score of 75.18. The results indicate that TWBiL effectively reduces ambiguity and captures context more accurately, enhancing entity recognition. Future research should priorities reducing computational time when handling larger datasets without compromising overall NER performance. This study underscores the potential of integrating advanced deep learning techniques to tackle the unique challenges of Indonesian NER, thus providing a solid foundation for further advancements in the field.

The identification of medical concepts from clinical narratives has a large interest in the biomedical scientific community due to its importance in treatment improvements or drug development research. Biomedical named entity recognition (NER) in clinical texts is crucial for automated information extraction, facilitating patient record analysis, drug development, and medical research. Traditional approaches often focus on single-class NER tasks, yet recent advancements emphasize the necessity of addressing multi-class scenarios, particularly in complex biomedical domains. This paper proposes a strategy to integrate a multi-head conditional random field (CRF) classifier for multi-class NER in Spanish clinical documents. Our methodology overcomes overlapping entity instances of different types, a common challenge in traditional NER methodologies, by using a multi-head CRF model. This architecture enhances computational efficiency and ensures scalability for multi-class NER tasks, maintaining high performance. By combining four diverse datasets, SympTEMIST, MedProcNER, DisTEMIST, and PharmaCoNER, we expand the scope of NER to encompass five classes: symptoms, procedures, diseases, chemicals, and proteins. To the best of our knowledge, these datasets combined create the largest Spanish multi-class dataset focusing on biomedical entity recognition and linking for clinical notes, which is important to train a biomedical model in Spanish. We also provide entity linking to the multi-lingual Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) vocabulary, with the eventual goal of performing biomedical relation extraction. Through experimentation and evaluation of Spanish clinical documents, our strategy provides competitive results against single-class NER models. For NER, our system achieves a combined micro-averaged F1-score of 78.73, with clinical mentions normalized to SNOMED CT with an end-to-end F1-score of 54.51. The code to run our system is publicly available at https://github.com/ieeta-pt/Multi-Head-CRF. Database URL: https://github.com/ieeta-pt/Multi-Head-CRF.

BackgroundFamily history information, including information on family members, side of the family of family members, living status of family members, and observations of family members, plays an important role in disease diagnosis and treatment. Family member information extraction aims to extract family history information from semistructured/unstructured text in electronic health records (EHRs), which is a challenging task regarding named entity recognition (NER) and relation extraction (RE), where named entities refer to family members, living status, and observations, and relations refer to relations between family members and living status, and relations between family members and observations.ObjectiveThis study aimed to introduce the system we developed for the 2019 n2c2/OHNLP track on family history extraction, which can jointly extract entities and relations about family history information from clinical text.MethodsWe proposed a novel graph-based model with biaffine attention for family history extraction from clinical text. In this model, we first designed a graph to represent family history information, that is, representing NER and RE regarding family history in a unified way, and then introduced a biaffine attention mechanism to extract family history information in clinical text. Convolution neural network (CNN)-Bidirectional Long Short Term Memory network (BiLSTM) and Bidirectional Encoder Representation from Transformers (BERT) were used to encode the input sentence, and a biaffine classifier was used to extract family history information. In addition, we developed a postprocessing module to adjust the results. A system based on the proposed method was developed for the 2019 n2c2/OHNLP shared task track on family history information extraction.ResultsOur system ranked first in the challenge, and the F1 scores of the best system on the NER subtask and RE subtask were 0.8745 and 0.6810, respectively. After the challenge, we further fine tuned the parameters and improved the F1 scores of the two subtasks to 0.8823 and 0.7048, respectively.ConclusionsThe experimental results showed that the system based on the proposed method can extract family history information from clinical text effectively.

Background: Understanding human language is a part of the research in Natural Language Processing (NLP) known as Natural Language Understanding (NLU). It becomes a crucial part of some NLP applications such as chatbots, that interpret the user intent and important entities. NLU systems depend on intent classification and named entity recognition (NER) which is crucial for understanding the user input to extract meaningful information. Not only important in chatbots, NLU also provides a pivotal function in other applications for efficient and precise text understanding. Objective: The aim of this study is to introduce multitask learning techniques to improve the application's performance on NLU tasks, especially intent classification and NER in specific domains. Methods: To achieve the language understanding capability, a strategy is to combine the intent classification and entity recognition tasks by using a shared model based on the shared representation and task dependencies. This approach is known as multitask learning and leverages the collaborative interaction between these related tasks to enhance performance. The proposed learning architecture is designed to be adaptable to various NLU-based applications, but in this work are discussed use cases in chatbots. Results: The results show the effectiveness of the proposed approach by following several experiments, both from intent classification and named entity recognitions. The multitask learning capabilities highlight the potential of multi-task learning in chatbot systems for close domains. The optimal hyperparameters consist of a warm-up step of 60, an early stopping probability of 10, a weight decay of 0.001, a Named Entity Recognition (NER) loss weight of 0.58, and an intention classification loss weight of 0.4. Conclusion: The performance of Dual Intent and Entity Transformer (DIET) for both tasks—intent classification and named entity recognition—is highly dependent on the data. This leads to various capabilities for the hyperparameter combinations. Our proposed model architecture significantly outperforms previous studies based on common evaluation metrics. Keywords: Natural Language Understanding, Chatbot, Multi-task Learning, Named Entity Recognition