AGNER: Agile governance-oriented unified named entity recognition for continual learning with diffusion adaptation.

Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be categorized into the flat NER, nested NER, and discontinuous NER subtasks. These subtasks have been mainly solved by the token-level sequence labelling or span-level classification. However, these solutions can hardly tackle the three kinds of NER subtasks concurrently. To that end, we propose to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework. Based on our unified framework, we can leverage the pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the special design of the tagging schema or ways to enumerate spans. We exploit three types of entity representations to linearize entities into a sequence. Our proposed framework is easy-to-implement and achieves state-of-the-art (SoTA) or near SoTA performance on eight English NER datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets.

CMiNER: Named entity recognition on imperfectly annotated data via confidence and meta weight adaptation

The task of named entity recognition (NER) is normally divided into nested NER and flat NER depending on whether named entities are nested or not. Models are usually separately developed for the two tasks, since sequence labeling models, the most widely used backbone for flat NER, are only able to assign a single label to a particular token, which is unsuitable for nested NER where a token may be assigned several labels. In this paper, we propose a unified framework that is capable of handling both flat and nested NER tasks. Instead of treating the task of NER as a sequence labeling problem, we propose to formulate it as a machine reading comprehension (MRC) task. For example, extracting entities with the \textsc{per} label is formalized as extracting answer spans to the question "{\it which person is mentioned in the text?}". This formulation naturally tackles the entity overlapping issue in nested NER: the extraction of two overlapping entities for different categories requires answering two independent questions. Additionally, since the query encodes informative prior knowledge, this strategy facilitates the process of entity extraction, leading to better performances for not only nested NER, but flat NER. We conduct experiments on both {\em nested} and {\em flat} NER datasets. Experimental results demonstrate the effectiveness of the proposed formulation. We are able to achieve vast amount of performance boost over current SOTA models on nested NER datasets, i.e., +1.28, +2.55, +5.44, +6.37, respectively on ACE04, ACE05, GENIA and KBP17, along with SOTA results on flat NER datasets, i.e.,+0.24, +1.95, +0.21, +1.49 respectively on English CoNLL 2003, English OntoNotes 5.0, Chinese MSRA, Chinese OntoNotes 4.0.

Humans have the ability to incrementally learn, accumulate, update, and apply knowledge from dynamic environments. This capability, known as continual learning or lifelong learning, is also a long-term goal in the development of artificial intelligence. However, neural network-based continual learning suffers from catastrophic forgetting: the acquisition of new knowledge typically disrupts previously learned knowledge, leading to partial forgetting and a decline in the model’s overall performance. Most current continual learning methods can only mitigate catastrophic forgetting and fail to incrementally improve the overall performance. In this work, we aim to incrementally improve performance within sample incremental context by utilizing inter-stage edges as a pathway for explicit knowledge transfer in continual graph learning. Building on this pathway, we propose a knowledge-augmented replay method by leveraging evolving subgraphs of important nodes. This method enhances the distinction between patterns associated with different node classes and consolidates previously learned knowledge. Experiments on phishing detection in Ethereum transaction networks validate the effectiveness of the proposed method, demonstrating effective knowledge retention and augmentation while overcoming catastrophic forgetting and incrementally improving performance. The results also reveal the relationship between average accuracy and average forgetting. Lastly, we identify the key factor to incremental performance improvement, which lays a foundation for convergence of continual graph learning.

Named entity recognition as a fundamental task plays a crucial role in accomplishing some of the tasks and applications in natural language processing. In the age of Internet information, as far as computer applications are concerned, a huge proportion of information is stored in structured and unstructured forms and used for language and text processing. Before neural networks were widely used in natural language processing tasks, research in the field of named entity recognition usually focused on leveraging lexical and syntactic knowledge to improve the performance of models or methods. To promote the development of named entity recognition, researchers have been creating named entity recognition datasets through conferences, projects, and competitions for many years, based on various research goals, and training entity recognition models with increasing accuracy on this basis. However, there has not been much exploration of named entity recognition datasets. Particularly, there have been many datasets available since the introduction of the named entity recognition task, but there is no clear framework to summarize the development of these seemingly independent datasets. A closer look at the context of the development of each dataset and the features it contains reveals that these datasets share some common features to varying degrees. In this thesis, we review the development of named entity recognition datasets over the years and describe them in terms of the language of the dataset, the domain of research, the type of entity, the granularity of the entity, and the annotation of the entity. Finally, we provide an idea for the creation of subsequent named entity recognition datasets.

Named Entity Recognition (NER) is a task of Natural Language Processing (NLP) that deals with finding and categorizing relevant entities (i.e., word n-grams) in a text, assigning them to predefined semantic categories. The availability of annotated datasets is crucial for developing NER models and assessing their quality. This becomes an issue considering underrepresented languages and specific domains. Furthermore, the word-level annotation required by NER datasets is laborious and prone to inconsistencies. Aiming to contribute to more resources for Portuguese, this paper compiled PetroGeoNER, a NER dataset in the Oil & Gas domain. The process of creating our dataset involved unifying, revising, and solving inconsistencies in two existing datasets. PetroGeoNER was used to train accurate NER models. Both the models and the dataset were made publicly available.

Catastrophic forgetting remains an outstanding challenge in continual learning. Recently, methods inspired by the brain, such as continual representation learning and memory replay, have been used to combat catastrophic forgetting. Associative learning (retaining associations between inputs and outputs, even after good representations are learned) plays an important function in the brain; however, its role in continual learning has not been carefully studied. Here, we identified a two-layer neural circuit in the fruit fly olfactory system that performs continual associative learning between odors and their associated valences. In the first layer, inputs (odors) are encoded using sparse, high-dimensional representations, which reduces memory interference by activating nonoverlapping populations of neurons for different odors. In the second layer, only the synapses between odor-activated neurons and the odor's associated output neuron are modified during learning; the rest of the weights are frozen to prevent unrelated memories from being overwritten. We prove theoretically that these two perceptron-like layers help reduce catastrophic forgetting compared to the original perceptron algorithm, under continual learning. We then show empirically on benchmark data sets that this simple and lightweight architecture outperforms other popular neural-inspired algorithms when also using a two-layer feedforward architecture. Overall, fruit flies evolved an efficient continual associative learning algorithm, and circuit mechanisms from neuroscience can be translated to improve machine computation.

In the natural language processing (NLP) field, it is fairly common that an entity is nested in another entity. Most existing named entity recognition (NER) models focus on flat entities but ignore nested entities. In this paper, we propose a neural model for nested named entity recognition. Our model employs a multi-label boundary detection module to detect entity boundaries, avoiding boundary detection conflict existing in the boundary-aware model. Besides, our model with a boundary detection module and a category detection module detects entity boundaries and entity categories simultaneously, avoiding the error propagation problem existing in current pipeline models. Furthermore, we introduce multitask learning to train the boundary detection module and the category detection module to capture the underlying association between entity boundary information and entity category information. In this way, our model achieves better performance of entity extraction. In evaluations on two nested NER datasets and a flat NER dataset, we show that our model outperforms previous state-of-the-art models on nested and flat NER.

In the digital era, where data generation is incessant and often presents non-stationary distributions, intelligent agents face the imperative challenge of emulating human-like learning and adaptation. Handling non-stationary data streams effectively is essential for intelligent agents, enabling them to adapt and make accurate predictions in dynamic environments. Among the diverse research domains in machine learning, continual learning emerges as a crucial paradigm, enabling networks to accumulate knowledge over sequential tasks without retraining from scratch in ever-evolving data environments. A paramount challenge in continual learning is catastrophic forgetting, characterized by the performance degradation of neural networks on previously acquired tasks when subsequently trained on a new task. This problem stems from the stability-plasticity dilemma. The stability–plasticity dilemma depicts a spectrum in artificial neural networks. While stability focuses on retaining learned knowledge, plasticity is essential for adapting to new data distributions. Furthermore, handling concept drifts and label scarcity in never-ending data streams is vital for effective learning in real-world scenarios, where true class labels may be delayed or unavailable. The former may render a learning agent obsolete due to shifting parameters, while the latter is prevalent in real-world scenarios where true class labels are delayed or unavailable. Traditional deep learning paradigms, anchored in offline learning, necessitate retraining the network from scratch to accommodate new information. Such approach is not only computationally and memory-intensive but also poses significant privacy concerns. In non-stationary environments, continual learning models must also focus on resource efficiency, ideally updating the network with only new training instances. These challenges are common in the broader context of non-stationary data streams. Therefore, addressing the pivotal problems in continual learning is a crucial step toward effectively handling non-stationary environments. This thesis endeavors to develop advanced algorithms adept at managing complex environments in the continual learning manner. Four pivotal contributions are presented, each addressing specific aspects of non-stationary data stream processing. These include both online learning techniques and continual learning approaches to cover a broader range of challenges in non-stationary environments. Firstly, an innovative online learning technique, Parsimonious Network++ (ParsNet++), is proposed to tackle the online quality monitoring under label scarcity. Utilizing limited labeled samples, ParsNet++ significantly reduces manual labor in product quality inspection, embodying a data-driven, weakly-supervised approach. For effective adjustment} to environmental changes, Autonomous Clustering Mechanism (ACM), a flexible density estimation method, is adopted for constructing complex probability densities that steer the structural learning process within its dynamic hidden layer. Secondly, we study the challenge of cross-domain multistream classification under extreme label scarcity in the source domain and the absence of labeled target domain data. Learning Streaming Process from Partial Ground Truth} (LEOPARD) is proposed, which diverges from traditional static transfer learning settings and is specifically tailored for streaming data. LEOPARD foster a clustering-friendly latent space via an adaptive structure and achieves cross-domain alignment through a domain-invariant network, together in response to asynchronous drifts and domain discrepancies. More importantly, it operates without the need for continuous label availability, relying only on a limited prerecorded labeled samples from the source stream to establish class-to-cluster relationships. Thirdly, we investigate continual learning problems. Different from previous tasks, continual learning aims to solve both forward and backward transfer. Its core lies in utilizing prior experiences to assimilate new knowledge over time, necessitating the capability to overcome catastrophic forgetting. Regularization approaches often overlook the mapping inter-task synaptic relevance, potentially harboring shared information across tasks within certain neurons. Furthermore, the importance matrix in conventional regularization approaches tends to explode along with the accumulation of tasks. ISYANA is introduced to address these problems via task-to-synapses and task-to-task modules, enhancing the importance matrix with a per-parameter learning rate implementation. Finally, Continual learning approach for many processes (CLAMP) is designed for efficient deployment in cross-domain multistream continual learning with unlabeled target domain. A distinctive feature of CLAMP is its reliance on sequence-aware assessors, which produce a set of weights for every sample. Dual assessors are trained in the meta-learning approach using random transformation techniques and similar samples of the source process to address the noisy pseudo-label problem. This approach not only controls the sample's influences addressing the issues of negative transfers and noisy pseudo labels but also the interactions of the multiple loss functions to achieve a proper tradeoff between the stability and the plasticity thus preventing catastrophic forgetting. Overall, CLAMP marks a significant stride in cross-domain continual learning, adeptly integrating adversarial domain adaptation to effectively address the challenges of label scarcity and domain shift.

A high-quality domain-oriented dataset is crucial for the domain-specific named entity recognition (NER) task. In this study, we introduce a novel education-oriented Chinese NER dataset (EduNER). To provide representative and diverse training data, we collect data from multiple sources, including textbooks, academic papers, and education-related web pages. The collected documents span ten years (2012–2021). A team of domain experts is invited to accomplish the education NER schema definition, and a group of trained annotators is hired to complete the annotation. A collaborative labeling platform is built for accelerating human annotation. The constructed EduNER dataset includes 16 entity types, 11k+ sentences, and 35,731 entities. We conduct a thorough statistical analysis of EduNER and summarize its distinctive characteristics by comparing it with eight open-domain or domain-specific NER datasets. Sixteen state-of-the-art models are further utilized for NER tasks validation. The experimental results can enlighten further exploration. To the best of our knowledge, EduNER is the first publicly available dataset for NER task in the education domain, which may promote the development of education-oriented NER models.

Named entity recognition is an important task in the field of natural language processing, used to identify entities in text and classify them into predefined types.The research on Yi language named entity recognition is still in its early stages both domestically and internationally.Currently, there is no publicly available comprehensive dataset for Yi language named entity recognition, which has hindered the progress in this field.This paper constructed a named entity recognition dataset (Yi language news named entity recognition, YNNER), and manually annotates the names of person, places, and institutions.Then, the named entity recognition model is used to carry out experimental and comparative analysis on the dataset.The experimental results show that the F1 values of all models are above 70%, which proved the validity and availability of the dataset constructed.This paper aims to promote the research and development of Yi language named entity recognition, provide dataset and baseline models for this field, and expand related research.

Continual learning models allow them to learn and adapt to new changes and tasks over time. However, in continual and sequential learning scenarios, in which the models are trained using different data with various distributions, neural networks (NNs) tend to forget the previously learned knowledge. This phenomenon is often referred to as catastrophic forgetting. The catastrophic forgetting is an inevitable problem in continual learning models for dynamic environments. To address this issue, we propose a method, called continual Bayesian learning networks (CBLNs), which enables the networks to allocate additional resources to adapt to new tasks without forgetting the previously learned tasks. Using a Bayesian NN, CBLN maintains a mixture of Gaussian posterior distributions that are associated with different tasks. The proposed method tries to optimize the number of resources that are needed to learn each task and avoids an exponential increase in the number of resources that are involved in learning multiple tasks. The proposed method does not need to access the past training data and can choose suitable weights to classify the data points during the test time automatically based on an uncertainty criterion. We have evaluated the method on the MNIST and UCR time-series data sets. The evaluation results show that the method can address the catastrophic forgetting problem at a promising rate compared to the state-of-the-art models.

Over the last several years, there has been a growing focus on the CL field in the context of machine learning and its goal to create models capable of learning new tasks step by step without loss of prior knowledge. Among these, catastrophic forgetting is especially challenging in real-world settings where the data experience changes over time. To this effect, what has become pivotal for models is mechanisms for memory update to enable the models to learn information as well as update what has been previously learned easily. This survey specifically investigates the memory update strategy in the continual learning setup wherein new categories and domains are continuously added in the text datasets including sentiment analysis, named entity recognition, text classification tasks etc. Moving on, three primary memory update strategies of memory replay, memory consolidation, and parameter isolation are discussed; this paper further addresses certain adaptations of the proposed methods for text-based applications. Memory replay means that part of previous data is stored to be replayed when new tasks are learned while memory consolidation strengthens only significant memories. Parameter isolation helps avoid masking previous tasks or overwriting the parameters when the machine learning algorithm is trained to accomplish new tasks. In this paper, we discuss the latest in these techniques and offer a thorough insight into their use in text datasets such as Amazon Reviews and Yelp Reviews. Further, we outline the primary drawbacks of existing solutions for memory updates such as capacity limitations, domain variation, and continually learning without having access to new task information. In addition, a summary table of literature review identifying the most relevant works within the field is offered. Lastly, we discuss the remaining issues and potential research directions where more focus and development should be given in CL for text data by noting the importance of efficient and adaptive update policies towards the memory.

Poor quality or noisy annotations in Named Entity Recognition (NER), as in any other NLP task, make it challenging to achieve state-ofthe-art performance.In this paper, we present a multi-step framework to enhance the annotation quality of NER datasets by employing automated techniques.We propose a frequencybased iterative approach that leverages selftraining and a dual-threshold mechanism to enhance inference confidence.Experimental evaluations on different NER datasets demonstrate significant improvements in NER performance with respect to the original datasets.This work further explores the potential of generative Large Language Models (LLMs) to perform NER for low-resource languages.

Species descriptions are stored in textual form in corpora such as in floras and faunas, but this large amount of information cannot be used directly by algorithms, nor can it be linked to other data sources. The production of knowledge bases expressing structured data can benefit from collaborative and easy-to-use platforms like Xper3 (Vignes-Lebbe et al. 2017, Kerner and Vignes 2019, Saucède et al. 2021) but is very time-consuming at the human level. It is therefore mandatory for this task to make the information contained in species descriptions measurable and compatible with computer techniques. One of the most used data structures on the web and by the deep learning community is the triplet structure. Each piece of information is represented by a set of 3 elements (subject, predicate, object). One of the first steps towards species information accessibility is developing a text-to-triplet model, also known as text-to-graph, for monograph descriptions. In this work, we developed NEARSIDE, a text-to-graph model adapted to biology corpora to create normalized morphological characteristic knowledge bases for species descriptions. In Natural Language Processing, deep learning models have proven to be effective in extracting knowledge from open domain corpora (Lample et al. 2016, Sutskever et al. 2014), especially since the emergence of attention-based models (Devlin et al. 2019b, Devlin et al. 2019a). Several works have been made also on biomedical corpora (Fries et al. 2017,Cho and Lee 2019). In our case, we propose a model adapted to floras. Fully supervised deep learning models require a large amount of annotated data for training, nevertheless, the annotation process for the text-to-triplet task implies an expensive human intervention. Distant supervision is a technique that can be used to reduce this cost. This paradigm uses a small annotated glossary to project classes at the word level on a new complex and longer text (see Fig. 1). Named Entity Recognition (NER) is an Natural Language Processing (NLP) task that consists of extracting and classifying words of interest from a text (Sutskever et al. 2014, Devlin et al. 2019b, Lample et al. 2016), while triplet extraction can be compared to the Relation Extraction task (RE) which consists of extracting the words and the semantic relations between pairs of words. Distantly supervised NER is an often studied subject in the literature in comparison to distantly supervised RE (Liang et al. 2020, Meng et al. 2021) simply because NER is a subtask to RE and distant annotations generation is less expensive for the NER task (see Fig. 2). Our first contribution is creating a distantly annotated species description dataset for Named Entity Recognition with a well-balanced test set that allows us to bypass several biases that can be induced by the distant annotation and that are often observed in NER datasets (Taillé et al. 2021). In this dataset, each word of interest will be classified into one of 15 classes, each class being a specific kind of organ or descriptor. Our second contribution is proposing a distantly supervised model trained on our dataset, since fauna and flora corpora are particularly long and use a very specific technical vocabulary. We develop a context-oriented model adapted to this data by pretraining the language model. Thus the encoder of our model provides contextualized vectors for each extracted word that can be used to measure description similarities between different species. Our model reaches 96% accuracy in named entity classification on the test set. Our third contribution is the triplet construction module that can directly be applied to our model's outputs. This module is based on class dependency rules that are inspired by Xper3’s data representation format (see Fig. 3). Finally, NEARSIDE is an end-to-end structured knowledge extraction framework from unstructured species description corpora, that can be applied to several data sources. Thus making species descriptions from different corpora easily linked, compared and measured.