Natural Language Processing Tasks Research Articles

Symbol ungrounding: what the successes (and failures) of large language models reveal about human cognition.

Large language models can handle sophisticated natural language processing tasks. This raises the question of how their understanding of semantic meaning compares to that of human beings. Supporters of embodied cognition often point out that because these models are trained solely on text, their representations of semantic content are not grounded in sensorimotor experience. This paper contends that human cognition exhibits capabilities that fit with both the embodied and artificial intelligence approaches. Evidence suggests that semantic memory is partially grounded in sensorimotor systems and dependent on language-specific learning. From this perspective, large language models demonstrate the richness of language as a source of semantic information. They show how our experience with language might scaffold and extend our capacity to make sense of the world. In the context of an embodied mind, language provides access to a valuable form of ungrounded cognition.This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Large Language Models Meet Next-Generation Networking Technologies: A Review

The evolution of network technologies has significantly transformed global communication, information sharing, and connectivity. Traditional networks, relying on static configurations and manual interventions, face substantial challenges such as complex management, inefficiency, and susceptibility to human error. The rise of artificial intelligence (AI) has begun to address these issues by automating tasks like network configuration, traffic optimization, and security enhancements. Despite their potential, integrating AI models in network engineering encounters practical obstacles including complex configurations, heterogeneous infrastructure, unstructured data, and dynamic environments. Generative AI, particularly large language models (LLMs), represents a promising advancement in AI, with capabilities extending to natural language processing tasks like translation, summarization, and sentiment analysis. This paper aims to provide a comprehensive review exploring the transformative role of LLMs in modern network engineering. In particular, it addresses gaps in the existing literature by focusing on LLM applications in network design and planning, implementation, analytics, and management. It also discusses current research efforts, challenges, and future opportunities, aiming to provide a comprehensive guide for networking professionals and researchers. The main goal is to facilitate the adoption and advancement of AI and LLMs in networking, promoting more efficient, resilient, and intelligent network systems.

NLPLego: Assembling Test Generation for Natural Language Processing Applications

With the development of Deep Learning, Natural Language Processing (NLP) applications have reached or even exceeded human-level capabilities in certain tasks. Although NLP applications have shown good performance, they can still have bugs like traditional software and even lead to serious consequences. Inspired by Lego blocks and syntax structure analysis, we propose an assembling test generation method for NLP applications or models and implement it in NLPLego . The key idea of NLPLego is to assemble the sentence skeleton and adjuncts in order by simulating the building of Lego blocks to generate multiple grammatically and semantically correct sentences based on one seed sentence. The sentences generated by NLPLego have derivation relations and different degrees of variation. These characteristics make it well-suited for integration with metamorphic testing theory, addressing the challenge of test oracle absence in NLP application testing. To validate NLPLego , we conduct experiments on three commonly used NLP tasks (i.e., machine reading comprehension, sentiment analysis, and semantic similarity measures), focusing on the efficiency of test generation and the quality and effectiveness of generated tests. We select five advanced NLP models and one popular industrial NLP software as the tested subjects. Given seed tests from SQuAD 2.0, SST, and QQP, NLPLego successfully detects 1,732, 3,140, and 261,879 incorrect behaviors with around 93.1% precision in three tasks, respectively. The experiment results show that NLPLego can efficiently generate high-quality tests for multiple NLP tasks to detect erroneous behaviors effectively. In the case study, we analyze the testing results provided by NLPLego to obtain intuitive representations of the different NLP capabilities of the tested subjects. The case study confirms that NLPLego can provide developers with clarity on the direction to improve NLP models or applications, laying the foundation for enhancing performance.

Economic aspects of error identification in semi-structured publications in the state language

Due to the rapid growth of information on the Internet and social networks, research in the field of computational linguistics has become very relevant. The volume of information that people and machines create in natural language needs to be processed, analyzed and verified. Information retrieval systems, dialog systems, and machine translation tools are used for this. The range of automatic text processing systems is very wide, it covers various tasks. Finding errors in texts and words, identifying and correcting incorrect words is one of the most important tasks of natural language processing (NLP). The article provides an overview of semi-structured data, methods and technologies for identifying incorrect words in natural languages. The paper gives an overview of semi-structured data, methods and techniques for detecting incorrect words in natural languages. The aim of the research is to develop an effective approach for detecting and correcting errors occurring in Kazakh-language texts, especially in the context of limited resources and unstructured data. The research includes the use of machine learning techniques as well as economic analysis of the costs of developing and implementing such solutions. The proposed approach facilitates the automation of text verification, which can significantly reduce the cost of manual data processing and improve the quality of information in various spheres, including business and public administration.

Extracting Sentence Embeddings from Pretrained Transformer Models

Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT’s hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT’s sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.

TS‐Mixer: A lightweight text representation model based on context awareness

AbstractLarge pre‐trained models (PTMs) have shown their powerful ability in multiple natural language processing tasks. However, using them in practical application remains a challenge due to the significant computational cost and memory requirements. In order to achieve the balance of computational cost and accuracy, MLP architecture can be used as an alternative to the self‐attention module, such as pNLP‐Mixer and Hyper‐Mixer. Experiments indicate that, MLP‐based models can attain competitive performance with low cost. They maintain the balance of computation cost and accuracy successfully, yet, this is at the expense of not being able to capture short‐range dependencies. In this paper, a novel MLP‐based model, termed TS‐Mixer, is proposed which can capture local dependencies by shifting operation. Compared with other MLP‐based models, the parameters of TS‐Mixer are decoupled from the sequence length, hence it has a smaller model size in long sequence tasks. In addition, TS‐Mixer has linear computational complexity, therefore it can be used as a lightweight alternative to the self‐attention model. Experiments show that the TS‐Mixer outperforms other MLP‐based models, which achieves higher accuracy with fewer parameters in multiple downstream tasks. Notably, compared with pre‐trained models, TS‐Mixer can reach more than 90% of their accuracy with 1% or even one thousandth of the parameters (0.174 ~ 1.2 M). These results demonstrate that TS‐Mixer can achieve a better balance between the computing resources and accuracy. Code is available at: https://github.com/wyl-privacy-project/TS-Mixer.

Description-Enhanced Label Embedding Contrastive Learning for Text Classification.

Text classification is one of the fundamental tasks in natural language processing, which requires an agent to determine the most appropriate category for input sentences. Recently, deep neural networks have achieved impressive performance in this area, especially pretrained language models (PLMs). Usually, these methods concentrate on input sentences and corresponding semantic embedding generation. However, for another essential component: labels, most existing works either treat them as meaningless one-hot vectors or use vanilla embedding methods to learn label representations along with model training, underestimating the semantic information and guidance that these labels reveal. To alleviate this problem and better exploit label information, in this article, we employ self-supervised learning (SSL) in model learning process and design a novel self-supervised relation of relation ( [Formula: see text]) classification task for label utilization from a one-hot manner perspective. Then, we propose a novel relation of relation learning network( [Formula: see text]-Net) for text classification, in which text classification and [Formula: see text] classification are treated as optimization targets. Meanwhile, triplet loss is employed to enhance the analysis of differences and connections among labels. Moreover, considering that one-hot usage is still short of exploiting label information, we incorporate external knowledge from WordNet to obtain multiaspect descriptions for label semantic learning and extend [Formula: see text]-Net to a novel description-enhanced label embedding network(DELE) from a label embedding perspective. One step further, since these fine-grained descriptions may introduce unexpected noise, we develop a mutual interaction module to select appropriate parts from input sentences and labels simultaneously based on contrastive learning (CL) for noise mitigation. Extensive experiments on different text classification tasks reveal that [Formula: see text]-Net can effectively improve the classification performance and DELE can make better use of label information and further improve the performance. As a byproduct, we have released the codes to facilitate other research.

Hypergraph Convolutional Networks with Multi-Ordering Relations for Cross-Document Event Coreference Resolution

Recognizing the coreference relationship between different event mentions in the text (i.e., event coreference resolution) is an important task in natural language processing. It helps to understand the association between various events in the text, and plays an important role in information extraction, question answering systems, and reading comprehension. Existing research has made progress in improving the performance of event coreference resolution, but there are also some shortcomings. For example, most of the existing methods analyze the event data in the document in a serial processing mode, without considering the complex relationship between events, and it is difficult to mine the deep semantics of events. To solve these problems, this paper proposes a cross-document event co-reference resolution method (HGCN-ECR) based on hypergraph convolutional neural networks. Firstly, the BiLSTM-CRF model was used to label the semantic role of the events extracted from a number of scientific and technical kinds of literature. According to the labeling results, the trigger words and non-trigger words of the event were determined, and the multi-document event hypergraph was constructed around the event trigger words. Then hypergraph convolutional neural networks are used to learn higher-order semantic information in multi-document event hypergraphs, and multi-head attention mechanisms are introduced to understand the hidden features of different event relationship types by treating each event relationship as a set of separate attention mechanisms. Finally, the feed-forward neural network and the average link clustering method are used to calculate the coreference score of events and complete the coreference event clustering, and the cross-document event coreference resolution is realized. The experimental results show that the cross-document event co-reference resolution method is superior to the baseline model.

Causal-relationship representation enhanced joint extraction model for elements and relationships

The joint extraction of elements such as entities, relations, events, and their arguments, along with their specific interrelationships, is a critical task in natural language processing. Existing research often employs implicit handling of interactions between tasks through techniques like shared encodings or parameter sharing, lacking explicit modeling of the specific relationships between tasks. This limitation hinders the full utilization of inter-task correlation information and impacts effective collaboration between tasks. To address this, we propose a model for joint extraction of elements and relations based on causal relationship representation enhancement (CRE). This model aims to capture specific relationships between tasks in multiple stages, facilitating finer adjustments and optimization of subtasks and thereby enhancing the overall model performance. Specifically, CRE comprises three key modules: feature adaptation, feature interaction, and feature fusion. The feature adaptation module selects and adjusts features from shared encodings based on the requirements of specific tasks to better adapt to semantic differences between different tasks. The feature interaction module employs causal reasoning to comprehensively capture the causal relationships between tasks while mitigating negative transfer brought about by interference from semantic information. The feature fusion module further integrates features to obtain optimized task-specific representations. Ultimately, CRE exhibits a significant improvement in average performance across multiple information extraction tasks.

A review of reinforcement learning for natural language processing and applications in healthcare.

Reinforcement learning (RL) represents a pivotal avenue within natural language processing (NLP), offering a potent mechanism for acquiring optimal strategies in task completion. This literature review studies various NLP applications where RL has demonstrated efficacy, with notable applications in healthcare settings. To systematically explore the applications of RL in NLP, focusing on its effectiveness in acquiring optimal strategies, particularly in healthcare settings, and provide a comprehensive understanding of RL's potential in NLP tasks. Adhering to the PRISMA guidelines, an exhaustive literature review was conducted to identify instances where RL has exhibited success in NLP applications, encompassing dialogue systems, machine translation, question-answering, text summarization, and information extraction. Our methodological approach involves closely examining the technical aspects of RL methodologies employed in these applications, analyzing algorithms, states, rewards, actions, datasets, and encoder-decoder architectures. The review of 93 papers yields insights into RL algorithms, prevalent techniques, emergent trends, and the fusion of RL methods in NLP healthcare applications. It clarifies the strategic approaches employed, datasets utilized, and the dynamic terrain of RL-NLP systems, thereby offering a roadmap for research and development in RL and machine learning techniques in healthcare. The review also addresses ethical concerns to ensure equity, transparency, and accountability in the evolution and application of RL-based NLP technologies, particularly within sensitive domains such as healthcare. The findings underscore the promising role of RL in advancing NLP applications, particularly in healthcare, where its potential to optimize decision-making and enhance patient outcomes is significant. However, the ethical challenges and technical complexities associated with RL demand careful consideration and ongoing research to ensure responsible and effective implementation. By systematically exploring RL's applications in NLP and providing insights into technical analysis, ethical implications, and potential advancements, this review contributes to a deeper understanding of RL's role for language processing.

Exploring the potential of large language models for author profiling tasks in digital text forensics

The rapid advancement of large language models (LLMs) has opened up new possibilities for various natural language processing tasks. This study explores the potential of LLMs for author profiling in digital text forensics, which involves identifying characteristics such as age and gender from writing style—a crucial task in forensic investigations of anonymous or pseudonymous communications. Experiments were conducted using state-of-the-art LLMs, including Polyglot, EEVE, and Bllossom, to evaluate their performance in author profiling. Different fine-tuning strategies, such as full fine-tuning, Low-Rank Adaptation (LoRA), and Quantized LoRA (QLoRA), were compared to determine the most effective methods for adapting LLMs to the specific needs of this task. The results show that fine-tuned LLMs can effectively predict authors’ age and gender based on their writing styles, with Polyglot-based models generally outperforming EEVE and Bllossom models. Additionally, LoRA and QLoRA strategies significantly reduce computational costs and memory requirements while maintaining performance comparable to full fine-tuning. However, error analysis reveals limitations in the current LLM-based approach, including difficulty in capturing subtle linguistic variations across age groups and potential biases from pre-training data. These challenges are discussed and future research directions to address them are proposed. This study underscores the potential of LLMs in author profiling for digital text forensics, suggesting promising avenues for further exploration and refinement.

Engineering flexible machine learning systems by traversing functionally invariant paths

Contemporary machine learning algorithms train artificial neural networks by setting network weights to a single optimized configuration through gradient descent on task-specific training data. The resulting networks can achieve human-level performance on natural language processing, image analysis and agent-based tasks, but lack the flexibility and robustness characteristic of human intelligence. Here we introduce a differential geometry framework—functionally invariant paths—that provides flexible and continuous adaptation of trained neural networks so that secondary tasks can be achieved beyond the main machine learning goal, including increased network sparsification and adversarial robustness. We formulate the weight space of a neural network as a curved Riemannian manifold equipped with a metric tensor whose spectrum defines low-rank subspaces in weight space that accommodate network adaptation without loss of prior knowledge. We formalize adaptation as movement along a geodesic path in weight space while searching for networks that accommodate secondary objectives. With modest computational resources, the functionally invariant path algorithm achieves performance comparable with or exceeding state-of-the-art methods including low-rank adaptation on continual learning, sparsification and adversarial robustness tasks for large language models (bidirectional encoder representations from transformers), vision transformers (ViT and DeIT) and convolutional neural networks.

Fine-grained detoxification framework via instance-level prefixes for large language models

Large Language Models (LLMs) have demonstrated remarkable performance across various natural language processing (NLP) tasks. However, their practical usability is often compromised by a propensity to generate toxic content, such as insults, threats, and profanity, particularly in response to adversarial prompts. Several fine-tuning and decoding approaches have been employed to address this challenge to mitigate toxicity. Nevertheless, these methods typically necessitate additional resources, such as high-quality training data or auxiliary models, thereby incurring extra costs. In this paper, we propose a novel detoxification framework, Fine-Grained Detoxification via Instance-Level Prefixes (FGDILP), which effectively mitigates the generation of toxic text without incurring additional training costs. Specifically, FGDILP leverages contextualized representations in attention space by contrasting a positive prefix-prepended prompt against multiple negative prefix-prepended prompts at the instance level. This methodology facilitates the construction of fine-grained subtoxicity vectors, which are subsequently fused to adjust the original generation pathway when responding to raw prompts. Our results demonstrate that FGDILP enables controlled text generation concerning detoxification at both the utterance and context levels. While our method slightly impacts generation fluency and diversity, it significantly outperforms prompt-based baselines regarding detoxification effectiveness. Our code is available at https://github.com/xinykou/FGDILP.

Learning to generate text with auxiliary tasks

Text generation is a vital natural language processing task that generates a sequence given an input text. Most recent text generation models have mainly utilized the power of pre-trained language models (PLMs) and achieved promising results. However, while working on this problem, we observed that important indicators like salient words and sentences from downstream generation tasks can still be useful and beneficial for enhancing the performance. This paper aims to fill this gap by exploring and investigating the contribution of auxiliary supervised learning tasks to improve the performance of PLMs for text generation. To this end, we observe the nature of text generation under three scenarios: incomplete utterance restoration, question generation, and summarization. The observation suggests that the indicators can be beneficial for text generation. We reflect this nature by defining a set of auxiliary tasks. Specifically, we design a model that considers both auxiliary tasks and the main text generation task under the joint training manner. The efficiency of the model is validated in the three generation scenarios. The results of the extensive experiments on 10 datasets for the three scenarios and languages show that the auxiliary tasks significantly contribute to improving the performance of PLMs for the main generation task. The ablation study facilitates the next studies of text generation by showing the behavior of the model in different settings. The model is simple, effective, and flexible for other text generation tasks.

Research of multi-label text classification based on label attention and correlation networks.

Multi-Label Text Classification (MLTC) is a crucial task in natural language processing. Compared to single-label text classification, MLTC is more challenging due to its vast collection of labels which include extracting local semantic information, learning label correlations, and solving label data imbalance problems. This paper proposes a model of Label Attention and Correlation Networks (LACN) to address the challenges of classifying multi-label text and enhance classification performance. The proposed model employs the label attention mechanism for a more discriminative text representation and uses the correlation network based on label distribution to enhance the classification results. Also, a weight factor based on the number of samples and a modulation function based on prediction probability are combined to alleviate the label data imbalance effectively. Extensive experiments are conducted on the widely-used conventional datasets AAPD and RCV1-v2, and extreme datasets EUR-LEX and AmazonCat-13K. The results indicate that the proposed model can be used to deal with extreme multi-label data and achieve optimal or suboptimal results versus state-of-the-art methods. For the AAPD dataset, compared with the suboptimal method, it outperforms the second-best method by 2.05% ∼ 5.07% in precision@k and by 2.10% ∼ 3.24% in NDCG@k for k = 1, 3, 5. The superior outcomes demonstrate the effectiveness of LACN and its competitiveness in dealing with MLTC tasks.

Natural Language Inference with Transformer Ensembles and Explainability Techniques

Natural language inference (NLI) is a fundamental and quite challenging task in natural language processing, requiring efficient methods that are able to determine whether given hypotheses derive from given premises. In this paper, we apply explainability techniques to natural-language-inference methods as a means to illustrate the decision-making procedure of its methods. First, we investigate the performance and generalization capabilities of several transformer-based models, including BERT, ALBERT, RoBERTa, and DeBERTa, across widely used datasets like SNLI, GLUE Benchmark, and ANLI. Then, we employ stacking-ensemble techniques to leverage the strengths of multiple models and improve inference performance. Experimental results demonstrate significant improvements of the ensemble models in inference tasks, highlighting the effectiveness of stacking. Specifically, our best-performing ensemble models surpassed the best-performing individual transformer by 5.31% in accuracy on MNLI-m and MNLI-mm tasks. After that, we implement LIME and SHAP explainability techniques to shed light on the decision-making of the transformer models, indicating how specific words and contextual information are utilized in the transformer inferences procedures. The results indicate that the model properly leverages contextual information and individual words to make decisions but, in some cases, find difficulties in inference scenarios with metaphorical connections which require deeper inferential reasoning.

Hcnnxgboost: A Hybrid Cnn-Xgboost Approach for Effective Emotion Detection in Textual Data

in recent years, emotional analysis has become a key focus in computational studies, driven by the need to understand societal sentiments. this exploration is motivated by its many promising applications, such as community well-being evaluation, human-computer interaction, suicide prevention, and personalized recommendations. even with progress in other areas like identifying expressions from facial cues and speech, the study of text-based emotion recognition remains a fascinating field of research because machines struggle to interpret context, especially compared to human capabilities. our research addresses this by introducing a multiclass text-based emotion detection system that combines a cnn architecture with xgboost for improved classification in natural language processing. we pre-process publicly available datasets and use glove pre-trained word embeddings for better text representation. a major contribution of our work is enhancing the feature space by combining cnn probabilities with the original text data. the proposed hcnnxgboost model outperforms all other machine learning and deep learning algorithms across the emoint, isear, and crowdflower datasets, achieving f-scores of 90.1%, 87.4%, and 62.2%, respectively. experimental evaluations on benchmark datasets show better f-scores, confirming the effectiveness of our approach. comparisons with other classifiers highlight the enhanced performance and effectiveness of our hybrid cnnxgboost (hcnnxgboost) model, making it one of the best solutions for emotion classification in natural language processing tasks.

A Novel Optimization Scheme for Named Entity Recognition with Pre-trained Language Models

Named Entity Recognition (NER) is crucial for extracting structured information from text. While traditional methods rely on rules, Conditional Random Fields (CRFs), or deep learning, the advent of large-scale Pre-trained Language Models (PLMs) offers new possibilities. PLMs excel at contextual learning, potentially simplifying many natural language processing tasks. However, their application to NER remains underexplored. This paper investigates leveraging the GPT-3 PLM for NER without fine-tuning. We propose a novel scheme that utilizes carefully crafted templates and context examples selected based on semantic similarity. Our experimental results demonstrate the feasibility of this approach, suggesting a promising direction for harnessing PLMs in NER.

Relation measurement between semantic fields by metric approach.

The article considers a numerical research of approach for semantic metric between lexical units calculation. Received a set of statistical characteristics of the lexicographic semantic trees. Simplified representation of tree as a semantic field is proposed and operations for relation measurement between fields is described. This approach can be used for explainable language model creation for natural language processing tasks.

Named Entity Recognition Using BERT

Abstract: In our current era where technology keeps increasing, where data driven applications are reign supreme, Mamed Entity Recognition (NER) emerges as an important tool, aiding in better understanding of text and powering various natural language processing tasks downstream. This paper takes into the use of pre trained BERT model for NER tasks, Skipping the need for further adjustments. Our aim is to boost the accuracy and speed of NER systems by leveraging BERT’s pre trained representations. Our approach involves leveraging built in capabilities to spot key entity labels like names, dates, and people, streaming the NER process while achieving competitive results. This research takes a step forward in maximizing the potential of pre trained models for NER applications ultimately making text analysis easier and more efficient in today’s digital world.