Field Of Natural Language Processing Research Articles

Backdoor Attacks and Defenses Targeting Multi-Domain AI Models: A Comprehensive Review

Since the emergence of security concerns in artificial intelligence (AI), there has been significant attention devoted to the examination of backdoor attacks. Attackers can utilize backdoor attacks to manipulate model predictions, leading to significant potential harm. However, current research on backdoor attacks and defenses in both theoretical and practical fields still has many shortcomings. To systematically analyze these shortcomings and address the lack of comprehensive reviews, this paper presents a comprehensive and systematic summary of both backdoor attacks and defenses targeting multi-domain AI models. Simultaneously, based on the design principles and shared characteristics of triggers in different domains and the implementation stages of backdoor defense, this paper proposes a new classification method for backdoor attacks and defenses. We use this method to extensively review backdoor attacks in the fields of computer vision and natural language processing, and also examine the current applications of backdoor attacks in audio recognition, video action recognition, multimodal tasks, time series tasks, generative learning, and reinforcement learning, while critically analyzing the open problems of various backdoor attack techniques and defense strategies. Finally, this paper builds upon the analysis of the current state of AI security to further explore potential future research directions for backdoor attacks and defenses.

Quantum mechanics and statistical physics: Novel frameworks for enhancing natural language processing

Abstract. This article explores the pioneering application of principles from quantum mechanics and statistical mechanics to the field of natural language processing (NLP). By drawing analogies between physical phenomena such as quantum entanglement, phase transitions, and statistical ensembles, and linguistic concepts like semantic relationships, language use dynamics, and lexical diversity, we offer a novel perspective on language analysis and processing. Quantum linguistic models, leveraging the intricacies of entanglement and quantum probability, provide a framework for understanding complex semantic networks and enhancing computational efficiency through quantum computing. Meanwhile, statistical mechanics inspires models for capturing lexical diversity and understanding the evolution of language patterns, akin to phase transitions in physical systems. This interdisciplinary approach not only deepens our understanding of linguistic phenomena but also introduces advanced mathematical and computational techniques to improve NLP tasks.

A review of research on training responsible large language models

Abstract. In recent years, there has been a growing acceptance of large language models (LLM) as a mainstream method in the field of natural language processing. Consequently, numerous studies have been conducted on this topic. Training responsible Large Language Models have become a prominent subject of research in the past few years. This type of research mainly focuses on the examination of bias, morality and other aspects of LLM. There are certain similarities in the methodologies employed in those studies. This article presents a comprehensive overview of numerous recent investigations, analyzing and categorizing the methodologies employed in these studies, and offering a literature review. This review examines the three perspectives of LLM bias data set construction, bias detection and bias elimination It provides a comparative analysis of the advantages and disadvantages of different methods. After completing relevant evaluations, a comprehensive examination of the research on training responsible LLM is conducted and potential future research directions are proposed in this article.

Causal Inference in Natural Language Processing: Application Status and Future Outlook

Abstract. Causal reasoning has received much attention in recent years. Unlike statistical learning that focuses on the correlation between variables, causal inference can analyze the causality between variables and avoid false causal relationships caused by confounding factors. The use of causal inference to help solve Natural Language Processing (NLP) problems has made great progress. Most existing studies have focused on exploring the application of causal relationships in downstream tasks of NLP, achieving good results. This article conducted a comprehensive survey, focusing on investigating the advantages of causal inference methods in solving specific NLP problems and applications, as well as their practical advantages compared to deep learning methods. Intended to provide researchers in the NLP field with more detailed perspectives and recommendations. Specifically, this article first introduces the relevant concepts of causal inference, including causal inference and related concepts of causal relationships, as well as the differences between causal inference and statistical machine learning; Then, existing research on the use of causal relationships in downstream applications was summarized, including bias removal, stock price prediction, document dialogue based, and continuous few shot learning. Finally, a summary was made on the development of future causal relationships.

Limitations of Feature Attribution in Long Text Classification of Standards

Managing complex AI systems requires insight into a model's decision-making processes. Understanding how these systems arrive at their conclusions is essential for ensuring reliability. In the field of explainable natural language processing, many approaches have been developed and evaluated. However, experimental analysis of explainability for text classification has been largely constrained to short text and binary classification. In this applied work, we study explainability for a real-world task where the goal is to assess the technological suitability of standards. This prototypical use case is characterized by large documents, technical language, and a multi-label setting, making it a complex modeling challenge. We provide an analysis of approx. 1000 documents with human-annotated evidence. We then present experimental results with two explanation methods evaluating plausibility and runtime of explanations. We find that the average runtime for explanation generation is at least 5 minutes and that the model explanations do not overlap with the ground truth. These findings reveal limitations of current explanation methods. In a detailed discussion, we identify possible reasons and how to address them on three different dimensions: task, model and explanation method. We conclude with risks and recommendations for the use of feature attribution methods in similar settings.

Artificial Intelligence Techniques for Sustainable Reconfigurable Manufacturing Systems: An AI-Powered Decision-Making Application Using Large Language Models

Artificial intelligence (AI) offers a promising avenue for developing sustainable reconfigurable manufacturing systems. Although there has been significant progress in these research areas, there seem to be no studies devoted to exploring and evaluating AI techniques for such systems. To address this gap, the current study aims to present a deliberation on the subject matter, with a particular focus on assessing AI techniques. For this purpose, an AI-enabled methodological approach is developed in Python, integrating fuzzy logic to effectively navigate the uncertainties inherent in evaluating the performance of techniques. The incorporation of sensitivity analysis further enables a thorough evaluation of how input variations impact decision-making outcomes. To conduct the assessment, this study provides an AI-powered decision-making application using large language models in the field of natural language processing, which has emerged as an influential branch of artificial intelligence. The findings reveal that machine learning and big data analytics as well as fuzzy logic and programming stand out as the most promising AI techniques for sustainable reconfigurable manufacturing systems. The application confirms that using fuzzy logic programming in Python as the computational foundation significantly enhances precision, efficiency, and execution time, offering critical insights that enable more timely and informed decision-making in the field. Thus, this study not only addresses a critical gap in the literature but also offers an AI-driven approach to support complex decision-making processes.

Exploring Sentiment Analysis for the Indonesian Presidential Election Through Online Reviews Using Multi-Label Classification with a Deep Learning Algorithm

Presidential elections are an important political event that often trigger intense debate. With more than 139 million users, YouTube serves as a significant platform for understanding public opinion through sentiment analysis. This study aimed to implement deep learning techniques for a multi-label sentiment analysis of comments on YouTube videos related to the 2024 Indonesian presidential election. Offering a fresh perspective compared to previous research that primarily employed traditional classification methods, this study classifies comments into eight emotional labels: anger, anticipation, disgust, joy, fear, sadness, surprise, and trust. By focusing on the emotional spectrum, this study provides a more nuanced understanding of public sentiment towards presidential candidates. The CRISP-DM method is applied, encompassing stages of business understanding, data understanding, data preparation, modeling, evaluation, and deployment, ensuring a systematic and comprehensive approach. This study employs a dataset comprising 32,000 comments, obtained via YouTube Data API, from the KPU and Najwa Shihab channels. The analysis is specifically centered on comments related to presidential candidate debates. Three deep learning models—Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (Bi-LSTM), and a hybrid model combining CNN and Bi-LSTM—are assessed using confusion matrix, Area Under the Curve (AUC), and Hamming loss metrics. The evaluation results demonstrate that the Bi-LSTM model achieved the highest accuracy with an AUC value of 0.91 and a Hamming loss of 0.08, indicating an excellent ability to classify sentiment with high precision and a low error rate. This innovative approach to multi-label sentiment analysis in the context of the 2024 Indonesian presidential election expands the insights into public sentiment towards candidates, offering valuable implications for political campaign strategies. Additionally, this research contributes to the fields of natural language processing and data mining by addressing the challenges associated with multi-label sentiment analysis.

Uncovering suggestions in MOOC discussion forums: a transformer-based approach

The field of natural language processing has experienced significant advances in recent years, but these advances have not yet resulted in improved analytics for instructors on MOOC platforms. Valuable information, such as suggestions, is generated in the comment forums of these courses, but due to their volume, manual processing is often impractical. This study examines the feasibility of fine-tuning and effectively utilizing state-of-the-art deep learning models to identify comments that contain suggestions in MOOC forums. The main challenges encountered are the lack of labeled datasets from the MOOC context for fine-tuning classification models and the soaring computational cost of this training. For this study, we manually collected and labeled 2228 comments in Spanish and English from 5 MOOCs and scraped 1.4 million MOOC reviews from 3 platforms. We fine-tuned and evaluated 4 pretrained models based on the transformer architecture and 3 traditional machine learning models to compare their effectiveness in the suggestion mining task in this domain. Transformer-based models proved to be highly effective in this task/domain combination, achieving performance levels that matched or exceeded those deemed appropriate in other contexts and were significantly greater than those achieved by traditional models. Domain adaptation led to improved linguistic understanding of the target domain; however, in this project, this approach did not translate into an observable improvement in suggestion mining. The automated identification of comments that can be labeled as suggestions can result in considerable time savings for instructors, especially considering that less than a quarter of the analyzed comments contain suggestions.

Exploring the effectiveness of instruction tuning in biomedical language processing

Large Language Models (LLMs), particularly those similar to ChatGPT, have significantly influenced the field of Natural Language Processing (NLP). While these models excel in general language tasks, their performance in domain-specific downstream tasks such as biomedical and clinical Named Entity Recognition (NER), Relation Extraction (RE), and Medical Natural Language Inference (NLI) is still evolving. In this context, our study investigates the potential of instruction tuning for biomedical language processing, applying this technique to two general LLMs of substantial scale. We present a comprehensive, instruction-based model trained on a dataset that consists of approximately 200,000 instruction-focused samples. This dataset represents a carefully curated compilation of existing data, meticulously adapted and reformatted to align with the specific requirements of our instruction-based tasks. This initiative represents an important step in utilising such models to achieve results on par with specialised encoder-only models like BioBERT and BioClinicalBERT for various classical biomedical NLP tasks. Our work includes an analysis of the dataset’s composition and its impact on model performance, providing insights into the intricacies of instruction tuning. By sharing our codes, models, and the distinctively assembled instruction-based dataset, we seek to encourage ongoing research and development in this area.22Our code repository is available at https://github.com/nlpie-research/BioInstTune-LLM.

Electric Vehicle Sentiment Analysis Using Large Language Models

Sentiment analysis is a technique used to understand the public’s opinion towards an event, product, or organization. For example, sentiment analysis can be used to understand positive or negative opinions or attitudes towards electric vehicle (EV) brands. This provides companies with valuable insight into the public’s opinion of their products and brands. In the field of natural language processing (NLP), transformer models have shown great performance compared to traditional machine learning algorithms. However, these models have not been explored extensively in the EV domain. EV companies are becoming significant competitors in the automotive industry and are projected to cover up to 30% of the United States light vehicle market by 2030 In this study, we present a comparative study of large language models (LLMs) including bidirectional encoder representations from transformers (BERT), robustly optimised BERT (RoBERTa), and a generalised autoregressive pre-training method (XLNet) using Lucid Motors and Tesla Motors YouTube datasets. Results evidenced that LLMs like BERT and her variants are off-the-shelf algorithms for sentiment analysis, specifically when fine-tuned. Furthermore, our findings present the need for domain adaptation whilst utilizing LLMs. Finally, the experimental results showed that RoBERTa achieved consistent performance across the EV datasets with an F1 score of at least 92%.

The fluency-based semantic network of LLMs differs from humans

Modern Large Language Models (LLMs) exhibit complexity and granularity similar to humans in the field of natural language processing, challenging the boundaries between humans and machines in language understanding and creativity. However, whether the semantic network of LLMs is similar to humans is still unclear. We examined the representative closed-source LLMs, GPT-3.5-Turbo and GPT-4, with open-source LLMs, LLaMA-2-70B, LLaMA-3-8B, LLaMA-3-70B using semantic fluency tasks widely used to study the structure of semantic networks in humans. To enhance the comparability of semantic networks between humans and LLMs, we innovatively employed role-playing to generate multiple agents, which is equivalent to recruiting multiple LLM participants. The results indicate that the semantic network of LLMs has poorer interconnectivity, local association organization, and flexibility compared to humans, which suggests that LLMs have lower search efficiency and more rigid thinking in the semantic space and may further affect their performance in creative writing and reasoning.

EEG Emotion Recognition Based on GADF and AMB‐CNN Model

ABSTRACTDeep learning has achieved better results in natural language processing, computer vision, and other fields. Nowadays, more deep learning algorithms have also been applied in brain‐based emotion recognition. In the studies on brain‐based emotion recognition, deep learning models typically use one‐dimensional time series as the input and cannot fully leverage the advantages of the models in image classification or recognition. To address this issue, based on the publicly available SEED and DEAP datasets, the Gramian angular difference field (GADF) method was proposed to construct two‐dimensional image representation datasets: SEED‐GADF and DEAP‐GADF datasets, in the paper. Additionally, a convolutional attention mechanism model (AMB‐CNN) was introduced and its classification performance was validated on SEED‐GADF and DEAP‐GADF datasets. AMB‐CNN achieved an average accuracy of 90.8%, a recall rate of 90%, and AUC of 96.86% on SEED‐GADF. On DEAP‐GADF, the average accuracy, recall rate, and AUC respectively reached 96.06%, 96.06%, and 98.58% in the valence dimension and 96.11%, 96.11%, and 98.73% in the arousal dimension. Finally, the comparison results with various algorithms and ablation experiments proved the superiority of the proposed model.

Performance and sustainability of BERT derivatives in dyadic data

In recent years, the Natural Language Processing (NLP) field has experienced a revolution, where numerous models –based on the Transformer architecture– have emerged to process the ever-growing volume of online text-generated data. This architecture has been the basis for the rise of Large Language Models (LLMs). Enabling their application to many diverse tasks in which they excel with just a fine-tuning process that comes right after a vast pre-training phase. However, their sustainability can often be overlooked, especially regarding computational and environmental costs. Our research aims to compare various BERT derivatives in the context of a dyadic data task while also drawing attention to the growing need for sustainable AI solutions. To this end, we utilize a selection of transformer models in an explainable recommendation setting, modeled as a multi-label classification task originating from a social network context, where users, restaurants, and reviews interact.

Enhancing Public Health: training an AI chatbot for complex tasks support

Abstract Background The evolution of Large Language Models (LLMs) such as GPT-4, Gemini, and Copilot has transformed the field of Natural Language Processing, demonstrating human-level capabilities and potential applications in Public Health. Here, we report on the training of a chatbot model based on GPT-4 designed to assist operators with relevant and complex public health tasks. Methods A scoping review of existing studies on the application of LLMs in Public Health settings was conducted to understand the landscape and identify gaps in their current implementations. We then proceeded to analyse the necessary skills and competencies required for public health professionals to adequately instruct the model on required tasks. Finally, we selected the Italian regulations and guidelines with the greatest relevance in the public health field to provide the knowledge base to the model. Results The review revealed no relevant documents on the use of LLMs in areas related to Public Health. Skills and competencies were retrieved from the Guidelines of the Italian Society of Hygiene for Public Health operators and used to instruct the model on how to address public health tasks. As for the knowledge base, we retrieved 25 documents regulating specific areas of the public health field, such as vaccinations, hygiene and safety at work, infection prevention, and school hygiene. Such documents were used to train the model, after formatting them appropriately. Conclusions Despite challenges with non-standardized and technically complex documentation, preliminary testing of our model has shown its versatility and utility in supporting daily public health operations. A validation phase is under definition and will involve a DELPHI process, in which products of the custom GPT model will be evaluated by a panel of public health operators to evaluate its performance in real-world settings, ensuring compliance with national health regulations. Key messages • Training a GPT-4-based chatbot to support complex tasks in Public Health, tailored by relevant national guidelines. • Our custom model, informed by public health guidelines, showed potential in preliminary tests.

Knowledge-injected Prompt Learning for Chinese Biomedical Entity Normalization

The Biomedical Entity Normalization (BEN) task aims to align raw, unstructured medical entities to standard entities, thus promoting data coherence and facilitating better downstream medical applications. Recently, prompt learning methods have shown promising results in the natural language processing field. However, existing research falls short in tackling the more complex Chinese BEN task, especially in the few-shot scenario with limited medical data, and the vast potential of the external medical knowledge base has not yet been fully exploited. To address these challenges, this article proposes a novel Knowledge-injected Prompt Learning (PL-Knowledge) method. Specifically, the approach consists of five stages: candidate entity matching, knowledge extraction, knowledge encoding, knowledge injection, and prediction output. By effectively encoding the knowledge items contained in medical entities and incorporating them into tailor-made knowledge-injected templates, the additional knowledge enhances the model’s ability to capture latent relationships between medical entities, thus achieving a better match with the standard entities. Comprehensive experiments are conducted on a benchmark dataset in both few-shot and full-scale settings. This method outperforms existing baselines, with an average accuracy improvement of 12.96 percentage points in few-shot and 0.94 percentage points in full-data cases, showcasing its excellence in the BEN task.

Deep Learning Applications in Natural Language Processing and Optimization Strategies

In recent years, deep learning has made significant advancements in the field of natural language processing (NLP), driving the development of language understanding and generation tasks. Deep learning utilizes multi-layer neural networks to automatically extract and represent complex data features. This paper first introduces the basic principles of deep learning and mainstream frameworks (such as TensorFlow and PyTorch) and discusses core NLP tasks, including word embedding, language modeling, and text generation. Subsequently, it analyzes specific practices of deep learning applications in text classification, machine translation, automatic summarization, dialogue systems, and speech recognition. Further discussion covers model optimization methods, including structural optimization (such as RNN, LSTM, GRU, and Transformer), data preprocessing and feature engineering, hyperparameter tuning, and accelerated computation. Finally, it explores cutting-edge optimization strategies like federated learning, model compression, self-supervised learning, and transfer learning, proposing future research directions and challenges. This paper aims to systematically analyze the applications and optimization strategies of deep learning in NLP.

A Hybrid Ensemble Approach for Greek Text Classification Based on Multilingual Models

The present study explores the field of text classification in the Greek language. A novel ensemble classification scheme based on generated embeddings from Greek text made by the multilingual capabilities of the E5 model is presented. Our approach incorporates partial transfer learning by using pre-trained models to extract embeddings, enabling the evaluation of classical classifiers on Greek data. Additionally, we enhance the predictive capability while maintaining the costs low by employing a soft voting combination scheme that exploits the strengths of XGBoost, K-nearest neighbors, and logistic regression. This method significantly improves all classification metrics, demonstrating the superiority of ensemble techniques in handling the complexity of Greek textual data. Our study contributes to the field of natural language processing by proposing an effective ensemble framework for the categorization of Greek texts, leveraging the advantages of both traditional and modern machine learning techniques. This framework has the potential to be applied to other less-resourced languages, thereby broadening the impact of our research beyond Greek language processing.

A Transformer-Based Yoruba to English Machine Translation (TYEMT) System with Rouge Score

Automated translation systems for some indigenous Nigerian languages like the Yoruba, have historically been limited by the lack of large, high- quality bilingual text and effective approaches to modeling. This paper presents introduces an approach to bi-directional Yoruba-English text-to-text machine translation utilizing deep learning technique, specifically Transformer models. Transformer models, which utilizes self-attention mechanisms to improve translation quality and efficiency. The system was trained and evaluated on a newly curated Yoruba- English parallel corpus, which significantly augments existing resources. Experimental results demonstrate that the Transformer-based model performs translation accurately and fluently, achieving a ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score improvement of 0.4649. This work not only advances the frontiers of Yoruba-English machine translation but also enriches a wider domain in the field of multilingual Natural Language processing (NLP) by addressing challenges associated with translating between languages with limited resources. Future studies include enhancing the available parallel corpus and exploring hybrid models that combine the strengths of both RNN and Transformer architectures.

A REVIEW ON MACHINE LEARNING AND DEEP LEARNING TECHNIQUES FOR TEXTUAL EMOTION ANALYSIS ON SOCIAL NETWORKS

In recent times emotion detection has achieved significant attention in the field of Natural Language Processing (NLP) due to the abundance of text data available on different social network platforms like Twitter, LinkedIn and Reddit. This paper presents a thorough review of existing emotion detection techniques on text analysis. The methodology involves a comparative analysis of different machinelearning and deep learning models, approaches and datasets utilised for emotion detection. The article discusses the limitations and challenges of traditional methods and delves into the theoretical foundations of machine learning and deep learning techniques such as SVM, CNN and BERT. By exploring the current state-of-the-art in emotion detection from social networks, this review aims to provide insight for researchers to advance the field of emotion detection.

Evaluation-Focused Multidimensional Score for Turkish Abstractive Text Summarization

Despite the inherent complexity of Abstractive Text Summarization, which is widely acknowledged as one of the most challenging tasks in the field of natural language processing, transformer-based models have emerged as an effective solution capable of delivering highly accurate and coherent summaries. In this study, the effectiveness of transformer-based text summarization models for Turkish language is investigated. For this purpose, we utilize BERTurk, mT5 and mBART as transformer-based encoder-decoder models. Each of the models was trained separately with MLSUM, TR-News, WikiLingua and Fırat_DS datasets. While obtaining experimental results, various optimizations were made in the summary functions of the models. Our study makes an important contribution to the limited Turkish text summarization literature by comparing the performance of different language models on existing Turkish datasets. We first evaluate ROUGE, BERTScore, FastText-based Cosine Similarity and Novelty Rate metrics separately for each model and dataset, then normalize and combine the scores we obtain to obtain a multidimensional score. We validate our innovative approach by comparing the summaries produced with the human evaluation results.