Natural Language Processing Tasks Research Articles

Design of an efficient Transformer-XL model for enhanced pseudo code to Python code conversion

The landscape of programming has long been challenged by the task of transforming pseudo code into executable Python code, a process traditionally marred by its labor-intensive nature and the necessity for a deep understanding of both logical frameworks and programming languages. Existing methodologies often grapple with limitations in handling variable-length sequences and maintaining context over extended textual data. Addressing these challenges, this study introduces an innovative approach utilizing the Transformer-XL model, a significant advancement in the domain of deep learning. The Transformer-XL architecture, an evolution of the standard Transformer, adeptly processes variable-length sequences and captures extensive contextual dependencies, thereby surpassing its predecessors in handling natural language processing (NLP) and code synthesis tasks. The proposed model employs a comprehensive process involving data preprocessing, model input encoding, a self-attention mechanism, contextual encoding, language modeling, and a meticulous decoding process, followed by post-processing. The implications of this work are far-reaching, offering a substantial leap in the automation of code conversion. As the field of NLP and deep learning continues to evolve, the Transformer-XL based model is poised to become an indispensable tool in the realm of programming, setting a new benchmark for automated code synthesis.

A Geometric Approach to Textual Augmented Data Filtering

Abstract Data augmentation is necessary if the amount of training data is insufficient for supervised learning. For natural language processing tasks, obtaining good quality augmented data is not easy. This paper introduces GATFilter, a novel method for filtering out inappropriate augmented textual data for text classification (TC). Utilizing geometric concepts, more specifically the principle component and convex hull analyses, this method adeptly preserves the semantic integrity of words within augmented texts. GATFilter is versatile and applicable across various types of textual augmentation methods. Experiments using several datasets and augmentation strategies showed that classifiers trained with GATFilter-filtered augmented data sets showed improvements in key performance metrics, including accuracy, precision, recall, and F1 score. The method’s efficacy is notably influenced by the quality of the underlying augmentation techniques, indicating its potential to complement and refine various text augmentation strategies. Furthermore, our analysis showed that GATFilter is particularly able to amplify the effectiveness of methods that generate good quality augmented data. GATFilter is openly available online on Github1, and as a Python package2.

An application of machine learning to classify food waste interventions from a food supply chain perspective

PurposeThe purpose of this paper is to demonstrate the efficacy of machine learning (ML) in managing natural language processing tasks, specifically by developing two ML models to systematically classify a substantial number of food waste interventions.Design/methodology/approachA literature review was undertaken to gather global food waste interventions. Subsequently, two ML models were designed and trained to classify these interventions into predefined supply chain-related groups and intervention types. To demonstrate the use of the models, a meta-analysis was performed to uncover patterns amongst the interventions.FindingsThe performance of the two classification models underscores the capabilities of ML in natural language processing, significantly enhancing the efficiency of text classification. This facilitated the rapid and effective classification of a large dataset consisting of 2,469 food waste interventions into six distinct types and assigning them to seven involved supply chain stakeholder groups. The meta-analysis reveals the most dominant intervention types and the strategies most widely adopted: 672 interventions are related to “Process and Operations Optimisation”, 457 to “Awareness and Behaviour Interventions” and 403 to “Technological and Engineering Solutions”. Prominent stakeholder groups, including “Processing and Manufacturing”, “Retail” “Government and Local Authorities” and “NGOs, Charitable Organisations and Research and Advocacy Groups”, are actively involved in over a thousand interventions each.Originality/valueThis study bridges a notable gap in food waste intervention research, a domain previously characterised by fragmentation and incomprehensive classification of the full range of interventions along the whole food supply chain. To the best of the authors’ knowledge, this is the first study to systematically classify a broad spectrum of food waste interventions while demonstrating ML capabilities. The study provides a clear, systematic framework for interventions to reduce food waste, offering valuable insight for practitioners in the food system, policymakers and consumers. Additionally, it lays the foundation for future in-depth research in the food waste reduction domain.

Textual emotion classification using MPNet and cascading broad learning

As one of the most important tasks of natural language processing, textual emotion classification (TEC) aims to recognize and detect all emotions contained in texts. However, most existing methods are implemented using deep learning approaches, which may suffer from long training time and low convergence. Motivated by these challenges, in this paper, we provide a new solution for TEC by using cascading broad learning (CBL) and sentence embedding using a masked and permuted pre-trained language model (MPNet), named CBLMP. Texts are input into MPNet to generate sentence embedding containing emotional semantic information. CBL is adopted to improve the ability of feature extraction in texts and to enhance model performance for general broad learning, by cascading feature nodes and cascading enhancement nodes, respectively. The L-curve model is adopted to ensure the balance between under-regularization and over-regularization for regularization parameter optimization. Extensive experiments have been carried out on datasets of SMP2020-EWECT and SemEval-2019 Task 3, and the results show that CBLMP outperforms the baseline methods in TEC.

Multi-Task Learning in Natural Language Processing: An Overview

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this article, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Harnessing large language models' zero-shot and few-shot learning capabilities for regulatory research.

Large language models (LLMs) are sophisticated AI-driven models trained on vast sources of natural language data. They are adept at generating responses that closely mimic human conversational patterns. One of the most notable examples is OpenAI's ChatGPT, which has been extensively used across diverse sectors. Despite their flexibility, a significant challenge arises as most users must transmit their data to the servers of companies operating these models. Utilizing ChatGPT or similar models online may inadvertently expose sensitive information to the risk of data breaches. Therefore, implementing LLMs that are open source and smaller in scale within a secure local network becomes a crucial step for organizations where ensuring data privacy and protection has the highest priority, such as regulatory agencies. As a feasibility evaluation, we implemented a series of open-source LLMs within a regulatory agency's local network and assessed their performance on specific tasks involving extracting relevant clinical pharmacology information from regulatory drug labels. Our research shows that some models work well in the context of few- or zero-shot learning, achieving performance comparable, or even better than, neural network models that needed thousands of training samples. One of the models was selected to address a real-world issue of finding intrinsic factors that affect drugs' clinical exposure without any training or fine-tuning. In a dataset of over 700000 sentences, the model showed a 78.5% accuracy rate. Our work pointed to the possibility of implementing open-source LLMs within a secure local network and using these models to perform various natural language processing tasks when large numbers of training examples are unavailable.

Evaluating Word Sense Disambiguation Techniques for Punjabi Language: A Comparative Analysis

Word Sense Disambiguation (WSD) is a fundamental task in natural language processing (NLP) that focuses on determining the precise meaning of a word by analyzing its contextual usage.This paper presents a comprehensive analysis of various WSD techniques applied to the Punjabi language, including supervised, unsupervised, and knowledge-based methods. We compare the accuracy, performance, benefits, drawbacks, and resource requirements of these techniques.The study aims to provide a detailed overview of the state of WSD for Punjabi, with visual representations such as tables and graphs to illustrate comparative performance. Key Words: Word Sense Disambiguation, Punjabi Language, Natural Language Processing, Supervised Learning, Unsupervised Learning, Knowledge-Based Approach

CLESR: Context-Based Label Knowledge Enhanced Span Recognition for Named Entity Recognition

Named entity recognition (NER) stands as a pivotal task in natural language processing, bearing profound implications for diverse downstream applications. Recent scholarship underscores the substantial performance gains attainable by integrating label knowledge into token representations, effectively treating named entity recognition as a machine reading comprehension task. Nevertheless, extant methodologies often inadequately leverage the potential of label knowledge. In response to this limitation, we introduce the Context-based Label Knowledge Enhanced Span Recognition (CLESR) architecture, designed to augment label knowledge through the assimilation of contextual information. We formulate an annotation paradigm tailored to nested scenarios, concurrently training external context-based label knowledge using conventional word association learning algorithms. The ensuing context-based label knowledge is seamlessly integrated into the model via a dedicated label attention module, thereby fortifying label learning capabilities during training. To adeptly manage both flat and nested entities, we implement a global pointer as our decoding strategy, enabling direct predictions of the positions and corresponding categories of named entities. Rigorous experimentation across six widely recognized benchmarks substantiates the efficacy of CLESR in both flat and nested tasks. Impressively, our model achieves performance enhancements surpassing those of the baseline BERT-MRC model, with gains of + 0.29%, + 0.77%, + 0.39%, + 1.9%, and + 0.47% on English CoNLL2003, English OntoNotes 4.0, Chinese MSRA, English ACE2004, and English ACE2005, respectively.

Fault transformer: An automatic fault detection algorithm on seismic images using a transformer enhanced neural network

Seismic fault detection is a key step in seismic interpretation and reservoir characterization that often requires a large amount of human labor and interpretation time. Therefore, automatic seismic fault detection is critical for improving the efficiency of seismic data processing and interpretation. Existing artificial intelligence methods are mostly based on convolutional neural networks with a U-shaped encoder-decoder structure, known as U-net. However, the convolution is limited in modeling long-range correlative features. Instead, transformers, using self-attention mechanisms, avoid the local nature of the convolution, which has the potential to extract long-distance correlations. Transformers are proven to perform well in natural language processing, image classification, and segmentation tasks in precision and recall. Here, we develop a new deep neural network with transformers and a U-net-like structure: a fault transformer to perform the fault detection task. The new network outperforms the traditional U-net in the application with synthetic data sets.

Evolution and advancements in deep learning models for Natural Language Processing

This paper provides a comprehensive review of the evolution and advancements in deep learning models for Natural Language Processing (NLP). It explores the transition from statistical models to neural networks, highlighting the paradigm shift towards data-driven methodologies and the implications for NLP tasks. The emergence of neural network architectures, such as Recurrent Neural Networks (RNNs) and transformer-based models like BERT and GPT, has revolutionized language understanding and generation. Furthermore, the integration of deep learning in traditional NLP tasks, such as part-of-speech tagging and named entity recognition, has led to significant improvements in accuracy and efficiency. The paper also discusses the quantitative analysis of deep learning models, including performance metrics, computational efficiency, and mathematical modeling of language tasks. Case studies and applications, including sentiment analysis, machine translation, and automated content generation, exemplify the transformative impact of deep learning in NLP.

MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model

Natural language-processing tasks have been improved greatly by large language models (LLMs). However, numerous parameters make their execution computationally expensive and difficult on resource-constrained devices. For this problem, as well as maintaining accuracy, some techniques such as distillation and quantization have been proposed. Unfortunately, current methods fail to integrate model pruning with downstream tasks and overlook sentence-level semantic modeling, resulting in reduced efficiency of distillation. To alleviate these limitations, we propose a novel distilled lightweight model for BERT named MicroBERT. This method can transfer the knowledge contained in the “teacher” BERT model to a “student” BERT model. The sentence-level feature alignment loss (FAL) distillation mechanism, guided by Mixture-of-Experts (MoE), captures comprehensive contextual semantic knowledge from the “teacher” model to enhance the “student” model’s performance while reducing its parameters. To make the outputs of “teacher” and “student” models comparable, we introduce the idea of a generative adversarial network (GAN) to train a discriminator. Our experimental results based on four datasets show that all steps of our distillation mechanism are effective, and the MicroBERT (101.14%) model outperforms TinyBERT (99%) by 2.24% in terms of average distillation reductions in various tasks on the GLUE dataset.

Simple Data Transformations for Mitigating the Syntactic Similarity to Improve Sentence Embeddings at Supervised Contrastive Learning

Contrastive learning of sentence representations has achieved great improvements in several natural language processing tasks. However, the supervised contrastive learning model trained on the natural language inference (NLI) dataset is insufficient to elucidate the semantics of sentences since it is prone to make a prediction based on heuristics. Herein, by using the ParsEVAL and the word overlap metric, it is shown that sentence pairs in the NLI dataset have strong syntactic similarity and propose a framework to compensate for this problem in two aspects. 1) Apply simple syntactic transformations to the hypothesis and 2) expand the objective to SupCon Loss to leverage variants of sentences. The method is evaluated on semantic textual similarity (STS) tasks and transfer tasks. The proposed methods improve the performance of the BERT‐based baseline in STS Benchmark and SICK Relatedness by 1.48% and 2.2%. Furthermore, the model achieves 82.65% on the HANS benchmark dataset, to the best of our knowledge, which is a state‐of‐the‐art performance demonstrating that our approach is effective in grasping semantics without heuristics in the NLI dataset at supervised contrastive learning. The code is available at https://github.com/whnhch/Break-the-Similarity.

Improving automatic cyberbullying detection in social network environments by fine-tuning a pre-trained sentence transformer language model

The internet use among children and adolescents has increased massively recently. This situation has promoted harmful situations such as cyberbullying, which is becoming a worldwide problem that entails serious consequences for well-being. The detection of these attitudes is essential to prevent and act accordingly. Groundbreaking techniques based on deep learning, like pre-trained language models, have achieved state-of-the-art results in many downstream Natural Language Processing tasks. This paper presents a simple but effective approach to improve the detection of cyberbullying situations by fine-tuning a pre-trained sentence transformer language model. We experimented on three datasets, and the results surpassed the state-of-the-art results. The approach could help prevent cyberbullying, filter those messages, and detect those children involved in the situation, providing opportunities to develop intervention programs to address this problem.

TransLSTM: A hybrid LSTM-Transformer model for fine-grained suggestion mining

Digital platforms on the internet are invaluable for collecting user feedback, suggestions, and opinions about various topics, such as company products and services. This data is instrumental in shaping business strategies, enhancing product development, and refining service delivery. Suggestion mining is a key task in natural language processing, which focuses on extracting and analysing suggestions from these digital sources. Initially, suggestion mining utilized manually crafted features, but recent advancements have highlighted the efficacy of deep learning models, which automatically learn features. Models like Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representations from Transformers (BERT) have been employed in this field. However, considering the relatively small datasets and the faster training time of LSTM compared to BERT, we introduce TransLSTM, a novel LSTM-Transformer hybrid model for suggestion mining. This model aims to automatically pinpoint and extract suggestions by harnessing both local and global text dependencies. It combines the sequential dependency handling of LSTM with the contextual interaction capabilities of the Transformer, thus effectively identifying and extracting suggestions. We evaluated our method against state-of-the-art approaches using the SemEval Task-9 dataset, a benchmark for suggestion mining. Our model shows promising performance, surpassing existing deep learning methods by 6.76% with an F1 score of 0.834 for SubTask A and 0.881 for SubTask B. Additionally, our paper presents an exhaustive literature review on suggestion mining from digital platforms, covering both traditional and state-of-the-art text classification techniques.

How Can Recommender Systems Benefit from Large Language Models: A Survey

With the rapid development of online services and web applications, recommender systems (RS) have become increasingly indispensable for mitigating information overload and matching users’ information needs by providing personalized suggestions over items. Although the RS research community has made remarkable progress over the past decades, conventional recommendation models (CRM) still have some limitations, e.g. , lacking open-domain world knowledge, and difficulties in comprehending users’ underlying preferences and motivations. Meanwhile, large language models (LLM) have shown impressive general intelligence and human-like capabilities for various natural language processing (NLP) tasks, which mainly stem from their extensive open-world knowledge, logical and commonsense reasoning abilities, as well as their comprehension of human culture and society. Consequently, the emergence of LLM is inspiring the design of recommender systems and pointing out a promising research direction, i.e. , whether we can incorporate LLM and benefit from their common knowledge and capabilities to compensate for the limitations of CRM. In this paper, we conduct a comprehensive survey on this research direction, and draw a bird’s-eye view from the perspective of the whole pipeline in real-world recommender systems. Specifically, we summarize existing research works from two orthogonal aspects: where and how to adapt LLM to RS. For the “ WHERE ” question, we discuss the roles that LLM could play in different stages of the recommendation pipeline, i.e. , feature engineering, feature encoder, scoring/ranking function, user interaction, and pipeline controller. For the “ HOW ” question, we investigate the training and inference strategies, resulting in two fine-grained taxonomy criteria, i.e. , whether to tune LLM or not during training, and whether to involve conventional recommendation models for inference. Detailed analysis and general development paths are provided for both “WHERE” and “HOW” questions, respectively. Then, we highlight the key challenges in adapting LLM to RS from three aspects, i.e. , efficiency, effectiveness, and ethics. Finally, we summarize the survey and discuss the future prospects. To further facilitate the research community of LLM-enhanced recommender systems, we actively maintain a GitHub repository for papers and other related resources in this rising direction 1 .

FedExLSA

Every government in the world has multiple departments that must function and operate to address the various inquiries raised by the population. The government's diverse range of websites offers citizens a platform to submit inquiries, thereby facilitating the fulfilling of their requirements. Comprehending the subjects addressed in People Query is essential for government services. Unstructured query data is analyzed using extracting information from text techniques such as allocation of Latent diffuser (LDA) and analysis of hidden semantics (LSA). LSA outperforms other methods in terms of performance because of its minimal complexity and quick installation process. Research on decentralized learning techniques for natural language processing (NLP) is necessary due to concerns about limited data availability and privacy. Federated learning (FL) employs methods that enable different users to collectively train an integrated broad model while maintaining their information regionally stored and accessible. Nevertheless, the current body of literature lacks a thorough examination and evaluation of FL techniques. Data federation is an approach to data integration that allows the government to access and query data from multiple diverse sources as if they were a single, unified repository. Functioning as a form of data virtualization, it facilitates the creation of a comprehensive representation of data, thereby enhancing operational efficiency and the accuracy of decision-making. FedEx utilizes Federated Learning to apply topic modelling techniques to common NLP tasks. The proposed structure integrates the FL Methodology with Latent Semantic Analysis to deliver outcomes for intelligent data analysis and management.

Navigating the currents of natural language processing: A comprehensive overview of modern techniques and applications

Natural Language Processing (NLP) has been a trend over the years, and many young researchers are inspired to study this field. In this paper, the author conducts a comprehensive overview of NLP from rule-based systems to machine learning techniques. Besides, the core principles of NLP and the impact of deep learning architectures, such as Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and Transformers, are explored. Additionally, this paper delves into significant NLP tasks, for instance, text classification, sentiment analysis, and machine translation. At the same time, the ethical considerations and challenges in NLP, including issues related to data bias and privacy, are also discussed. As a result, this research concludes that NLP is versatile and valuable, but it also needs to overcome challenges related to language ambiguity, data bias, privacy concerns, and so on. The ethical use of AI is necessary in order to ensure progress in NLP in the future.

From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers

The expressive power and effectiveness of large language models (LLMs) is going to increasingly push intelligent agents towards sub-symbolic models for natural language processing (NLP) tasks in human–agent interaction. However, LLMs are characterised by a performance vs. transparency trade-off that hinders their applicability to such sensitive scenarios. This is the main reason behind many approaches focusing on local post-hoc explanations, recently proposed by the XAI community in the NLP realm. However, to the best of our knowledge, a thorough comparison among available explainability techniques is currently missing, as well as approaches for constructing global post-hoc explanations leveraging the local information. This is why we propose a novel framework for comparing state-of-the-art local post-hoc explanation mechanisms and for extracting logic programs surrogating LLMs. Our experiments—over a wide variety of text classification tasks—show how most local post-hoc explainers are loosely correlated, highlighting substantial discrepancies in their results. By relying on the proposed novel framework, we also show how it is possible to extract faithful and efficient global explanations for the original LLM over multiple tasks, enabling explainable and resource-friendly AI techniques.

You only compress once: Towards effective and elastic BERT compression via exploit–explore stochastic nature gradient

Despite superior performance on various natural language processing tasks, pre-trained models such as BERT are challenged by deploying on resource-constraint devices. Most existing model compression approaches require re-compression or fine-tuning across diverse constraints to accommodate various hardware deployments. This practically limits the further application of model compression. Moreover, the ineffective training and searching process of existing elastic compression paradigms (Wang et al., 2020; Cai et al., 2020) prevents the direct migration to BERT compression. Motivated by the necessity of efficient inference across various constraints on BERT, we propose a novel approach, YOCO-BERT, to achieve compress once and deploy everywhere. Specifically, we first construct a huge search space with 1013 architectures, which covers nearly all configurations in BERT model. Then, we propose a novel stochastic nature gradient optimization method to guide the generation of optimal candidate architecture which could keep a balanced trade-off between explorations and exploitation. When a certain resource constraint is given, a lightweight distribution optimization approach is utilized to obtain the optimal network for target deployment without fine-tuning. Compared with state-of-the-art algorithms, YOCO-BERT provides more compact models, yet achieving 2.1%–4.5% average accuracy improvement on the GLUE benchmark. Besides, YOCO-BERT is also more effective, e.g., the training complexity is O(1) for N different devices. Codes available https://github.com/MAC-AutoML/YOCO-BERT.

Transformers meets neoantigen detection: a systematic literature review.

Cancer immunology offers a new alternative to traditional cancer treatments, such as radiotherapy and chemotherapy. One notable alternative is the development of personalized vaccines based on cancer neoantigens. Moreover, Transformers are considered a revolutionary development in artificial intelligence with a significant impact on natural language processing (NLP) tasks and have been utilized in proteomics studies in recent years. In this context, we conducted a systematic literature review to investigate how Transformers are applied in each stage of the neoantigen detection process. Additionally, we mapped current pipelines and examined the results of clinical trials involving cancer vaccines.