Pre-trained Language Models Research Articles

ENHANCING NATURAL LANGUAGE PROCESSING USING WALRUS OPTIMIZER WITH SELF-ATTENTION DEEP LEARNING MODEL IN APPLIED LINGUISTICS

Sentiment analysis (SA) is the popular natural language processing (NLP) problem utilized for analyzing texts, including uploaded reviews and user posts on e-commerce portals, forums, and social media platforms, regarding their opinions about the event or person, product, and service. The SA task comprises analyzing text to determine whether the sentiment expressed is negative, positive, or neutral, aiming for precise subjective data analysis. The deep learning (DL) technique enables various architectures to model SA tasks and has surpassed other machine learning (ML) techniques as the first method to perform SA tasks. Recent advancements in the DL model depart from the growing dominance of transformer language algorithms and convolutional neural network (CNN) and recurrent neural network (RNN) models. Utilizing pre-trained transformer language models to transfer knowledge to downstream tasks has emerged as a cutting-edge model in NLP. Therefore, the study designs a fractal walrus optimizer with self-attention DL-based SA in applied linguistics (WOSADL-SAAL) method. The WOSADL-SAAL method aims to recognize and classify the presence of sentiments in social media content. In the WOSADL-SAAL technique, data preprocessing is initially performed to transform the input dataset into a meaningful format. In addition, the WOSADL-SAAL technique uses the bag of words (BoW) model for extracting features. The self-attention bidirectional long short-term memory (SA-BiLSTM) network is applied to classify sentiment. The walrus optimizer (WO) model performs the hyperparameter tuning model to boost the detection outcomes of the SA-BiLSTM network. The performance evaluation of the WOSADL-SAAL method takes place under the benchmark SA dataset. The experimental analysis of the WOSADL-SAAL method exhibited a superior accuracy value of 99.07% and 99.24% under TUSA and IMDB datasets over existing approaches.

Fine-tuned PhoBERT for sentiment analysis of Vietnamese phone reviews

This paper presents an exploration of sentiment analysis applied to Vietnamese phone reviews, leveraging the PhoBERT model. While significant advancements have been made in sentiment analysis for English and other widely spoken languages, Vietnamese remains relatively under investigated. Our study addresses this gap by constructing a comprehensive dataset that integrates data from the UIT-ViSFD dataset and data collected through web scraping. We experimented with various models including naive Bayes, Support Vector Machine, and PhoBERT, utilizing multiple data preprocessing techniques. PhoBERT, a state-of-the-art pre-trained language model specifically designed for Vietnamese, demonstrated superior performance. The final PhoBERT model with optimized preprocessing achieved an accuracy of 92.74%, highlighting its efficacy in accurately identifying sentiments.

Prediction of human O-linked glycosylation sites using stacked generalization and embeddings from pre-trained protein language model.

O-linked glycosylation, an essential post-translational modification process in Homo sapiens, involves attaching sugar moieties to the oxygen atoms of serine and/or threonine residues. It influences various biological and cellular functions. While threonine or serine residues within protein sequences are potential sites for O-linked glycosylation, not all serine and/or threonine residues undergo this modification, underscoring the importance of characterizing its occurrence. This study presents a novel approach for predicting intracellular and extracellular O-linked glycosylation events on proteins, which are crucial for comprehending cellular processes. Two base multi-layer perceptron models were trained by leveraging a stacked generalization framework. These base models respectively employ ProtT5 and Ankh O-linked glycosylation site-specific embeddings whose combined predictions are used to train the meta-multi-layer perceptron model. Trained on extensive O-linked glycosylation datasets, the stacked-generalization model demonstrated high predictive performance on independent test datasets. Furthermore, the study emphasizes the distinction between nucleocytoplasmic and extracellular O-linked glycosylation, offering insights into their functional implications that were overlooked in previous studies. By integrating the protein language model's embedding with stacked generalization techniques, this approach enhances predictive accuracy of O-linked glycosylation events and illuminates the intricate roles of O-linked glycosylation in proteomics, potentially accelerating the discovery of novel glycosylation sites. Stack-OglyPred-PLM produces Sensitivity, Specificity, Matthews Correlation Coefficient, and Accuracy of 90.50%, 89.60%, 0.464, and 89.70%, respectively on a benchmark NetOGlyc-4.0 independent test dataset. These results demonstrate that Stack-OglyPred-PLM is a robust computational tool to predict O-linked glycosylation sites in proteins. The developed tool, programs, training, and test dataset are available at https://github.com/PakhrinLab/Stack-OglyPred-PLM. Supplementary data are available at Bioinformatics online.

PPTopicPLM: plug-and-play topic-enhanced pre-trained language model for short-text rumor detection

PPTopicPLM: plug-and-play topic-enhanced pre-trained language model for short-text rumor detection

Accelerating and Compressing Transformer-Based PLMs for Enhanced Comprehension of Computer Terminology

Pretrained language models (PLMs) have significantly advanced natural language processing (NLP), establishing the "pretraining + fine-tuning" paradigm as a cornerstone approach in the field. However, the vast size and computational demands of transformer-based PLMs present challenges, particularly regarding storage efficiency and processing speed. This paper addresses these limitations by proposing a novel lightweight PLM optimized for accurately understanding domain-specific computer terminology. Our method involves a pipeline parallelism algorithm designed to accelerate training. It is paired with an innovative mixed compression strategy that combines pruning and knowledge distillation to effectively reduce the model size while preserving its performance. The model is further fine-tuned using a dataset that mixes source and target languages to enhance its versatility. Comprehensive experimental evaluations demonstrate that the proposed approach successfully achieves a balance between model efficiency and performance, offering a scalable solution for NLP tasks involving specialized terminology.

Few-Shot Methods for Aspect-Level Sentiment Analysis

In this paper, we explore the approaches to the problem of cross-domain few-shot classification of sentiment aspects. By cross-domain few-shot, we mean a setting where the model is trained on large data in one domain (for example, hotel reviews) and is intended to perform on another (for example, restaurant reviews) with only a few labelled examples in the target domain. We start with pre-trained monolingual language models. Using the Polish language dataset AspectEmo, we compare model training using standard gradient-based learning to a zero-shot approach and two dedicated few-shot methods: ProtoNet and NNShot. We find both dedicated methods much superior to both gradient learning and zero-shot setup, with a small advantage held by NNShot. Overall, we find few-shot to be a compelling alternative, achieving a surprising amount of performance compared to gradient training on full-size data.

Descriptive Case Analysis on the Application of Prompt Engineering in Business Management

In recent years, the application of AI has been witnessed in various domains and disciplines, mainly AI in terms of large-scale pre-trained language models (LMs) and Visual Language models which are the prompt engineering types that have gained increasing attention in the field of Business and Management field. The prompt engineering approach is considered vital leverage in the areas of BM tasks. In this AI-generated review, we will try to discuss various aspects of prompt engineering encompassing the domain of management and also highlight various applications in this area. This paper also tries to examine the challenges of applying prompt engineering in the context of business management. This is a descriptive case study research exploring various facets of prompt engineering and application on the spectrum of business management. Through this exploration, this paper intends to contribute to the evolving dialogue on the integration of human creativity with generative AI capabilities, offering insights into the future of effective and ethical AI interaction. Prompt engineering, Artificial intelligence, large language models, Visual language models, Business Management, Human-AI interaction, Chat GPT, Generative AI

Reconstructing representations using diffusion models for multimodal sentiment analysis through reading comprehension

The primary challenge in multimodal sentiment analysis (MSA), which utilizes textual, audio, and visual information to analyze speakers' emotions, lies in constructing representation vectors that incorporate both unimodal semantic and multimodal interaction information. While existing research has extensively focused on multimodal fusion strategies, there remains insufficient exploration of the intrinsic potential within concurrently enhancing unimodal and multimodal representations. To address this gap, we introduce two additional steps to traditional three-step MSA: text modality enhancement through the machine reading comprehension (MRC) framework and multimodal representation reconstruction via proposing the diverse diffusion denoising autoencoder (D3AE) module. The MRC queries are integrated to locate sentiment-related prior knowledge, thereby deepening textual semantic understanding from a pretrained language model. Meanwhile, D3AE employs a single-step denoising strategy along with diffusion models across multiple time intervals, enabling efficient reconstruction and enhancement of multimodal representations. Extensive experiments conducted on two benchmark datasets, CMU-MOSI and CMU-MOSEI, validate that our model, MRC-D3AE, achieves state-of-the-art performance. The superiority of our model over existing baselines is primarily attributed to integrating MRC for enhancing text modality and D3AE for reconstructing multimodal representations.

Chinese word segmentation with heterogeneous graph convolutional network

Abstract Recently, deep learning methods have achieved remarkable success in the Chinese word segmentation (CWS) task. Some of them enhance the CWS model by utilizing contextual features and external resources (e.g., sub-words, lexicon, and syntax). However, existing approaches fail to fully use the heterogeneous features and their structural information. Therefore, in this paper, we propose a heterogeneous information learning framework for CWS, named heterogeneous graph neural segmenter (HGNSeg), which exploits heterogeneous features with the graph convolutional networks and the pretrained language model. Experimental results on six benchmark datasets (e.g., SIGHAN 2005 and SIGHAN 2008) confirm that HGNSeg can effectively improve the performance of CWS. Importantly, HGNSeg also demonstrates an excellent ability to alleviate the out-of-vocabulary (OOV) issue in cross-domain scenarios.

On Elastic Language Models

Large-scale pretrained language models have achieved compelling performance in a wide range of language understanding and information retrieval tasks. While their large scales ensure capacity, they also hinder deployment. Knowledge distillation offers an opportunity to compress a large language model to a small one, in order to reach a reasonable latency-performance tradeoff. However, for scenarios where the number of requests (e.g., queries submitted to a search engine) is highly variant, the static tradeoff attained by the compressed language model might not always fit. Once a model is assigned with a static tradeoff, it could be inadequate in that the latency is too high when the number of requests is large, or the performance is too low when the number of requests is small. To this end, we propose an elastic language model ( ElasticLM ) that elastically adjusts the tradeoff according to the request stream. The basic idea is to introduce a compute elasticity to the compressed language model, so that the tradeoff could vary on-the-fly along a scalable and controllable compute. Specifically, we impose an elastic structure to equip ElasticLM with compute elasticity and design an elastic optimization method to learn ElasticLM under compute elasticity. To serve ElasticLM , we apply an elastic schedule. Considering the specificity of information retrieval, we adapt ElasticLM to dense retrieval and reranking, and present an ElasticDenser and an ElasticRanker, respectively. Offline evaluation is conducted on a language understanding benchmark GLUE, and several information retrieval tasks including Natural Question, Trivia QA and MS MARCO. The results show that ElasticLM along with ElasticDenser and ElasticRanker can perform correctly and competitively compared with an array of static baselines. Furthermore, an online simulation with concurrency is also carried out. The results demonstrate that ElasticLM can provide elastic tradeoffs with respect to varying request stream.

Zero-Shot Relation Classification Through Inference on Category Attributes.

The goal of relationship classification (RC) is to predict the semantic relationship between two entities in a given sentence. With the advent of deep learning and pretrained language models, RC research has progressed by leaps and bounds. However, the current studies are focused mainly on predicting semantic relationships from a predefined set. How to recognize unseen relationships remains a challenge, which is also known as the zero-shot RC (ZSRC) task. Some ZSRC-related methods directly map relationship categories to numerical indices, constraining the model's ability to autonomously infer and understand these relationships, while others rely heavily on manual definitions. To address these issues and inspired by the way of reasoning in which humans perform RC tasks, we propose a new framework to handle the ZSRC task through inference on category attributes (ICAs). The main idea of ICA is to detect the semantic relationship between promises, which are RC sentences, and hypotheses, which are relational sentences of entities created by templates. Specifically, instead of manual design, we introduce two hypothesis templates derived from the label words (LWs) and descriptions (LDs) associated with each relationship. These templates are used to automatically convert the RC data into the textual entailment (TE) format. Furthermore, they are fine-tuned with a pretrained TE model, facilitating the acquisition of relational knowledge and enabling the generalization of semantic reasoning rules learned from seen classes to unseen classes. Moreover, to implement multirelationship semantic inference for all unseen classes, we propose an entailment difference mechanism to enhance the reasoning capability of the model. Besides the current ZSRC test setting, we also examine our method in an even more challenging setting to deal with data scarcity in real-world applications. The outstanding performance of ICA on the FewRel and Wiki-ZSL datasets demonstrates its effectiveness in the ZSRC task.

A transformer-based approach to Nigerian Pidgin text generation

AbstractThis paper describes the development of a transformer-based text generation model for Nigerian Pidgin also known as Naijá, a popular language in West Africa. Despite its wide use, Nigerian Pidgin remains under-resourced, particularly in areas related to text generation and natural language processing. These difficulties are primarily due to technological constraints rather than the language’s fundamental attributes. There is currently a demand for Nigerian Pidgin-specific solutions because it is used in everyday communication and has a unique linguistic blend. This paper aims to close this gap by exploring the application of state-of-the-art transformer technology to develop a text generation model for Nigerian Pidgin. This work uses the public Afriberta-corpus dataset to optimize the Generative Pre-trained Transformer (GPT-2) model across a sizeable dataset. The performance evaluators, BLEU and Perplexity metrics provide a detailed breakdown of the model’s text quality and predictive accuracy. Despite the difficulties caused by a limited amount of training data, preliminary evaluations show that the model can generate coherent Nigerian Pidgin text. The performance evaluation yielded perplexity scores of 43.56 for variable target reference length and 43.26 for fixed text length. BLEU scores of 0.15 for fixed max length and 0.56 for variable reference target length. This highlights the quality of generated text and the significant improvement when the generated text length is aligned with the reference target. Our work was benchmarked against African American Vernacular (AAVE) revealing that BLEU scores for AAVE are significantly lower than those for Standard American English, with BLEU given as 0.26. Our Nigerian Pidgin model, with a BLEU score of 0.56, shows a better performance. However, both results suggest that both dialects are challenging for language models. Leveraging the pre-trained transformer-based language model and evaluation metrics, we showcase the model’s capacity for coherent Nigerian Pidgin text generation. For future research, the research work can serve as a good foundation for advancement and progress in the Nigerian Pidgin language generation and other low-resource languages.

Toward the Adoption of Explainable Pre-Trained Large Language Models for Classifying Human-Written and AI-Generated Sentences

Pre-trained large language models have demonstrated impressive text generation capabilities, including understanding, writing, and performing many tasks in natural language. Moreover, with time and improvements in training and text generation techniques, these models are proving efficient at generating increasingly human-like content. However, they can also be modified to generate persuasive, contextual content weaponized for malicious purposes, including disinformation and novel social engineering attacks. In this paper, we present a study on identifying human- and AI-generated content using different models. Precisely, we fine-tune different models belonging to the BERT family, an open-source version of the GPT model, ELECTRA, and XLNet, and then perform a text classification task using two different labeled datasets—the first one consisting of 25,000 sentences generated by both AI and humans and the second comprising 22,929 abstracts that are ChatGPT-generated and written by humans. Furthermore, we perform an additional phase where we submit 20 sentences generated by ChatGPT and 20 sentences randomly extracted from Wikipedia to our fine-tuned models to verify the efficiency and robustness of the latter. In order to understand the prediction of the models, we performed an explainability phase using two sentences: one generated by the AI and one written by a human. We leveraged the integrated gradients and token importance techniques, analyzing the words and subwords of the two sentences. As a result of the first experiment, we achieved an average accuracy of 99%, precision of 98%, recall of 99%, and F1-score of 99%. For the second experiment, we reached an average accuracy of 51%, precision of 50%, recall of 52%, and F1-score of 51%.

Behavioral Nudging With Generative AI for Content Development in SMS Health Care Interventions: Case Study.

Brief message interventions have demonstrated immense promise in health care, yet the development of these messages has suffered from a dearth of transparency and a scarcity of publicly accessible data sets. Moreover, the researcher-driven content creation process has raised resource allocation issues, necessitating a more efficient and transparent approach to content development. This research sets out to address the challenges of content development for SMS interventions by showcasing the use of generative artificial intelligence (AI) as a tool for content creation, transparently explaining the prompt design and content generation process, and providing the largest publicly available data set of brief messages and source code for future replication of our process. Leveraging the pretrained large language model GPT-3.5 (OpenAI), we generate a collection of messages in the context of medication adherence for individuals with type 2 diabetes using evidence-derived behavior change techniques identified in a prior systematic review. We create an attributed prompt designed to adhere to content (readability and tone) and SMS (character count and encoder type) standards while encouraging message variability to reflect differences in behavior change techniques. We deliver the most extensive repository of brief messages for a singular health care intervention and the first library of messages crafted with generative AI. In total, our method yields a data set comprising 1150 messages, with 89.91% (n=1034) meeting character length requirements and 80.7% (n=928) meeting readability requirements. Furthermore, our analysis reveals that all messages exhibit diversity comparable to an existing publicly available data set created under the same theoretical framework for a similar setting. This research provides a novel approach to content creation for health care interventions using state-of-the-art generative AI tools. Future research is needed to assess the generated content for ethical, safety, and research standards, as well as to determine whether the intervention is successful in improving the target behaviors.

Enhanced Prompt Learning for Few-shot Text Classification Method

An enhanced prompt learning method (EPL4FTC) for few-shot text classification task is proposed. This algorithm first converts the text classification task into the form of prompt learning based on natural language inference. Thus, the implicit data enhancement is achieved based on the prior knowledge of pre-training language models and the algorithm is optimized by two losses with different granularities. Moreover, to capture the category information of specific downstream tasks, the triple loss is used for joint optimization. The masked-language model is incorporated as a regularizer to improve the generalization ability. We evaluated our method on four Chinese and three English text classification datasets. The experimental results show that the classification accuracy of the proposed EPL4FTC is significantly better than the other compared baselines.

Hazard Analysis for Massive Civil Aviation Safety Oversight Reports Using Text Classification and Topic Modeling

There are massive amounts of civil aviation safety oversight reports collected each year in the civil aviation of China. The narrative texts of these reports are typically short texts, recording the abnormal events detected during the safety oversight process. In the construction of an intelligent civil aviation safety oversight system, the automatic classification of safety oversight texts is a key and fundamental task. However, all safety oversight reports are currently analyzed and classified into categories by manual work, which is time consuming and labor intensive. In recent years, pre-trained language models have been applied to various text mining tasks and have proven to be effective. The aim of this paper is to apply text classification to the mining of these narrative texts and to show that text classification technology can be a critical element of the aviation safety oversight report analysis. In this paper, we propose a novel method for the classification of narrative texts in safety oversight reports. Through extensive experiments, we validated the effectiveness of all the proposed components. The experimental results demonstrate that our method outperforms existing methods on the self-built civil aviation safety oversight dataset. This study undertakes a thorough examination of the precision and associated outcomes of the dataset, thereby establishing a solid basis for furnishing valuable insights to enhance data quality and optimize information.

Semantic-aware frame-event fusion based pattern recognition via large vision–language models

Pattern recognition through the fusion of RGB frames and Event streams has emerged as a novel research area in recent years. Current methods typically employ backbone networks to individually extract the features of RGB frames and event streams, and subsequently fuse these features for pattern recognition. However, we posit that these methods may suffer from two key issues: (1). They attempt to directly learn a mapping from the input vision modality to the semantic labels. This approach often leads to sub-optimal results due to the disparity between the input and semantic labels; (2). They utilize small-scale backbone networks for the extraction of RGB and Event input features, thus these models fail to harness the recent performance advancements of large-scale visual-language models. In this study, we introduce a novel pattern recognition framework that consolidates the semantic labels, RGB frames, and event streams, leveraging pre-trained large-scale vision–language models. Specifically, given the input RGB frames, event streams, and all the predefined semantic labels, we employ a pre-trained large-scale vision model (CLIP vision encoder) to extract the RGB and event features. To handle the semantic labels, we initially convert them into language descriptions through prompt engineering and polish using ChatGPT, and then obtain the semantic features using the pre-trained large-scale language model (CLIP text encoder). Subsequently, we integrate the RGB/Event features and semantic features using multimodal Transformer networks. The resulting frame and event tokens are further amplified using self-attention layers. Concurrently, we propose to enhance the interactions between text tokens and RGB/Event tokens via cross-attention. Finally, we consolidate all three modalities using self-attention and feed-forward layers for recognition. Comprehensive experiments on the HARDVS and PokerEvent datasets fully substantiate the efficacy of our proposed SAFE model. The source code has been released at https://github.com/Event-AHU/SAFE_LargeVLM.

An ontology-based text mining dataset for extraction of process-structure-property entities

While large language models learn sound statistical representations of the language and information therein, ontologies are symbolic knowledge representations that can complement the former ideally. Research at this critical intersection relies on datasets that intertwine ontologies and text corpora to enable training and comprehensive benchmarking of neurosymbolic models. We present the MaterioMiner dataset and the linked materials mechanics ontology where ontological concepts from the mechanics of materials domain are associated with textual entities within the literature corpus. Another distinctive feature of the dataset is its eminently fine-grained annotation. Specifically, 179 distinct classes are manually annotated by three raters within four publications, amounting to 2191 entities that were annotated and curated. Conceptual work is presented for the symbolic representation of causal composition-process-microstructure-property relationships. We explore the annotation consistency between the three raters and perform fine-tuning of pre-trained language models to showcase the feasibility of training named entity recognition models. Reusing the dataset can foster training and benchmarking of materials language models, automated ontology construction, and knowledge graph generation from textual data.

Dual syntax aware graph attention networks with prompt for aspect-based sentiment analysis

Aspect-based sentiment analysis (ABSA) is a challenging task due to the presence of multiple aspect words with different sentiment polarities in a sentence. Recently, pre-trained language models like BERT have been widely used as context encoders in ABSA. Graph neural networks have also been employed to extract syntactic and semantic information from sentence parsing trees, resulting in superior results. However, dependency trees may establish irrelevant dependencies for sentences with irregular syntax and complex structures. Additionally, previous methods have not fully utilized recent developments in pre-trained language models. Therefore, we propose a Dual Syntax aware Graph attention networks with Prompt (DSGP) model to address these issues. Our model utilizes prompt templates to maximize the potential of pre-trained models and masked vector outputs of templates as supplementary aspect feature representations. We also leverage both dependency trees and constituent trees with graph attention networks to extract different types of syntactic information. The dependency tree captures syntactic correlation between words, while the constituent tree provides a high-level formation of the sentence. Finally, the output from the prompt and parsing trees is fused and fed into a standard classifier. Experimental results on four public datasets demonstrate the competitive performance of our model.

Efficient Adaptation: Enhancing Multilingual Models for Low-Resource Language Translation

This study focuses on the neural machine translation task for the TR-EN language pair, which is considered a low-resource language pair. We investigated fine-tuning strategies for pre-trained language models. Specifically, we explored the effectiveness of parameter-efficient adapter methods for fine-tuning multilingual pre-trained language models. Various combinations of LoRA and bottleneck adapters were experimented with. The combination of LoRA and bottleneck adapters demonstrated superior performance compared to other methods. This combination required only 5% of the pre-trained language model to be fine-tuned. The proposed method enhances parameter efficiency and reduces computational costs. Compared to the full fine-tuning of the multilingual pre-trained language model, it showed only a 3% difference in the BLEU score. Thus, nearly the same performance was achieved at a significantly lower cost. Additionally, models using only bottleneck adapters performed worse despite having a higher parameter count. Although adding LoRA to pre-trained language models alone did not yield sufficient performance, the proposed method improved machine translation. The results obtained are promising, particularly for low-resource language pairs. The proposed method requires less memory and computational load while maintaining translation quality.