Natural Language Processing Tasks Research Articles

From large language models to small logic programs: building global explanations from disagreeing local post-hoc explainers

The expressive power and effectiveness of large language models (LLMs) is going to increasingly push intelligent agents towards sub-symbolic models for natural language processing (NLP) tasks in human–agent interaction. However, LLMs are characterised by a performance vs. transparency trade-off that hinders their applicability to such sensitive scenarios. This is the main reason behind many approaches focusing on local post-hoc explanations, recently proposed by the XAI community in the NLP realm. However, to the best of our knowledge, a thorough comparison among available explainability techniques is currently missing, as well as approaches for constructing global post-hoc explanations leveraging the local information. This is why we propose a novel framework for comparing state-of-the-art local post-hoc explanation mechanisms and for extracting logic programs surrogating LLMs. Our experiments—over a wide variety of text classification tasks—show how most local post-hoc explainers are loosely correlated, highlighting substantial discrepancies in their results. By relying on the proposed novel framework, we also show how it is possible to extract faithful and efficient global explanations for the original LLM over multiple tasks, enabling explainable and resource-friendly AI techniques.

You only compress once: Towards effective and elastic BERT compression via exploit–explore stochastic nature gradient

Despite superior performance on various natural language processing tasks, pre-trained models such as BERT are challenged by deploying on resource-constraint devices. Most existing model compression approaches require re-compression or fine-tuning across diverse constraints to accommodate various hardware deployments. This practically limits the further application of model compression. Moreover, the ineffective training and searching process of existing elastic compression paradigms (Wang et al., 2020; Cai et al., 2020) prevents the direct migration to BERT compression. Motivated by the necessity of efficient inference across various constraints on BERT, we propose a novel approach, YOCO-BERT, to achieve compress once and deploy everywhere. Specifically, we first construct a huge search space with 1013 architectures, which covers nearly all configurations in BERT model. Then, we propose a novel stochastic nature gradient optimization method to guide the generation of optimal candidate architecture which could keep a balanced trade-off between explorations and exploitation. When a certain resource constraint is given, a lightweight distribution optimization approach is utilized to obtain the optimal network for target deployment without fine-tuning. Compared with state-of-the-art algorithms, YOCO-BERT provides more compact models, yet achieving 2.1%–4.5% average accuracy improvement on the GLUE benchmark. Besides, YOCO-BERT is also more effective, e.g., the training complexity is O(1) for N different devices. Codes available https://github.com/MAC-AutoML/YOCO-BERT.

Transformers meets neoantigen detection: a systematic literature review.

Cancer immunology offers a new alternative to traditional cancer treatments, such as radiotherapy and chemotherapy. One notable alternative is the development of personalized vaccines based on cancer neoantigens. Moreover, Transformers are considered a revolutionary development in artificial intelligence with a significant impact on natural language processing (NLP) tasks and have been utilized in proteomics studies in recent years. In this context, we conducted a systematic literature review to investigate how Transformers are applied in each stage of the neoantigen detection process. Additionally, we mapped current pipelines and examined the results of clinical trials involving cancer vaccines.

Evaluating Generative Artificial Intelligence on Multilingual Sentiment Analysis

This study shows that natural language processing (NLP), large language models (LLM), and generative Artificial Intelligence are becoming the most important. LLM is considered essential for many NLP tasks, including question-answering, classification, interpretation, and writing. New LLMs must be able to analyze and create documents in different languages as they know how to train training materials in different languages such as ChatGPT, BLOOMZ, and others. Considering how frequently LLMs are used, it is critical to assess the effectiveness in multi-lingual environments. In a zero-shot context, the present generative models are still useful in producing text in Indian languages. On the other hand, generative models routinely outperform human quality-based evaluation when it comes to English language generation and Indian languages. LLMs are not meant to be used in a zero-shot manner in downstream applications due to poor generating performance.

Is Boundary Annotation Necessary? Evaluating Boundary-Free Approaches to Improve Clinical Named Entity Annotation Efficiency: Case Study.

Named entity recognition (NER) is a fundamental task in natural language processing. However, it is typically preceded by named entity annotation, which poses several challenges, especially in the clinical domain. For instance, determining entity boundaries is one of the most common sources of disagreements between annotators due to questions such as whether modifiers or peripheral words should be annotated. If unresolved, these can induce inconsistency in the produced corpora, yet, on the other hand, strict guidelines or adjudication sessions can further prolong an already slow and convoluted process. The aim of this study is to address these challenges by evaluating 2 novel annotation methodologies, lenient span and point annotation, aiming to mitigate the difficulty of precisely determining entity boundaries. We evaluate their effects through an annotation case study on a Japanese medical case report data set. We compare annotation time, annotator agreement, and the quality of the produced labeling and assess the impact on the performance of an NER system trained on the annotated corpus. We saw significant improvements in the labeling process efficiency, with up to a 25% reduction in overall annotation time and even a 10% improvement in annotator agreement compared to the traditional boundary-strict approach. However, even the best-achieved NER model presented some drop in performance compared to the traditional annotation methodology. Our findings demonstrate a balance between annotation speed and model performance. Although disregarding boundary information affects model performance to some extent, this is counterbalanced by significant reductions in the annotator's workload and notable improvements in the speed of the annotation process. These benefits may prove valuable in various applications, offering an attractive compromise for developers and researchers.

Incorporating target-aware knowledge into prompt-tuning for few-shot stance detection

Stance detection, a fundamental task in natural language processing, identifies user stances in texts towards specific targets. The diverse targets and ever-changing expressions make it challenging to attain comprehensive knowledge from limited data. Existing methods focus on incorporating supplementary knowledge, neglecting fusion consistency during training, which is critical for preserving the rationality of the inference. In this paper, we introduce TAP, a novel approach for few-shot stance detection. TAP extends the verbalizer hierarchically, a mapping function in prompt-tuning. Constructed using a log-odds ratio of topics and targets, the verbalizer refines candidates with prior knowledge, forming the foundation for subsequent hierarchical distillation. The hierarchical distillation, a technique based on pilot experiments on the hierarchical verbalizer, ensures the fusion of diverse knowledge during prompt-tuning, maintaining consistency throughout the training process. Notably, TAP constructs verbalizers without external knowledge augmentation. The hierarchical distillation involves a joint loss function, contributing to the model’s robustness and training consistency. Extensive experiments are conducted on SemEval2016t6 and ArgMin datasets with 13 different targets. The proposed method is evaluated on various few-shot and full-data settings with F1-Macro and F1-Micro scores. On average, TAP achieves overall improvements of 4.71% and 3.76% over state-of-the-art baselines on Semeval2016t6 and ArgMin datasets, respectively, in few-shot scenarios.

Machine Transliteration of Handwritten MODI Script to Devanagari using Deep Neural Networks

The transliteration process involves transcribing words from the source language into the target language that uses a different script. Language and scriptural hurdles can be overcome via transliteration systems. There is a demand for automated transliteration systems due to the existence of several languages and the growing number of multilingual speakers. This study focuses on the Machine Transliteration of handwritten MODI script to Devanagari. MODI script was the official script for Marathi till 1950. Although Devanagari has, since then, taken over as the Marathi language's official script, the MODI script has historical significance as large volumes of its manuscripts are preserved in libraries across different parts of India. However, MODI into Devanagari transliteration is a difficult task because MODI script documents are complex in nature and there is no standard dataset available for the experiment. Machine Transliteration can be approached either as a Natural Language Processing task or as a pattern recognition task. In this research work, the transliteration task is carried out using the pattern recognition technique. The transliteration of MODI script to Devanagari is implemented using Convolutional Recurrent Neural Network (CRNN) based Calamari OCR, which is open-source software. An accuracy of 88.14% is achieved in character level matching of each word in the MODI to Devanagari transliteration process. When considering the entire word matching, the accuracy achieved is 61%. Machine Transliteration of

Cloud-based machine learning algorithms for anomalies detection

Gradient boosting machines harnesses the inherent capabilities of decision trees and meticulously corrects their errors in a sequential fashion, culminating in remarkably precise predictions. Word2Vec, a prominent word embedding technique, occupies a pivotal role in natural language processing (NLP) tasks. Its proficiency lies in capturing intricate semantic relationships among words, thereby facilitating applications such as sentiment analysis, document classification, and machine translation to discern subtle nuances present in textual data. Bayesian networks introduce probabilistic modeling capabilities, predominantly in contexts marked by uncertainty. Their versatile applications encompass risk assessment, fault diagnosis, and recommendation systems. Gated recurrent units (GRU), a variant of recurrent neural networks, emerges as a formidable asset in modeling sequential data. Both training and testing are crucial to the success of an intrusion detection system (IDS). During the training phase, several models are created, each of which can recognize typical from anomalous patterns within a given dataset. To acquire passwords and credit card details, "phishing" usually entails impersonating a trusted company. Predictions of student performance on academic tasks are improved by hyper parameter optimization of the gradient boosting regression tree using the grid search approach.

SENSE DIFFERENTIATION OF TEXTS AS A COMPONENT OF NEURAL NETWORK MODELLING

The article argues that the most productive for linguistic research at the present stage is the use of Artificial Neural Networks (ANNs) due to their productivity, representativeness, etc. It is emphasized that the basis for such use should be sense differentiation, thanks to which linguists can optimize the search, analysis, etc. of data for their research. In particular, taking into account semantic, morphological, syntactic, etc. features will allow the production of more reliable, fundamental results in various tasks of Natural Language Processing (NLP). The author emphasizes that the above will result in a qualitative leap in the scientific research of Ukrainian linguists, the possibility of presenting their results to world science, and further fruitful cooperation with foreign colleagues within the framework of grant programs. Thus, the semantic differentiation of texts is an integral part of the actualization of Artificial Neural Networks (ANNs) (in particular, Bidirectional Long Short-Term Memory Network (BiLSTM), Convolutional Neural Networks (CNNs), Deep Learning Networks (DLNs), Deep Neural Networks (DNNs), Graph Neural Networks (GNNs), Recurrent Neural Networks (RNNs), etc.) in modern linguistic research within the digital humanities. In the author’s opinion, the latter is produced by the focus on practical results, localization of implementation (in particular, Natural Language Processing (NLP), sentiment analysis, etc. Therefore, further study, improvement, and optimization of the existing innovative tools (in particular, neural network modelling of linguistic units) include work on more effective methods of working with context (through a combination of different types of Artificial Neural Networks (ANNs) with different layers, which is presented in the work of foreign colleagues), localization of language styles (essential in the process of fact-checking initiatives – as a milestone for validating text data), etc. without manual intervention in the above.

Let's Speak Trajectories: A Vision to Use NLP Models for Trajectory Analysis Tasks

The availability of trajectory data combined with various real-life practical applications has sparked the interest of the research community to design a plethora of algorithms for various trajectory analysis techniques. However, there is an apparent lack of full-fledged systems that provide the infrastructure support for trajectory analysis techniques, which hinders the applicability of most of the designed algorithms. Inspired by the tremendous success of the Bidirectional Encoder Representations from Transformers (BERT) deep learning model in solving various Natural Language Processing tasks, our vision is to have a BERT-like system for trajectory analysis tasks. We envision that in a few years, we will have such system where no one needs to worry again about each specific trajectory analysis operation. Whether it is trajectory imputation, similarity, clustering, or whatever, it would be one system that researchers, developers, and practitioners can deploy to get high accuracy for their trajectory operations. Our vision stands on a solid ground that trajectories in a space are highly analogous to statements in a language. We outline the challenges and the road to our vision. Exploratory results confirm the promise and possibility of our vision.

SPECE: Subject Position Encoder in Complex Embedding for Relation Extraction

As a crucial component of many natural language processing tasks, extracting entities and relations transforms unstructured text information into structured data, providing essential support for constructing knowledge graphs (KGs). However, current entity relation extraction models often prioritize the extraction of richer semantic features or the optimization of relation extraction methods, overlooking the significance of positional information and subject characteristics in this task. To solve this problem, we introduce the subject position-based complex exponential embedding for entity relation extraction model (SPECE). The encoder module of this model ingeniously combines a randomly initialized dilated convolutional network with a BERT encoder. Notably, it determines the initial position of the predicted subject based on semantic cues. Furthermore, it achieves a harmonious integration of positional encoding features and textual features through the adoption of the complex exponential embedding method. The experimental outcomes on both the NYT and WebNLG datasets reveal that, when compared to other baseline models, our proposed SPECE model demonstrates significant improvements in the F1 score on both datasets. This further validates its efficacy and superiority.

Author Profiling Approach: Predicting Personality Traits on Twitter Data using Combined BERT and SimCSE Embeddings

Abstract: Author profiling involves predicting various characteristics of an author from their writing style, including age, gender, native language, and personality traits. The PAN2015 shared task concentrated on author profiling within social media, challenging participants to predict the personality traits of Twitter users based on their tweets. In recent years, deep learning methods have risen to prominence in author profiling. Researchers frequently employ several notable models such as Word2Vec, doc2vec, GloVe, and FastText for generating word embeddings. These models have consistently shown effectiveness across various natural language processing tasks. For the PAN2015 task, participants employed a range of deep learning models to generate word embeddings, aiming to predict the age, gender, and personality traits of Twitter users. In this study, our focus was on enhancing the accuracy of personality traits classification using the PAN2015 dataset, a renowned benchmark corpus for author profiling. We employed pre-trained deep learning models, namely BERT and SimCSE, to generate word embeddings and sentence embeddings. For classification, we utilized Long Short-Term Memory (LSTM) and Convolution Neural Network (CNN) classifiers. Our findings revealed that the LSTM model, integrated with combined BERT and SimCSE embeddings, achieved an accuracy of 87.53% for personality traits classification, while the CNN model, similarly equipped, attained 80.48%. Additionally, utilizing BERT alone with LSTM yielded an accuracy of 78.45%, and with CNN, 75.32%. Our findings highlight the versatility of these models in addressing a range of natural language processing tasks, indicating their potential utility in diverse author profiling applications.

Systematic Study of Prompt Engineering

Abstract: Now a days Generative Artificial Intelligence is the buzz in the field of technology and science it is the implementation of the Artificial intelligence to generate different types of contents with the help of its models and ease the human life to a extend. Prompt Engineering is one of the arts of crafting instructions to guide large language models (LLMs), and has emerged as a critical technique in natural language processing (NLP). This systematic study delves into the intricacies of prompt engineering, exploring its techniques, evaluation methods, and applications. The study categorizes prompt engineering techniques into instruction-based, information-based, reformulation, and metaphorical prompts. It emphasizes the importance of evaluating prompt effectiveness using metrics like accuracy, fluency, and relevance. Additionally, the study investigates factors influencing prompt effectiveness, including prompt length, complexity, specificity, phrasing, vocabulary choice, framing, and context. The study highlights the impact of prompt engineering in enhancing LLM performance for NLP tasks like machine translation, question answering, summarization, and text generation. It underscores the role of prompt engineering in developing domainspecific LLM applications, enabling knowledge extraction, creative content generation, and addressing domain-specific challenges. The study concludes by addressing ethical considerations in prompt engineering, emphasizing the need to mitigate bias and discrimination while ensuring transparency

Advancements in Deep Learning Architectures for Natural Language Processing Tasks

Advancements in Deep Learning Architectures for Natural Language Processing Tasks

Fine-tuning BERT, DistilBERT, XLM-RoBERTa and Ukr-RoBERTa models for sentiment analysis of ukrainian language reviews

Sentiment analysis is one of the crucial tasks of natural language processing, which includes recognizing emotions expressed in textual data from various fields of activity. Automated tonality detection impacts businesses and helps increase profits by analyzing customer sentiment and responding quickly to their level of satisfaction with products or services. Therefore, the development of tools that will allow qualitative classification of text sentiment is significant, considering that users leave many reviews on various social networks, platforms, and websites in today's world. The study examines the fine-tuning of BERT, DistilBERT, XLM-RoBERTa, and Ukr-RoBERTa models for sentiment analysis of reviews in the Ukrainian language, as transformer models demonstrate a better understanding of the context and show high efficiency in solving natural language processing tasks. The dataset used in this study comprised about 11,000 user comments in Ukrainian, covering a range of topics such as shops, restaurants, hotels, medical facilities, fitness clubs, and the provision of various services. The textual data was categorized into two classes: positive and negative. Following text preprocessing, the dataset was divided into training and test samples in an 80:20 ratio. The hyperparameters were selected to optimize the performance of the pre-trained models for comment sentiment classification, and their effectiveness was evaluated using metrics such as accuracy, recall, precision, and F1-score. The results show that DistilBERT requires significantly fewer computing resources and is faster than other models. The XLM-RoBERTa model achieved the highest accuracy of 91.32%. However, considering the time needed to train the model and all the classification metrics, Ukr-RoBERTa is the optimal choice.

A BERT-GRU Model for Measuring the Similarity of Arabic Text

Semantic Textual Similarity (STS) aims to assess the semantic similarity between two pieces of text. As a challenging task in natural language processing, various approaches for STS in high-resource languages, such as English, have been proposed. In this paper, we are concerned with STS in low resource languages such as Arabic. A baseline approach for STS is based on vector embedding of the input text and application of similarity metric on the embedding space. In this contribution, we propose a cross-encoder neural network (Cross-BERT-GRU) to handle semantic similarity of Arabic sentences that benefits from both the strong contextual understanding of BERT and the sequential modeling capabilities of GRU. The architecture begins by inputting the BERT word embeddings for each word into a GRU cell to model long-term dependencies. Then, max pooling and average pooling are applied to the hidden outputs of the GRU cell, serving as the sentence -pair encoder. Finally, a softmax layer is utilized to predict the degree of similarity. The experiment results show a Spearman correlation coefficient of around 0.9 and that Cross-BERT-GRU outperforms the other BERT models in predicting the semantic textual similarity of Arabic sentences. The experimentation results also indicate that the performance improves by integrating data augmentation techniques.

Revolutionising Translation Technology: A Comparative Study of Variant Transformer Models - BERT, GPT, and T5

Recently, transformer-based models have reshaped the landscape of Natural Language Processing (NLP), particularly in the domain of Machine Translation (MT). this study explores three revolutionary transformer models: Bidirectional Encoder Representations from Transformers (BERT), Generative Pretrained Transformer (GPT), and Text-to-Text Transfer Transformer (T5). The study delves into their architecture, capabilities, and applications in the context of translation technology. The study begins by discussing the evolution of machine translation from rule-based to statistical machine translation and finally to transformer models. The models have distinct architectures and purposes which pushed the limits of MT and have been instrumental in revolutionising the field. The study found significant contributions of the models in the advancement of NLP tasks including translation technology. Using comparative approach, the study further elaborates on each model’s design and utility. BERT is strong in excelling in tasks requiring a deep understanding of the context. GPT is excellent for tasks such as text generation, translation and creative writing. While the strengths of T5 is text-to-text framework by simplifying the taskspecific architectures, making it easy to perform different NLP tasks. Recognising these models’ unique features allows translators to select the best one for particular translation tasks and adjust them for better accuracy, fluency, and cultural relevance in translations. The study concludes that the models bridge language barriers, improve cross-cultural communication and pave way for more accurate and natural translations in the future. The study also points out that language processing models are continually evolving but understanding BERT, GPT, and T5’s specific features is key for ongoing development in translation technology.

Data Set and Benchmark (MedGPTEval) to Evaluate Responses From Large Language Models in Medicine: Evaluation Development and Validation.

Large language models (LLMs) have achieved great progress in natural language processing tasks and demonstrated the potential for use in clinical applications. Despite their capabilities, LLMs in the medical domain are prone to generating hallucinations (not fully reliable responses). Hallucinations in LLMs' responses create substantial risks, potentially threatening patients' physical safety. Thus, to perceive and prevent this safety risk, it is essential to evaluate LLMs in the medical domain and build a systematic evaluation. We developed a comprehensive evaluation system, MedGPTEval, composed of criteria, medical data sets in Chinese, and publicly available benchmarks. First, a set of evaluation criteria was designed based on a comprehensive literature review. Second, existing candidate criteria were optimized by using a Delphi method with 5 experts in medicine and engineering. Third, 3 clinical experts designed medical data sets to interact with LLMs. Finally, benchmarking experiments were conducted on the data sets. The responses generated by chatbots based on LLMs were recorded for blind evaluations by 5 licensed medical experts. The evaluation criteria that were obtained covered medical professional capabilities, social comprehensive capabilities, contextual capabilities, and computational robustness, with 16 detailed indicators. The medical data sets include 27 medical dialogues and 7 case reports in Chinese. Three chatbots were evaluated: ChatGPT by OpenAI; ERNIE Bot by Baidu, Inc; and Doctor PuJiang (Dr PJ) by Shanghai Artificial Intelligence Laboratory. Dr PJ outperformed ChatGPT and ERNIE Bot in the multiple-turn medical dialogues and case report scenarios. Dr PJ also outperformed ChatGPT in the semantic consistency rate and complete error rate category, indicating better robustness. However, Dr PJ had slightly lower scores in medical professional capabilities compared with ChatGPT in the multiple-turn dialogue scenario. MedGPTEval provides comprehensive criteria to evaluate chatbots by LLMs in the medical domain, open-source data sets, and benchmarks assessing 3 LLMs. Experimental results demonstrate that Dr PJ outperforms ChatGPT and ERNIE Bot in social and professional contexts. Therefore, such an assessment system can be easily adopted by researchers in this community to augment an open-source data set.

When Large Language Models Meet Optical Networks: Paving the Way for Automation

Since the advent of GPT, large language models (LLMs) have brought about revolutionary advancements in all walks of life. As a superior natural language processing (NLP) technology, LLMs have consistently achieved state-of-the-art performance in numerous areas. However, LLMs are considered to be general-purpose models for NLP tasks, which may encounter challenges when applied to complex tasks in specialized fields such as optical networks. In this study, we propose a framework of LLM-empowered optical networks, facilitating intelligent control of the physical layer and efficient interaction with the application layer through an LLM-driven agent (AI-Agent) deployed in the control layer. The AI-Agent can leverage external tools and extract domain knowledge from a comprehensive resource library specifically established for optical networks. This is achieved through user input and well-crafted prompts, enabling the generation of control instructions and result representations for autonomous operation and maintenance in optical networks. To improve LLM’s capability in professional fields and stimulate its potential on complex tasks, the details of performing prompt engineering, establishing domain knowledge library, and implementing complex tasks are illustrated in this study. Moreover, the proposed framework is verified on two typical tasks: network alarm analysis and network performance optimization. The good response accuracies and semantic similarities of 2400 test situations exhibit the great potential of LLM in optical networks.

Domain-Specific Few-Shot Table Prompt Question Answering via Contrastive Exemplar Selection

As a crucial task in natural language processing, table question answering has garnered significant attention from both the academic and industrial communities. It enables intelligent querying and question answering over structured data by translating natural language into corresponding SQL statements. Recently, there have been notable advancements in the general domain table question answering task, achieved through prompt learning with large language models. However, in specific domains, where tables often have a higher number of columns and questions tend to be more complex, large language models are prone to generating invalid SQL or NoSQL statements. To address the above issue, this paper proposes a novel few-shot table prompt question answering approach. Specifically, we design a prompt template construction strategy for structured SQL generation. It utilizes prompt templates to restructure the input for each test data and standardizes the model output, which can enhance the integrity and validity of generated SQL. Furthermore, this paper introduces a contrastive exemplar selection approach based on the question patterns and formats in domain-specific contexts. This enables the model to quickly retrieve the relevant exemplars and learn characteristics about given question. Experimental results on the two datasets in the domains of electric energy and structural inspection show that the proposed approach outperforms the baseline models across all comparison settings.