Natural Language Processing Tasks Research Articles

Toward accurate Amazigh part-of-speech tagging

<span lang="EN-US">Part-of-speech (POS) tagging is the process of assigning to each word in a text its corresponding grammatical information POS. It is an important pre-processing step in other natural language processing (NLP) tasks, so the objective of finding the most accurate one. The previous approaches were based on traditional machine learning algorithms, later with the development of deep learning, more POS taggers were adopted. If the accuracy of POS tagging reaches 97%, even with the traditional machine learning, for high resourced language like English, French, it’s far the case in low resource language like Amazigh. The most used approaches are traditional machine learning, and the results are far from those for rich language. In this paper, we present a new POS tagger based on bidirectional long short-term memory for Amazigh language and the experiments that have been done on real dataset shows that it outperforms the existing machine learning methods.</span>

How Is a “Kitchen Chair” like a “Farm Horse”? Exploring the Representation of Noun-Noun Compound Semantics in Transformer-based Language Models

Abstract Despite the success of Transformer-based language models in a wide variety of natural language processing tasks, our understanding of how these models process a given input in order to represent task-relevant information remains incomplete. In this work, we focus on semantic composition and examine how Transformer-based language models represent semantic information related to the meaning of English noun-noun compounds. We probe Transformer-based language models for their knowledge of the thematic relations that link the head nouns and modifier words of compounds (e.g., KITCHEN CHAIR: a chair located in a kitchen). Firstly, using a dataset featuring groups of compounds with shared lexical or semantic features, we find that token representations of six Transformer-based language models distinguish between pairs of compounds based on whether they use the same thematic relation. Secondly, we utilize fine-grained vector representations of compound semantics derived from human annotations, and find that token vectors from several models elicit a strong signal of the semantic relations used in the compounds. In a novel “compositional probe” setting, where we compare the semantic relation signal in mean-pooled token vectors of compounds to mean-pooled token vectors when the two constituent words appear in separate sentences, we find that the Transformer-based language models that best represent the semantics of noun-noun compounds also do so substantially better than in the control condition where the two constituent works are processed separately. Overall, our results shed light on the ability of Transformer-based language models to support compositional semantic processes in representing the meaning of noun-noun compounds.

Taiyi: a bilingual fine-tuned large language model for diverse biomedical tasks.

Most existing fine-tuned biomedical large language models (LLMs) focus on enhancing performance in monolingual biomedical question answering and conversation tasks. To investigate the effectiveness of the fine-tuned LLMs on diverse biomedical natural language processing (NLP) tasks in different languages, we present Taiyi, a bilingual fine-tuned LLM for diverse biomedical NLP tasks. We first curated a comprehensive collection of 140 existing biomedical text mining datasets (102 English and 38 Chinese datasets) across over 10 task types. Subsequently, these corpora were converted to the instruction data used to fine-tune the general LLM. During the supervised fine-tuning phase, a 2-stage strategy is proposed to optimize the model performance across various tasks. Experimental results on 13 test sets, which include named entity recognition, relation extraction, text classification, and question answering tasks, demonstrate that Taiyi achieves superior performance compared to general LLMs. The case study involving additional biomedical NLP tasks further shows Taiyi's considerable potential for bilingual biomedical multitasking. Leveraging rich high-quality biomedical corpora and developing effective fine-tuning strategies can significantly improve the performance of LLMs within the biomedical domain. Taiyi shows the bilingual multitasking capability through supervised fine-tuning. However, those tasks such as information extraction that are not generation tasks in nature remain challenging for LLM-based generative approaches, and they still underperform the conventional discriminative approaches using smaller language models.

Leveraging pre-trained language models for code generation

Code assistance refers to the utilization of various tools, techniques, and models to help developers in the process of software development. As coding tasks become increasingly complex, code assistant plays a pivotal role in enhancing developer productivity, reducing errors, and facilitating a more efficient coding workflow. This assistance can manifest in various forms, including code autocompletion, error detection and correction, code generation, documentation support, and context-aware suggestions. Language models have emerged as integral components of code assistance, offering developers the capability to receive intelligent suggestions, generate code snippets, and enhance overall coding proficiency. In this paper, we propose new hybrid models for code generation by leveraging pre-trained language models BERT, RoBERTa, ELECTRA, and LUKE with the Marian Causal Language Model. Selecting these models based on their strong performance in various natural language processing tasks. We evaluate the performance of these models on two datasets CoNaLa and DJANGO and compare them to existing state-of-the-art models. We aim to investigate the potential of pre-trained transformer language models to revolutionize code generation, offering improved precision and efficiency in navigating complex coding scenarios. Additionally, conducting error analysis and refining the generated code. Our results show that these models, when combined with the Marian Decoder, significantly improve code generation accuracy and efficiency. Notably, the RoBERTaMarian model achieved a maximum BLEU score of 35.74 and an exact match accuracy of 13.8% on CoNaLa, while LUKE-Marian attained a BLEU score of 89.34 and an exact match accuracy of 78.50% on DJANGO. Implementation of this work is available at https://github.com/AhmedSSoliman/Leveraging-Pretrained-Language-Models-for-Code-Generation.

A Comparative Study of Text Comprehension in IELTS Reading Exam using GPT-3

This paper discusses the capabilities and limitations of GPT-3 (0), a state-of-the-art language model, in the context of text understanding. We begin by describing the architecture and training process of GPT-3, and provide an overview of its impressive performance across a wide range of natural language processing tasks, such as language translation, question-answering, and text completion. Throughout this research project, a summarizing tool was also created to help us retrieve content from any types of document, specifically IELTS (0) Reading Test data in this project. We also aimed to improve the accuracy of the summarizing, as well as question-answering capabilities of GPT-3 (0) via long text.

Adapted large language models can outperform medical experts in clinical text summarization.

Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven. Here we applied adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes and doctor-patient dialogue. Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.

Using word embeddings for immigrant and refugee stereotype quantification in a diachronic and multilingual setting

Word embeddings are efficient machine-learning-based representations of human language used in many Natural Language Processing tasks nowadays. Due to their ability to learn underlying word association patterns present in large volumes of data, it is possible to observe various sociolinguistic phenomena in the embedding semantic space, such as social stereotypes. The use of stereotypical framing in discourse can be detrimental and induce misconceptions about certain groups, such as immigrants and refugees, especially when used by media and politicians in public discourse. In this paper, we use word embeddings to investigate immigrant and refugee stereotypes in a multilingual and diachronic setting. We analyze the Danish, Dutch, English, and Spanish portions of four different multilingual corpora of political discourse, covering the 1997–2018 period. Then, we measure the effect of sociopolitical variables such as the number of offences committed and the size of the refugee and immigrant groups in the host country over our measurements of stereotypical association using the Bayesian multilevel framework. Our results indicate the presence of stereotypical associations towards both immigrants and refugees for all 4 languages, and that the immigrants are overall more strongly associated with the stereotypical frames than refugees.

A Comparative Analysis of Generative Artificial Intelligence Tools for Natural Language Processing

Generative artificial intelligence tools have recently attracted a great deal of attention. This is because of their huge advantages, which include ease of usage, quick generation of answers to requests, and the human-like intelligence they possess. This paper presents a vivid comparative analysis of the top 9 generative artificial intelligence (AI) tools, namely ChatGPT, Perplexity AI, YouChat, ChatSonic, Google's Bard, Microsoft Bing Assistant, HuggingChat, Jasper AI, and Quora's Poe, paying attention to the Pros and Cons each of the AI tools presents. This comparative analysis shows that the generative AI tools have several Pros that outweigh the Cons. Further, we explore the transformative impact of generative AI in Natural Language Processing (NLP), focusing on its integration with search engines, privacy concerns, and ethical implications. A comparative analysis categorizes generative AI tools based on popularity and evaluates challenges in development, including data limitations and computational costs. The study highlights ethical considerations such as technology misuse and regulatory challenges. Additionally, we delved into AI Planning techniques in NLP, covering classical planning, probabilistic planning, hierarchical planning, temporal planning, knowledge-driven planning, and neural planning models. These planning approaches are vital in achieving specific goals in NLP tasks. In conclusion, we provide a concise overview of the current state of generative AI, including its challenges, ethical considerations, and potential applications, contributing to the academic discourse on human-computer interaction.

Automated Scoring of Translations with BERT Models: Chinese and English Language Case Study

With the wide application of artificial intelligence represented by deep learning in natural language-processing tasks, the automated scoring of translations has also advanced and improved. This study aims to determine if the BERT-assist system can reliably assess translation quality and identify high-quality translations for potential recognition. It takes the Han Suyin International Translation Contest as a case study, which is a large-scale and influential translation contest in China, with a history of over 30 years. The experimental results show that the BERT-assist system is a reliable second rater for massive translations in terms of translation quality, as it can effectively sift out high-quality translations with a reliability of r = 0.9 or higher. Thus, the automated translation scoring system based on BERT can satisfactorily predict the ranking of translations according to translation quality and sift out high-quality translations potentially shortlisted for prizes.

Tabular text embedding for Vietnamese text-based person search

Vietnamese text-based person search is still a challenging problem with the limited dataset of Vietnamese descriptions. The current popular approach to this problem is Deep Neural Networks (DNNs), and recently, transformer networks have been more favored because of their outperformance over CNN and RNN networks for both vision and natural language processing tasks. However, DNN, or transformer networks, require a large amount of training data and computing time for efficient learning of visual and textual features. This brings a burden for implementing Vietnamese text-based person search by DNN, or transformer networks. Towards building a Vietnamese text-based person search system on a scarce resource dataset of Vietnamese descriptive sentences with low computing cost, in this work, we propose to apply the transformer-based architecture named TabTransformer for contextual embedding of the noun phrases chunked from the Vietnamese descriptive sentences. This is the first time the TabTransformer network has been deployed together with CNN and RNN architectures for Vietnamese text-based person search. The experimental results on a limited dataset of 3000VnPersonSearch show the better recognition accuracy of the proposed method compared to the baseline method by about 7.5% at Rank 1. In addition, the computing time of our method is more effective than the baseline method.

A concise analysis of low-rank adaptation

Recent years the pre-trained language models have been proved to be a transformative technology within the domain of Natural Language Processing (NLP). From early word embeddings to modern transformer-based architectures, the success of models like BERT, GPT-3, and their variants has led to remarkable advancements in various NLP tasks. This paper is based on the Transformer model and explores and summarizes the application of the lightweight fine-tuning technique LoRA in pretrained language models, as well as improvements and derived technologies based on LoRA. Moreover, this paper categorizes these techniques into two main directions according to the advancements: enhancing training efficiency and improving training performance. Under these two major directions, several representative optimization and derived techniques are summarized and analyzed. Furthermore, this paper offers a perspective on the hot topics and future prospects of this research subject, and summarizes and proposes several directions that hold exploration value for the future, such as the possible avenues for further optimization and integration with other lightweight technologies.

Comparative study of sequence-to-sequence models: From RNNs to transformers

In this comprehensive exploration of sequence-to-sequence models in Natural Language Processing (NLP), we have traced the trajectory of their evolution and contributions. Starting from foundational Recurrent Neural Networks (RNNs) to the revolutionary capabilities of Long Short-Term Memory (LSTM), In this comprehensive exploration of sequence-to-sequence models in Natural Language Processing (NLP), we have meticulously traced the trajectory of their evolution and impactful contributions. From the foundational Recurrent Neural Networks (RNNs) to the revolutionary capabilities of Long Short-Term Memory (LSTM), as well as the transformative innovations brought forth by Transformers and BERT, this review eloquently highlights the monumental advancements that have fundamentally reshaped our understanding and generation of language. The crux of our comparative analysis lies in its ability to spotlight the distinctive strengths and limitations inherent in each model. Through an intricate examination, we uncover their nuanced applications across a diverse spectrum of NLP tasks. Particularly noteworthy is the pivotal role played by Transformers and the transformative Bidirectional Encoder Representations from Transformers (BERT). The paper concludes with a summary and outlook of the entire paper.

The evolution of transformer models from unidirectional to bidirectional in Natural Language Processing

Transformer models have revolutionized Natural Language Processing (NLP), transitioning from traditional sequential models to innovative architectures based on attention mechanisms. The shift from unidirectional to bidirectional models has been a remarkable development in NLP. This paper mainly focuses on the evolution of NLP caused by Transformer models, with the transition from unidirectional to bidirectional modeling. This paper explores how the transformer model has revolutionized NLP, and the evolution from traditional sequential models to innovative attention-driven architectures. In this paper, it mainly discusses the limitations of traditional NLP models like RNNs, LSTMs and CNN when handling lengthy text sequences and complex dependencies, highlighting how transformer models, employing self-attention mechanisms and bidirectional modeling (e.g., BERT and GPT), have significantly improved NLP tasks. It provides a thorough review of the shift from unidirectional to bidirectional transformer models, offering insights into their utilization and development. Finally, this paper concludes with a summary and outlook for the entire study.

Automatic Text Labeling Method Based on Large Language Models

With the increasing demand for large amounts of training data for model development, this paper proposes LLM4Label, an automatic text labeling method based on large language models, to assist human labelers in annotating text data. LLM4Label first selects the most representative seed data using a clustering algorithm based on text similarity. It then constructs prompt dialogues with few-shot prompts to stimulate the language model’s performance on entity labeling tasks, enabling it to automatically and efficiently label more data. Finally, LLM4Label introduces human feedback to correct un- certain labeling results and retrains the model with the corrected annotations. Experiments show that LLM4Label achieves high- quality labeled data at low human labeling cost. The proposed method provides an effective way to obtain sizable and high- quality annotated datasets with minimal manual effort, which can strongly support downstream natural language processing tasks.

APRE: Annotation-Aware Prompt-Tuning for Relation Extraction

AbstractPrompt-tuning has been successfully applied to support classification tasks in natural language processing and has achieved promising performance. The main characteristic of prompt-tuning based classification is to verbalize class labels and predict masked tokens like a cloze-like task. It has the advantage to make use of knowledge in pre-trained language models (PLMs). Because prompt templates are manually designed, they are more prone to overfitting. Furthermore, traditional prompt templates are appended in the tail of an original sentence. They are far from some semantic units in a sentence. It is weak to decode semantic information of an input relevant to PLMs. To aggregate more semantic information from PLMs for masked token prediction, we propose an annotation-aware prompt-tuning model for relation extraction. In our method, entity type representations are used as entity annotations. They are implanted near the site of entities in a sentence for decoding semantic information of PLMs. It is effective to make full use of knowledge in PLMs for relation extraction. In the experiment section, our method is validated on the Chinese literature text and SemEval 2010 Task datasets and achieves 89.3% and 90.6% in terms of F1-score, respectively. It achieves the state-of-the-art performance on two public datasets. The result further demonstrates the effectiveness of our model to decode semantic information in PLMs.

Prompt for extraction: Multiple templates choice model for event extraction

Event Extraction (EE) is an essential task in natural language processing that aims to mine events occurring in event mentions represent events using event records, which usually consist of event types, trigger words, argument elements corresponding to roles in the event types. Recently, prompt-based generative models have been developed to extract events. However, these prompt-based generative studies have ignored the fact that the strong language comprehension capability of the pre-trained language model (PLM) can analyze extract the potential role relationships in multiple templates for more information that can help extract argument elements. To determine the extended templates that can help the model for event extraction, we propose the multiple template choice model (MTCM), which designs an extended event type mining module to automatically mine the extended event types in the event mention uses the templates corresponding to the extended event types to interact with the template, corresponding to the currently to-be-extracted event type of event mention, in a multi-template information interaction, which gives the model more information guides the PLM for event extraction. To validate our model, we used two widely used datasets in the event extraction domain, ACE2005-EN ERE-EN. The experimental results show that our model achieves state-of-the-art performance on the ACE2005-EN dataset significantly improves the ERE dataset. In addition, according to the results, our model can be effectively adapted to low-resource environments.

Self-augmented sequentiality-aware encoding for aspect term extraction

Aspect Term Extraction (ATE) is a natural language processing task which recognizes and extracts aspect terms from sentences. The recent study in this field successfully leverages Pretrained Language Model (PLM) and data augmentation, which contributes to the construction of knowledgeable encoder for ATE. In this paper, we propose a novel method to strengthen ATE encoder, using self-augmented mechanism and sequentiality-aware network. Specifically, self augmentation expands training data with paraphrases and fuses correlated information using PLM during encoding. Back translation is used for obtaining pragmatically-diverse paraphrases. On the basis, BiGRU is utilized as a supplementary neural layer for extra-encoding, so as to involve sequence features into the output hidden states of PLM. We refer to our method as Self-augmented Sequentiality-sensitive Encoding (SSE for short). We carry out experiments on the benchmark SemEval datasets, including L-14, R-14, R-15 and R-16. Experimental results show that SSE yields substantial improvements, compared to BERT-based baselines. In particular, it is demonstrated that SSE is able to collaborate with other data augmentation approaches to produce more significant improvements, where the resultant ATE performance is up to 86.74%, 88.91%, 77.43% and 82.42% F1-scores (for L-14, R-14, R-15 and R-16 respectively).

An Overview of the Application of Convolutional Neural Networks inSentiment Analysis

The field of natural language processing, or NLP, uses its understanding of human language to find practical solutions to issues. It mainly includes two parts: the core task and the application. The core task represents the common problem that needs to be solved in various natural language application directions. It includes language models, morphology, grammar analysis, semantic analysis, etc. At the same time, the application section focuses on specific natural language processing tasks such as machine translation, information retrieval, question-answering systems, dialogue systems, etc. Natural language processing has made a significant contribution to the development of human society and the economy and provides strong support for all aspects of research work. Opinion mining, or sentiment analysis, is a subfield of natural language processing that develops systems for identifying and extracting ideas from text. Sentiment analysis is a hot topic since it has many practical applications. Many opinion-expressing texts are available on review sites, forums, blogs, and social media as the amount of publicly available information on the Internet grows. This unstructured information can then be automatically transformed into structured data about products, services, brands, politics, or other topics on which people can express their opinions using sentiment analysis systems. This information can be used for marketing analytics, public relations, product reviews, network sponsor ratings, product feedback, and customer service. With the rapid growth of labeled sample data sets and the notable enhancement in graphics processor (GPU) performance, convolutional neural network research has advanced rapidly and achieved remarkable leads to various computer vision tasks. By reviewing the application of CNN, we see that convolutional operations are naturally suitable for some text processing and, thus, naturally suitable for the background of sentiment analysis.

Measuring and Mitigating Gender Bias in Legal Contextualized Language Models

Transformer-based contextualized language models constitute the state-of-the-art in several natural language processing (NLP) tasks and applications. Despite their utility, contextualized models can contain human-like social biases, as their training corpora generally consist of human-generated text. Evaluating and removing social biases in NLP models has been a major research endeavor. In parallel, NLP approaches in the legal domain, namely, legal NLP or computational law, have also been increasing. Eliminating unwanted bias in legal NLP is crucial, since the law has the utmost importance and effect on people. In this work, we focus on the gender bias encoded in BERT-based models. We propose a new template-based bias measurement method with a new bias evaluation corpus using crime words from the FBI database. This method quantifies the gender bias present in BERT-based models for legal applications. Furthermore, we propose a new fine-tuning-based debiasing method using the European Court of Human Rights (ECtHR) corpus to debias legal pre-trained models. We test the debiased models’ language understanding performance on the LexGLUE benchmark to confirm that the underlying semantic vector space is not perturbed during the debiasing process. Finally, we propose a bias penalty for the performance scores to emphasize the effect of gender bias on model performance.

Sentiment Analysis Through the Application of Machine Learning Algorithms

Sentiment analysis, also called opinion mining. It is a natural language processing (NLP) task that involves fixing the sentiment tone expressed in a piece of text including a review, tweet, post, comment, or news article, using machine learning algorithms. There are various social networking sites where people express their views, opinions, and emotions freely. All such posts are recorded and analyzed to determine emotions of the people. Present study recovers posts and finds emotions and polarity of posts. To determine emotions and polarity of posts, various techniques are used.