Field Of Natural Language Processing Research Articles

A Survey of Visual Transformers.

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

Shakespeare Machine: New AI-Based Technologies for Textual Analysis

Abstract This article demonstrates a method using tools from the field of Natural Language Processing (NLP) to aid in analyzing theatrical texts and similar works. The method deploys pre-trained large language model neural networks to gather metadata for a text that is amenable to downstream statistical analyses surfacing patterns of interest in character dialogue. We specifically focus on Shakespeare’s works, collecting metadata in the form of sentiment and emotion scores for each line of his plays. In addition to sentiment and emotion scores produced by NLP models, we also directly gather metadata such as genre, line length, and character gender. We show how these metadata may be used to illuminate a number of interesting patterns in Shakespearean character which may be difficult to detect from a direct reading of the texts. We use these metadata to expose statistically significant relationships in Shakespeare between character gender and the emotional content of that character’s dialogue, controlling for genre. We also present here the publicly available dataset that we have compiled to perform these analyses. The data collects text from Shakespeare’s plays along with a variety of metadata useful for this and other forms of analysis of Shakespeare’s works. The methodology demonstrated here may be extended to other varieties of metadata provided by large NLP models.

The Role of Typological Feature Prediction in NLP and Linguistics

Abstract Computational typology has gained traction in the field of Natural Language Processing (NLP) in recent years, as evidenced by the increasing number of papers on the topic and the establishment of a Special Interest Group on the topic (SIGTYP), including the organization of successful workshops and shared tasks. A considerable amount of work in this sub-field is concerned with prediction of typological features, for example, for databases such as the World Atlas of Language Structures (WALS) or Grambank. Prediction is argued to be useful either because (1) it allows for obtaining feature values for relatively undocumented languages, alleviating the sparseness in WALS, in turn argued to be useful for both NLP and linguistics; and (2) it allows us to probe models to see whether or not these typological features are encapsulated in, for example, language representations. In this article, we present a critical stance concerning prediction of typological features, investigating to what extent this line of research is aligned with purported needs—both from the perspective of NLP practitioners, and perhaps more importantly, from the perspective of linguists specialized in typology and language documentation. We provide evidence that this line of research in its current state suffers from a lack of interdisciplinary alignment. Based on an extensive survey of the linguistic typology community, we present concrete recommendations for future research in order to improve this alignment between linguists and NLP researchers, beyond the scope of typological feature prediction.

From RNN techniques to pre-trained models: emphasis on the use in Arabic machine translation

<p>Neural Machine Translation (NMT) has gained increasing attention in recent years due to its promising performance over conventional approaches such as Statistical Machine Translation. Nevertheless, when applied to languages with different structures, such as the pair (English, Arabic) that interests us in this work, its efficiency is degraded.<br />Alternatively, the use of unsupervised pre-training of large neural models has recently made a significant leap forward in the field of Natural Language Processing (NLP). By warm starting from published checkpoints, NLP practitioners have been able to push the state-of-the-art boundaries on multiple benchmarks while simultaneously saving considerable computational time. The emphasis so far has been mainly on natural language understanding problems. In this paper, we show the efficacy of pre-trained checkpoints for Arabic Machine Translation. We have evolved a transformer-based sequence-to-sequence model which is compatible with pre-trained checkpoints publicly available for Arabic Bidirectional Encoder Representations from Transformers (AraBERT) and Arabic Generative Pre-trained Transformer (AraGPT) and conducted a thorough empirical study on the usefulness of initializing our Arabic MT model, both encoder and decoder, with these checkpoints. Our models yield new state-of-the-art results in Arabic MT.</p>

Evaluating the machine learning models based on natural language processing tasks

<span lang="EN-US">In the realm of natural language processing (NLP), a diverse array of language models has emerged, catering to a wide spectrum of tasks, ranging from speaker recognition and auto-correction to sentiment analysis and stock prediction. The significance of language models in enabling the execution of these NLP tasks cannot be overstated. This study proposes an approach to enhance accuracy by leveraging a hybrid language model, combining the strengths of long short-term memory (LSTM) and gated recurrent unit (GRU). LSTM excels in preserving long-term dependencies in data, while GRU's simpler gating mechanism expedites the training process. The research endeavors to evaluate four variations of this hybrid model: LSTM, GRU, <a name="_Hlk158992630"></a>bidirectional long short-term memory (Bi-LSTM), and a combination of LSTM with GRU. These models are subjected to rigorous testing on two distinct datasets: one focused on IBM stock price prediction, and the other on Jigsaw toxic comment classification (sentiment analysis). This work represents a significant stride towards democratizing NLP capabilities, ensuring that even in resource-constrained settings, NLP models can exhibit improved performance. The anticipated implications of these findings span a wide spectrum of real-world applications and hold the potential to stimulate further research in the field of NLP. </span>

Comparative Analysis of Deep Learning Models for Part of Speech Tagging in the Malay Language

Despite the widespread use of Malay, under-resourced languages like Malay face challenges in Natural Language Processing (NLP), particularly in Part-of-Speech (POS) tagging. The scarcity of annotated corpora poses a primary obstacle to POS tagging in Malay. This study aims to enhance the effectiveness and reliability of POS tagging models explicitly tailored for under-resourced languages within the field of NLP, focusing on Malay. Existing models, which rely on Conditional Random Fields and Hidden Markov Models, exhibit limitations, underscoring the need for more robust approaches. The research conducts a comparative analysis of various deep-learning models with different encoders for POS tagging in Malay sentences. The experimental analysis demonstrates that the Bidirectional Long Short-Term Memory (Bi-LSTM) model, leveraging a pre-trained Bidirectional Encoder Representations from Transformers (BERT) embedding model, achieves exceptional accuracy, precision, recall, and F1 scores in predicting tags. Notably, the BERT + Bi-LSTM model, boasting an accuracy of 98.82%, outperforms other models, showcasing superior performance across all evaluated metrics. Additionally, this combined model effectively handles known and unknown words, yielding highly accurate POS tagging results for Malay sentences. Doi: 10.28991/HIJ-2024-05-02-04 Full Text: PDF

Optimization techniques for sentiment analysis based on LLM (GPT-3)

With the rapid development of natural language processing (NLP) technology, large-scale pre-trained language models such as GPT-3 have become a popular research object in NLP field. This paper aims to explore sentiment analysis optimization techniques based on large pre-trained language models such as GPT-3 to improve model performance and effect and further promote the development of natural language processing (NLP). By introducing the importance of sentiment analysis and the limitations of traditional methods, GPT-3 and Fine-tuning techniques are introduced in this paper, and their applications in sentiment analysis are explained in detail. The experimental results show that the Fine-tuning technique can optimize GPT-3 model and obtain good performance in sentiment analysis task. This study provides an important reference for future sentiment analysis using large-scale language models.

Predicting multi-label emojis, emotions, and sentiments in code-mixed texts using an emojifying sentiments framework

In the era of social media, the use of emojis and code-mixed language has become essential in online communication. However, selecting the appropriate emoji that matches a particular sentiment or emotion in the code-mixed text can be difficult. This paper presents a novel task of predicting multiple emojis in English-Hindi code-mixed sentences and proposes a new dataset called SENTIMOJI, which extends the SemEval 2020 Task 9 SentiMix dataset. Our approach is based on exploiting the relationship between emotion, sentiment, and emojis to build an end-to-end framework. We replace the self-attention sublayers in the transformer encoder with simple linear transformations and use the RMS-layer norm instead of the normal layer norm. Moreover, we employ Gated Linear Unit and Fully Connected layers to predict emojis and identify the emotion and sentiment of a tweet. Our experimental results on the SENTIMOJI dataset demonstrate that the proposed multi-task framework outperforms the single-task framework. We also show that emojis are strongly linked to sentiment and emotion and that identifying sentiment and emotion can aid in accurately predicting the most suitable emoji. Our work contributes to the field of natural language processing and can help in the development of more effective tools for sentiment analysis and emotion recognition in code-mixed languages. The codes and data will be available at https://www.iitp.ac.in/~ai-nlp-ml/resources.html#SENTIMOJI to facilitate research.

Sample-Adaptive Classification Inference Network

Existing pre-trained models have yielded promising results in terms of computational time reduction. However, these models only focus on pruning simple sentences or less salient words, while neglecting the treatment of relatively complex sentences. It is frequently these sentences that cause the loss of model accuracy. This shows that the adaptation of the existing models is one-sided. To address this issue, in this paper, we propose a sample-adaptive training and inference model. Specifically, complex samples are extracted from the training datasets and a dedicated data augmentation module is trained to extract global and local semantic information of complex samples. During inference, simple samples can exit the model via the Sample Adaptive Exit Mechanism, Normal samples pass through the whole backbone model before inference, while complex samples are processed by the Characteristic Enhancement Module after passing through the backbone model. In this way, all samples are processed adaptively. Our extensive experiments on classification tasks datasets in the field of Natural Language Processing demonstrate that our method enhances model accuracy and reduces model inference time for multiple datasets. Moreover, our method is transferable and can be applied to multiple pre-trained models.

Identifying widespread and recurrent variants of genetic parts to improve annotation of engineered DNA sequences.

Engineered plasmids have been workhorses of recombinant DNA technology for nearly half a century. Plasmids are used to clone DNA sequences encoding new genetic parts and to reprogram cells by combining these parts in new ways. Historically, many genetic parts on plasmids were copied and reused without routinely checking their DNA sequences. With the widespread use of high-throughput DNA sequencing technologies, we now know that plasmids often contain variants of common genetic parts that differ slightly from their canonical sequences. Because the exact provenance of a genetic part on a particular plasmid is usually unknown, it is difficult to determine whether these differences arose due to mutations during plasmid construction and propagation or due to intentional editing by researchers. In either case, it is important to understand how the sequence changes alter the properties of the genetic part. We analyzed the sequences of over 50,000 engineered plasmids using depositor metadata and a metric inspired by the natural language processing field. We detected 217 uncatalogued genetic part variants that were especially widespread or were likely the result of convergent evolution or engineering. Several of these uncatalogued variants are known mutants of plasmid origins of replication or antibiotic resistance genes that are missing from current annotation databases. However, most are uncharacterized, and 3/5 of the plasmids we analyzed contained at least one of the uncatalogued variants. Our results include a list of genetic parts to prioritize for refining engineered plasmid annotation pipelines, highlight widespread variants of parts that warrant further investigation to see whether they have altered characteristics, and suggest cases where unintentional evolution of plasmid parts may be affecting the reliability and reproducibility of science.

What does the language system look like in pre-trained language models? A study using complex networks

Pre-trained language models has advanced the fields of natural language processing. The exceptional capabilities exhibited by PLMs in NLP tasks have been attracting researchers to explore the underlying factors responsible for their success. However, most of work primarily focus on studying some certain linguistic knowledge encoded in PLMs, rather than investigating how these models comprehend language from a holistic perspective. Furthermore, they cannot point out how PLMs organize the whole language system. Therefore, we adopt the complex network approach to represent the language system, and investigate how language elements are organized within the system. Specifically, we take the attention relationships among words as the research object, which are generated by attention heads within BERT models. Then, the words are treated as nodes, and the connections between words and their most-attending words are represented as edges. After obtaining these "words' attention networks", we analyze the network properties from various perspectives by calculating the network metrics. Many constructive conclusions are summarized, including: (1) The English attention networks demonstrate exceptional performance in organizing words; (2) Most words’ attention networks exhibit small-world property and scale-free behavior; (3) Some networks generated by multilingual BERT can reflect typological information well, achieving preferable clustering performance among language groups; (4) In cross-layer analysis, the networks from 8 to 10 layers in Chinese BERT and from 6 to 9 layers in English BERT exhibit more consistent characteristics. Our study provides a comprehensive explanation of how PLMs organize language systems, which can be utilized to evaluate and develop improved models.

Textual Compression Using Lamini-LM

In the age of digital overload, the ability to efficiently summarize large amount of text has become a critical capability. This project focuses on the development of a document summarization system that uses the power of the LaMini-LM model, a member of the Langchain family of large language models (LLMs). By using the natural language processing (NLP) techniques through the Langchain LLM technology it creates a robust and practical solution for automatically generating concise and informative summaries of text documents. The methodological approach of this project combines the strengths of NLP techniques, including the transformer architecture, the T5 model, and specialized components such as the T5 tokenizer and the Pipeline API. This strategic integration of technologies allows to create a comprehensive and efficient document summarization system that can effectively process and summarize a diverse range of text documents. The successful development and implementation of the document summarization system using the LaMini-LM model represents a significant advancement in the field of natural language processing.

Optimization Strategies for Deep Learning Models in Natural Language Processing

Deep learning models have achieved remarkable performance in the field of natural language processing (NLP), but they still face many challenges in practical applications, such as data heterogeneity and complexity, the black-box nature of models, and difficulties in transfer learning across multilingual and cross-domain scenarios. In this paper, corresponding improvement measures are proposed from four perspectives: model structure, loss functions, regularization methods, and optimization strategies, to address these issues. Extensive experiments on three tasks including text classification, named entity recognition, and reading comprehension confirm the feasibility and effectiveness of the proposed optimization solutions. The experimental results demonstrate that introducing innovative mechanisms like Multi-Head Attention and Focal Loss, and judiciously applying techniques such as LayerNorm and AdamW, can significantly improve model performance. Finally, this paper also explores model compression techniques, providing new insights for deploying deep models in resource-constrained scenarios.

ADLBiLSTM: A Semantic Generation Algorithm for Multi-Grammar Network Access Control Policies

Semantic generation of network access control policies can help network administrators accurately implement policies to achieve desired security objectives. Current semantic generation research mainly focuses on semantic generation of single grammar and lacks work on automatically generating semantics for different grammatical strategies. Generating semantics for different grammars is a tedious, inefficient, and non-scalable task. Inspired by sequence labeling in the field of natural language processing, this article models automatic semantic generation as a sequence labeling task. We propose a semantic generation algorithm named ADLBiLSTM. The algorithm uses a self-attention mechanism and double-layer BiLSTM to extract the features of security policies from different aspects, so that the algorithm can flexibly adapt to policies of different complexity without frequent modification. Experimental results showed that the algorithm has good performance and can achieve high accuracy in semantic generation of access control list (ACL) and firewall data and can accurately understand and generate the semantics of network access control policies.

#2924 Comparison of large language models and traditional natural language processing techniques in predicting arteriovenous fistula failure

Abstract Background and Aims Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP), marking a shift from traditional techniques like Term Frequency-Inverse Document Frequency (TF-IDF). We developed a traditional NLP model to predict arteriovenous fistula (AVF) failure within next 30 days using clinical notes. The goal of this analysis was to investigate whether LLMs would outperform traditional NLP techniques, specifically in the context of predicting AVF failure within the next 30 days using clinical notes. Method We defined AVF failure as the change in status from active to permanently unusable status or temporarily unusable status. We used data from a large kidney care network from January 2021 to December 2021. Two models were created using LLMs and traditional TF-IDF technique. We used “distilbert-base-uncased”, a distilled version of BERT base model [1], and compared its performance with traditional TF-IDF-based NLP techniques. The dataset was randomly divided into 60% training, 20% validation and 20% test dataset. The test data, comprising of unseen patients’ data was used to evaluate the performance of the model. Both models were evaluated using metrics such as area under the receiver operating curve (AUROC), accuracy, sensitivity, and specificity. Results The incidence of 30 days AVF failure rate was 2.3% in the population. Both LLMs and traditional showed similar overall performance as summarized in Table 1. Notably, LLMs showed marginally better performance in certain evaluation metrics. Both models had same AUROC of 0.64 on test data. The accuracy and balanced accuracy for LLMs were 72.9% and 59.7%, respectively, compared to 70.9% and 59.6% for the traditional TF-IDF approach. In terms of specificity, LLMs scored 73.2%, slightly higher than the 71.2% observed for traditional NLP methods. However, LLMs had a lower sensitivity of 46.1% compared to 48% for traditional NLP. However, it is worth noting that training on LLMs took considerably longer than TF-IDF. Moreover, it also used higher computational resources such as utilization of graphics processing units (GPU) instances in cloud-based services, leading to higher cost. Conclusion In our study, we discovered that advanced LLMs perform comparably to traditional TF-IDF modeling techniques in predicting the failure of AVF. Both models demonstrated identical AUROC. While specificity was higher in LLMs compared to traditional NLP, sensitivity was higher in traditional NLP compared to LLMs. LLM was fine-tuned with a limited dataset, which could have influenced its performance to be similar to that of traditional NLP methods. This finding suggests that while LLMs may excel in certain scenarios, such as performing in-depth sentiment analysis of patient data for complex tasks, their effectiveness is highly dependent on the specific use case. It is crucial to weigh the benefits against the resources required for LLMs, as they can be significantly more resource-intensive and costly compared to traditional TF-IDF methods. This highlights the importance of a use-case-driven approach in selecting the appropriate NLP technique for healthcare applications.

Using corpora from Natural Language Processing for investigating crosslinguistic influence

Language transfer or crosslinguistic influence (CLI), referring to the influence of an L1 on the learning of an L2, is a significant aspect of Second Language Acquisition (SLA). Much work in this area is data-driven, and consequently, large L2 corpora have been constructed for use in CLI analyses. The field of Natural Language Processing, and in particular the specific task of Grammatical Error Correction (GEC), also has corpora that can be of use in these kinds of analyses. In this paper, we take the FCE corpus, a popular dataset of English as a Second Language (ESL) learner texts used for Grammatical Error Correction model training, and use it to analyse the relationship between the distributions of errors and the first languages of the ESL learners. We carry out a detailed analysis of three error types, and demonstrate that the errors made by ESL learners have a statistically significant relationship with linguistic characteristics of their first languages, suggesting the existence of both positive and negative transfer. The analysis aligns with results from the SLA literature, and validates the use of GEC corpora for use in CLI analysis.

Semantic Verification in Large Language Model-based Retrieval Augmented Generation

This position paper presents a novel approach of semantic verification in Large Language Model-based Retrieval Augmented Generation (LLM-RAG) systems, focusing on the critical need for factually accurate information dissemination during public debates, especially prior to plebiscites e.g. in direct democracies, particularly in the context of Switzerland. Recognizing the unique challenges posed by the current generation of Large Language Models (LLMs) in maintaining factual integrity, this research proposes an innovative solution that integrates retrieval mechanisms with enhanced semantic verification processes. The paper outlines a comprehensive methodology following a Design Science Research approach, which includes defining user personas, designing conversational interfaces, and iteratively developing a hybrid dialogue system. Central to this system is a robust semantic verification framework that leverages a knowledge graph for fact-checking and validation, ensuring the correctness and consistency of information generated by LLMs. The paper discusses the significance of this research in the context of Swiss direct democracy, where informed decision-making is pivotal. By improving the accuracy and reliability of information provided to the public, the proposed system aims to support the democratic process, enabling citizens to make well-informed decisions on complex issues. The research contributes to advancing the field of natural language processing and information retrieval, demonstrating the potential of AI and LLMs in enhancing civic engagement and democratic participation.

AMPD: an Analects-Mandarin parallel dataset for bidirectional translation

ABSTRACT In the field of natural language processing, there is no specialized dataset for the Analects, which makes it difficult to assess whether language models can find the semantic relevance between the Analects and modern Mandarin. To address this issue, this paper proposes a dataset named AMPD (Analects-Mandarin Parallel Dataset), which includes the Analects and its corresponding modern Mandarin, keywords and their annotations in the Analects, as well as sentiment. Additionally, we propose four baseline tasks and benchmark them by implementing currently popular algorithms respectively.

How Factually Accurate is GPT-3? A Focused Case Study on Helping Malaysia’s B40s Through e-Commerce

GPT-3 (Generative Pre-trained Transformer 3) is an advanced natural language processing model utilizing unsupervised learning to generate sophisticated human-like text. GPT-3 has been lauded for its potential to revolutionize the field of natural language processing, with its capacity to generate a variety of text with a high degree of fluency and accuracy. We examine the ability of GPT-3’s to produce text related to a focused subject matter: alleviating poverty in Malaysia through e-Commerce. We especially examine GPT-3’s ability to produce factual responses within this narrow context. It was discovered that GPT-3 could produce plausible statements, albeit some of them being factually debatable or incorrect due to how its training data was sourced. We also discuss how GPT-3 could be used unscrupulously to either produce academic-sounding responses that appear to be a product of research, but possibly untrue or inaccurate and discuss its potential ramifications (such as propaganda and disinformation). We end the paper with some suggestions to the brilliant team at OpenAI to further improve GPT-3 for the advancement of humankind.

Exploring named‐entity recognition techniques for academic books

Recent advances in the natural language processing (NLP) field have achieved impressive results in various tasks. However, NLP techniques are underrepresented in the analysis of Humanities and Social Science texts and in languages other than English. In particular, academic books are a highly valuable source of information that has not been exploited by these techniques at all. The recognition of named entities (person names, organizations or locations) and their semantic annotation over books could enrich the visibility and discoverability of the information by users. This is an opportunity for academia and the academic publishing industry in which semantic search is a central task and now books can be queried by named entities of interest that are in their content. This work proposes a methodology to apply named‐entity recognition to publish the results into an ontological semantic‐web format. The work has been performed over a corpus of academic books provided by UNE (Unión de Editoriales Universitarias Españolas, Union of Spanish University Presses). Results show an enrichment of the information extracted over the books and of the possibilities of querying them at the individual level but also within the whole set of books, increasing the possibilities for books to be discovered or retrieved beyond metadata.