Field Of Natural Language Processing Research Articles

From RNN techniques to pre-trained models: emphasis on the use in Arabic machine translation

<p>Neural Machine Translation (NMT) has gained increasing attention in recent years due to its promising performance over conventional approaches such as Statistical Machine Translation. Nevertheless, when applied to languages with different structures, such as the pair (English, Arabic) that interests us in this work, its efficiency is degraded.<br />Alternatively, the use of unsupervised pre-training of large neural models has recently made a significant leap forward in the field of Natural Language Processing (NLP). By warm starting from published checkpoints, NLP practitioners have been able to push the state-of-the-art boundaries on multiple benchmarks while simultaneously saving considerable computational time. The emphasis so far has been mainly on natural language understanding problems. In this paper, we show the efficacy of pre-trained checkpoints for Arabic Machine Translation. We have evolved a transformer-based sequence-to-sequence model which is compatible with pre-trained checkpoints publicly available for Arabic Bidirectional Encoder Representations from Transformers (AraBERT) and Arabic Generative Pre-trained Transformer (AraGPT) and conducted a thorough empirical study on the usefulness of initializing our Arabic MT model, both encoder and decoder, with these checkpoints. Our models yield new state-of-the-art results in Arabic MT.</p>

Evaluating the machine learning models based on natural language processing tasks

<span lang="EN-US">In the realm of natural language processing (NLP), a diverse array of language models has emerged, catering to a wide spectrum of tasks, ranging from speaker recognition and auto-correction to sentiment analysis and stock prediction. The significance of language models in enabling the execution of these NLP tasks cannot be overstated. This study proposes an approach to enhance accuracy by leveraging a hybrid language model, combining the strengths of long short-term memory (LSTM) and gated recurrent unit (GRU). LSTM excels in preserving long-term dependencies in data, while GRU's simpler gating mechanism expedites the training process. The research endeavors to evaluate four variations of this hybrid model: LSTM, GRU, <a name="_Hlk158992630"></a>bidirectional long short-term memory (Bi-LSTM), and a combination of LSTM with GRU. These models are subjected to rigorous testing on two distinct datasets: one focused on IBM stock price prediction, and the other on Jigsaw toxic comment classification (sentiment analysis). This work represents a significant stride towards democratizing NLP capabilities, ensuring that even in resource-constrained settings, NLP models can exhibit improved performance. The anticipated implications of these findings span a wide spectrum of real-world applications and hold the potential to stimulate further research in the field of NLP. </span>

AMPD: an Analects-Mandarin parallel dataset for bidirectional translation

ABSTRACT In the field of natural language processing, there is no specialized dataset for the Analects, which makes it difficult to assess whether language models can find the semantic relevance between the Analects and modern Mandarin. To address this issue, this paper proposes a dataset named AMPD (Analects-Mandarin Parallel Dataset), which includes the Analects and its corresponding modern Mandarin, keywords and their annotations in the Analects, as well as sentiment. Additionally, we propose four baseline tasks and benchmark them by implementing currently popular algorithms respectively.

How Factually Accurate is GPT-3? A Focused Case Study on Helping Malaysia’s B40s Through e-Commerce

GPT-3 (Generative Pre-trained Transformer 3) is an advanced natural language processing model utilizing unsupervised learning to generate sophisticated human-like text. GPT-3 has been lauded for its potential to revolutionize the field of natural language processing, with its capacity to generate a variety of text with a high degree of fluency and accuracy. We examine the ability of GPT-3’s to produce text related to a focused subject matter: alleviating poverty in Malaysia through e-Commerce. We especially examine GPT-3’s ability to produce factual responses within this narrow context. It was discovered that GPT-3 could produce plausible statements, albeit some of them being factually debatable or incorrect due to how its training data was sourced. We also discuss how GPT-3 could be used unscrupulously to either produce academic-sounding responses that appear to be a product of research, but possibly untrue or inaccurate and discuss its potential ramifications (such as propaganda and disinformation). We end the paper with some suggestions to the brilliant team at OpenAI to further improve GPT-3 for the advancement of humankind.

Exploring named‐entity recognition techniques for academic books

Recent advances in the natural language processing (NLP) field have achieved impressive results in various tasks. However, NLP techniques are underrepresented in the analysis of Humanities and Social Science texts and in languages other than English. In particular, academic books are a highly valuable source of information that has not been exploited by these techniques at all. The recognition of named entities (person names, organizations or locations) and their semantic annotation over books could enrich the visibility and discoverability of the information by users. This is an opportunity for academia and the academic publishing industry in which semantic search is a central task and now books can be queried by named entities of interest that are in their content. This work proposes a methodology to apply named‐entity recognition to publish the results into an ontological semantic‐web format. The work has been performed over a corpus of academic books provided by UNE (Unión de Editoriales Universitarias Españolas, Union of Spanish University Presses). Results show an enrichment of the information extracted over the books and of the possibilities of querying them at the individual level but also within the whole set of books, increasing the possibilities for books to be discovered or retrieved beyond metadata.

Neural Machine Translation for Low-Resource Languages from a Chinese-centric Perspective: A Survey

Machine translation—the automatic transformation of one natural language (source language) into another (target language) through computational means—occupies a central role in computational linguistics and stands as a cornerstone of research within the field of Natural Language Processing (NLP). In recent years, the prominence of Neural Machine Translation (NMT) has grown exponentially, offering an advanced framework for machine translation research. It is noted for its superior translation performance, especially when tackling the challenges posed by low-resource language pairs that suffer from a limited corpus of data resources. This article offers an exhaustive exploration of the historical trajectory and advancements in NMT, accompanied by an analysis of the underlying foundational concepts. It subsequently provides a concise demarcation of the unique characteristics associated with low-resource languages and presents a succinct review of pertinent translation models and their applications, specifically within the context of languages with low-resources. Moreover, this article delves deeply into machine translation techniques, highlighting approaches tailored for Chinese-centric low-resource languages. Ultimately, it anticipates upcoming research directions in the realm of low-resource language translation.

Potential of Large Language Models in Health Care: Delphi Study.

A large language model (LLM) is a machine learning model inferred from text data that captures subtle patterns of language use in context. Modern LLMs are based on neural network architectures that incorporate transformer methods. They allow the model to relate words together through attention to multiple words in a text sequence. LLMs have been shown to be highly effective for a range of tasks in natural language processing (NLP), including classification and information extraction tasks and generative applications. The aim of this adapted Delphi study was to collect researchers' opinions on how LLMs might influence health care and on the strengths, weaknesses, opportunities, and threats of LLM use in health care. We invited researchers in the fields of health informatics, nursing informatics, and medical NLP to share their opinions on LLM use in health care. We started the first round with open questions based on our strengths, weaknesses, opportunities, and threats framework. In the second and third round, the participants scored these items. The first, second, and third rounds had 28, 23, and 21 participants, respectively. Almost all participants (26/28, 93% in round 1 and 20/21, 95% in round 3) were affiliated with academic institutions. Agreement was reached on 103 items related to use cases, benefits, risks, reliability, adoption aspects, and the future of LLMs in health care. Participants offered several use cases, including supporting clinical tasks, documentation tasks, and medical research and education, and agreed that LLM-based systems will act as health assistants for patient education. The agreed-upon benefits included increased efficiency in data handling and extraction, improved automation of processes, improved quality of health care services and overall health outcomes, provision of personalized care, accelerated diagnosis and treatment processes, and improved interaction between patients and health care professionals. In total, 5 risks to health care in general were identified: cybersecurity breaches, the potential for patient misinformation, ethical concerns, the likelihood of biased decision-making, and the risk associated with inaccurate communication. Overconfidence in LLM-based systems was recognized as a risk to the medical profession. The 6 agreed-upon privacy risks included the use of unregulated cloud services that compromise data security, exposure of sensitive patient data, breaches of confidentiality, fraudulent use of information, vulnerabilities in data storage and communication, and inappropriate access or use of patient data. Future research related to LLMs should not only focus on testing their possibilities for NLP-related tasks but also consider the workflows the models could contribute to and the requirements regarding quality, integration, and regulations needed for successful implementation in practice.

Multi-Task Learning in Natural Language Processing: An Overview

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Natural Language Reasoning, A Survey

This survey paper proposes a clearer view of natural language reasoning in the field of Natural Language Processing (NLP), both conceptually and practically. Conceptually, we provide a distinct definition for natural language reasoning in NLP, based on both philosophy and NLP scenarios, discuss what types of tasks require reasoning, and introduce a taxonomy of reasoning. Practically, we conduct a comprehensive literature review on natural language reasoning in NLP, mainly covering classical logical reasoning, natural language inference, multi-hop question answering, and commonsense reasoning. The paper also identifies and views backward reasoning, a powerful paradigm for multi-step reasoning, and introduces defeasible reasoning as one of the most important future directions in natural language reasoning research. We focus on single-modality unstructured natural language text, excluding neuro-symbolic research and mathematical reasoning.

STABLE DIFFUSION TEXT TO IMAGE USING AI

The Stable Diffusion Text-to-Image Generation Project is an innovative endeavor in the field of generative adversarial networks (GANs) and natural language processing (NLP). This project aims to bridge the semantic gap between textual descriptions and visual content by utilizing the Stable Diffusion training framework to generate highly realistic and coherent images from text prompts. The project leverages recent advancements in deep learning techniques to tackle the challenging task of text-to image synthesis. The project introduces an innovative approach at the crossroads of generative adversarial networks (GANs) and natural language processing (NLP). It aims to bridge the semantic gap between textual descriptions and visual content by utilizing the Stable Diffusion training framework. The key goal is to generate highly realistic and coherent images from text prompts, leveraging recent deep learning Stable Diffusion. The Stable Diffusion training framework plays a central role in this project. It’s a sophisticated training methodology for GANs, designed to stabilize the training process. GANs have exhibited great potential in generating images but often suffer from issues like mode collapse and training instability. Keywords: Stable Diffusion, Text-to-Image Generation, Image Synthesis, Natural Language Processing(NLP), Generative Adversarial Networks (GANs)

A comprehensive analysis of static word embeddings for Turkish

Word embeddings are fixed-length, dense and distributed word representations that are used in natural language processing (NLP) applications. There are basically two types of word embedding models which are non-contextual (static) models and contextual models. The former method generates a single embedding for a word regardless of its context, while the latter method produces distinct embeddings for a word based on the specific contexts in which it appears. There are plenty of works that compare contextual and non-contextual embedding models within their respective groups in different languages. However, the number of studies that compare the models in these two groups with each other is very few and there is no such study in Turkish. This process necessitates converting contextual embeddings into static embeddings. In this paper, we compare and evaluate the performance of several contextual and non-contextual models in both intrinsic and extrinsic evaluation settings for Turkish. We make a fine-grained comparison by analyzing the syntactic and semantic capabilities of the models separately. The results of the analyses provide insights about the suitability of different embedding models in different types of NLP tasks. We also build a Turkish word embedding repository comprising the embedding models used in this work, which may serve as a valuable resource for researchers and practitioners in the field of Turkish NLP. We make the word embeddings, scripts, and evaluation datasets publicly available.

AI Call Assistant – Extractive Summarization of Call Recordings

In today's fast-paced business landscape, effective communication is a key for success. With the increasing volume of phone calls in various industries, there is a growing need for efficient call management and analysis tools. The AI Call Assistant project aims to address this need by leveraging artificial intelligence (AI) technology to summarize call recordings, enhancing productivity and decision-making processes. The core functionality of this project revolves around its ability to process audio data from call recordings and extract key insights and information. Using advanced natural language processing (NLP) techniques, the system identifies important topics, sentiments, and actions discussed during the call. The call transcripts attained from call recording pose unique challenges that are not adequately addressed by most open-source automatic text summarizers. This project aims to contribute to the field of artificial intelligence by providing efficient and effective methods to recognize call recording and summarize, offering a valuable tool for extracting insights from spoken audio efficiently.Top of Form This research aligns with the broader fields of natural language processing (NLP), machine learning, and artificial intelligence (AI), particularly in the domain of speech recognition and understanding. It also intersects with communication technology and data analytics, focusing on optimizing call management processes and extracting actionable insights from conversational data. Moreover, it aligns with the goal of improving efficiency and decision-making in various industries through the application of AI-driven solutions. Overall, this project offers a novel approach to improving call management processes through AI-driven extractive summarization, contributing to advancements in natural language processing and communication technologies. Key Words: Artificial Intelligence, BERT, Textual Summarization, Transformers, Natural Language Processing, Speech Recognition, Speech-To-Text.

Text Summarization Based on Semantic Similarity

In the contemporary information age, the sheer volume of textual data poses a significant challenge for efficient comprehension and utilization. This project endeavors to address this challenge by developing a Text Summarization System grounded in semantic similarities. The primary goal is to create a robust and intuitive tool that extracts key information from large textual datasets, offering users a concise and meaningful summary. The proposed system employs advanced Natural Language Processing (NLP) techniques to analyze the semantic relationships within the text. Rather than relying solely on syntactic structures, the model identifies and leverages semantic similarities, such as shared concepts, themes, and contextual relationships, to distill the essential content. This approach enhances the summarization process by ensuring that the generated summaries reflect a deeper understanding of the underlying semantics, thereby capturing the core meaning of the text. Throughout the development of this project, the B.Tech student will delve into the intricacies of semantic analysis, exploring techniques to recognize and prioritize key concepts. The system's effectiveness will be evaluated through rigorous testing on diverse textual datasets, assessing its ability to generate coherent and relevant summaries across various domains. This project not only contributes to the field of NLP but also has practical applications in information retrieval, document summarization, and content curation. By providing an innovative solution to the challenges of information overload, the Text Summarization System based on semantic similarities offers a valuable tool for enhancing efficiency in information processing and decision-making. Index terms Text Summarization, Semantic Similarities, Natural Language Processing (NLP), Semantic Analysis, Information Retrieval, Document Summarization, Content Curation, Information Overload, Decision Making, Textual Data Analysis, Key Concept Recognition, Conceptual Relationships, Syntactic Structures, Semantic Understanding, Textual Datasets Evalution.

ADVANCMENTS IN TEXT SUMMARIZATION AND EXTRACTIVE QUESTION- ANSWERING : A MACHINE LEARNING APPROACH

In the era of social media platforms, the rapid expansion of data mining in the fields of information retrieval and natural language processing emphasizes the crucial need for automated text summarization. At the current time, pretrained word embedding techniques and sequence to sequence models can be effectively repurposed in the realm of social network summarization to efficiently condense significant information with strong encoding capabilities. However, dealing with the challenge of extended text dependency and efficiently utilizing latent topic mapping presents an increasingly significant obstacle for these computational models. In this document, we propose a topic- focused approach for both extractive and abstractive summarization, integrating Question Answering features by employing BERT and Pegasus pretrained models. A comprehensive analysis of the architectural complexities of the pre-existing models utilized in the creation of our model has been presented. The evaluation of our Text Summarization & Question Answering Model was conducted meticulously on a variety of datasets including Multi news, XSum, and CNN/Daily Mail, with experimental results indicating its achievement of state-of-the-art performance levels based on ROGUE scores. Following this, the verification of our results through human evaluation confirms that the summaries generated by our model are in alignment with the performance standards set by humans across various datasets. Furthermore, we have integrated an Extractive Question Answering task into our model, proposing an architecture leveraging BERT and conducting comparative analyses against alternative language models to gauge its efficacy. Keywords—Natural Language Processing (NLP), Text Summarization, Question & Answering (Q&A), Transformer Models, Pegasus, BERT (Bidirectional Encoder Representations from Transformers), Neural Topic Modeling, Extractive Summarization, Abstractive Summarization, Semantic Representation, Topic Embedding

Mental health prediction from social media connections

ABSTRACT The relation between mental health and the language employed by social media users has been the focus of a wide range of studies in Natural Language Processing and related fields intended to predict early signs of a mental disorder. In many cases, however, text data alone is considered to be a sufficient source of learning features, and other possible sources also available from social media are often neglected. To shed light on these issues, the present work addresses the use of social media connections information — represented by the identities of Twitter friends, followers and mentions — to predict depression and anxiety disorder. Using a large multimodal dataset of over 31K unique Twitter timelines (555 million words) and associated network data, we built a number of network-based models (using network embeddings representations), and text-based alternatives (bag-of-words and LIWC baseline models) to these tasks. Results suggest that not only network connections may act as strong predictors of mental health, but that these models may actually outperform standard baseline alternatives that rely on text data alone, which may be seen as a first step towards more sophisticated architectures that may actually combine textual and non-textual information.

A new framework for building agricultural domain-based ontologies from text documents using natural language processing and artificial intelligence techniques

The advancements in technology, particularly in the field of Natural Language Processing (NLP) and Artificial Intelligence (AI) can be advantageous for the agricultural sector to enhance the yield. Establishing an agricultural ontology as part of the development would spur the expansion of cross-domain agriculture. Semantic and syntactic knowledge of the domain data is required for building such a domain-based ontology. To process the data from text documents, a standard technique with syntactic and semantic features are needed because the availability of pre-determined agricultural domain-based data is insufficient. In this research work, an Agricultural Ontologies Construction framework (AOC) is proposed for creating the agricultural domain ontology from text documents using NLP techniques with Robustly Optimized BERT Approach (RoBERTa) model and Graph Convolutional Network (GCN). The anaphora present in the documents are resolved to produce precise ontology from the input data. In the proposed AOC work, the domain terms are extracted using the RoBERTa model with Regular Expressions (RE) and the relationships between the domain terms are retrieved by utilizing the GCN with RE. When compared to other current systems, the efficacy of the proposed AOC method achieves an exceptional result, with precision and recall of 99.6% and 99.1% respectively.

Human Language Data Processing using NLTK

Natural Language Toolkit (NLTK) is a comprehensive Python library designed to facilitate the exploration, analysis, and processing of human language data. With its extensive collection of tools, NLTK provides researchers, developers, and educators with a powerful platform for tasks ranging from basic text processing to advanced natural language understanding and machine learning. The toolkit includes modules for tokenization, stemming, lemmatization, part-of-speech tagging, named entity recognition, syntactic parsing, semantic analysis, and more. Furthermore, NLTK offers access to numerous linguistic resources such as corpora, lexicons, and treebanks, making it an invaluable resource for both learning and research in the field of natural language processing (NLP). NLTK serves as an indispensable tool for unlocking the complexities of human language

R-Cut: Enhancing Explainability in Vision Transformers with Relationship Weighted Out and Cut.

Transformer-based models have gained popularity in the field of natural language processing (NLP) and are extensively utilized in computer vision tasks and multi-modal models such as GPT4. This paper presents a novel method to enhance the explainability of transformer-based image classification models. Our method aims to improve trust in classification results and empower users to gain a deeper understanding of the model for downstream tasks by providing visualizations of class-specific maps. We introduce two modules: the "Relationship Weighted Out" and the "Cut" modules. The "Relationship Weighted Out" module focuses on extracting class-specific information from intermediate layers, enabling us to highlight relevant features. Additionally, the "Cut" module performs fine-grained feature decomposition, taking into account factors such as position, texture, and color. By integrating these modules, we generate dense class-specific visual explainability maps. We validate our method with extensive qualitative and quantitative experiments on the ImageNet dataset. Furthermore, we conduct a large number of experiments on the LRN dataset, which is specifically designed for automatic driving danger alerts, to evaluate the explainability of our method in scenarios with complex backgrounds. The results demonstrate a significant improvement over previous methods. Moreover, we conduct ablation experiments to validate the effectiveness of each module. Through these experiments, we are able to confirm the respective contributions of each module, thus solidifying the overall effectiveness of our proposed approach.

Research Paper Summarization and Recommendation

The exponential growth of academic literature has made it increasingly challenging for researchers to keep up with the latest developments in their respective fields. This paper presents an approach for summarizing and recommending research papers to help researchers efficiently manage and navigate the vast amount of academic literature. We propose using a transformer-based model for generating abstractive summaries of research papers and a cosine similarity-based recommendation system for suggesting similar papers to a given paper. The proposed approach is evaluated on a dataset of research papers, and the results show that it is effective in generating coherent and concise summaries and recommending relevant research papers. The findings of this study have important implications for researchers and practitioners in the field of natural language processing and information retrieval.

IDENTIFICATION OF EMOTIONAL WORDS IN TEXT COMMUNICATION

The study of the identification of emotions in text communication is an actual direction of research in the field of natural language processing and machine learning. The main goal of the research is to develop algorithms and models that can automatically determine a person's emotional state based on text messages. This work is devoted to the review of some models and an algorithm for improving data processing in the middle of text communication of users.