Natural Language Processing Domain Research Articles

Application of the Nadaraya-Watson estimator based attention mechanism to the field of predictive maintenance

Attention mechanism has recently gained immense importance in the natural language processing (NLP) world. This technique highlights parts of the input text that the NLP task (such as translation) must pay “attention” to. Inspired by this, some researchers have recently applied the NLP domain, deep-learning based, attention mechanism techniques to predictive maintenance. In contrast to the deep-learning based solutions, Industry 4.0 predictive maintenance solutions that often rely on edge-computing, demand lighter predictive models. With this objective, we have investigated the adaptation of a simpler, incredibly fast and compute-resource friendly, “Nadaraya-Watson estimator based” attention method. We develop a method to predict tool-wear of a milling machine using this attention mechanism and demonstrate, with the help of heat-maps, how the attention mechanism highlights regions that assist in predicting onset of tool-wear. We validate the effectiveness of this adaptation on the benchmark IEEEDataPort PHM Society dataset, by comparing against other comparatively “lighter” machine learning techniques – Bayesian Ridge, Gradient Boosting Regressor, SGD Regressor and Support Vector Regressor. Our experiments indicate that the proposed Nadaraya-Watson attention mechanism performed best with an MAE of 0.069, RMSE of 0.099 and R2 of 83.40 %, when compared to the next best technique Gradient Boosting Regressor with figures of 0.100, 0.138, 66.51 % respectively. Additionally, it produced a lighter and faster model as well.•We propose a Nadaraya-Watson estimator based “attention mechanism”, applied to a predictive maintenance problem.•Unlike the deep-learning based attention mechanisms from the NLP domain, our method creates fast, light and high-performance models, suitable for edge computing devices and therefore supports the Industry 4.0 initiative.•Method validated on real tool-wear data of a milling machine.

Enhanced Linear and Vision Transformer-Based Architectures for Time Series Forecasting

Time series forecasting has been a challenging area in the field of Artificial Intelligence. Various approaches such as linear neural networks, recurrent linear neural networks, Convolutional Neural Networks, and recently transformers have been attempted for the time series forecasting domain. Although transformer-based architectures have been outstanding in the Natural Language Processing domain, especially in autoregressive language modeling, the initial attempts to use transformers in the time series arena have met mixed success. A recent important work indicating simple linear networks outperform transformer-based designs. We investigate this paradox in detail comparing the linear neural network- and transformer-based designs, providing insights into why a certain approach may be better for a particular type of problem. We also improve upon the recently proposed simple linear neural network-based architecture by using dual pipelines with batch normalization and reversible instance normalization. Our enhanced architecture outperforms all existing architectures for time series forecasting on a majority of the popular benchmarks.

Analyse the Enhancement of Sentiment Analysis in Arabic by doing a Comparative Study of Several Machine Learning Techniques

Abstract: Sentiment analysis is a crucial component of natural language processing that seeks to determine the emotional sentiment expressed in a given text. This study investigates sentiment analysis in the Arabic language through a comprehensive approach that integrates traditional machine learning methods with sophisticated deep learning models. We examine the efficacy of conventional algorithms such as Support Vector Machines (SVM) and Naive Bayes, as well as sophisticated neural network architectures such as Convolutional Neural Networks (CNNs), Long Short-Term Memory networks (LSTMs), and the Arabic variant of Bidirectional Encoder Representations from Transformers (BERT). The primary novelty of our approach is in the ensemble method, which combines many approaches to enhance the precision of sentiment categorization in Arabic text. To address the particular challenges presented by Arabic sentiment analysis, such as the intricate structure of the language and the diverse regional variations, we utilize a tailored preprocessing pipeline to effectively handle the nuances of Arabic text. Our comprehensive analysis of various datasets demonstrates that the ensemble technique outperforms individual model benchmarks and offers novel insights into the interplay between different machine learning paradigms in Arabic NLP. The results emphasize the ability of hybrid approaches to improve Arabic sentiment analysis, providing a solid basis for future research and practical applications in understanding the sentiments of Arabic consumers. This study is a significant addition to the expanding domain of Arabic Natural Language Processing (NLP). This resource offers a comprehensive and advanced methodology for utilizing machine learning and deep learning methods to comprehend and analyse the intricate aspects of sentiment in the Arabic language

Design and Implementation of English Writing Aids Based on Natural Language Processing

The design and implementation of English writing aids based on natural language processing (NLP) involve leveraging advanced algorithms and techniques to assist users in improving their writing skills. These aids can encompass various functionalities such as grammar and spell checking, style suggestions, vocabulary enhancement, and plagiarism detection. By analyzing the context, structure, and semantics of the text, NLP models can provide intelligent feedback and recommendations to help users express themselves more effectively. Additionally, interactive features such as real-time editing and personalized suggestions contribute to a seamless user experience. The paper presents a comprehensive exploration of the Sentimental Random Field Point (SRFP) approach in the domain of natural language processing (NLP). Through a series of experiments and analyses, we investigate the effectiveness and versatility of SRFP across various NLP tasks, including sentiment analysis, named entity recognition, text classification, and machine translation. Our findings demonstrate the robustness of SRFP-based algorithms and models in accurately classifying sentiments, enhancing grammatical correctness, expanding vocabulary, improving writing style, and achieving high performance in classification tasks. Through results it is observed that an accuracy of 92.7% for sentiment analysis using the BERT model, a precision of 85.3% and recall of 88.9% for vocabulary enhancement, and an accuracy of 93.5% for machine translation using the Transformer model.

LLaMA-LoRA Neural Prompt Engineering: A Deep Tuning Framework for Automatically Generating Chinese Text Logical Reasoning Thinking Chains

Abstract The exption of Chinese natural language processing (NLP) has stimulated research in the broader NLP domain. However, existing large language models have limitations in comprehending and reasoning in Chinese. This paper addresses these limitations by enhancing Chinese language models comprehension and reasoning capabilities while minimizing resource requirements. We propose LLaMA-LoRA, a neural prompt engineering framework that builds upon the LLaMA-13B model and incorporates the Low-Rank Adaptation(LoRA) of Large Language Models technique for refinement. Chain-of-Thought(CoT) are crucial for generating intermediate reasoning chains in language models, but their effectiveness can be limited by isolated language patterns. Erroneous reasoning resulting from conventional prompts negatively impacts model performance. Automatic prompts are introduced to encourage reasoning chain generation and accurate answer inference. Training the model with an extensive corpus of Chinese CoT data enhances its comprehension and reasoning abilities. The LLaMA-LoRA model demonstrates exceptional performance across numerous Chinese language tasks, surpassing benchmark performance achieved by related language models such as GPT-3.5, Chat-GLM, and OpenAssistant, delivering accurate, comprehensive, and professional answers. The availability of our open-source model code facilitates further research in the field of Chinese text logical reasoning thinking chains.

Translation of English Videos to Indian Regional Languages

In spite of many languages being spoken in India, it is difficult for the people to understand Indian regional languages like English, Gujrati, Kannada, Tamil, Telugu, Punjabi, Malayalam, etc. The recognition and synthesis of speech are prominent emerging technologies in natural language processing and communication domains. This paper aims to leverage the open-source applications of these technologies, machine translation, text-to-speech system (TTS), and speech-to-text system (STT) to convert available online resources to Indian languages. This application takes an English language video as an input and separates the audio from video. It then divides the audio file into several smaller chunks based on the timestamps. These audio chunks are then individually converted into text using WhisperAI speech recognition model.And facebooks M2M100 model for translation. After this translation, a TTS system is required to convert the text into the desired audio output. Not many open source TTS systems are available for Indian regional languages. This application is beneficial to visually impaired people as well as individuals who are not capable of reading text to acquire knowledge in their native language. In future, this application aims to achieve ubiquitous communication enabling people of different regions to communicate with each other breaking the language barriers. Index Terms- Translation of English to Regional indian languages, ,translation and transcription using WhisperAI and FacebookM2M100, Jupiter Notebook

Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning

As the societal impact of Deep Neural Networks (DNNs) grows, the goals for advancing DNNs become more complex and diverse, ranging from improving a conventional model accuracy metric to infusing advanced human virtues such as fairness, accountability, transparency, and unbiasedness. Recently, techniques in Explainable Artificial Intelligence (XAI) have been attracting considerable attention and have tremendously helped Machine Learning (ML) engineers in understand AI models. However, at the same time, we started to witness the emerging need beyond XAI among AI communities; based on the insights learned from XAI, how can we better empower ML engineers in steering their DNNs so that the model’s reasonableness and performance can be improved as intended? This article provides a timely and extensive literature overview of the field Explanation-Guided Learning (EGL), a domain of techniques that steer the DNNs’ reasoning process by adding regularization, supervision, or intervention on model explanations. In doing so, we first provide a formal definition of EGL and its general learning paradigm. Second, an overview of the key factors for EGL evaluation, as well as summarization and categorization of existing evaluation procedures and metrics for EGL are provided. Finally, the current and potential future application areas and directions of EGL are discussed, and an extensive experimental study is presented aiming at providing comprehensive comparative studies among existing EGL models in various popular application domains, such as Computer Vision and Natural Language Processing domains. Additional resources related to event prediction are included in the article website: https://kugaoyang.github.io/EGL/

Regular paper Comparative Analysis of Deep Learning Models for Sentiment Analysis on IMDB Reviews

Within the domain of natural language processing, sentiment analysis assumes a fundamental position, facilitating the comprehension of societal perspectives and opinions, thereby playing a pivotal role in understanding public sentiment. In this study, an examination and comparison of deep learning architectures was conducted on IMDB movie reviews. We evaluated the performance of Basic Recurrent Neural Networks, Long Short-Term Memory (LSTM), Gated Recurrent Neural Unit (GRU), Bidirectional LSTM, Bidirectional GRU and 1D Convolutional Neural Networks (Conv1D) based on their training, validation, and testing accuracies. Our results indicate that while LSTM achieved the highest accuracy of 99.91% on the training data, GRU demonstrated superior performance (88.28%) on the validation dataset. Interestingly, Bidirectional GRU emerged as the top-performing model (87.54%) on the testing data, showcasing its robustness in generalizing to unseen instances. These findings highlight the importance of evaluating model performance across multiple datasets to assess their real-world effectiveness. Furthermore, our comparative analysis provides valuable understanding of the advantages and limitations of each model, offering practical guidance for selecting the optimal framework for sentiment analysis endeavors. Overall, this research contributes to the progress of such methodologies and deep learning approaches in natural language processing.

Crosslingual Transfer Learning for Arabic Story Ending Generation

In the field of natural language processing, the task of generating story endings (SEG) requires not only a deep understanding of the narrative context but also the ability to formulate coherent conclusions. This study delves into the use of crosslingual transfer learning to address the challenges posed by the scarcity of Arabic data in SEG, proposing the utilization of extensive English story corpora as a solution. We evaluated the efficacy of multilingual models, such as mBART, mT5, and mT0, in generating Arabic story endings, assessing their performance in both zero-shot and few-shot scenarios. Despite the linguistic complexities of Arabic and the inherent challenges of crosslingual transfer, our findings demonstrate the potential of these multilingual models to transcend linguistic barriers, significantly contributing to the domain of natural language processing across different languages. This research has significant implications for generating creative text and improving multilingual natural language processing in resource-limited language contexts

TEXT GENERATION MODELS FOR PARAPHRASE ON KAZAKH LANGUAGE

This study delves into the relatively unexplored domain of natural language processing for the Kazakh language—a language with limited computational resources. The paper dissects the effectiveness of diffusion models and transformers in generating text, specifically paraphrases, which is a critical aspect of machine learning applications such as chatbots, virtual assistants, and automated translation services.
 The researchers methodically adapt these advanced models to understand and generate Kazakh text, tackling the unique challenges posed by the language's complex morphology. The paper is comprehensive in its approach, covering everything from the initial adaptation of the models to the Kazakh language context, to the creation of specialized tokenizer tools, to the translation and preparation of datasets for effective training.
 Through rigorous testing and performance analysis, the study identifies the strengths and weaknesses of each model type. This is critical as it informs the direction of future research and model development, with the goal of enhancing the fluency and accuracy of automated Kazakh text generation. The paper also discusses the broader impact of its findings, suggesting that the methodologies and insights gained could inform similar efforts in other low-resource languages, thereby contributing to the global field of NLP.
 The research concludes with reflections on the implications of their findings for the ongoing development of machine learning technologies, asserting the potential of these technologies to accommodate the intricacies of any language, given the right approach and resources. This work not only advances the technical capabilities for Kazakh text generation but also serves as a testament to the potential of machine learning to bridge language gaps and foster greater digital inclusivity.

Text data augmentation and pre-trained Language Model for enhancing text classification of low-resource languages.

In the domain of natural language processing (NLP), the development and success of advanced language models are predominantly anchored in the richness of available linguistic resources. Languages such as Azerbaijani, which is classified as a low-resource, often face challenges arising from limited labeled datasets, consequently hindering effective model training. The primary objective of this study was to enhance the effectiveness and generalization capabilities of news text classification models using text augmentation techniques. In this study, we solve the problem of working with low-resource languages using translations using the Facebook mBart50 model, as well as the Google Translate API and a combination of mBart50 and Google Translate thus expanding the capabilities when working with text. The experimental outcomes reveal a promising uptick in classification performance when models are trained on the augmented dataset compared with their counterparts using the original data. This investigation underscores the immense potential of combined data augmentation strategies to bolster the NLP capabilities of underrepresented languages. As a result of our research, we have published our labeled text classification dataset and pre-trained RoBERTa model for the Azerbaijani language.

Aiding ICD-10 Encoding of Clinical Health Records Using Improved Text Cosine Similarity and PLM-ICD

The International Classification of Diseases, 10th edition (ICD-10), has been widely used for the classification of patient diagnostic information. This classification is usually performed by dedicated physicians with specific coding training, and it is a laborious task. Automatic classification is a challenging task for the domain of natural language processing. Therefore, automatic methods have been proposed to aid the classification process. This paper proposes a method where Cosine text similarity is combined with a pretrained language model, PLM-ICD, in order to increase the number of probably useful suggestions of ICD-10 codes, based on the Medical Information Mart for Intensive Care (MIMIC)-IV dataset. The results show that a strategy of using multiple runs, and bucket category search, in the Cosine method, improves the results, providing more useful suggestions. Also, the use of a strategy composed by the Cosine method and PLM-ICD, which was called PLM-ICD-C, provides better results than just the PLM-ICD.

Literature mining discerns latent disease-gene relationships.

Dysregulation of a gene's function, either due to mutations or impairments in regulatory networks, often triggers pathological states in the affected tissue. Comprehensive mapping of these apparent gene-pathology relationships is an ever-daunting task, primarily due to genetic pleiotropy and lack of suitable computational approaches. With the advent of high throughput genomics platforms and community scale initiatives such as the Human Cell Landscape project, researchers have been able to create gene expression portraits of healthy tissues resolved at the level of single cells. However, a similar wealth of knowledge is currently not at our finger-tip when it comes to diseases. This is because the genetic manifestation of a disease is often quite diverse and is confounded by several clinical and demographic covariates. To circumvent this, we mined ∼18 million PubMed abstracts published till May 2019 and automatically selected ∼4.5 million of them that describe roles of particular genes in disease pathogenesis. Further, we fine-tuned the pretrained bidirectional encoder representations from transformers (BERT) for language modeling from the domain of natural language processing to learn vector representation of entities such as genes, diseases, tissues, cell-types, etc., in a way such that their relationship is preserved in a vector space. The repurposed BERT predicted disease-gene associations that are not cited in the training data, thereby highlighting the feasibility of in silico synthesis of hypotheses linking different biological entities such as genes and conditions. PathoBERT pretrained model: https://github.com/Priyadarshini-Rai/Pathomap-Model. BioSentVec-based abstract classification model: https://github.com/Priyadarshini-Rai/Pathomap-Model. Pathomap R package: https://github.com/Priyadarshini-Rai/Pathomap.

Comparative study on personalized open-domain generative dialogue systems

Developing personalized dialogue systems is a monumental challenge in the rapidly evolving domain of Natural Language Processing (NLP). These systems are engineered to facilitate more natural, intuitive, and user-friendly interactions between humans and machines. Their theoretical significance extends to real-world applications, making them an invaluable asset in today's digital age. Over recent years, this area has garnered immense attention from researchers and academics, becoming a hotbed for innovation and study. This paper aims to contribute to this burgeoning field by conducting an in-depth comparative analysis of the current leading personalized language models. Specifically, this paper will scrutinize models grounded in deep learning algorithms and those that employ reinforcement learning techniques. This research objective is to dissect each approach's unique advantages and limitations. By doing so, the paper hopes to identify actionable avenues for improvement and optimization. Furthermore, the paper will offer a forward-looking perspective, outlining potential advancements and innovations that could shape the future landscape of personalized dialogue systems in Natural Language Processing.

Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention

Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. However, the enigmatic ``black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. While past approaches, such as attention visualization, pivotal subnetwork extraction, and concept-based analyses, offer some insight, they often focus on either local or global explanations within a single dimension, occasionally falling short in providing comprehensive clarity. In response, we propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs. Our framework, termed SparseCBM, innovatively integrates sparsity to elucidate three intertwined layers of interpretation: input, subnetwork, and concept levels. In addition, the newly introduced dimension of interpretable inference-time intervention facilitates dynamic adjustments to the model during deployment. Through rigorous empirical evaluations on real-world datasets, we demonstrate that SparseCBM delivers a profound understanding of LLM behaviors, setting it apart in both interpreting and ameliorating model inaccuracies. Codes are provided in supplements.

Improving Automated Labeling for ATT&CK Tactics in Malware Threat Reports

Once novel malware is detected, threat reports are written by security companies that discover it. The reports often vary in the terminology describing the behavior of the malware making comparisons of reports of the same malware from different companies difficult. To aid in the automated discovery of novel malware, it was recently proposed that novel malware could be detected by identifying behaviors. This assumes that a core set of behaviors are present in most, if not all, malware variants. However, there is a lack of malware datasets that are labeled with behaviors. Motivated by a need to label malware with a common set of behaviors, this work examines automating the process of labeling malware with behaviors identified in malware threat reports despite the variability of terminology. To do so, we examine several techniques from the natural language processing (NLP) domain. We find that most state-of-the-art word embedding NLP methods require large amounts of data and are trained on generic corpora of text data—missing the nuances related to information security. To address this, we use simple feature selection techniques. We find that simple feature selection techniques generally outperform word embedding methods and achieve an increase of 6% in the F .5 -score over prior work when used to predict MITRE ATT&CK tactics in threat reports. Our work indicates that feature selection, which has commonly been overlooked by sophisticated methods in NLP tasks, is beneficial for information security related tasks, where more sophisticated NLP methodologies are not able to pick out relevant information security terms.

Image Captioning: A Comprehensive Review

Abstract: The Image Caption Generator is an intriguing project that bridges the domains of computer vision and natural language processing, aiming to automatically generate descriptive text for images. In the last decade, considerable progress has been made, yet key challenges in optimising Convolutional Neural Networks (CNNs) for precise image feature extraction and refining Long Short-Term Memory (LSTM) networks for coherent text generation persist. Challenges include overfitting, limited context understanding, and difficulties in modelling long-range dependencies. Solutions have emerged in the form of attention mechanisms, which help focus on relevant image regions, and advanced LSTM architectures like the use of Gated Recurrent Units (GRUs) to improve sequential modelling. These innovations have significantly enhanced the overall performance, leading to more accurate and contextually relevant image captions, ultimately advancing the synergy between computer vision and natural language processing in the realm of image captioning

A concise analysis of low-rank adaptation

Recent years the pre-trained language models have been proved to be a transformative technology within the domain of Natural Language Processing (NLP). From early word embeddings to modern transformer-based architectures, the success of models like BERT, GPT-3, and their variants has led to remarkable advancements in various NLP tasks. This paper is based on the Transformer model and explores and summarizes the application of the lightweight fine-tuning technique LoRA in pretrained language models, as well as improvements and derived technologies based on LoRA. Moreover, this paper categorizes these techniques into two main directions according to the advancements: enhancing training efficiency and improving training performance. Under these two major directions, several representative optimization and derived techniques are summarized and analyzed. Furthermore, this paper offers a perspective on the hot topics and future prospects of this research subject, and summarizes and proposes several directions that hold exploration value for the future, such as the possible avenues for further optimization and integration with other lightweight technologies.

A Novel Approach to Text Summarization Using Machine Learning

Text summarization is a key strategy in the domains of information retrieval and natural language processing (NLP). Its objective is to reduce a lengthy written document into a clearer, more succinct summary of the information it contains. When a text document is too lengthy or intricate to analyse completely, as in news stories, academic papers, or legal documents, this approach is extremely helpful. The major challenge of text summarising is to take the most important and relevant information from the original text and convey it in an understandable and concise way. In this study, extractive and abstractive summarising techniques are the two primary categories of text summary methods. The paper also presents several algorithms that have been proposed for text summarization, including TextRank, Seq2Seq, and BART. TextRank is a simple and fast algorithm that works well for short documents, Seq2Seq is a deep learning-based approach that generates high-quality summaries, and BART is a transformer-based algorithm that provides the best results on benchmark datasets. The obtained ROUGE Score after passing TextRank, BART, and Seq2SEq algorithm significant also.

PReFormer: A memory-efficient transformer for point cloud semantic segmentation

The success of transformer networks in the natural language processing and 2D vision domains has encouraged the adaptation of transformers to 3D computer vision tasks. However, most of the existing approaches employ standard backpropagation (SBP). SBP requires the storage of model activations on a forward pass for use during the backward pass, making their memory complexity linearly proportional to model depth, hence, inefficient. Furthermore, most 3D point transformers use the classic QKV matrix multiplication design which comes with a memory bottleneck. To address these issues, we propose a memory-efficient point transformer that makes use of reversible functions and linearized self-attention to minimize SBP and transformer memory complexities, respectively. Additionally, rather than the usual UNet architectural design for segmentation, we adopt a ∇-shaped design to capture multi-size/resolution feature representations towards a finely detailed segmentation. Experimental results on benchmark datasets (Toronto3D, DALES, and CSPC) from different sensor platforms (vehicle, aerial, and backpack) show that our approach uses less than half the number of model parameters compared to its SBP counterpart. It also takes more than twice the input sequence and uses less than half the memory compared to most of the traditional approaches. Additionally, our use of a ∇-shaped architectural design yielded about a 5% increment in model performance over the U-shaped approach. Overall, the proposed PReFormer attained competitive performance compared to the state-of-the-art.