Natural Language Processing Problems Research Articles

Implications of Minimum Description Length for Adversarial Attack in Natural Language Processing.

Investigating causality to establish novel criteria for training robust natural language processing (NLP) models is an active research area. However, current methods face various challenges such as the difficulties in identifying keyword lexicons and obtaining data from multiple labeled environments. In this paper, we study the problem of robust NLP from a complementary but different angle: we treat the behavior of an attack model as a complex causal mechanism and quantify its algorithmic information using the minimum description length (MDL) framework. Specifically, we use masked language modeling (MLM) to measure the "amount of effort" needed to transform from the original text to the altered text. Based on that, we develop techniques for judging whether a specified set of tokens has been altered by the attack, even in the absence of the original text data.

Biomedical semantic text summarizer

BackgroundText summarization is a challenging problem in Natural Language Processing, which involves condensing the content of textual documents without losing their overall meaning and information content, In the domain of bio-medical research, summaries are critical for efficient data analysis and information retrieval. While several bio-medical text summarizers exist in the literature, they often miss out on an essential text aspect: text semantics.ResultsThis paper proposes a novel extractive summarizer that preserves text semantics by utilizing bio-semantic models. We evaluate our approach using ROUGE on a standard dataset and compare it with three state-of-the-art summarizers. Our results show that our approach outperforms existing summarizers.ConclusionThe usage of semantics can improve summarizer performance and lead to better summaries. Our summarizer has the potential to aid in efficient data analysis and information retrieval in the field of biomedical research.

Neurocomputer System of Semantic Analysis of the Text in the Kazakh Language

The purpose of the study is to solve an extreme mathematical problem—semantic analysis of natural language, which can be used in various fields, including marketing research, online translators, and search engines. When training the neural network, data training methods based on the latent Dirichlet allocation model and vector representation of words were used. This study presents the development of a neurocomputer system used for the purpose of semantic analysis of the text in the Kazakh language, based on machine learning and the use of the latent Dirichlet allocation model. In the course of the study, the stages of system development were considered, regarding the text recognition algorithm. The Python programming language was used as a tool using libraries that greatly simplify the process of creating neural networks, including the Keras library. An experiment was conducted with the involvement of experts to test the effectiveness of the system, the results of which confirmed the reliability of the data provided by the system. The papers of modern computer linguists dealing with the problems of natural language processing using various technologies and methods are considered.

Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges

Recent years have witnessed a substantial increase in the use of deep learning to solve various natural language processing (NLP) problems. Early deep learning models were constrained by their sequential or unidirectional nature, such that they struggled to capture the contextual relationships across text inputs. The introduction of bidirectional encoder representations from transformers (BERT) leads to a robust encoder for the transformer model that can understand the broader context and deliver state-of-the-art performance across various NLP tasks. This has inspired researchers and practitioners to apply BERT to practical problems, such as information retrieval (IR). A survey that focuses on a comprehensive analysis of prevalent approaches that apply pretrained transformer encoders like BERT to IR can thus be useful for academia and the industry. In light of this, we revisit a variety of BERT-based methods in this survey, cover a wide range of techniques of IR, and group them into six high-level categories: (i) handling long documents, (ii) integrating semantic information, (iii) balancing effectiveness and efficiency, (iv) predicting the weights of terms, (v) query expansion, and (vi) document expansion. We also provide links to resources, including datasets and toolkits, for BERT-based IR systems. Additionally, we highlight the advantages of employing encoder-based BERT models in contrast to recent large language models like ChatGPT, which are decoder-based and demand extensive computational resources. Finally, we summarize the comprehensive outcomes of the survey and suggest directions for future research in the area.

Deep learning based Arabic short answer grading in serious games

Automatic short answer grading (ASAG) has become part of natural language processing problems. Modern ASAG systems start with natural language preprocessing and end with grading. Researchers started experimenting with machine learning in the preprocessing stage and deep learning techniques in automatic grading for English. However, little research is available on automatic grading for Arabic. Datasets are important to ASAG, and limited datasets are available in Arabic. In this research, we have collected a set of questions, answers, and associated grades in Arabic. We have made this dataset publicly available. We have extended to Arabic the solutions used for English ASAG. We have tested how automatic grading works on answers in Arabic provided by schoolchildren in 6th grade in the context of serious games. We found out those schoolchildren providing answers that are 5.6 words long on average. On such answers, deep learning-based grading has achieved high accuracy even with limited training data. We have tested three different recurrent neural networks for grading. With a transformer, we have achieved an accuracy of 95.67%. ASAG for school children will help detect children with learning problems early. When detected early, teachers can solve learning problems easily. This is the main purpose of this research.

Research on Retrieval Method of Online English Grammar Questions Based on Natural Language Processing

The rapid development of information technology is driving the advancement of natural language processing. The retrieval of grammatical problems in natural language processing is one of its specific tasks, particularly in the context of online learning. Therefore, a retrieval method based on fuzzy tree matching is proposed to tackle the issue of grammatical multiple-choice questions (MCQs) in online English, and its effectiveness is validated through experiments. The experimental results indicate that for incomplete queries, the MRR value STPK of the grammatical MCQ questions STP is increased by 7.9% compared to the proposed method. Compared to the traditional POS sorting algorithm, this algorithm demonstrates a 2.1% improvement. When the recall rate is 0.1, the accuracy rate of other methods is below 0.4, while the method proposed in the study surpasses 0.4. In the case of a comprehensive query, STPK the MRR value for t is STP increases by 29.6%. The proposed method in the research generally maintains an accuracy rate between 0.2 and 1.0. However, when other methods achieve an accuracy rate of 0.2, the proposed method’s accuracy rate drops below 0.2. Overall, the proposed method effectively enhances the retrieval accuracy of online English grammar MCQs compared to existing statistical and grammatical analysis methods. This improvement holds great significance for the actual retrieval of online English grammar multiple-choice questions.

Identifying Mentions of Pain in Mental Health Records Text: A Natural Language Processing Approach.

Pain is a common reason for accessing healthcare resources and is a growing area of research, especially in its overlap with mental health. Mental health electronic health records are a good data source to study this overlap. However, much information on pain is held in the free text of these records, where mentions of pain present a unique natural language processing problem due to its ambiguous nature. This project uses data from an anonymised mental health electronic health records database. A machine learning based classification algorithm is trained to classify sentences as discussing patient pain or not. This will facilitate the extraction of relevant pain information from large databases. 1,985 documents were manually triple-annotated for creation of gold standard training data, which was used to train four classification algorithms. The best performing model achieved an F1-score of 0.98 (95% CI 0.98-0.99).

Revolutionizing Duplicate Question Detection: A Deep Learning Approach for Stack Overflow

This study provides a novel way to detect duplicate questions in the Stack Overflow community, posing a daunting problem in natural language processing. Our proposed method leverages the power of deep learning by seamlessly merging Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to capture both local nuances and long-term relationships inherent in textual input. Word embeddings, notably Google’s Word2Vec and GloVe, raise the bar for text representation to new heights. Extensive studies on the Stack Overflow dataset demonstrate the usefulness of our approach, generating excellent results. The combination of CNN and LSTM models improves performance while streamlining preprocessing, establishing our technology as a viable piece in the arsenal for duplicate question detection. Aside from Stack Overflow, our technique has promise for various question-and-answer platforms, providing a robust solution for finding similar questions and paving the path for advances in natural language processing

СПЕЦІАЛІЗОВАНЕ ПОПЕРЕДНЄ НАВЧАННЯ НЕЙРОМЕРЕЖЕВИХ МОДЕЛЕЙ НА СИНТЕТИЧНИХ ДАНИХ ДЛЯ ПОКРАЩЕННЯ ГЕНЕРАЦІЇ ПЕРЕФРАЗУВАННЯ

SPECIALIZED PRE-TRAINING OF NEURAL NETWORKS ON SYNTHETIC DATA FOR IMPROVING PARAPHRASE GENERATION Abstract. Generating paraphrases is a fundamental problem in natural language processing. In light of the significant success of transfer learning technology, the “pre-training fine-tuning” approach has become the standard. However, popular general-purpose pre-training methods typically require large datasets and computational resources, and available pre-trained models are limited by fixed architecture and size. We propose a simple and effective approach for pre-training specifically for paraphrase generation, which significantly improves model quality and matches the quality level of general-purpose models. Both existing public data and new data generated by large language models were used. The impact of this procedure on neural networks of different architectures was investigated, and it was shown to work for all of them. Keywords: artificial intelligence, machine learning, neural networks, paraphrase generation, pre-training, fine-tuning.

Can Pretrained English Language Models Benefit Non-English NLP Systems in Low-Resource Scenarios?

Pretrained language models have achieved great success in a wide range of natural language processing (NLP) problems, because they learn language representations from large-scale text corpora and can adapt to downstream tasks by finetuning them on annotated task data. However, such success relies on both large-scale text and annotated data, so the lack of training data is a major practical problem for many languages, especially low-resource languages. In this paper, we explore whether a pretrained English language model can benefit non-English NLP systems in low-resource scenarios, i.e., with limited text corpora or annotated data. To achieve this, we first propose cross-lingual knowledge transfer methods and then validate our methods in low-resource scenarios. Specifically, our cross-lingual knowledge transfer methods are applied in the training stages of language model pretraining or downstream finetuning. At the two stages, the methods are designed for the transfer of upstream general knowledge or downstream task-specific knowledge, respectively. In the experiments, we perform pretraining and finetuning with limited non-English data to simulate the low-resource scenarios. We evaluate our methods on ten downstream tasks over a wide range of languages, and present systematic comparisons among various knowledge transfer methods. Experimental results show that our methods successfully leverage a pretrained English language model to improve task performance in other languages. Besides, we demonstrate the multilinguality of the English language model in various application scenarios. Our findings imply the possibility to improve low-resource-language NLP systems with large-scale English language models.

Quantifying confidence shifts in a BERT-based question answering system evaluated on perturbed instances.

Recent work on transformer-based neural networks has led to impressive advances on multiple-choice natural language processing (NLP) problems, such as Question Answering (QA) and abductive reasoning. Despite these advances, there is limited work still on systematically evaluating such models in ambiguous situations where (for example) no correct answer exists for a given prompt among the provided set of choices. Such ambiguous situations are not infrequent in real world applications. We design and conduct an experimental study of this phenomenon using three probes that aim to 'confuse' the model by perturbing QA instances in a consistent and well-defined manner. Using a detailed set of results based on an established transformer-based multiple-choice QA system on two established benchmark datasets, we show that the model's confidence in its results is very different from that of an expected model that is 'agnostic' to all choices that are incorrect. Our results suggest that high performance on idealized QA instances should not be used to infer or extrapolate similarly high performance on more ambiguous instances. Auxiliary results suggest that the model may not be able to distinguish between these two situations with sufficient certainty. Stronger testing protocols and benchmarking may hence be necessary before such models are deployed in front-facing systems or ambiguous decision making with significant human impact.

Analysis of transformer-based models for time series data, natural language processing, and computer vision

The birth of the Transformer revolutionarily signalled the start of a new epic chapter in the deep learning era. Through an encoder-decoder architecture, including residual connection, multi-head self-attention, etc., it completely reformed the deep models and unified the models used in traditional computer vision (CV) and natural language processing (NLP) problems. In recent years, many papers published have adapted the original Transformer model to better complete tasks in time series analysis, CV, and NLP. In the area of natural language processing, Bidirectional Encoder Representations from Transformers (BERT) employs a two-way transformer structure to learn context-based language representation, whereas Generative Pre-trained Transformer (GPT) employs a one-way transformer but enhances corpus training to enhance the model effect. The Vision Transformer model is the cornerstone of computer vision. It separates the input image into various patches, projects each patch into vectorized features, and then passes the them to Transformer. Based on the idea of the Vision Transformer, Swin Transformer and Biformer further optimized the Transformer and achieved better results. Time series combines the ideas embodied in CV and NLP, and in doing so, improves the specificity and various difficulties of time series problems to lower algorithm complexity and increase prediction accuracy. This article summarizes the uses and improvements of the Transformer in NLP, CV and time series, explores the development history and ideas on algorithm optimization, and predicts the potential developments of Transformer in these three fields.

Towards Developing Word Sense Disambiguation System for Kashmiri Language

Background: A word, phrase, sentence or other communication is “ambiguous” if interpreted in multiple ways. The process of assigning the correct meaning to a word with respect to its context is known as Word Sense Disambiguation (WSD). WSD is intended to be a very imperious problem in Natural Language Processing (NLP) that requires proper attention as it impacts the performance of various NLP applications.Objectives: In this paper first attempt is made to propose a supervised machine learning Kashmiri WSD system.Material & Methods: The dataset comprising of 500K tokens for this research study has been collected from different resources. A sense annotated corpus for fifty commonly used ambiguous Kashmiri words has been created using the manual annotation method. Kashmiri WordNet is used to extract senses for the target words. Decision-tree based classifier is trained using the features extracted from annotated corpus for carrying out WSD task. We have used context widow of ±3 to extract features that are used to train the classifier.Results: The proposed system is tested on all fifty target words and evaluation is carried using accuracy, precision, recall and F-1 measures. The proposed system reported 81.831% accuracy, 0.834 precision,0.816 recall and 0.824 F1-measure.Conclusions: This was the initial step towards developing the WSD system for Kashmir and it has shown good results. In the future we expect to use other algorithms to carry out this task with greater language coverage

A scalable parallel algorithm for global sequence alignment with customizable scoring scheme

SummarySequence alignment is a critical computational problem in various domains, including genomics, proteomics, and natural language processing. The Needleman‐Wunsch (NW) algorithm is a classical dynamic programming approach for finding the optimal global alignment between two sequences. However, its quadratic time and space complexity make it impractical for aligning large‐scale sequences, which are increasingly common in modern applications. In this article, we propose a parallel variation of the NW algorithm that enables scalable global sequence alignment with customizable scoring schemes. Our approach re‐formulates the dependencies in the NW algorithm to enable parallel execution, thereby leveraging the computational power of modern parallel architectures, such as graphics processing unit (GPU). Furthermore, our algorithm supports arbitrary linear scoring schemes, which allows us to use domain‐specific knowledge to improve alignment accuracy. We establish the correctness of our algorithm and evaluate its performance using real DNA and user trajectory sequences on GPUs. Our parallel algorithm has shown impressive results in our experiments, with a peak performance of 27.99 GCUPS (giga cell updates per second) and a maximum speedup of 48.18 times compared to the traditional sequential implementation. Additionally, our algorithm demonstrates remarkable scalability, enabling the alignment of sequences of any length while ensuring balanced work distribution and optimal utilization of resources. Our primary objective is to harness the computational capabilities of a single GPU and fully utilize the processing power of multi‐core CPUs.

Mô hình BiDAF trong bài toán phân tích cảm xúc

Sentiment analysis is a critical job in natural language processing. Controlling and evaluating customer feedback on their goods is a task that companies are especially interested in. For reading comprehension problems including attention processes, the BiDAF model is developed. Attention processes have recently been expanded and effectively used for natural language processing problems. In this study, we use the BiDAF model to perform sentiment analysis on Amazon product evaluations at the sentence level. The BiDAF model is a multilayered processing model that reflects context at multiple levels and uses the BiLSTM model. Furthermore, we investigate the sentence's attention weight distribution using the attention mechanism. With a recall measure, the model achieves an accuracy of up to 99.9%. We discovered that the attention weights of important phrases are equivalent to, if not higher than, the attention weights of sentiment words in the sentence.

ChatGPT: Jack of all trades, master of none

OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. The first contact with the chatbot reveals its ability to provide detailed and precise answers in various areas. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool’s usefulness to society and how the learning and validation procedures for such systems should be established.

A New Method of Time-Series Event Prediction Based on Sequence Labeling

In the existing research on time-series event prediction (TSEP) methods, most of the work is focused on improving the algorithm for classifying subsequence sets (sets composed of multiple adjacent subsequences). However, these prediction methods ignore the timing dependence between the subsequence sets, nor do they capture the mutual transition relationship between events, the prediction effect on a small sample data set is very poor. Meanwhile, the sequence labeling problem is one of the common problems in natural language processing and image segmentation. To solve this problem, this paper proposed a new framework for time-series event prediction, which transforms the event prediction problem into a labeling problem, to better capture the timing relationship between the subsequence sets. Specifically, the framework used a sequence clustering algorithm for the first time to identify representative patterns in the time series, then represented the set of subsequences as a weighted combination of patterns, and used the eXtreme gradient boosting algorithm (XGBoost) for feature selection. After that, the selected pattern feature was used as the input of the long-term short-term memory model (LSTM) to obtain the preliminary prediction value. Furthermore, the fully-linked conditional random field (CRF) was used to smooth and refine the preliminary prediction value to obtain the final prediction result. Finally, the experimental results of event prediction on five real data sets show that the CX-LC method has a certain improvement in prediction accuracy compared with the other six models.

Improving the Performance of Sentiment Analysis by Ensemble Hybrid Learning Algorithm With NLP And Cascaded Feature Extraction

Sentiment analysis is a challenging problem in Natural Language Processing since every language has its own character within several difficulties such as ambiguity, synonymy, negative suffixes…etc. Since words with ambiguity can have different sentiment scores depending on the meaning they have in their corresponding context, we accomplished a study on Turkish language to determine whether the polarity scores of these polysemous words may differ according to their meaning. For a word with ambiguity, we first made a polarity calculation module to calculate its polarity score. This way, we calculated the polarity scores of 100 Turkish polysemous words. Then, since negation directly affects the correct meaning of the word in the sentiment analysis, a negation handler module is also implemented. After that, we prepared a sentiment polarity corpus which consists of 159,876 Turkish words including 100 Turkish polysemous words. Actually, the main purpose of this study is to detect sentiment polarity of Turkish texts by considering and building a specialized module for polysemous words. In short, we built a system for Turkish sentiment polarity detection task including these modules: 1) Pre-processing, 2) Polarity Calculation Module, 3) Negation Handling Module, 4) Feature Generation Module, and 5) Classification Module. According to our knowledge, this is the first study which includes all of these modules in one Turkish sentiment analysis task. Finally, we conducted this corpus using an ensemble hybrid regularized learning algorithm on two self-collected Twitter-datasets. Experimental results show that the suggested approach improves the classification performance on Turkish sentiment analysis task.

The quarks of attention: Structure and capacity of neural attention building blocks

Attention plays a fundamental role in both natural and artificial intelligence systems. In deep learning, attention-based neural architectures, such as transformer architectures, are widely used to tackle problems in natural language processing and beyond. Here we investigate the most fundamental building blocks of attention and their computational properties within the standard model of deep learning. We first derive a systematic taxonomy of all possible attention mechanisms within, or as extensions of the standard model into 18 classes depending on the origin of the attention signal, the target of the attention signal, and whether the interaction is additive or multiplicative. Second, using this taxonomy, we identify three key attention mechanisms: additive activation attention (multiplexing), multiplicative output attention (output gating), and multiplicative synaptic attention (synaptic gating). Output gating and synaptic gating are proper extensions of the standard model and all current attention-based architectures, including transformers, use either output gating or synaptic gating, or a combination of both. Third, we develop a theory of attention capacity and derive mathematical results about the capacity of basic attention networks comprising linear or polynomial threshold gates. For example, the output gating of a linear threshold gate of n variables by another linear threshold gate of the same n variables has capacity 2n2(1+o(1)), achieving the maximal doubling of the capacity for a doubling of the number of parameters. Perhaps surprisingly, multiplexing attention is used in the proofs of these results. Synaptic and output gating provide computationally efficient extensions of the standard model enabling sparse quadratic activation functions. They can also be viewed as primitives for collapsing several layers of processing in the standard model into shallow compact representations.

A Chinese Few-Shot Text Classification Method Utilizing Improved Prompt Learning and Unlabeled Data

Insufficiently labeled samples and low-generalization performance have become significant natural language processing problems, drawing significant concern for few-shot text classification (FSTC). Advances in prompt learning have significantly improved the performance of FSTC. However, prompt learning methods typically require the pre-trained language model and tokens of the vocabulary list for model training, while different language models have different token coding structures, making it impractical to build effective Chinese prompt learning methods from previous approaches related to English. In addition, a majority of current prompt learning methods do not make use of existing unlabeled data, thus often leading to unsatisfactory performance in real-world applications. To address the above limitations, we propose a novel Chinese FSTC method called CIPLUD that combines an improved prompt learning method and existing unlabeled data, which are used for the classification of a small amount of Chinese text data. We used the Chinese pre-trained language model to build two modules: the Multiple Masks Optimization-based Prompt Learning (MMOPL) module and the One-Class Support Vector Machine-based Unlabeled Data Leveraging (OCSVM-UDL) module. The former generates prompt prefixes with multiple masks and constructs suitable prompt templates for Chinese labels. It optimizes the random token combination problem during label prediction with joint probability and length constraints. The latter, by establishing an OCSVM model in the trained text vector space, selects reasonable pseudo-label data for each category from a large amount of unlabeled data. After selecting the pseudo-label data, we mixed them with the previous few-shot annotated data to obtain brand new training data and then repeated the steps of the two modules as an iterative semi-supervised optimization process. The experimental results on the four Chinese FSTC benchmark datasets demonstrate that our proposed solution outperformed other prompt learning methods with an average accuracy improvement of 2.3%.