Text Classification Tasks Research Articles

TextJuggler: Fooling text classification tasks by generating high-quality adversarial examples

Deep neural networks are vulnerable to adversarial attacks, despite performing well in a variety of tasks. In the current black-box word-level text adversarial attacks on various classification tasks, the main problems are the relatively low success rate and the need to improve the quality of the adversarial examples generated. These problems mainly involve two aspects: first, the key to effectively conducting adversarial attacks is accurately determining the key words in a sentence that significantly affect the model’s judgment. Only by precisely finding these words can the attack be effectively performed. Second, to generate high-quality adversarial examples, it is essential to mislead the classification model while minimizing changes to words in the sentence. It is essential to ensure that adversarial examples are as semantically and grammatically similar to the original samples as possible. Therefore, accurately determining key words and minimally altering them to produce high-quality adversarial examples presents a significant challenge. To address these challenges, we introduce TextJuggler, a new black-box word-level text adversarial attack method, inspired by occlusion and language modeling concepts. By using the Bert model to sample and replace words in sentences, the key words that influence classifier decisions can be efficiently determined. To ensure efficiency in the search for key words, our method reduces queries via crafted locality-sensitive hashing. For the determined key words, we adopt the robust and optimized Bert model, to generate high-quality adversarial examples through insertion or substitution operations for different text classification tasks while ensuring semantic similarity and text fluency. Extensive experiments and API experiments show that TextJuggler outperforms the baselines in attack success rate, textual similarity, and fluency.

Analysis of Sentiment Based on Opinions from the 2019 Presidential Election

Twitter is a popular social media platform where the public is free to comment and write about anything. It is common for people to post comments containing harsh words and even hate speech. The 2019 presidential election in Indonesia generated a significant amount of comments, with some users praising the candidates, others criticizing them, and some even resorting to insults. To extract meaningful information from these comments and classify the text, sentiment analysis is essential. In this research, sentiment analysis involves the process of categorizing textual documents into two classes: negative and positive sentiment. The opinion data was collected from the Twitter social network in the form of tweets related to the 2019 presidential election. The dataset used in the study consisted of 3,337 tweets, which were divided into 70% training data and 30% test data. The training data comprised tweets whose sentiment was already known, serving as a foundation for the model to learn and make predictions. The primary objective of this research is to determine whether the tweets, written in Indonesian, express positive or negative sentiments. The Naive Bayes Classifier algorithm was employed to classify the tweet data. This algorithm is well-suited for text classification tasks due to its simplicity and efficiency in handling large datasets. The classification results on the test data demonstrated that the Naive Bayes Classifier algorithm achieved an overall accuracy of 71%. Specifically, the accuracy for negativesentiment classification was 71%, while the accuracy for positive sentiment classification was 70%. These results indicate that the Naive Bayes Classifier is effective in distinguishing between positive and negative sentiments in tweets related to the presidential election

Language Identifier

This project presents a language identifier system focused on achieving high accuracy in identifying a specific set of 22 languages: Estonian, Swedish, English, Russian, Romanian, Persian, Pashto, Spanish, Hindi, Korean, Chinese, French, Portuguese, Indonesian, Urdu, Latin, Turkish, Japanese, Dutch, Tamil, Thai, and Arabic. Existing LI systems might struggle with the nuances of these languages, often prioritizing identification of more common languages. Our targeted approach allows for tailored optimization to achieve superior accuracy. We employ the Multinomial Naive Bayes (MNB) algorithm due to its effectiveness in text classification tasks and its ability to handle the high-dimensional, sparse features characteristic of LI based on character and word frequencies. The system achieves a promising accuracy of 95% using an 80/20 split for training and testing data. Keywords: Natural Language Processing (NLP), Language Identification (LI), Multinomial Naive Bayes (MNB), Text Classification, Language Barriers, Accuracy, Text Analysis.

Research on Feature Fusion Method Based on Graph Convolutional Networks

This paper proposes an enhanced BertGCN-Fusion (BGF) model aimed at addressing the limitations of Graph Convolutional Networks (GCN) in processing global text features for text categorization tasks. While traditional GCN effectively capture local structural features, they face challenges when integrating global semantic features. Issues such as the potential loss of global semantic information due to local feature fusion and limited depth of information propagation are prevalent. To overcome these challenges, the BGF model introduces improvements based on the BertGCN framework: (1) Feature fusion mechanism: Introducing a linear layer to fuse BERT outputs with traditional features facilitates the integration of fine-grained local semantic features from BERT with traditional global features. (2) Multilayer fusion approach: Employing a multilayer fusion technique enhances the integration of textual semantic features, thereby comprehensively and accurately capturing text semantic information. Experimental results demonstrate that the BGF model achieves notable performance improvements across multiple datasets. On the R8 and R52 datasets, the BGF model achieves accuracies of 98.45% and 93.77%, respectively, marking improvements of 0.28% to 0.90% compared to the BertGCN model. These findings highlight the BGF model’s efficacy in overcoming the deficiencies of traditional GCN in processing global semantic features, presenting an efficient approach for handling text data.

MizBERT: A Mizo BERT Model

This research investigates the utilization of pre-trained BERT transformers within the context of the Mizo language. BERT, an abbreviation for Bidirectional Encoder Representations from Transformers, symbolizes Google’s forefront neural network approach to Natural Language Processing (NLP), renowned for its remarkable performance across various NLP tasks. However, its efficacy in handling low-resource languages such as Mizo remains largely unexplored. In this study, we introduce MizBERT , a specialized Mizo language model. Through extensive pre-training on a corpus collected from diverse online platforms, MizBERT has been tailored to accommodate the nuances of the Mizo language. Evaluation of MizBERT’s capabilities is conducted using two primary metrics: masked language modeling and perplexity, yielding scores of 76.12% and 3.2565, respectively. Additionally, its performance in a text classification task is examined. Results indicate that MizBERT outperforms both the Multilingual BERT model and the Support Vector Machine algorithm, achieving an accuracy of 98.92%. This underscores MizBERT’s proficiency in understanding and processing the intricacies inherent in the Mizo language.

Comparing human text classification performance and explainability with large language and machine learning models using eye-tracking

To understand the alignment between reasonings of humans and artificial intelligence (AI) models, this empirical study compared the human text classification performance and explainability with a traditional machine learning (ML) model and large language model (LLM). A domain-specific noisy textual dataset of 204 injury narratives had to be classified into 6 cause-of-injury codes. The narratives varied in terms of complexity and ease of categorization based on the distinctive nature of cause-of-injury code. The user study involved 51 participants whose eye-tracking data was recorded while they performed the text classification task. While the ML model was trained on 120,000 pre-labelled injury narratives, LLM and humans did not receive any specialized training. The explainability of different approaches was compared based on the top words they used for making classification decision. These words were identified using eye-tracking for humans, explainable AI approach LIME for ML model, and prompts for LLM. The classification performance of ML model was observed to be relatively better than zero-shot LLM and non-expert humans, overall, and particularly for narratives with high complexity and difficult categorization. The top-3 predictive words used by ML and LLM for classification agreed with humans to a greater extent as compared to later predictive words.

Visual Analytics for Fine‐grained Text Classification Models and Datasets

AbstractIn natural language processing (NLP), text classification tasks are increasingly fine‐grained, as datasets are fragmented into a larger number of classes that are more difficult to differentiate from one another. As a consequence, the semantic structures of datasets have become more complex, and model decisions more difficult to explain. Existing tools, suited for coarse‐grained classification, falter under these additional challenges. In response to this gap, we worked closely with NLP domain experts in an iterative design‐and‐evaluation process to characterize and tackle the growing requirements in their workflow of developing fine‐grained text classification models. The result of this collaboration is the development of SemLa, a novel Visual Analytics system tailored for 1) dissecting complex semantic structures in a dataset when it is spatialized in model embedding space, and 2) visualizing fine‐grained nuances in the meaning of text samples to faithfully explain model reasoning. This paper details the iterative design study and the resulting innovations featured in SemLa. The final design allows contrastive analysis at different levels by unearthing lexical and conceptual patterns including biases and artifacts in data. Expert feedback on our final design and case studies confirm that SemLa is a useful tool for supporting model validation and debugging as well as data annotation.

Comparing Natural Language Processing and Quantum Natural Processing approaches in text classification tasks

Quantum physics and mechanics have demonstrated significant advances and promising results in different areas using the current near-term devices. One emerging subarea in quantum machine learning is quantum natural language processing, which combines quantum computing advantages and speedups with language processing algorithms to create and perform natural language tasks such as text classification or generation. The libraries and toolboxes used in this subarea include DisCoPy and lambeq, which are used to transform sentences into string diagrams or monoidal functors, convert these diagrams into quantum circuits or ansatz and embed it into a quantum model. In this study, we used both libraries with different text-based datasets to perform sentiment analysis via classification. To do so, we create synthetic datasets to train the different models. After we obtain satisfactory results, we test the resulting models with known datasets. Despite its promising results, quantum natural language processing is far from achieving its full potential. To achieve this potential, the quantum software and hardware must be improved to make them suitable for use with more extensive and complex datasets and other tasks.

Automatic text classification of drug-induced liver injury using document-term matrix and XGBoost.

Regulatory agencies generate a vast amount of textual data in the review process. For example, drug labeling serves as a valuable resource for regulatory agencies, such as U.S. Food and Drug Administration (FDA) and Europe Medical Agency (EMA), to communicate drug safety and effectiveness information to healthcare professionals and patients. Drug labeling also serves as a resource for pharmacovigilance and drug safety research. Automated text classification would significantly improve the analysis of drug labeling documents and conserve reviewer resources. We utilized artificial intelligence in this study to classify drug-induced liver injury (DILI)-related content from drug labeling documents based on FDA's DILIrank dataset. We employed text mining and XGBoost models and utilized the Preferred Terms of Medical queries for adverse event standards to simplify the elimination of common words and phrases while retaining medical standard terms for FDA and EMA drug label datasets. Then, we constructed a document term matrix using weights computed by Term Frequency-Inverse Document Frequency (TF-IDF) for each included word/term/token. The automatic text classification model exhibited robust performance in predicting DILI, achieving cross-validation AUC scores exceeding 0.90 for both drug labels from FDA and EMA and literature abstracts from the Critical Assessment of Massive Data Analysis (CAMDA). Moreover, the text mining and XGBoost functions demonstrated in this study can be applied to other text processing and classification tasks.

Evaluating text classification: A benchmark study

This paper presents an impartial and extensive benchmark for text classification involving five different text classification tasks, 20 datasets, 11 different model architectures, and 42,800 algorithm runs. The five text classification tasks are fake news classification, topic detection, emotion detection, polarity detection, and sarcasm detection. While in practice, especially in Natural Language Processing (NLP), research tends to focus on the most sophisticated models, we hypothesize that this is not always necessary. Therefore, our main objective is to investigate whether the largest state-of-the-art (SOTA) models are always preferred, or in what cases simple methods can compete with complex models, i.e. for which dataset specifications and classification tasks. We assess the performance of different methods with varying complexity, ranging from simple statistical and machine learning methods to pretrained transformers like robustly optimized BERT (Bidirectional Encoder Representations from Transformers) pretraining approach (RoBERTa). This comprehensive benchmark is lacking in existing literature, with research mainly comparing similar types of methods. Furthermore, with increasing awareness of the ecological impacts of extensive computational resource usage, this comparison is both critical and timely.We find that overall, bidirectional long short-term memory (LSTM) networks are ranked as the best-performing method albeit not statistically significantly better than logistic regression and RoBERTa. Overall, we cannot conclude that simple methods perform worse although this depends mainly on the classification task. Concretely, we find that for fake news classification and topic detection, simple techniques are the best-ranked models and consequently, it is not necessary to train complicated neural network architectures for these classification tasks. Moreover, we also find a negative correlation between F1 performance and complexity for the smallest datasets (with dataset size less than 10,000). Finally, the different models’ results are analyzed in depth to explain the model decisions, which is an increasing requirement in the field of text classification.

Shift-insensitive perceptual feature of quadratic sum of gradient magnitude and LoG signals for image quality assessment and image classification

Most existing full-reference (FR) Image quality assessment (IQA) models work in the premise of that the two images should be well registered. Shifting an image would lead to an inaccurate evaluation of image quality, because small spatial shifts are far less noticeable than structural distortion for human observers. To this regard, we propose to study an IQA feature that is shift-insensitive to the basic primitive structure of images, i.e., image edge. According to previous studies, the image gradient magnitude (GM) and the Laplacian of Gaussian (LoG) operator that depict the edge profiles of natural images are highly efficient structural features in IQA tasks. In this paper, we find that the Quadratic sum of the normalized GM and the LoG signals (QGL) has excellent shift-insensitive property in representing image edges after theoretically solving the selection problem of a ratio parameter to balance the GM and LoG signals. Based on the proposed QGL feature, two FR-IQA models can be built directly by measuring the similarity map with mean and standard deviation pooling strategies, named mQGL and sQGL, respectively. Experimental results show that the proposed sQGL and mQGL work robustly on four benchmark IQA databases, and QGL-based models show great shift-insensitive property to spatial translation and image rotation while judging the image quality. In addition, we explore the feasibility of combining QGL feature with deep neural networks, and verify that it can help to promote image pattern recognition in texture classification tasks.

Federated deep long-tailed learning: A survey

The federated learning privacy-preserving framework has achieved fruitful results in training deep models across clients. This survey aims to provide a systematic overview of federated deep long-tailed learning. We analyze the problems of federated deep long-tailed learning of class imbalance/missing, different long-tailed distributions, and biased training, and summarize the current approaches that fall into the following three categories: information enhancement, model component optimization, and algorithm-based calibration. Meanwhile, we also sort out the representative open-source datasets for different tasks. We conduct abundant experiments on CIFAR-10/100-LT using LeNet-5/ResNet-8/ResNet-34 and evaluate the model performance with multiple metrics. We also consider a text classification task and evaluate the performance of multiple methods using LSTM on the 20NewsGroups-LT. We discuss the challenges posed by data heterogeneity, model heterogeneity, fairness, and security, and identify future research directions for the follow-up studies.

PIRAP: A Study on Optimized Multi-Language Classification and Text Categorization Using Supervised Hybrid Machine Learning Approaches

Nowadays, all the records in various languages are accessible with their advanced structures. For simple recovery of these digitized records, these reports should be ordered into a class as indicated by their content. Text Categorization is an area of Text Mining which helps to overcome this challenge. Text Classification is a demonstration of allotting classes to records. This paper investigates Text Classification works done in foreign Languages, regional languages and a list of books’ content. Messages available in different languages force the difficulties of NLP approaches. This study shows that supervised ML algorithms such as Logistic regression, Naive Bayes classifier, [Formula: see text]-Nearest-Neighbor classifier, Decision Tree and SVMs performed better for Text Classification tasks. The automated document classification technique is useful in our day-to-day life to find out the type of language and different department books based on their text content. We have been using different foreign and regional languages here to classify such as Tamil, Telugu, Kannada, Bengali, English, Spanish, French, Russian and German. Here, we utilize one versus all SVMs for multi-characterization with 3-crease Cross Validation in all cases and see that SVMs outperform different classifiers. This implementation is done by using hybrid classifiers and it depicts analyses with delicate edge straight SVMs as well as bit-based SVMs.

Federated Learning for Exploiting Annotators’ Disagreements in Natural Language Processing

Abstract The annotation of ambiguous or subjective NLP tasks is usually addressed by various annotators. In most datasets, these annotations are aggregated into a single ground truth. However, this omits divergent opinions of annotators, hence missing individual perspectives. We propose FLEAD (Federated Learning for Exploiting Annotators’ Disagreements), a methodology built upon federated learning to independently learn from the opinions of all the annotators, thereby leveraging all their underlying information without relying on a single ground truth. We conduct an extensive experimental study and analysis in diverse text classification tasks to show the contribution of our approach with respect to mainstream approaches based on majority voting and other recent methodologies that also learn from annotator disagreements.

A typical sample-driven learning framework for automatic disease diagnosis

Disease diagnosis mainly depends on the doctor’s medical knowledge and clinical experience, which can be treated as a medical text classification task. We observe that existing data-driven methods always suffer from the distribution bias since a small amount of common diseases appear high-frequently, while most diseases are infrequent in real-world, which leads to an unbalanced data distribution in the disease diagnosis task. To address this problem, we propose a new learning framework, Typical sample-Driven Graph Neural Network (TD-GNN) for disease knowledge representation and classification. In our framework, different from previous methods, each disease (label) is concretized and learned from several corresponding well-representative samples rather than full imbalance data. In addition, the contrastive learning strategy is utilized to enhance the distinguishable features learning between different diseases. In this study, we construct a real-world dataset covering 350 common diseases to evaluate the proposed learning method. The experimental results demonstrate that the proposed TD-GNN significantly outperforms the state-of-the-art baselines, especially for the majority of diseases in which only small samples can be collected from the real world. Additionally, our method can provide a sample-based interpretation for disease prediction learning.

Exploration of Word Embeddings with Graph-Based Context Adaptation for Enhanced Word Vectors

In the aspect of information storage, text assumes a central role, necessitating streamlined and effective methods for swift retrieval. Among various text representations, the vector form stands out for its remarkable efficiency, especially when dealing with expansive datasets. Arranging words that are similar in meaning close to each other in the vectorized representation helps improve how well the system performs in different Natural Language Processing related tasks. Previous methods, primarily centered on capturing word context through neural language models, have fallen short in delivering high scores for word similarity problems. This paper investigates the connection between representing words in vector form and the improved performance and accuracy observed in Natural Language Processing tasks. It introduces a method to represent words as a graph, aiming to preserve their inherent relationships and to enhance overall capabilities in semantic representation. Experimental deployment of this technique across diverse text corpora underscores its superiority over conventional word embedding approaches. The findings contribute to the evolving landscape of semantic representation learning but also illuminates their implications for text classification tasks, especially within the context of dynamic embedding models.

Comparison between parameter-efficient techniques and full fine-tuning: A case study on multilingual news article classification.

Adapters and Low-Rank Adaptation (LoRA) are parameter-efficient fine-tuning techniques designed to make the training of language models more efficient. Previous results demonstrated that these methods can even improve performance on some classification tasks. This paper complements existing research by investigating how these techniques influence classification performance and computation costs compared to full fine-tuning. We focus specifically on multilingual text classification tasks (genre, framing, and persuasion techniques detection; with different input lengths, number of predicted classes and classification difficulty), some of which have limited training data. In addition, we conduct in-depth analyses of their efficacy across different training scenarios (training on the original multilingual data; on the translations into English; and on a subset of English-only data) and different languages. Our findings provide valuable insights into the applicability of parameter-efficient fine-tuning techniques, particularly for multilabel classification and non-parallel multilingual tasks which are aimed at analysing input texts of varying length.

InA: Inhibition Adaption on pre-trained language models

Fine-tuning pre-trained language models (LMs) may not always be the most practical approach for downstream tasks. While adaptation fine-tuning methods have shown promising results, a clearer explanation of their mechanisms and further inhibition of the transmission of information is needed. To address this, we propose an Inhibition Adaptation (InA) fine-tuning method that aims to reduce the number of added tunable weights and appropriately reweight knowledge derived from pre-trained LMs. The InA method involves (1) inserting a small trainable vector into each Transformer attention architecture and (2) setting a threshold to directly eliminate irrelevant knowledge. This approach draws inspiration from the shunting inhibition, which allows the inhibition of specific neurons to gate other functional neurons. With the inhibition mechanism, InA achieves competitive or even superior performance compared to other fine-tuning methods on BERT−large, RoBERTa−large, and DeBERTa−large for text classification and question-answering tasks.

AMPLIFY: attention-based mixup for performance improvement and label smoothing in transformer

Mixup is an effective data augmentation method that generates new augmented samples by aggregating linear combinations of different original samples. However, if there are noises or aberrant features in the original samples, mixup may propagate them to the augmented samples, leading to over-sensitivity of the model to these outliers. To solve this problem, this paper proposes a new mixup method called AMPLIFY. This method uses the attention mechanism of Transformer itself to reduce the influence of noises and aberrant values in the original samples on the prediction results, without increasing additional trainable parameters, and the computational cost is very low, thereby avoiding the problem of high resource consumption in common mixup methods such as Sentence Mixup. The experimental results show that, under a smaller computational resource cost, AMPLIFY outperforms other mixup methods in text classification tasks on seven benchmark datasets, providing new ideas and new ways to further improve the performance of pre-trained models based on the attention mechanism, such as BERT, ALBERT, RoBERTa, and GPT. Our code can be obtained at https://github.com/kiwi-lilo/AMPLIFY.

Bayesian Classification for SMS Spam Detection in Mobile Devices

Abstract: The abundance of unwanted spam messages complicates the use of Short Message Service (SMS) for efficient communication in modern times. This study investigates developing and utilizing a Naive Bayes Theorem-based Ham/Spam detection system. Because of its ease of use and effectiveness in text classification tasks, the Naive Bayes classifier is used. A collection of SMS messages labeled as” spam” or” ham” (non-spam) makes up the dataset that was used for testing and training. Preprocessing methods, including tokenization, stop-word elimination, and stemming, are employed to extract pertinent features from the text messages. The Naive Bayes classifier learns how words relate to whether they’re in a spam or non-spam message by looking at some examples from the dataset. Utilizing criteria such as accuracy, precision, and confusion matrix on a separate testing set, the classifier’s performance is evaluated. Additionally, the impact of varying parameters such as smoothing techniques and feature selection methods on the classifier’s performance is analyzed. The experimental results used to distinguishing between ham and spam messages in SMS communication