Text Classification Tasks Research Articles

Modelling long medical documents and code associations for explainable automatic ICD coding

Quick and accurate International Classification of Diseases (ICD) code assignment is vital for billing, reimbursement and medical research. Owing to the labour-intensive and error-prone nature of manual coding, automatic ICD coding using deep learning methods has flourished. However, this task is still challenging because of (1) the interpretability of coding, (2) lengthy clinical documents and (3) long-tail label distribution. In the current study, we propose a novel automatic ICD coding framework to address these issues. First, a biomedical-specific pre-trained language model, Clinical-Longformer, is used as an encoder, which generates meaningful representations of long clinical documents by injecting rich medical knowledge and capturing long-distance dependence among tokens. Second, a decoding architecture that combines the multi-synonym attention mechanism, hierarchical curriculum learning and distribution-balance loss is designed to perform ICD code prediction. The decoder improves the tail-end performance by fully capturing code associations in terms of semantics, structure and co-occurrence. In addition, the label-wise attention mechanism provides the interpretability of prediction. Experimental results on benchmark MIMIC-III datasets indicate that our model achieves higher F1 scores than previous state-of-the-art baselines. Our suggested model is expected to serve as an aid in improving the efficiency of manual ICD coding and to offer insights for other long text classification tasks with multiple label associations.

Text classification by BERT-Capsules

This paper presents a model that integrates a BERT encoder with a Capsule network, eliminating the traditional fully connected layer designed for downstream classification tasks in BERT in favor of a capsule layer. This capsule layer consists of three main modules: the representation module, the probability module, and the reconstruction module. It transforms the final hidden layer output of BERT into the final activation capsule probabilities to classify the text. By applying the model to sentiment analysis and text classification tasks, and comparing the test results with various BERT variants, the performance across all metrics was found to be superior. Observing the model’s handling of multiple entities and complex relationships, sentences with high ambiguity were extracted to observe the probability distribution of all capsules and compared with RNN-Capsule. It was found that the activation capsule probabilities for BERT-Capsule were significantly higher than the rest, and more pronounced than RNN-Capsule, indicating the model’s exceptional ability to process ambiguous information.

Improvements to adversarial training for text classification tasks

Although deep learning models show powerful performance, they are still easily deceived by adversarial samples. Some methods for generating adversarial samples have the drawback of high time loss, which is problematic for adversarial training, and the existing adversarial training methods are difficult to adapt to the dynamic nature of the model, so it is still challenging to study an efficient adversarial training method. In this paper, we propose an adversarial training method, the core of which is the improved adversarial sample generation method AGFAT for adversarial training and the improved dynamic adversarial training method AGFAT-DAT. AGFAT uses a word frequency-based approach to identify significant words, filter replacement candidates, and use an efficient semantic constraint module as a means to reduce the time of adversarial sample generation; AGFAT-DAT is a dynamic adversarial training approach that uses a cyclic attack on the model after adversarial training and generates adversarial samples for adversarial training again. It is demonstrated that the proposed method can significantly reduce the generation time of adversarial samples, and the adversarial-trained model can also effectively defend against other types of word-level adversarial attacks.

Federated Freeze BERT for text classification

Pre-trained BERT models have demonstrated exceptional performance in the context of text classification tasks. Certain problem domains necessitate data distribution without data sharing. Federated Learning (FL) allows multiple clients to collectively train a global model by sharing learned models rather than raw data. However, the adoption of BERT, a large model, within a Federated Learning framework incurs substantial communication costs. To address this challenge, we propose a novel framework, FedFreezeBERT, for BERT-based text classification. FedFreezeBERT works by adding an aggregation architecture on top of BERT to obtain better sentence embedding for classification while freezing BERT parameters. Keeping the model parameters frozen, FedFreezeBERT reduces the communication costs by a large factor compared to other state-of-the-art methods. FedFreezeBERT is implemented in a distributed version where the aggregation architecture only is being transferred and aggregated by FL algorithms such as FedAvg or FedProx. FedFreezeBERT is also implemented in a centralized version where the data embeddings extracted by BERT are sent to the central server to train the aggregation architecture. The experiments show that FedFreezeBERT achieves new state-of-the-art performance on Arabic sentiment analysis on the ArSarcasm-v2 dataset with a 12.9% and 1.2% improvement over FedAvg/FedProx and the previous SOTA respectively. FedFreezeBERT also reduces the communication cost by 5×\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ imes$$\\end{document} compared to the previous SOTA.

TG468: a text graph convolutional network for predicting clinical response to immune checkpoint inhibitor therapy.

Enhancing cancer treatment efficacy remains a significant challenge in human health. Immunotherapy has witnessed considerable success in recent years as a treatment for tumors. However, due to the heterogeneity of diseases, only a fraction of patients exhibit a positive response to immune checkpoint inhibitor (ICI) therapy. Various single-gene-based biomarkers and tumor mutational burden (TMB) have been proposed for predicting clinical responses to ICI; however, their predictive ability is limited. We propose the utilization of the Text Graph Convolutional Network (GCN) method to comprehensively assess the impact of multiple genes, aiming to improve the predictive capability for ICI response. We developed TG468, a Text GCN model framing drug response prediction as a text classification task. By combining natural language processing (NLP) and graph neural network techniques, TG468 effectively handles sparse and high-dimensional exome sequencing data. As a result, TG468 can distinguish survival time for patients who received ICI therapy and outperforms single gene biomarkers, TMB and some classical machine learning models. Additionally, TG468's prediction results facilitate the identification of immune status differences among specific patient types in the Cancer Genome Atlas dataset, providing a rationale for the model's predictions. Our approach represents a pioneering use of a GCN model to analyze exome data in patients undergoing ICI therapy and offers inspiration for future research using NLP technology to analyze exome sequencing data.

Exploring the Efficacy of Natural Language Processing and Supervised Learning in the Classification of Fake News Articles

This research article investigates the effectiveness of natural language processing (NLP) and supervised learning in classifying fake news articles. With the increasing prevalence of fake news in online media, it has become critical to identify and categorize such articles accurately. In this study, we apply NLP techniques to extract features from textual data, and use a supervised learning algorithm to train a classification model. We use a dataset of fake news articles to evaluate the performance of our model in terms of accuracy, precision, recall, and F1 score. Our results demonstrate that our approach achieved high accuracy and robustness in the classification of fake news articles. Furthermore, we perform a feature importance analysis to identify the most significant features that contribute to the classification of fake news. The findings of this study have practical implications for identifying and combating fake news in online media, and also provide insights into the effectiveness of NLP and supervised learning for text classification tasks.

Towards Robust Learning with Noisy and Pseudo Labels for Text Classification

Unlike Positive Training (PT), Negative Training (NT) is an indirect learning technique that trains the model on a combination of clean and noisy data using complementary labels, which are randomly generated from the label space except for the actual label. Although clean samples have identical distributions to the test samples, they are treated with the same level of uncertainty as noisy samples because of the complementary labeling of NT. Consequently, their contribution to the overall performance is relatively lower. We propose a Learning with Noisy and Pseudo Label (LNPL) framework, which jointly trains the model using PT and NT on clean and noisy data, respectively. We aim to enable direct learning on clean samples while leveraging the robustness of NT against noise in a unified framework. To mitigate the abundance of noisy instances, we leverage a gradient reversal layer at the top of LNPL as a regularization term to mislead the recognition of the source of the instance (e.g., clean or noisy). Moreover, we introduce a self-training LNPL that performs a semi-supervised text classification task as a learning with noisy pseudo-label problem. Extensive experiments on various textual benchmark datasets demonstrate that LNPL is robust and consistently outperforms the alternatives. The code is available on GitHub.1

A novel redistribution-based feature selection for text classification

Text classification is the process of automatically categorizing text documents into predefined labels, gaining increasing importance with the growing volume of data. In the vector space model, terms within documents must be quantified to be used with classifiers. The abundance of unique terms associated with documents can lead to unwieldy vectors. To address this issue, feature selection, reducing the number of terms by removing irrelevant terms, is a common solution. In this paper, a novel feature selection method called Amount of ReDistribution to Establish Neutrality, ARDEN, is proposed for the task of text classification. ARDEN, is designed with statistical distance perspective by measuring the distance of a term to its neutral counterpart, which represents the least distinguishing term having uniform document frequencies across all classes. ARDEN introduces a method to measure the related distance, providing insights into the degree of deviation of the term from neutrality, thus capturing its distinguishing power. ARDEN is experimentally tested against state-of-the-art feature selection methods. The results suggest that it is a competitive feature selection method, exhibiting superior performance in terms of summary statistics, μ and δ′. Furthermore, ARDEN is designed for multi-class text classification problems, which does not require a globalization function. Additionally, the proposed method is compared against Wasserstein statistical distance metric, and clearly outperforms it. Finally, we also propose δ′ evaluation criterion in order to facilitate the interpretation of experimental outcomes across multiple feature sizes.

Short text classification with Soft Knowledgeable Prompt-tuning

Over the past few decades, short text classification has emerged as a critical downstream task in natural language processing (NLP). One crucial classification research issue is how to advance semantic understanding considering the short length, feature sparsity, and high ambiguity in short texts. Recently, prompt-tuning has been proposed to insert a template into the input and convert text classification tasks into equivalent cloze-style tasks. However, among most of the previous approaches, either the crafted template methods are time-consuming and labor-intensive, or automatic prompt generation methods cannot achieve satisfied performance. In this paper, we introduce a novel approach called Soft Knowledgeable Prompt-tuning for short text classification. Our method considers both the template generation and classification performance to construct prompts for label prediction. We employ five different strategies to expand the label words space for modifying soft prompts, and the integration of these strategies is used as the final verbalizer. Despite being automatic, experimental results show that our method achieved more desirable performance even than the crafted template methods, outperforming the state-of-the-art by more than 14 Accuracy points on four well-known benchmarks.

Innovative deep learning techniques for monitoring aggressive behavior in social media posts

The study aims to evaluate and compare the performance of various machine learning (ML) classifiers in the context of detecting cyber-trolling behaviors. With the rising prevalence of online harassment, developing effective automated tools for aggression detection in digital communications has become imperative. This research assesses the efficacy of Random Forest, Light Gradient Boosting Machine (LightGBM), Logistic Regression, Support Vector Machine (SVM), and Naive Bayes classifiers in identifying cyber troll posts within a publicly available dataset. Each ML classifier was trained and tested on a dataset curated for the detection of cyber trolls. The performance of the classifiers was gauged using confusion matrices, which provide detailed counts of true positives, true negatives, false positives, and false negatives. These metrics were then utilized to calculate the accuracy, precision, recall, and F1 scores to better understand each model’s predictive capabilities. The Random Forest classifier outperformed other models, exhibiting the highest accuracy and balanced precision-recall trade-off, as indicated by the highest true positive and true negative rates, alongside the lowest false positive and false negative rates. LightGBM, while effective, showed a tendency towards higher false predictions. Logistic Regression, SVM, and Naive Bayes displayed identical confusion matrix results, an anomaly suggesting potential data handling or model application issues that warrant further investigation. The findings underscore the effectiveness of ensemble methods, with Random Forest leading in the cyber troll detection task. The study highlights the importance of selecting appropriate ML algorithms for text classification tasks in social media contexts and emphasizes the need for further scrutiny into the anomaly observed among the Logistic Regression, SVM, and Naive Bayes results. Future work will focus on exploring the reasons behind this occurrence and the potential of deep learning techniques in enhancing detection performance.

Multi-Label Text Classification Model Based on Multi-Level Constraint Augmentation and Label Association Attention

In the multi-label text classification task, a text usually corresponds to multiple label categories, and the labels have correlation and hierarchical structure. However, when the label hierarchy is unknown, the number of various labels is not balanced, which makes it difficult for the model to classify low-frequency labels. In addition, labels have semantic similarities that make it difficult for the model to distinguish between them. In this article, we propose a multi-label text classification model based on multi-level constraint augmentation and label association attention. Compared with traditional methods, our method has two contributions: (1) In order to alleviate the problem of unbalanced number of different label categories and ensure the rationality of sample generation, we propose a data augmentation method based on multi-level constraints. In the process of sample generation, this method uses historical generation information, sample original text information, and sample topic to constrain the generated text. (2) In order to make the model recognize the associated labels accurately, we propose an interaction mechanism based on label association attention and filter gate. This method combines text information and label weight information. At the same time, our classification model considers the important weights of text sentences and effectively utilizes the co-occurrence relationship between labels. Experimental results on three benchmark datasets show that our model outperforms state-of-the-art methods on all main evaluation metrics, especially on low-frequency label prediction with sparse samples.

BVMHA: Text classification model with variable multihead hybrid attention based on BERT

Text classification is an important tasks in natural language processing. Multilayer attention networks have achieved excellent performance in text classification tasks, but they also face challenges such as high temporal and spatial complexity levels and low-rank bottleneck problems. This paper incorporates spatial attention into a neural network architecture that utilizes fewer encoder layers. The proposed model aims to enhance the spatial information of semantic features while addressing the high temporal and spatial demands of traditional multilayer attention networks. This approach utilizes spatial attention to selectively weigh the relevance of the spatial locations in the input feature maps, thereby enabling the model to focus on the most informative regions while ignoring the less important regions. By incorporating spatial attention into a shallower encoder network, the proposed model achieves improved performance on spatially oriented tasks while reducing the computational overhead associated with deeper attention-based models. To alleviate the low-rank bottleneck problem of multihead attention, this paper proposes a variable multihead attention mechanism, which changes the number of attention heads in a layer-by-layer manner with the encoder, achieving a balance between expression power and computational efficiency. We use two Chinese text classification datasets and an English sentiment classification dataset to verify the effectiveness of the proposed model.

An empirical assessment of different word embedding and deep learning models for bug assignment

Bug assignment, or bug triage, focuses on identifying the appropriate developers to repair newly discovered bugs, thereby managing them more effectively. Several deep learning-based approaches have been proposed for automated bug assignment. These approaches view automated bug assignment as a text classification task — the textual description of a bug report is utilized as the input and the potential fixers are regarded as the output labels. Such approaches typically depend on the classification performance of natural language processing and machine learning techniques. Various word embedding and deep learning models have emerged continuously. The effectiveness of those approaches depends on the chosen deep learning model, used for classification, and the word embedding model, used for representing bug reports. However, prior research does not empirically evaluate the impacts of various word embedding and deep learning models for automated bug assignment. In this paper, we conduct an empirical study to analyze the performance variations among 35 deep learning-based automated bug assignment approaches. These approaches are based on five word embedding techniques, i.e., Word2Vec, GloVe, NextBug, ELMo, and BERT, and seven text classification models, i.e., TextCNN, LSTM, Bi-LSTM, LSTM with attention, Bi-LSTM with attention, MLP, and Naive Bayes. We evaluated these combinations across three benchmark datasets, namely Eclipse JDT, GCC, and Firefox, and their mergence i.e., a cross-project dataset. Our main observations are: (1) Bi-LSTM with attention and Bi-LSTM using ELMo are significantly superior to other deep learning models on bug assignment tasks in terms of top-k (k = 1, 5, 10) accuracy and MRR; (2) Both the summary and description of bug reports are useful for bug assignment, but the description is more useful than the summary; (3) The training corpus for word embedding models has a significant impact on the performance of deep learning-based bug assignment methods. Our results show the importance of tuning different components (e.g. word embedding model, classification model, and textual input) in deep learning-based automated bug assignment methods and provide important insights for practitioners and researchers.

Contrastive classification: A label-independent generalization model for text classification

Although existing text classification methods have achieved SOTA results on most tasks, their generalization ability is inferior to their excellent performance on a single task. For this issue, existing research mainly improves the generalization of text classification models by adding label semantics to model training and allowing models to see label semantic information. However, its generalization ability is still limited by the description and quantity of labels. This paper presents a Contrastive Classification method based on contrastive learning to enhance the generalization of text classification models. By transforming all text classification tasks into semantic understanding tasks, the method strengthens the correlation between pre-training tasks and downstream tasks, better preserving the generalization of pre-trained models. At the same time, it unifies the training objectives of all text classification tasks, reducing the differences between different tasks. We verified the model’s generalization ability on 17 different zero-shot tasks, verified the model’s cross-domain generalization ability and growth potential on 3 question–answer and 2 reading comprehension tasks, and verified the model’s performance on 12 supervised text categorization tasks. Experimental results show that Contrastive Classification frees text classification models from the limitations of classification labels and improves them in different dimensions: (1) Robustness, (2) Cross-domain generalization ability, and (3) Growth potential. We plan to release our code at https://github.com/liangyi-qianwan/Contrastive-classification-A-label-independent-generalization-model-for-text-classification/tree/main.

Improving long COVID-related text classification: a novel end-to-end domain-adaptive paraphrasing framework.

The emergence of long COVID during the ongoing COVID-19 pandemic has presented considerable challenges for healthcare professionals and researchers. The task of identifying relevant literature is particularly daunting due to the rapidly evolving scientific landscape, inconsistent definitions, and a lack of standardized nomenclature. This paper proposes a novel solution to this challenge by employing machine learning techniques to classify long COVID literature. However, the scarcity of annotated data for machine learning poses a significant obstacle. To overcome this, we introduce a strategy called medical paraphrasing, which diversifies the training data while maintaining the original content. Additionally, we propose a Data-Reweighting-Based Multi-Level Optimization Framework for Domain Adaptive Paraphrasing, supported by a Meta-Weight-Network (MWN). This innovative approach incorporates feedback from the downstream text classification model to influence the training of the paraphrasing model. During the training process, the framework assigns higher weights to the training examples that contribute more effectively to the downstream task of long COVID text classification. Our findings demonstrate that this method substantially improves the accuracy and efficiency of long COVID literature classification, offering a valuable tool for physicians and researchers navigating this complex and ever-evolving field.

Embedding-based alignment: combining protein language models with dynamic programming alignment to detect structural similarities in the twilight-zone.

Language models are routinely used for text classification and generative tasks. Recently, the same architectures were applied to protein sequences, unlocking powerful new approaches in the bioinformatics field. Protein language models (pLMs) generate high-dimensional embeddings on a per-residue level and encode a "semantic meaning" of each individual amino acid in the context of the full protein sequence. These representations have been used as a starting point for downstream learning tasks and, more recently, for identifying distant homologous relationships between proteins. In this work, we introduce a new method that generates embedding-based protein sequence alignments (EBA) and show how these capture structural similarities even in the twilight zone, outperforming both classical methods as well as other approaches based on pLMs. The method shows excellent accuracy despite the absence of training and parameter optimization. We demonstrate that the combination of pLMs with alignment methods is a valuable approach for the detection of relationships between proteins in the twilight-zone. The code to run EBA and reproduce the analysis described in this article is available at: https://git.scicore.unibas.ch/schwede/EBA and https://git.scicore.unibas.ch/schwede/eba_benchmark.

A security and privacy preserving approach based on social IoT and classification using DenseNet convolutional neural network

This method is able to synthesize fine-detailed images by the use of a global attention that gives more attention to the words in the textual descriptions. Also we have the deep attention multimodal similarity model (DAMSM) that calculates the matching loss in the generator. Though this work produced images of high quality, there was some loss while training the system and it takes enough time for training. Although there has been little study on applying character-level Dense Net algorithms for text classification tasks; the Dense Net structures we suggested in this paper have shown outstanding performance in image classification tasks. Extensive testing has revealed that they perform better when it comes to their ability to withstand interruption and that they can influence exerted many organizations implementing information usage and language information on the specifications of user privacy protection, framework implies, and regulatory requirements.

The hybrid of BERT and deep learning models for Indonesian sentiment analysis

<span>Artificial intelligence (AI) is one example of how data science innovation has advanced quickly in recent years and has greatly improved human existence. Neural networks, which are a type of machine learning model, are a fundamental component of deep learning in the field of AI. Deep learning models can carry out feature extraction and classification tasks in a single design because of their numerous neural network layers. Modern machine learning algorithms have been shown to perform worse than this model on tasks including text classification, audio recognition, imaginary, and pattern recognition. Deep learning models have outperformed AI-based methods in sentiment analysis and other text categorization tasks. Text data can originate from a number of places, including social media. Sentiment analysis is the computational examination of textual expressions of ideas and feelings. This study employs the convolutional neural network (CNN), long-short term memory (LSTM), CNN-LSTM, and LSTM-CNN models in a deep learning framework using bidirectional encoder representations from transformers (BERT) data representation to assess the performance of machine learning. The implementation of the model utilises YouTube discussion data pertaining to political films associated with the Indonesian presidential election of 2024. Confusion metrics, including as accuracy, precision, and recall, are then used to analyse the model’s performance.</span>

Cyberbullying Detection in Social Networks Using Deep Learning

Cyberbullying causes significant harm, especially among adolescents and young adults. With the growth of social media, online harassment through platforms like Facebook and Twitter has also proliferated rapidly. Though social networks have reporting mechanisms, the volume of user-generated content makes manual moderation infeasible. This necessitates automated detection systems that can accurately identify cyberbullying at scale. Recent advances in deep learning provide promising techniques for text classification tasks. This paper explores Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), and transformer models like Bidirectional Encoder Representations from Transformers (BERT) for cyberbullying detection in social networks. The models are evaluated on a benchmark dataset containing 11,000 Facebook comments labeled as clean or cyberbullying. Extensive experiments demonstrate that BERT achieves the highest accuracy of 87.3% followed by a hierarchical CNN-LSTM architecture with 86.5% accuracy. The former benefits from bidirectional context modeling using self-attention while the latter combines the strengths of convolutional layers and LSTMs. The results verify the effectiveness of deep learning methodologies for this problem. However, enhancements in multilingual, multimodal support and adversarial robustness are required. Testing on diverse platforms and content along with user privacy considerations remain as future research directions. This empirical study provides useful insights to build robust cyberbullying detection systems.

Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases.

This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results. We analyzed appraisals for RDTs (n=94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group. The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity=74.4 percent, specificity=92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio=1.44, 95 percent CI 1.09, 1.90, p=0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model. Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.