Short Text Classification Research Articles

Limitations of Feature Attribution in Long Text Classification of Standards

Managing complex AI systems requires insight into a model's decision-making processes. Understanding how these systems arrive at their conclusions is essential for ensuring reliability. In the field of explainable natural language processing, many approaches have been developed and evaluated. However, experimental analysis of explainability for text classification has been largely constrained to short text and binary classification. In this applied work, we study explainability for a real-world task where the goal is to assess the technological suitability of standards. This prototypical use case is characterized by large documents, technical language, and a multi-label setting, making it a complex modeling challenge. We provide an analysis of approx. 1000 documents with human-annotated evidence. We then present experimental results with two explanation methods evaluating plausibility and runtime of explanations. We find that the average runtime for explanation generation is at least 5 minutes and that the model explanations do not overlap with the ground truth. These findings reveal limitations of current explanation methods. In a detailed discussion, we identify possible reasons and how to address them on three different dimensions: task, model and explanation method. We conclude with risks and recommendations for the use of feature attribution methods in similar settings.

Application of Short Text Classification Model Based on GPT-3.5 in E-Commerce

In the realm of e-commerce, the crucial role of short text classification in enhancing user experience and platform operational efficiency has led to the development of effective. approaches encompassing feature-based algorithms and early deep learning models. However, challenges remain when dealing with complex semantics and vast amounts of unstructured data. To bridge this gap, This paper proposes a novel model of short text classification based on GPT-3.5, which significantly improves classification accuracy by combining conceptualization and character-level information. Our model's superior performence across all datasets was validated through experimentsal trials on five widely used datasets (TREC, AG News, Bing, and Movie Review).

Improving Text Classification in Agricultural Expert Systems with a Bidirectional Encoder Recurrent Convolutional Neural Network

With the rapid development of internet and AI technologies, Agricultural Expert Systems (AESs) have become crucial for delivering technical support and decision-making in agricultural management. However, traditional natural language processing methods often struggle with specialized terminology and context, and they lack the adaptability to handle complex text classifications. The diversity and evolving nature of agricultural texts make deep semantic understanding and integration of contextual knowledge especially challenging. To tackle these challenges, this paper introduces a Bidirectional Encoder Recurrent Convolutional Neural Network (AES-BERCNN) tailored for short-text classification in agricultural expert systems. We designed an Agricultural Text Encoder (ATE) with a six-layer transformer architecture to capture both preceding and following word information. A recursive convolutional neural network based on Gated Recurrent Units (GRUs) was also developed to merge contextual information and learn complex semantic features, which are then combined with the ATE output and refined through max-pooling to form the final feature representation. The AES-BERCNN model was tested on a self-constructed agricultural dataset, achieving an accuracy of 99.63% in text classification. Its generalization ability was further verified on the Tsinghua News dataset. Compared to other models such as TextCNN, DPCNN, BiLSTM, and BERT-based models, the AES-BERCNN shows clear advantages in agricultural text classification. This work provides precise and timely technical support for intelligent agricultural expert systems.

Heterogeneous graph contrastive learning with adaptive data augmentation for semi‐supervised short text classification

AbstractShort text classification has been widely used in many fields. Due to the scarcity of labelled data, implementing short text classification under semi‐supervised learning setting has become increasingly popular. Semi‐supervised short text classification methods based on graph neural networks can achieve state‐of‐the‐art classification performance by utilizing the expressive power of graph neural networks. However, these methods usually fail to mine the hidden patterns of a large amount of short text node data in the graph to optimize the short text node embeddings, which limits the semantic representation power of the short texts, thus leading to suboptimal classification performance. To overcome the limitation, this paper proposes a novel semi‐supervised short text classification method called the Heterogeneous Graph Contrastive Learning with Adaptive Data Augmentation (HGCLADA). In the knowledge bases guided soft prompt‐based data augmentation component, the related words of the tag words are used to optimize the soft prompts for generating diverse augmented samples. In the heterogeneous graph contrastive learning framework component, a heterogeneous graph that is constructed using short texts and keywords and an effective edge augmentation scheme based on a short text clustering algorithm are proposed. The optimized short text embeddings can be obtained to achieve the effective semi‐supervised short text classification. Extensive experiments on six benchmark datasets show that our HGCLADA method outperforms four classes of state‐of‐the‐art methods in terms of classification accuracy, especially with significant performance improvements of 8.74% on the TagMyNews dataset when each class only contains 20 labelled data.

Topic-aware cosine graph convolutional neural network for short text classification

Topic-aware cosine graph convolutional neural network for short text classification

Knowledge and separating soft verbalizer based prompt-tuning for multi-label short text classification

Knowledge and separating soft verbalizer based prompt-tuning for multi-label short text classification

A head-to-head attention with prompt text augmentation for text classification

Short-text classification is an important task in natural language processing (NLP). The classification results are unsatisfactory due to the sparsity of Chinese short texts, insufficient annotation data, single tasks, and classification imbalance problems. Therefore, we propose a method based on a self-attention mechanism to focus on the relationships between attention heads, which can improve the performance of models represented by self-attention for short-text classification tasks. In addition, we designed a text augmentation template based on prompt learning with embedded labels. This allows single-task classification to be transformed into multitask classification while allowing the model to focus on the semantic consistency of the labels with the text. We conducted experiments on the CHNSenticorp, COLD, and SST-2 datasets to achieve better results than several popular text classification methods.

ALBERT4Spam: A Novel Approach for Spam Detection on Social Networks

Engaging in social media browsing stands out as one of the most prevalent online activities. As social media increasingly integrates into our daily routines, it opens up numerous opportunities for spammers seeking to target individuals through these platforms. Given the concise and sporadic nature of messages exchanged on social networks, they fall within the realm of short text classification challenges. Effectively addressing such issues requires appropriately representing the text to enhance classifier efficiency.Accordingly, this study utilizes robust representations derived from contextualized models as a component of the feature extraction process within our deep neural network model, which is built upon the Bidirectional Long Short-Term Memory neural network (BLSTM). Introducing ALBERT4Spam, the study presents a deep learning methodology aimed at identifying spam on social networking platforms. It harnesses the proven ALBERT model to acquire contextualized word representations, thereby elevating the effectiveness of the suggested neural network framework.The random search method was used to fine-tune the ALBERT4Spam model's hyperparameters, which included the number of BLSTM layers, neuron count, layer count, activation function, weight initializer, learning rate, optimizer, and dropout, in order to obtain optimal performance. The experiments conducted on three benchmark datasets demonstrate that our innovative model surpasses widely used methods in social network spam detection. The precision results stand at 0.98, 0.96, and 0.98 for Twitter, YouTube, and SMS datasets, respectively, showcasing superior performance outcomes.

Edge-enhanced minimum-margin graph attention network for short text classification

With the rapid advancement of the internet, there has been a dramatic increase in short-text data. Due to the brevity of short texts, sparse features, and limited contextual information, short-text classification has become a challenging task in natural language processing. However, current methods primarily capture semantic information from locally-sequenced words in short text, which ignores the intricate feature relationships that pervade both the intra-text and inter-text. Therefore, this paper proposes a novel Edge-Enhanced Minimum-Margin Graph Attention Network (EMGAN) for short text classification to address this issue. Specifically, we construct a Heterogeneous Information Graph (HIG) to represent complex relationships among short text features. HIG mainly considers the relationship between document features and three attribute features, such as entities, topics, and keywords, and can represent short text features from multiple dimensions and levels. Then, to enhance the connectivity and expressiveness of the HIG for more effective propagation of feature information within it, we present a novel X-shaped structure edge-enhancement method. It enriches their relationships by reconstructing the edge structures. Furthermore, we design a Minimum Margin Graph Attention Network (MMGAN) for short text classification. Specifically, this method aims to explore the minimum margin between high-order neighbors and central nodes at the minimum cost, efficiently extracting and aggregating feature information. Extensive experimental results demonstrate that our proposed EMGAN model outperforms existing methods on five datasets, validating its effectiveness in short-text classification. Our code is submitted at https://github.com/w123yy/EMGAN.

Chinese medical short text classification model based on DPECNN

Medical short text classification is of great significance to medical information extraction and medical auxiliary diagnosis. However, medical short texts face challenges such as sparse features, semantic ambiguity, and the specialized nature of the medical field, resulting in relatively low accuracy in short text classification. Taking into consideration the characteristics of medical short texts, this paper proposes a Chinese medical short text classification model based on DPECNN. First, ERNIE is utilized to learn text knowledge and information in order to enhance the model’s semantic representation capabilities. Then, the DPECNN model is employed to extract rich feature information, and the classification results are generated through a fully connected layer. In the case of DPCNN, it only considers deep-level contextual semantic information, overlooking the correlation of adjacent semantic information between channels. To address this, ECA channel attention is introduced to account for adjacent semantic information. The use of a self-normalizing activation function helps avoid the problem of vanishing gradients. To enhance the model’s robustness and generalization ability, the FGM adversarial training algorithm is employed to perturb the data. The F1 values achieved on the THUCNews, KUAKE-QIC, and CHIP-CTC datasets are 95.00%, 79.45%, and 82.81%, respectively.

Improved Graph Contrastive Learning for Short Text Classification

Text classification occupies an important role in natural language processing and has many applications in real life. Short text classification, as one of its subtopics, has attracted increasing interest from researchers since it is more challenging due to its semantic sparsity and insufficient labeled data. Recent studies attempt to combine graph learning and contrastive learning to alleviate the above problems in short text classification. Despite their fruitful success, there are still several inherent limitations. First, the generation of augmented views may disrupt the semantic structure within the text and introduce negative effects due to noise permutation. Second, they ignore the clustering-friendly features in unlabeled data and fail to further utilize the prior information in few valuable labeled data. To this end, we propose a novel model that utilizes improved Graph contrastIve learning for short text classiFicaTion (GIFT). Specifically, we construct a heterogeneous graph containing several component graphs by mining from an internal corpus and introducing an external knowledge graph. Then, we use singular value decomposition to generate augmented views for graph contrastive learning. Moreover, we employ constrained kmeans on labeled texts to learn clustering-friendly features, which facilitate cluster-oriented contrastive learning and assist in obtaining better category boundaries. Extensive experimental results show that GIFT significantly outperforms previous state-of-the-art methods. Our code can be found in https://github.com/KEAML-JLU/GIFT.

Short text classification using semantically enriched topic model

Modelling short text is challenging due to the small number of word co-occurrence and insufficient semantic information that affects downstream Natural Language Processing (NLP) tasks, for example, text classification. Gathering information from external sources is expensive and may increase noise. For efficient short text classification without depending on external knowledge sources, we propose Expressive Short text Classification (EStC). EStC consists of a novel document context-aware semantically enriched topic model called the Short text Topic Model (StTM) that captures words, topics and documents semantics in a joint learning framework. In StTM, the probability of predicting a context word involves the topic distribution of word embeddings and the document vector as the global context, which obtains by weighted averaging of word embeddings on the fly simultaneously with the topic distribution of words without requiring an additional inference method for the document embedding. EStC represents documents in an expressive (number of topics × number of word embedding features) embedding space and uses a linear support vector machine (SVM) classifier for their classification. Experimental results demonstrate that EStC outperforms many state-of-the-art language models in short text classification using several publicly available short text data sets.

Heterogeneous Graph-Convolution-Network-Based Short-Text Classification

With the development of online interactive media platforms, a large amount of short text has appeared on the internet. Determining how to classify these short texts efficiently and accurately is of great significance. Graph neural networks can capture information dependencies in the entire short-text corpus, thereby enhancing feature expression and improving classification accuracy. However, existing works have overlooked the role of entities in these short texts. In this paper, we propose a heterogeneous graph-convolution-network-based short-text classification (SHGCN) method that integrates heterogeneous graph convolutional neural networks of text, entities, and words. Firstly, the model constructs a graph network of the text and extracts entity nodes and word nodes. Secondly, the relationship of the graph nodes in the heterogeneous graphs is determined by the mutual information between the words, the relationship between the documents and words, and the confidence between the words and entities. Then, the feature is represented through a word graph and combined with its BERT embedding, and the word feature is strengthened through BiLstm. Finally, the enhanced word features are combined with the document graph representation features to predict the document categories. To verify the performance of the model, experiments were conducted on the public datasets AGNews, R52, and MR. The classification accuracy of SHGCN reached 88.38%, 93.87%, and 82.87%, respectively, which is superior to that of some existing advanced classification methods.

A multi‐label social short text classification method based on contrastive learning and improved ml‐KNN

AbstractShort texts on social platforms often have the problems of diverse categories and semantic sparsity, making it challenging to identify the diverse intentions of users. To address this issue, this article proposes a multi‐label social short text classification method (IML‐CL) based on contrastive learning and improved ml‐KNN. First, a contrastive learning approach is employed to train a multi‐label text classification model. This approach improves semantic sparsity by leveraging the knowledge from the existing samples to enrich the feature representation of short texts. Simultaneously, an improved ml‐KNN algorithm is developed to enhance the accuracy of label prediction. This algorithm utilizes a two‐layer nearest neighbor rule and introduces a penalty function and weight optimization. Next, the model generates the feature representation for the test sample and predicts its label. Additionally, the improved ml‐KNN algorithm retrieves neighbors of the test sample and uses their label information for prediction. Finally, the two predictions are combined to obtain the final prediction, which accurately identifies the user's intention. The experimental results demonstrate that, on the dataset constructed in this article, the IML‐CL method effectively boosts the performance of the baseline model.

Short Text Classification Model based on Pre-trained Language Model with Feature Fusion

In response to the low accuracy of Chinese short text classification in the current data mining field and the defects of existing deep learning models with more model parameters and higher time complexity, this paper proposes a new text classification model - short text classification model (ACBSM) based on pre-trained language model with feature expansion. In ACBSM, to address the problem of high dimensionality of text data without accurate text representation, the Bert model is used to train word vector representation to solve the problem of multiple meanings of a word. From the parallelization acceleration level, a parallel acceleration strategy of two-channel neural network is designed to improve the efficiency of the algorithm in processing massive data. To address the sparsity of text data and the more complex semantics, an attention mechanism is introduced and a CNN model is used to enhance the extraction of keyword information; secondly, BiSRU is used to capture the contextual features of the text, and finally, experimental validation is conducted on a news dataset. The experimental results show that ACBSM improves the accuracy of text classification to 95.83% under the same environment and dataset, and its classification performance is better than other text classification methods.

Short text classification with Soft Knowledgeable Prompt-tuning

Over the past few decades, short text classification has emerged as a critical downstream task in natural language processing (NLP). One crucial classification research issue is how to advance semantic understanding considering the short length, feature sparsity, and high ambiguity in short texts. Recently, prompt-tuning has been proposed to insert a template into the input and convert text classification tasks into equivalent cloze-style tasks. However, among most of the previous approaches, either the crafted template methods are time-consuming and labor-intensive, or automatic prompt generation methods cannot achieve satisfied performance. In this paper, we introduce a novel approach called Soft Knowledgeable Prompt-tuning for short text classification. Our method considers both the template generation and classification performance to construct prompts for label prediction. We employ five different strategies to expand the label words space for modifying soft prompts, and the integration of these strategies is used as the final verbalizer. Despite being automatic, experimental results show that our method achieved more desirable performance even than the crafted template methods, outperforming the state-of-the-art by more than 14 Accuracy points on four well-known benchmarks.

Research on Short Text Information Mining and Classification Methods for Social Media

Research on Short Text Information Mining and Classification Methods for Social Media

A Multiscale Interactive Attention Short Text Classification Model Based on BERT

A Multiscale Interactive Attention Short Text Classification Model Based on BERT

Short Text Classification Based on Hybrid Semantic Expansion and Bidirectional GRU (BiGRU) based Method to Improve Hate Speech Detection

Short Text Classification Based on Hybrid Semantic Expansion and Bidirectional GRU (BiGRU) based Method to Improve Hate Speech Detection

A Hybrid Model with New Word Weighting for Fast Filtering Spam Short Texts.

Short message services (SMS), microblogging tools, instant message apps, and commercial websites produce numerous short text messages every day. These short text messages are usually guaranteed to reach mass audience with low cost. Spammers take advantage of short texts by sending bulk malicious or unwanted messages. Short texts are difficult to classify because of their shortness, sparsity, rapidness, and informal writing. The effectiveness of the hidden Markov model (HMM) for short text classification has been illustrated in our previous study. However, the HMM has limited capability to handle new words, which are mostly generated by informal writing. In this paper, a hybrid model is proposed to address the informal writing issue by weighting new words for fast short text filtering with high accuracy. The hybrid model consists of an artificial neural network (ANN) and an HMM, which are used for new word weighting and spam filtering, respectively. The weight of a new word is calculated based on the weights of its neighbor, along with the spam and ham (i.e., not spam) probabilities of short text message predicted by the ANN. Performance evaluations on benchmark datasets, including the SMS message data maintained by University of California, Irvine; the movie reviews, and the customer reviews are conducted. The hybrid model operates at a significantly higher speed than deep learning models. The experiment results show that the proposed hybrid model outperforms other prominent machine learning algorithms, achieving a good balance between filtering throughput and accuracy.