Multi-label Text Classification Research Articles

Deep active learning for multi label text classification.

Given a set of labels, multi-label text classification (MLTC) aims to assign multiple relevant labels for a text. Recently, deep learning models get inspiring results in MLTC. Training a high-quality deep MLTC model typically demands large-scale labeled data. And comparing with annotations for single-label data samples, annotations for multi-label samples are typically more time-consuming and expensive. Active learning can enable a classification model to achieve optimal prediction performance using fewer labeled samples. Although active learning has been considered for deep learning models, there are few studies on active learning for deep multi-label classification models. In this work, for the deep MLTC model, we propose a deep Active Learning method based on Bayesian deep learning and Expected confidence (BEAL). It adopts Bayesian deep learning to derive the deep model's posterior predictive distribution and defines a new expected confidence-based acquisition function to select uncertain samples for deep MLTC model training. Moreover, we perform experiments with a BERT-based MLTC model, where BERT can achieve satisfactory performance by fine-tuning in various classification tasks. The results on benchmark datasets demonstrate that BEAL enables more efficient model training, allowing the deep model to achieve training convergence with fewer labeled samples.

Multi-label text classification via secondary use of large clinical real-world data sets.

Procedural coding presents a taxing challenge for clinicians. However, recent advances in natural language processing offer a promising avenue for developing applications that assist clinicians, thereby alleviating their administrative burdens. This study seeks to create an application capable of predicting procedure codes by analysing clinicians' operative notes, aiming to streamline their workflow and enhance efficiency. We downstreamed an existing and a native German medical BERT model in a secondary use scenario, utilizing already coded surgery notes to model the coding procedure as a multi-label classification task. In comparison to the transformer-based architecture, we were levering the non-contextual model fastText, a convolutional neural network, a support vector machine and logistic regression for a comparative analysis of possible coding performance. About 350,000 notes were used for model adaption. By considering the top five suggested procedure codes from medBERT.de, surgeryBERT.at, fastText, a convolutional neural network, a support vector machine and a logistic regression, the mean average precision achieved was 0.880, 0.867, 0.870, 0.851, 0.870and0.805 respectively. Support vector machines performed better for surgery reports with a sequence length greater than 512, achieving a mean average precision of 0.872 in comparison to 0.840 for fastText, 0.837 for medBERT.de and 0.820 for surgeryBERT.at. A prototypical front-end application for coding support was additionally implemented. The problem of predicting procedure codes from a given operative report can be successfully modelled as a multi-label classification task, with a promising performance. Support vector machines as a classical machine learning method outperformed the non-contextual fastText approach. FastText with less demanding hardware resources has reached a similar performance to BERT-based models and has shown to be more suitable for explaining the predictions efficiently.

Residual diverse ensemble for long-tailed multi-label text classification

Residual diverse ensemble for long-tailed multi-label text classification

All is attention for multi-label text classification

All is attention for multi-label text classification

A Label Information Aware Model for Multi-label Text Classification

Multi-label text classification (MLTC) refers to that each document is associated with more than one label at the same time, which attract much attention from researchers in both academia and industry. Existing methods have difficulties in determining label-related components from documents, which cannot effectively establish the association between textual features and label information. In fact, there are some label information, such as label semantic information and co-occurrence relations among labels, could be used to improve the performance on multi-label text classification. In this paper, we propose a label information aware model to utilize these information. Our model makes use of label semantic information to determine label-related components from textual features for obtaining the label-specific textual presentation for each sample, and then take advantages of co-occurrence relations among labels to construct interaction among label-specific textual presentation. The superiority of our model has been proved through comparing our method with several existing models on two datasets.

Breaking the gap between label correlation and instance similarity via new multi-label contrastive learning

Multi-label text classification (MLTC) is a fundamental yet challenging task in natural language processing. Existing MLTC models mostly learn text representations and label correlations, separately; while the instance-level correlation, which is crucial for the classification is ignored. To rectify this, we propose a new multi-label contrastive learning model, that captures instance-level correlations, for the MLTC task. Specifically, we first learn label representations by using Graph Convolutional Network (GCN) on label co-occurrence graphs. We next learn text representations by taking label correlations into consideration. Through an attention mechanism, instance-level correlation can be established. To better utilize label correlations, we propose a new contrastive learning model, whose learning is guided by a new learning objective, to further refine label representations. We finally implement a k-NN mechanism, that identifies k nearest neighbors of a given text for final prediction. Intensive experimental studies over benchmark multi-label datasets demonstrate the effectiveness of our approach.

Research of multi-label text classification based on label attention and correlation networks.

Multi-Label Text Classification (MLTC) is a crucial task in natural language processing. Compared to single-label text classification, MLTC is more challenging due to its vast collection of labels which include extracting local semantic information, learning label correlations, and solving label data imbalance problems. This paper proposes a model of Label Attention and Correlation Networks (LACN) to address the challenges of classifying multi-label text and enhance classification performance. The proposed model employs the label attention mechanism for a more discriminative text representation and uses the correlation network based on label distribution to enhance the classification results. Also, a weight factor based on the number of samples and a modulation function based on prediction probability are combined to alleviate the label data imbalance effectively. Extensive experiments are conducted on the widely-used conventional datasets AAPD and RCV1-v2, and extreme datasets EUR-LEX and AmazonCat-13K. The results indicate that the proposed model can be used to deal with extreme multi-label data and achieve optimal or suboptimal results versus state-of-the-art methods. For the AAPD dataset, compared with the suboptimal method, it outperforms the second-best method by 2.05% ∼ 5.07% in precision@k and by 2.10% ∼ 3.24% in NDCG@k for k = 1, 3, 5. The superior outcomes demonstrate the effectiveness of LACN and its competitiveness in dealing with MLTC tasks.

Contrastive Enhanced Learning for Multi-Label Text Classification

Multi-label text classification (MLTC) aims to assign appropriate labels to each document from a given set. Prior research has acknowledged the significance of label information, but its utilization remains insufficient. Existing approaches often focus on either label correlation or label textual semantics, without fully leveraging the information contained within labels. In this paper, we propose a multi-perspective contrastive model (MPCM) with an attention mechanism to integrate labels and documents, utilizing contrastive methods to enhance label information from both textual semantic and correlation perspectives. Additionally, we introduce techniques for contrastive global representation learning and positive label representation alignment to improve the model’s perception of accurate labels. The experimental results demonstrate that our algorithm achieves superior performance compared to existing methods when evaluated on the AAPD and RCV1-V2 datasets.

Performance Analysis of Multi-Label Text Classification Using Beta Ant Colony Based Feature Selection

Performance Analysis of Multi-Label Text Classification Using Beta Ant Colony Based Feature Selection

Feature selection technique on convolutional neural network – multilabel classification task

Automated text-based recommendation, an artificial intelligence development, finds application in document analysis like job resumes. The classification of job resumes poses challenges due to the ambiguity in categorizing multiple potential jobs in a single application file, termed multi-label classification, deep learning, particularly convolutional neural networks (CNN), offers flexibility in enhancing feature representations. Despite its robust learning capabilities, the black-box design of deep learning lacks interpretability and demands a substantial number of parameters, requiring significant computational resources. The primary challenge in multilabel learning is the ambiguity of labels not fully explained by traditional equivalence relations. To address this, the research employs feature selection techniques, specifically the Chi-square method. The goal is to reduce features in deep learning models while considering label relevance in multi-label text classification, easing computational workload while preserving model performance. Experimental tests, both with and without the Chi-square feature selection technique on the dataset, underscore its substantial impact on the classification model's ability. The conclusion emphasizes the influence of the Chi-square feature selection technique on performance and computational time. In summary, the research underscores the importance of balancing computational efficiency and model interpretability, especially in complex multi-label classification tasks like job applications.

Text Classification Based on Natural Language Processing and Machine Learning in Multi-Label Corpus

The rapid development of the Internet has led to a geometric expansion of text information resources online. Among them, corpus, as the basic data source of natural language processing based on statistical language model, its construction and application have become a hot issue in current language processing research. After consulting a large number of relevant literature and materials, it was found that many researchers have provided new ideas for multi label corpus text classification methods. However, this article adds its own understanding and takes this as the direction and basis. In the introduction, the research significance of text classification was introduced, and then academic research and analysis were carried out on the two key sentences of corpus text classification and natural language processing in multi-tag corpus text classification. This article then utilizes an algorithm model to provide a theoretical basis for the study of multi-label corpus text classification methods; At the end of this article, a simulation comparative experiment is conducted, and the experiment is summarized and discussed; In the Enterprise L corpus, the difference in recall rates before and after the use of Entrance 1 was 5.5%, the difference in recall rates before and after the use of Entrance 2 was 7.8%, the difference in recall rates before and after the use of Entrance 3 was 3.3%, and the difference in recall rates before and after the use of Entrance 4 was 4.5%. At the same time, with the continuous research of natural language processing and machine learning, the research on text classification methods of multi tag corpus is also facing new opportunities and challenges.

ADVANCED TECHNIQUES IN MULTI-LABEL TEXT CLASSIFICATION: INTEGRATION OF BETA ANT COLONY AND DEEP LEARNING APPROACHES

The rapid growth of unstructured textual data in various domains has necessitated the development of sophisticated techniques for multi-label text classification. Traditional methods often struggle with handling the complexity and interdependence of multiple labels, leading to suboptimal performance. This paper presents an advanced approach that integrates the Beta Ant Colony Optimization (BACO) algorithm with deep learning techniques for multi-label text classification. The BACO algorithm effectively explores the feature space and selects relevant features by leveraging pheromone trails, while the deep learning model captures intricate patterns and relationships within the textual data. The integration of these two methodologies aims to enhance the efficiency and accuracy of multi-label classification tasks, particularly in domains where label dependency is prominent. Empirical evaluations on benchmark datasets demonstrate that the proposed hybrid approach outperforms existing state-of-the-art techniques in terms of precision, recall, F1-score, and computational efficiency. The findings suggest that combining heuristic optimization algorithms with deep learning can significantly improve multi-label text classification performance, providing a robust solution for real-world applications.

Dual-view graph convolutional network for multi-label text classification

Dual-view graph convolutional network for multi-label text classification

CoocNet: a novel approach to multi-label text classification with improved label co-occurrence modeling

CoocNet: a novel approach to multi-label text classification with improved label co-occurrence modeling

Knowledge and separating soft verbalizer based prompt-tuning for multi-label short text classification

Knowledge and separating soft verbalizer based prompt-tuning for multi-label short text classification

Hierarchy-Aware and Label Balanced Model for Hierarchical Text Classification

Hierarchical text classification, where labels can be modeled as a hierarchical structure, is a special multi-label text classification sub-task. Current methods mainly improve model performance by modeling label dependencies. The model HGCLR with excellent performance obtains hierarchy-aware text representations through contrastive learning, but the degree of hierarchical awareness is insufficient. To address this problem, a new multi-label negative supervision method is proposed to drive text representations of samples with more different labels farther away. In addition, to tackle the problem of label imbalance in hierarchical text classification, asymmetric loss is employed to compute the classification loss, so that the model focuses on learning from difficult samples and the contributions of positive and negative labels to the loss function tend to be balanced. In summary, based on HGCLR, by adding multi-label negative supervision and replacing the classification loss with asymmetric loss, we propose a hierarchy-aware and label balanced model (HALB) for hierarchical text classification. Experimental results demonstrate that HALB outperforms several classical models for hierarchical text classification. HALB achieves the best results on both Micro-F1 and Macro-F1 and obtains average improvements of 0.51% and 1.28% respectively on four datasets compared with the strongest baseline model HGCLR.

Variational Continuous Label Distribution Learning for Multi-Label Text Classification

Variational Continuous Label Distribution Learning for Multi-Label Text Classification

Multi-Label Text Classification model integrating Label Attention and Historical Attention

Multi-Label Text Classification (MLTC) is one of the most import research of natural language processing. Although Deep Learning (DL) models have been widely applied to MLTC, there still exist some drawbacks. First, traditional DL models utilize all the words in the document to construct the embedding vector, while there are many words that affect the classification results. Next, the labels in MLTC have specific semantics, while traditional DL models ignore fine-grained matching signals between words and labels. Then, traditional DL models have difficulty in handling the data imbalance issue in MLTC datasets. In addition, during the training process, small errors in a certain epoch may be amplified with the increase of the number of iterations, resulting in classification errors. To address the above problems, a MLTC model integrating Label Attention and Historical Attention (i.e. LAHA) is proposed. First, a word filter is set up to select important words based on the cosine similarity between words and labels. Next, Document Self Attention (DSA) and Label Attention (LA) are obtained, and DSA-attended LA co-attention (LA-co) and LA-attended DSA co-attention (DAS-co) networks are constructed. Then, the fine-grained matching signals between words and labels are integrated through the adaptive plus of LA-co and DAS-co. At last, Historical Attention is integrated to LAHA, which not only avoids mis-classification caused by minor errors of a certain epoch, but also reduces overfitting to high-frequency labels. Multiple comparative experiments on four benchmark datasets demonstrate that LAHA outperforms the state-of-the-art baseline models and can effectively solve the data imbalance issue in MLTC datasets. Our code is available at https://github.com/sgysgywaityou/LAHA.

Multi-label text classification on unbalanced Twitter with monolingual model and hyperparameter optimization for hate speech and abusive language detection

The increase in hate speech and abusive language on social media leads to uncomfortable interactions among users. Many datasets available publicly that address hate speech and abusive language are not balanced, particularly those from Indonesian Twitter. To develop a more effective classification model that also considers minority classes, we needed to optimize the hyperparameters of a monolingual model, use four different data preprocessing scenarios, and improve the treatment of slang words. We assessed the model's effectiveness by its accuracy, achieving 81.38%. This result came from optimizing hyperparameters, processing data without stemming and removing stop words, and enhancing the slang word data. The optimal hyperparameters were a learning rate of 4e-5, a batch size of 16, and a dropout rate of 0.1. However, using too much dropout can decrease the model’s performance and its ability to predict less common categories, such as physical- and gender-related hate speech.

Business text classification with imbalanced data and moderately large label spaces for digital transformation

Digital transformation refers to an organization’s use of digital technology to improve its products, services, and operations, aligning them with evolving business requirements. To demonstrate this transformative process, we present a real-life case study where a company seeks to automate the classification of their textual data rather than relying on manual methods. Transitioning to automated classification involves deploying machine learning models, which rely on pre-labeled datasets for training and making predictions on new data. However, upon receiving the dataset from the company, we faced challenges due to the imbalanced distribution of labels and moderately large label spaces. To tackle text classification with such a business dataset, we evaluated four distinct methods for multi-label text classification: fine-tuned Bidirectional Encoder Representations from Transformers (BERT), Binary Relevance, Classifier Chains, and Label Powerset. The results revealed that fine-tuned BERT significantly outperformed the other methods across key metrics like Accuracy, F1-score, Precision, and Recall. Binary Relevance also displayed competence in handling the dataset effectively, while Classifier Chains and Label Powerset exhibited comparatively less impressive performance. These findings highlight the remarkable effectiveness of fine-tuned BERT model and the Binary Relevance classifier in multi-label text classification tasks, particularly when dealing with imbalanced training datasets and moderately large label spaces. This positions them as valuable assets for businesses aiming to automate data classification in the digital transformation era.