Multi-label Text Classification Research Articles

Text classification for evaluating digital technology adoption maturity based on BERT: An evidence of Industrial AI from China

Natural language processing (NLP) models, such as GPT and BERT, are predictable to subvert the research paradigm of technology foresight and innovation management, for their good objectivity, robustness, and efficiency. This paper aims to apply an NLP model based on deep learning to realize the digital technology adoption maturity evaluation. Firstly, a 3-layer evaluation system, with a hierarchy of domain-indicator-level, is proposed. Meanwhile, a dataset on the deployment of Industrial AI in China is collected and provided to our evaluation system. After data annotation by experts with reference to domains and indicators, a BERT model is introduced to execute the multi-label text classification task. The experiment results prove that our high-performance BERT model has the ability to learn from human experts, and then benefits to mitigate biases and reduce cost in evaluation. In the area of Industrial AI deployments, our research points out the digital technologies adoption maturity trends over time, trickle-down effect across regions, and the flying geese pattern between industries.

LSPCL: Label-specific supervised prototype contrastive learning for multi-label text classification

LSPCL: Label-specific supervised prototype contrastive learning for multi-label text classification

HGBL: A Fine Granular Hierarchical Multi-Label Text Classification Model

Hierarchical multi-label text classification is vital for natural language processing (NLP). However, existing research rarely makes full use of the interaction between labels and text features that are crucial to hierarchical multi-label text classification. To address this issue, a novel model named hierarchy-guided BiLSTM guided contrastive learning classification (HGBL) is proposed, which successfully enhances the interaction between labels and text features by incorporating global context and embedding the idea of contrastive learning into this model. During modeling, Graphormer is adopted to model the dependencies between labels, and the bidirectional recurrent network (BiLSTM) is used to integrate global context including label features. Afterwards, the contrastive learning module embeds hierarchical awareness into the fine-tuned bidirectional encoder representations from transformers (BERT) by training the value of the loss. Experimental results on NYT, WOS and RCV1-V2 datasets show that HGBL exhibits significant competitive advantages compared with 19 competitors in terms of several indicators and can be used effectively for hierarchical multi-label text classification problems.

LAR-SiCo: recommending law articles based on multi-label text classification

LAR-SiCo: recommending law articles based on multi-label text classification

Deep active learning for multi label text classification

Given a set of labels, multi-label text classification (MLTC) aims to assign multiple relevant labels for a text. Recently, deep learning models get inspiring results in MLTC. Training a high-quality deep MLTC model typically demands large-scale labeled data. And comparing with annotations for single-label data samples, annotations for multi-label samples are typically more time-consuming and expensive. Active learning can enable a classification model to achieve optimal prediction performance using fewer labeled samples. Although active learning has been considered for deep learning models, there are few studies on active learning for deep multi-label classification models. In this work, for the deep MLTC model, we propose a deep Active Learning method based on Bayesian deep learning and Expected confidence (BEAL). It adopts Bayesian deep learning to derive the deep model’s posterior predictive distribution and defines a new expected confidence-based acquisition function to select uncertain samples for deep MLTC model training. Moreover, we perform experiments with a BERT-based MLTC model, where BERT can achieve satisfactory performance by fine-tuning in various classification tasks. The results on benchmark datasets demonstrate that BEAL enables more efficient model training, allowing the deep model to achieve training convergence with fewer labeled samples.

MFLSCI: Multi-granularity fusion and label semantic correlation information for multi-label legal text classification

Multi-label text classification tasks face challenges such as sample diversity, complexity, and the need for effective utilization of label correlations. In this paper, we propose a model that integrates multi-granularity fusion of text sequence features and label semantic correlation information. Our model leverages graph convolutional networks to extract label semantic correlation, which enhances classification performance for samples with similar labels and addresses label omission issues. Additionally, text convolutional neural networks are employed to extract multi-granularity sense group features from text sequences, calculate their similarity with semantic correlation label distributions, and dynamically adjust the similarity between text context and label information. This approach tackles the limitations of feature extraction in short texts and label confusion. We replace the original multi-hot label encoding in model training with a label distribution that fuses text multi-granularity sense group features and label correlation information, using a more precise encoding method for soft alignment based on label probability distributions. This enhances the model’s resilience to noisy data, avoiding the issue of assigning high-confidence probabilities to incorrect categories due to hard-coded supervision. Our model’s performance improvement on noisy datasets significantly surpasses that achieved by label smoothing. Extensive experiments on three legal text datasets and two generalized multi-label datasets demonstrate the model’s excellent performance. Our approach is applicable in various real-world scenarios, such as legal judgment prediction, news categorization, and recommendation systems, where accurate multi-label classification is crucial. Ablation and experiments on noisy datasets validate the model’s effectiveness and robustness.

Multi-label text classification via secondary use of large clinical real-world data sets.

Procedural coding presents a taxing challenge for clinicians. However, recent advances in natural language processing offer a promising avenue for developing applications that assist clinicians, thereby alleviating their administrative burdens. This study seeks to create an application capable of predicting procedure codes by analysing clinicians' operative notes, aiming to streamline their workflow and enhance efficiency. We downstreamed an existing and a native German medical BERT model in a secondary use scenario, utilizing already coded surgery notes to model the coding procedure as a multi-label classification task. In comparison to the transformer-based architecture, we were levering the non-contextual model fastText, a convolutional neural network, a support vector machine and logistic regression for a comparative analysis of possible coding performance. About 350,000 notes were used for model adaption. By considering the top five suggested procedure codes from medBERT.de, surgeryBERT.at, fastText, a convolutional neural network, a support vector machine and a logistic regression, the mean average precision achieved was 0.880, 0.867, 0.870, 0.851, 0.870and0.805 respectively. Support vector machines performed better for surgery reports with a sequence length greater than 512, achieving a mean average precision of 0.872 in comparison to 0.840 for fastText, 0.837 for medBERT.de and 0.820 for surgeryBERT.at. A prototypical front-end application for coding support was additionally implemented. The problem of predicting procedure codes from a given operative report can be successfully modelled as a multi-label classification task, with a promising performance. Support vector machines as a classical machine learning method outperformed the non-contextual fastText approach. FastText with less demanding hardware resources has reached a similar performance to BERT-based models and has shown to be more suitable for explaining the predictions efficiently.

Hierarchical multi-label text classification of tourism resources using a label-aware dual graph attention network

In the era of big data, classifying online tourism resource information can facilitate the matching of user needs with tourism resources and enhance the efficiency of tourism resource integration. However, most research in this field has concentrated on a simple classification problem with a single level of single labelling. In this paper, a Hierarchical Label-Aware Tourism-Informed Dual Graph Attention Network (HLT-DGAT) is proposed for the complex multi-level and multi-label classification presented by online textual information about Chinese tourism resources. This model integrates domain knowledge into a pre-trained language model and employs attention mechanisms to transform the text representation into the label-based representation. Subsequently, the model utilizes dual Graph Attention Network (GAT), with one component capturing vertical information and the other capturing horizontal information within the label hierarchy. The model's performance is validated on two commonly used public datasets as well as on a manually curated Chinese tourism resource dataset, which consists of online textual overviews of Chinese tourism resources above 3A level. Experimental results indicate that HLT-DGAT demonstrates superiority in threshold-based and area-under-curve evaluation metrics. Specifically, the AU(PRC‾) reaches 64.5 % on the Chinese tourism resource dataset with enforced leaf nodes, which is 3 % higher than the optimal corresponding metric of the baseline model. Furthermore, ablation studies show that (1) integrating domain knowledge, (2) combining local information, (3) considering label dependencies within the same level of label hierarchy, and (4) merging dynamic reconstruction can enhance overall model performance.

BoostXML: Gradient Boosting for Extreme Multilabel Text Classification With Tail Labels.

Multilabel learning involving hundreds of thousands or even millions of labels is referred to as extreme multilabel learning (XML), in which the labels often follow a power-law distribution with the majority occurring in very few data points as tail labels. Recent years have witnessed the intensive use of deep-learning methods for high-performance XML, but they are typically optimized for the head labels with abundant training instances and less consider the performance on tail labels, which, however, like the needles in haystacks, are often the focus of attention in real-life applications. In light of this, we present BoostXML, a deep learning-based XML method for extreme multilabel text classification, enhanced greatly by gradient boosting. In BoostXML, we pay more attention to tail labels in each Boosting Step by optimizing the residual mostly from unfitted training instances with tail labels. A Corrective Step is further proposed to avoid the mismatching between the text encoder and weak learners during optimization, which reduces the risk of falling into local optima and improves model performance. A Pretraining Step is also introduced in the initial stage of BoostXML to avoid exorbitant bias to tail labels. Extensive experiments on five benchmark datasets with state-of-the-art baselines demonstrate the advantage of BoostXML in tail-label prediction.

Breaking the gap between label correlation and instance similarity via new multi-label contrastive learning

Multi-label text classification (MLTC) is a fundamental yet challenging task in natural language processing. Existing MLTC models mostly learn text representations and label correlations, separately; while the instance-level correlation, which is crucial for the classification is ignored. To rectify this, we propose a new multi-label contrastive learning model, that captures instance-level correlations, for the MLTC task. Specifically, we first learn label representations by using Graph Convolutional Network (GCN) on label co-occurrence graphs. We next learn text representations by taking label correlations into consideration. Through an attention mechanism, instance-level correlation can be established. To better utilize label correlations, we propose a new contrastive learning model, whose learning is guided by a new learning objective, to further refine label representations. We finally implement a k-NN mechanism, that identifies k nearest neighbors of a given text for final prediction. Intensive experimental studies over benchmark multi-label datasets demonstrate the effectiveness of our approach.

Residual diverse ensemble for long-tailed multi-label text classification

Residual diverse ensemble for long-tailed multi-label text classification

All is attention for multi-label text classification

All is attention for multi-label text classification

A Label Information Aware Model for Multi-label Text Classification

Multi-label text classification (MLTC) refers to that each document is associated with more than one label at the same time, which attract much attention from researchers in both academia and industry. Existing methods have difficulties in determining label-related components from documents, which cannot effectively establish the association between textual features and label information. In fact, there are some label information, such as label semantic information and co-occurrence relations among labels, could be used to improve the performance on multi-label text classification. In this paper, we propose a label information aware model to utilize these information. Our model makes use of label semantic information to determine label-related components from textual features for obtaining the label-specific textual presentation for each sample, and then take advantages of co-occurrence relations among labels to construct interaction among label-specific textual presentation. The superiority of our model has been proved through comparing our method with several existing models on two datasets.

Research of multi-label text classification based on label attention and correlation networks.

Multi-Label Text Classification (MLTC) is a crucial task in natural language processing. Compared to single-label text classification, MLTC is more challenging due to its vast collection of labels which include extracting local semantic information, learning label correlations, and solving label data imbalance problems. This paper proposes a model of Label Attention and Correlation Networks (LACN) to address the challenges of classifying multi-label text and enhance classification performance. The proposed model employs the label attention mechanism for a more discriminative text representation and uses the correlation network based on label distribution to enhance the classification results. Also, a weight factor based on the number of samples and a modulation function based on prediction probability are combined to alleviate the label data imbalance effectively. Extensive experiments are conducted on the widely-used conventional datasets AAPD and RCV1-v2, and extreme datasets EUR-LEX and AmazonCat-13K. The results indicate that the proposed model can be used to deal with extreme multi-label data and achieve optimal or suboptimal results versus state-of-the-art methods. For the AAPD dataset, compared with the suboptimal method, it outperforms the second-best method by 2.05% ∼ 5.07% in precision@k and by 2.10% ∼ 3.24% in NDCG@k for k = 1, 3, 5. The superior outcomes demonstrate the effectiveness of LACN and its competitiveness in dealing with MLTC tasks.

Contrastive Enhanced Learning for Multi-Label Text Classification

Multi-label text classification (MLTC) aims to assign appropriate labels to each document from a given set. Prior research has acknowledged the significance of label information, but its utilization remains insufficient. Existing approaches often focus on either label correlation or label textual semantics, without fully leveraging the information contained within labels. In this paper, we propose a multi-perspective contrastive model (MPCM) with an attention mechanism to integrate labels and documents, utilizing contrastive methods to enhance label information from both textual semantic and correlation perspectives. Additionally, we introduce techniques for contrastive global representation learning and positive label representation alignment to improve the model’s perception of accurate labels. The experimental results demonstrate that our algorithm achieves superior performance compared to existing methods when evaluated on the AAPD and RCV1-V2 datasets.

Performance Analysis of Multi-Label Text Classification Using Beta Ant Colony Based Feature Selection

Performance Analysis of Multi-Label Text Classification Using Beta Ant Colony Based Feature Selection

Feature selection technique on convolutional neural network – multilabel classification task

Automated text-based recommendation, an artificial intelligence development, finds application in document analysis like job resumes. The classification of job resumes poses challenges due to the ambiguity in categorizing multiple potential jobs in a single application file, termed multi-label classification, deep learning, particularly convolutional neural networks (CNN), offers flexibility in enhancing feature representations. Despite its robust learning capabilities, the black-box design of deep learning lacks interpretability and demands a substantial number of parameters, requiring significant computational resources. The primary challenge in multilabel learning is the ambiguity of labels not fully explained by traditional equivalence relations. To address this, the research employs feature selection techniques, specifically the Chi-square method. The goal is to reduce features in deep learning models while considering label relevance in multi-label text classification, easing computational workload while preserving model performance. Experimental tests, both with and without the Chi-square feature selection technique on the dataset, underscore its substantial impact on the classification model's ability. The conclusion emphasizes the influence of the Chi-square feature selection technique on performance and computational time. In summary, the research underscores the importance of balancing computational efficiency and model interpretability, especially in complex multi-label classification tasks like job applications.

Text Classification Based on Natural Language Processing and Machine Learning in Multi-Label Corpus

The rapid development of the Internet has led to a geometric expansion of text information resources online. Among them, corpus, as the basic data source of natural language processing based on statistical language model, its construction and application have become a hot issue in current language processing research. After consulting a large number of relevant literature and materials, it was found that many researchers have provided new ideas for multi label corpus text classification methods. However, this article adds its own understanding and takes this as the direction and basis. In the introduction, the research significance of text classification was introduced, and then academic research and analysis were carried out on the two key sentences of corpus text classification and natural language processing in multi-tag corpus text classification. This article then utilizes an algorithm model to provide a theoretical basis for the study of multi-label corpus text classification methods; At the end of this article, a simulation comparative experiment is conducted, and the experiment is summarized and discussed; In the Enterprise L corpus, the difference in recall rates before and after the use of Entrance 1 was 5.5%, the difference in recall rates before and after the use of Entrance 2 was 7.8%, the difference in recall rates before and after the use of Entrance 3 was 3.3%, and the difference in recall rates before and after the use of Entrance 4 was 4.5%. At the same time, with the continuous research of natural language processing and machine learning, the research on text classification methods of multi tag corpus is also facing new opportunities and challenges.

ADVANCED TECHNIQUES IN MULTI-LABEL TEXT CLASSIFICATION: INTEGRATION OF BETA ANT COLONY AND DEEP LEARNING APPROACHES

The rapid growth of unstructured textual data in various domains has necessitated the development of sophisticated techniques for multi-label text classification. Traditional methods often struggle with handling the complexity and interdependence of multiple labels, leading to suboptimal performance. This paper presents an advanced approach that integrates the Beta Ant Colony Optimization (BACO) algorithm with deep learning techniques for multi-label text classification. The BACO algorithm effectively explores the feature space and selects relevant features by leveraging pheromone trails, while the deep learning model captures intricate patterns and relationships within the textual data. The integration of these two methodologies aims to enhance the efficiency and accuracy of multi-label classification tasks, particularly in domains where label dependency is prominent. Empirical evaluations on benchmark datasets demonstrate that the proposed hybrid approach outperforms existing state-of-the-art techniques in terms of precision, recall, F1-score, and computational efficiency. The findings suggest that combining heuristic optimization algorithms with deep learning can significantly improve multi-label text classification performance, providing a robust solution for real-world applications.

Dual-view graph convolutional network for multi-label text classification

Dual-view graph convolutional network for multi-label text classification