Hierarchical Text Classification Research Articles

Data Sorting Influence on Short Text Manual Labeling Quality for Hierarchical Classification

The precise categorization of brief texts holds significant importance in various applications within the ever-changing realm of artificial intelligence (AI) and natural language processing (NLP). Short texts are everywhere in the digital world, from social media updates to customer reviews and feedback. Nevertheless, short texts’ limited length and context pose unique challenges for accurate classification. This research article delves into the influence of data sorting methods on the quality of manual labeling in hierarchical classification, with a particular focus on short texts. The study is set against the backdrop of the increasing reliance on manual labeling in AI and NLP, highlighting its significance in the accuracy of hierarchical text classification. Methodologically, the study integrates AI, notably zero-shot learning, with human annotation processes to examine the efficacy of various data-sorting strategies. The results demonstrate how different sorting approaches impact the accuracy and consistency of manual labeling, a critical aspect of creating high-quality datasets for NLP applications. The study’s findings reveal a significant time efficiency improvement in terms of labeling, where ordered manual labeling required 760 min per 1000 samples, compared to 800 min for traditional manual labeling, illustrating the practical benefits of optimized data sorting strategies. Comparatively, ordered manual labeling achieved the highest mean accuracy rates across all hierarchical levels, with figures reaching up to 99% for segments, 95% for families, 92% for classes, and 90% for bricks, underscoring the efficiency of structured data sorting. It offers valuable insights and practical guidelines for improving labeling quality in hierarchical classification tasks, thereby advancing the precision of text analysis in AI-driven research. This abstract encapsulates the article’s background, methods, results, and conclusions, providing a comprehensive yet succinct study overview.

Hierarchical Text Classification and Its Foundations: A Review of Current Research

While collections of documents are often annotated with hierarchically structured concepts, the benefits of these structures are rarely taken into account by classification techniques. Within this context, hierarchical text classification methods are devised to take advantage of the labels’ organization to boost classification performance. In this work, we aim to deliver an updated overview of the current research in this domain. We begin by defining the task and framing it within the broader text classification area, examining important shared concepts such as text representation. Then, we dive into details regarding the specific task, providing a high-level description of its traditional approaches. We then summarize recently proposed methods, highlighting their main contributions. We also provide statistics for the most commonly used datasets and describe the benefits of using evaluation metrics tailored to hierarchical settings. Finally, a selection of recent proposals is benchmarked against non-hierarchical baselines on five public domain-specific datasets. These datasets, along with our code, are made available for future research.

Adaptive micro- and macro-knowledge incorporation for hierarchical text classification

Hierarchical text classification (HTC) aims to classify a text into multiple categories organized in a hierarchical structure. The state-of-the-art HTC methods usually employ graph networks, where label graphs are constructed and label representation is learned to interact with text representations for classification. In general, label graphs are built on the intrinsic label hierarchy, label semantic similarity, or label co-occurrence. Such graphs have been proven to be effective, but they only exploit knowledge from training data or simple label descriptions, without considering the vast external knowledge in the open sources. Actually, external knowledge from open sources could bring in complementary information to enhance the label graph’s representation power. Motivated by the above considerations, we explore the use of external knowledge for improving HTC in this paper. We categorize knowledge into micro-knowledge and macro-knowledge, which are defined as the fundamental concepts related to a single class label and the correlations among class labels, respectively. For tailor-made incorporation of the two types of knowledge into representation learning and classification, we propose Adaptive Micro- and Macro-Knowledge Incorporation for Hierarchical Text Classification (AMKI-HTC) model in this paper. The micro-knowledge incorporation helps capture class-relevant keywords in the text and hence produce discriminative representations, while the macro-knowledge incorporation improves the accuracy of label graphs. Finally, a confidence maximization fusion strategy is developed for adaptive aggregation of multi-view features. Extensive experiments on three benchmark HTC datasets demonstrate that AMKI-HTC consistently outperforms state-of-the-art models.

Hierarchical text classification with multi-label contrastive learning and KNN

Given the complicated label hierarchy, hierarchical text classification (HTC) has emerged as a challenging subtask in the realm of multi-label text classification. Existing methods enhance the quality of text representations by contrastive learning, but this supervised contrastive learning is designed for single-label setting and has two main limitations. On one hand, sample pairs with completely identical labels which should be treated as positive pairs are ignored. On the other hand, a simple pair is deemed as an absolutely positive or negative pair, which lacks consideration about the situation where sample pairs share some labels while having labels unique to each sample. Therefore, we propose a method combining multi-label contrastive learning with KNN (MLCL-KNN) for HTC. The proposed multi-label contrastive learning method can make text representations of sample pairs having more shared labels closer and separate those with no labels in common. During inference, we employ KNN to retrieve several neighbor samples and regard their labels as additional prediction, which is interpolated into the model output to further improve the performance of MLCL-KNN. Compared with the strongest baseline, MLCL-KNN achieves average improvements of 0.31%, 0.76%, 0.83%, and 0.43% on Micro-F1, Macro-F1, accuracy, and HiF respectively, which demonstrates its effectiveness.

A new sentence embedding framework for the education and professional training domain with application to hierarchical multi-label text classification

In recent years, Natural Language Processing (NLP) has made significant advances through advanced general language embeddings, allowing breakthroughs in NLP tasks such as semantic similarity and text classification. However, complexity increases with hierarchical multi-label classification (HMC), where a single entity can belong to several hierarchically organized classes. In such complex situations, applied on specific-domain texts, such as the Education and professional training domain, general language embedding models often inadequately represent the unique terminologies and contextual nuances of a specialized domain. To tackle this problem, we present HMCCCProbT, a novel hierarchical multi-label text classification approach. This innovative framework chains multiple classifiers, where each individual classifier is built using a novel sentence-embedding method BERTEPro based on existing Transformer models, whose pre-training has been extended on education and professional training texts, before being fine-tuned on several NLP tasks. Each individual classifier is responsible for the predictions of a given hierarchical level and propagates local probability predictions augmented with the input feature vectors to the classifier in charge of the subsequent level. HMCCCProbT tackles issues of model scalability and semantic interpretation, offering a powerful solution to the challenges of domain-specific hierarchical multi-label classification. Experiments over three domain-specific textual HMC datasets indicate the effectiveness of HMCCCProbT to compare favorably to state-of-the-art HMC algorithms in terms of classification accuracy and also the ability of BERTEPro to obtain better probability predictions, well suited to HMCCCProbT, than three other vector representation techniques.

An improved multi-stage framework for large-scale hierarchical text classification problems using a modified feature hashing and bi-filtering strategy

The classification of large-scale textual dataset is associated with a huge number of instances and millions of features which must be discriminated between large numbers of categories. The task requires the utilization of a defined hierarchy structure and tools that automatically classify instances within the hierarchy known as Large Scale Hierarchical Text Classification (LSHTC). Predicting the labels of instances by the employed classifiers is challenging due to the high number of features. Furthermore, the existing Dimensional Reduction (DR) approaches in cooperation with the LSHTC framework are still quite inefficient. In such a problem, an effective Hierarchical Dimensional Reduction approach can be advantageous in improving the performance of the LSHTC. Therefore, in this paper, we enhance the performance of LSHTC by proposing a Multi-stage Hierarchical Dimensional Reduction (MHDR) approach based on Modified Feature Hashing (MFH) and Hierarchical Bi-Filtering (HBF) method. In addition to alleviating bad collision and result discrepancy, experimental results show that the proposed approach has achieve the best performance in terms of micro-f1 and macro-f1 by recording average scores of 58.47% and 54.77% using TD-SVM, and average scores of 51.14% and 48.70% using TD-LR, respectively. The method also achieved 11% speed-up than the approaches compared.

Does the Order Matter? A Random Generative Way to Learn Label Hierarchy for Hierarchical Text Classification

Does the Order Matter? A Random Generative Way to Learn Label Hierarchy for Hierarchical Text Classification

Improve label embedding quality through global sensitive GAT for hierarchical text classification

Hierarchical text classification aims to assign text to multiple labels in a label set stored in a tree structure. The current algorithms mainly introduce the priori information of the label hierarchy, but the implicit correlation between labels in the hierarchy is rarely applied. At the same time, we also found that the inherent class imbalance of chain labels will also lead to poor classification effects of lower-level labels through a large number of studies. Therefore, a label structure enhanced hierarchy aware global model (LSE-HiAGM) is proposed. Firstly, the common density coefficient of labels is defined to measure the importance of a pair of labels in the hierarchical structure. Secondly, the common density coefficient is used as the weight of the label to update the topological structure features, so that the label can be linked with all labels globally. Finally, the topological structure feature, text features, and label hierarchical features are fused to make full use of all features to improve the embedding quality of low-level labels. In addition, to alleviate the class imbalance problem, a new loss function is used to constrain the model training. The probability of the label being sampled relative to all the labels of the sample is taken as the weight of the loss function. Therefore, a small penalty is imposed on the upper label and a large penalty on the lower label. A large number of experiments on datasets such as RCV1, WOS and NYT show that LSE-HiAGM performs better than the baseline models in hierarchical text classification.

A Comparative Study of Traditional, Ensemble and Neural Network-Based Natural Language Processing Algorithms

Accurate data analysis is an important part of data-driven financial audits. Given the increased data availability and various systems from which audit files are generated, RCSFI provides a way for standardization on behalf of analysis. This research attempted to automate this hierarchical text classification task in order to save financial auditors time and avoid errors. Several studies have shown that ensemble-based models and neural-network-based natural language processing (NLP) techniques achieved encouraging results for classification problems in various domains. However, there has been limited empirical research comparing the performance of both of the aforementioned techniques in a hierarchical multi-class classification setting. Moreover, neural-network- based NLP techniques have commonly been applied to English datasets and not to Dutch financial datasets. Additionally, this research took the implementation of hierarchical approaches into account for the traditional and ensemble-based models and found that the performance did not increase when implementing the included hierarchical approaches. DistilBERT achieved the highest scores on level 1-2-3-4 and outperformed the traditional and ensemble-based models. The model obtained a F1 of 94.50% for level 1-2-3-4. DistilBERT also outperformed BERTje at level 1-2-3-4 despite BERTje being specifically pre-trained on Dutch datasets.

Hierarchical Text Classification as Sub-hierarchy Sequence Generation

Hierarchical text classification (HTC) is essential for various real applications. However, HTC models are challenging to develop because they often require processing a large volume of documents and labels with hierarchical taxonomy. Recent HTC models based on deep learning have attempted to incorporate hierarchy information into a model structure. Consequently, these models are challenging to implement when the model parameters increase for a large-scale hierarchy because the model structure depends on the hierarchy size. To solve this problem, we formulate HTC as a sub-hierarchy sequence generation to incorporate hierarchy information into a target label sequence instead of the model structure. Subsequently, we propose the Hierarchy DECoder (HiDEC), which decodes a text sequence into a sub-hierarchy sequence using recursive hierarchy decoding, classifying all parents at the same level into children at once. In addition, HiDEC is trained to use hierarchical path information from a root to each leaf in a sub-hierarchy composed of the labels of a target document via an attention mechanism and hierarchy-aware masking. HiDEC achieved state-of-the-art performance with significantly fewer model parameters than existing models on benchmark datasets, such as RCV1-v2, NYT, and EURLEX57K.

Multi-Label Classification of Chinese Rural Poverty Governance Texts Based on XLNet and Bi-LSTM Fused Hierarchical Attention Mechanism

Hierarchical multi-label text classification (HMTC) is a highly relevant and widely discussed topic in the era of big data, particularly for efficiently classifying extensive amounts of text data. This study proposes the HTMC-PGT framework for poverty governance’s single-path hierarchical multi-label classification problem. The framework simplifies the HMTC problem into training and combination problems of multi-class classifiers in the classifier tree. Each independent classifier in this framework uses an XLNet pretrained model to extract char-level semantic embeddings of text and employs a hierarchical attention mechanism integrated with Bi-LSTM (BiLSTM + HA) to extract semantic embeddings at the document level for classification purposes. Simultaneously, this study proposes that the structure uses transfer learning (TL) between classifiers in the classifier tree. The experimental results show that the proposed XLNet + BiLSTM + HA + FC + TL model achieves micro-P, micro-R, and micro-F1 values of 96.1%, which is 7.5~38.1% higher than those of other baseline models. The HTMC-PGT framework based on XLNet, BiLSTM + HA, and transfer learning (TL) between classifier tree nodes proposed in this study solves the hierarchical multi-label classification problem of poverty governance text (PGT). It provides a new idea for solving the traditional HMTC problem.

Hierarchical graph-based text classification framework with contextual node embedding and BERT-based dynamic fusion

We propose a novel hierarchical graph-based text classification framework that leverages the power of contextual node embedding and BERT-based dynamic fusion to capture the complex relationships between the nodes in the hierarchical graph and generate a more accurate classification of text. The framework consists of seven stages: Linguistic Feature Extraction, Hierarchical Node Construction with Domain-Specific Knowledge, Contextual Node Embedding, Multi-Level Graph Learning, Dynamic Text Sequential Feature Interaction, Attention-Based Graph Learning, and Dynamic Fusion with BERT. The first stage, Linguistic Feature Extraction, extracts the linguistic features of the text, including part-of-speech tags, dependency parsing, and named entities. The second stage constructs a hierarchical graph based on the domain-specific knowledge, which is used to capture the relationships between nodes in the graph. The third stage, Contextual Node Embedding, generates a vector representation for each node in the hierarchical graph, which captures its local context information, linguistic features, and domain-specific knowledge. The fourth stage, Multi-Level Graph Learning, uses a graph convolutional neural network to learn the hierarchical structure of the graph and extract the features of the nodes in the graph. The fifth stage, Dynamic Text Sequential Feature Interaction, captures the sequential information of the text and generates dynamic features for each node. The sixth stage, Attention-Based Graph earning, uses an attention mechanism to capture the important features of the nodes in the graph. Finally, the seventh stage, Dynamic Fusion with BERT, combines the output from the previous stages with the output from a pre-trained BERT model to obtain the final integrated vector representation of the text. This approach leverages the strengths of both the proposed framework and BERT, allowing for better performance on the classification task. The proposed framework was evaluated on several benchmark datasets and compared to state-of-the-art methods, demonstrating significant improvements in classification accuracy.

UMP-MG: A Uni-directed Message-Passing Multi-label Generation Model for Hierarchical Text Classification

Hierarchical Text Classification (HTC) is a formidable task which involves classifying textual descriptions into a taxonomic hierarchy. Existing methods, however, have difficulty in adequately modeling the hierarchical label structures, because they tend to focus on employing graph embedding methods to encode the hierarchical structure while disregarding the fact that the HTC labels are rooted in a tree structure. This is significant because, unlike a graph, the tree structure inherently has a directive that ordains information flow from one node to another—a critical factor when applying graph embedding to the HTC task. But in the graph structure, message-passing is undirected, which will lead to the imbalance of message transmission between nodes when applied to HTC. To this end, we propose a unidirectional message-passing multi-label generation model for HTC, referred to as UMP-MG. Instead of viewing HTC as a classification problem as previous methods have done, this novel approach conceptualizes it as a sequence generation task, introducing prior hierarchical information during the decoding process. This further enables the blocking of information flow in one direction to ensure that the graph embedding method is better suited for the HTC task and thus resulted in the enhanced tree structure representation. Results obtained through experimentation on both the public WOS dataset and an E-commerce user intent classification dataset demonstrate that our proposed model can achieve superlative results.

Multilevel Classification of Users’ Needs in Chinese Online Medical and Health Communities: Model Development and Evaluation Based on Graph Convolutional Network

Online medical and health communities provide a platform for internet users to share experiences and ask questions about medical and health issues. However, there are problems in these communities, such as the low accuracy of the classification of users' questions and the uneven health literacy of users, which affect the accuracy of user retrieval and the professionalism of the medical personnel answering the question. In this context, it is essential to study more effective classification methods of users' information needs. Most online medical and health communities tend to provide only disease-type labels, which do not give a comprehensive summary of users' needs. The study aims to construct a multilevel classification framework based on the graph convolutional network (GCN) model for users' needs in online medical and health communities so that users can perform more targeted information retrieval. Using the Chinese online medical and health community "Qiuyi" as an example, we crawled questions posted by users in the "Cardiovascular Disease" section as the data source. First, the disease types involved in the problem data were segmented by manual coding to generate the first-level label. Second, the needs were identified by K-means clustering to generate the users' information needs label as the second-level label. Finally, by constructing a GCN model, users' questions were automatically classified, thus realizing the multilevel classification of users' needs. Based on the empirical research of questions posted by users in the "Cardiovascular Disease" section of Qiuyi, the hierarchical classification of users' questions (data) was realized. The classification models designed in the study achieved accuracy, precision, recall, and F1-score of 0.6265, 0.6328, 0.5788, and 0.5912, respectively. Compared with the traditional machine learning method naïve Bayes and the deep learning method hierarchical text classification convolutional neural network, our classification model showed better performance. At the same time, we also performed a single-level classification experiment on users' needs, which in comparison with the multilevel classification model exhibited a great improvement. A multilevel classification framework has been designed based on the GCN model. The results demonstrated that the method is effective in classifying users' information needs in online medical and health communities. At the same time, users with different diseases have different directions for information needs, which plays an important role in providing diversified and targeted services to the online medical and health community. Our method is also applicable to other similar disease classifications.

Using hierarchical text classification to investigate the utility of machine learning in automating online analyses of wildlife exploitation

Expanding digital data sources, including social media, online news articles and blogs, provide an opportunity to understand better the context and intensity of human-nature interactions, such as wildlife exploitation. However, online searches encompassing large taxonomic groups can generate vast datasets, which can be overwhelming to filter for relevant content without the use of automated tools. The variety of machine learning models available to researchers, and the need for manually labelled training data with an even balance of labels, can make applying these tools challenging. Here, we implement and evaluate a hierarchical text classification pipeline which brings together three binary classification tasks with increasingly specific relevancy criteria. Crucially, the hierarchical approach facilitates the filtering and structuring of a large dataset, of which relevant sources make up a small proportion. Using this pipeline, we also investigate how the accuracy with which text classifiers identify relevant and irrelevant texts is influenced by the use of different models, training datasets, and the classification task. To evaluate our methods, we collected data from Facebook, Twitter, Google and Bing search engines, with the aim of identifying sources documenting the hunting and persecution of bats (Chiroptera). Overall, the ‘state-of-the-art’ transformer-based models were able to identify relevant texts with an average accuracy of 90%, with some classifiers achieving accuracy of >95%. Whilst this demonstrates that application of more advanced models can lead to improved accuracy, comparable performance was achieved by simpler models when applied to longer documents and less ambiguous classification tasks. Hence, the benefits from using more computationally expensive models are dependent on the classification context. We also found that stratification of training data, according to the presence of key search terms, improved classification accuracy for less frequent topics within datasets, and therefore improves the applicability of classifiers to future data collection. Overall, whilst our findings reinforce the usefulness of automated tools for facilitating online analyses in conservation and ecology, they also highlight that the effectiveness and appropriateness of such tools is determined by the nature and volume of data collected, the complexity of the classification task, and the computational resources available to researchers.

Using hierarchical text classification to investigate the utility of machine learning in automating online analyses of wildlife exploitation

Using hierarchical text classification to investigate the utility of machine learning in automating online analyses of wildlife exploitation

Label-Correction Capsule Network for Hierarchical Text Classification

Hierarchical Text Classification (HTC) aims to predict the category of a document in a given label hierarchy. Considering a parent-child relationship among labels at different levels, previous works mainly leverage the parent-level label information to guide the child-level classification and achieve promising results. However, they still suffer from two drawbacks: (1) insufficient for distinguishing similar labels at the same level; (2) fail to consider the error propagation problem caused by the incorrect parent-level predictions. For this reason, we first propose a hierarchical capsule network for the HTC task, due to the ability of capsules to distinguish similar categories. To ease the error propagation problem, we further devise two novel mechanisms in the proposed hierarchical capsule framework, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Label Injection</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Label Re-Routing</i> , to enhance the tolerance of the model to the incorrect parent-level predictions. Experiments on two widely used datasets prove that our model achieves competitive performance. The ablation study further demonstrates the scalability of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Label Injection</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Label Re-Routing</i> .

On the Privacy–Utility Trade-Off in Differentially Private Hierarchical Text Classification

Hierarchical text classification consists of classifying text documents into a hierarchy of classes and sub-classes. Although Artificial Neural Networks have proved useful to perform this task, unfortunately, they can leak training data information to adversaries due to training data memorization. Using differential privacy during model training can mitigate leakage attacks against trained models, enabling the models to be shared safely at the cost of reduced model accuracy. This work investigates the privacy–utility trade-off in hierarchical text classification with differential privacy guarantees, and it identifies neural network architectures that offer superior trade-offs. To this end, we use a white-box membership inference attack to empirically assess the information leakage of three widely used neural network architectures. We show that large differential privacy parameters already suffice to completely mitigate membership inference attacks, thus resulting only in a moderate decrease in model utility. More specifically, for large datasets with long texts, we observed Transformer-based models to achieve an overall favorable privacy–utility trade-off, while for smaller datasets with shorter texts, convolutional neural networks are preferable.

LA-HCN: Label-based Attention for Hierarchical Multi-label Text Classification Neural Network

Hierarchical multi-label text classification (HMTC) has been gaining popularity in recent years thanks to its applicability to a plethora of real-world applications. The existing HMTC algorithms largely focus on the design of classifiers, such as the local, global, or a combination of them. However, very few studies have focused on hierarchical feature extraction and explore the association between the hierarchical labels and the text. In this paper, we propose a Label-based Attention for Hierarchical Multi-label Text Classification Neural Network (LA-HCN), where the novel label-based attention module is designed to hierarchically extract important information from the text based on the labels from different hierarchy levels. Besides, hierarchical information is shared across levels while preserving the hierarchical label-based information. Separate local and global document embeddings are obtained and used to facilitate the respective local and global classifications. In our experiments, LA-HCN outperforms other state-of-the-art neural network-based HMTC algorithms on four public HMTC datasets. The ablation study also demonstrates the effectiveness of the proposed label-based attention module as well as the novel local and global embeddings and classifications. By visualizing the learned attention (words), we find that LA-HCN is able to extract meaningful information corresponding to the different labels which provides explainability that may be helpful for the human analyst.

Hybrid embedding-based text representation for hierarchical multi-label text classification

Many real-world text classification tasks often deal with a large number of closely related categories organized in a hierarchical structure or taxonomy. Hierarchical multi-label text classification (HMTC) has become rather challenging when it requires handling large sets of closely related categories. The structural features of all categories in the entire hierarchy and the word semantics of their category labels are very helpful in improving text classification accuracy over large sets of closely related categories, which has been neglected in most of existing HMTC approaches. In this paper, we present a hybrid embedding-based text representation for HMTC with high accuracy. First, the hybrid embedding consists of both graph embedding of categories in the hierarchy and their word embedding of category labels. The Structural Deep Network Embedding-based graph embedding model is used to simultaneously encode the global and local structural features of a given category in the whole hierarchy for making the category structurally discriminable. We further use the word embedding technique to encode the word semantics of each category label in the hierarchy for making different categories semantically discriminable. Second, we presented a level-by-level HMTC approach based on the bidirectional Gated Recurrent Unit network model together with the hybrid embedding that is used to learn the representation of the text level-by-level. Last but not least, extensive experiments were made over five large-scale real-world datasets in comparison with the state-of-the-art hierarchical and flat multi-label text classification approaches, and the experimental results show that our approach is very competitive to the state-of-the-art approaches in classification accuracy, in particular maintaining computational costs while achieving superior performance.