Multi-label Text Classification Research Articles

CoocNet: a novel approach to multi-label text classification with improved label co-occurrence modeling

CoocNet: a novel approach to multi-label text classification with improved label co-occurrence modeling

Variational Continuous Label Distribution Learning for Multi-Label Text Classification

Variational Continuous Label Distribution Learning for Multi-Label Text Classification

Multi-Label Text Classification model integrating Label Attention and Historical Attention

Multi-Label Text Classification (MLTC) is one of the most import research of natural language processing. Although Deep Learning (DL) models have been widely applied to MLTC, there still exist some drawbacks. First, traditional DL models utilize all the words in the document to construct the embedding vector, while there are many words that affect the classification results. Next, the labels in MLTC have specific semantics, while traditional DL models ignore fine-grained matching signals between words and labels. Then, traditional DL models have difficulty in handling the data imbalance issue in MLTC datasets. In addition, during the training process, small errors in a certain epoch may be amplified with the increase of the number of iterations, resulting in classification errors. To address the above problems, a MLTC model integrating Label Attention and Historical Attention (i.e. LAHA) is proposed. First, a word filter is set up to select important words based on the cosine similarity between words and labels. Next, Document Self Attention (DSA) and Label Attention (LA) are obtained, and DSA-attended LA co-attention (LA-co) and LA-attended DSA co-attention (DAS-co) networks are constructed. Then, the fine-grained matching signals between words and labels are integrated through the adaptive plus of LA-co and DAS-co. At last, Historical Attention is integrated to LAHA, which not only avoids mis-classification caused by minor errors of a certain epoch, but also reduces overfitting to high-frequency labels. Multiple comparative experiments on four benchmark datasets demonstrate that LAHA outperforms the state-of-the-art baseline models and can effectively solve the data imbalance issue in MLTC datasets. Our code is available at https://github.com/sgysgywaityou/LAHA.

Multi-label text classification on unbalanced Twitter with monolingual model and hyperparameter optimization for hate speech and abusive language detection

The increase in hate speech and abusive language on social media leads to uncomfortable interactions among users. Many datasets available publicly that address hate speech and abusive language are not balanced, particularly those from Indonesian Twitter. To develop a more effective classification model that also considers minority classes, we needed to optimize the hyperparameters of a monolingual model, use four different data preprocessing scenarios, and improve the treatment of slang words. We assessed the model's effectiveness by its accuracy, achieving 81.38%. This result came from optimizing hyperparameters, processing data without stemming and removing stop words, and enhancing the slang word data. The optimal hyperparameters were a learning rate of 4e-5, a batch size of 16, and a dropout rate of 0.1. However, using too much dropout can decrease the model’s performance and its ability to predict less common categories, such as physical- and gender-related hate speech.

Business text classification with imbalanced data and moderately large label spaces for digital transformation

Digital transformation refers to an organization’s use of digital technology to improve its products, services, and operations, aligning them with evolving business requirements. To demonstrate this transformative process, we present a real-life case study where a company seeks to automate the classification of their textual data rather than relying on manual methods. Transitioning to automated classification involves deploying machine learning models, which rely on pre-labeled datasets for training and making predictions on new data. However, upon receiving the dataset from the company, we faced challenges due to the imbalanced distribution of labels and moderately large label spaces. To tackle text classification with such a business dataset, we evaluated four distinct methods for multi-label text classification: fine-tuned Bidirectional Encoder Representations from Transformers (BERT), Binary Relevance, Classifier Chains, and Label Powerset. The results revealed that fine-tuned BERT significantly outperformed the other methods across key metrics like Accuracy, F1-score, Precision, and Recall. Binary Relevance also displayed competence in handling the dataset effectively, while Classifier Chains and Label Powerset exhibited comparatively less impressive performance. These findings highlight the remarkable effectiveness of fine-tuned BERT model and the Binary Relevance classifier in multi-label text classification tasks, particularly when dealing with imbalanced training datasets and moderately large label spaces. This positions them as valuable assets for businesses aiming to automate data classification in the digital transformation era.

Multi-label text classification of cardiovascular drug attributes based on BERT and BiGRU

Extracting and digitizing drug attributes from medical literature is the first step to build a knowledge computing system for precision disease treatment. In order to build a cardiovascular drug knowledge base, this paper proposes a multi-label text classification method for cardiovascular drug attributes from the Chinese drug guideline. The drug attributes are characterized by a BERT pre-trained model, and a dual-feature extraction structure is proposed based on the BiGRU neural network to capture high-level semantic information. Label categorization of cardiovascular drug attributes, such as indications and mode of administration, is accomplished. The F1 score of 0.8431 was obtained using 5-fold cross-validation. Comparing KNN and Naïve bayes, and conducting CNN and BiGRU control experiments on the basis of Word2Vec characterization of medication guidelines, the proposed multi-label text classification method is effective and the F1 value is significantly improved. Proved by analysis of ablation and crossover experiments, the proposed method can achieve a high accuracy rate averaged at 0.8339.

Label-text bi-attention capsule networks model for multi-label text classification

Multi-label text classification (MLTC) is the process of establishing relationships between documents and their corresponding labels. Previous research has focused on mining textual information, treating labels as information-less vectors in classification. This ignores the semantic and dependency relationships of labels. In real-life scenarios, the neglect of label information contradicts the classification process, which presents significant challenges for MLTC tasks. Label embedding partially resolves label information loss. Efficiently exploring semantic and dependency relationships of labels and their text connections remains a new challenge. In this paper, we propose a Label-Text Bi-Attention Capsule Networks (LTBACN) model for in-depth exploration of the dependency relationships between labels and text. Specifically, we first incorporate label information into nodes through label embedding, construct a graph structure to represent the dependency relationships between labels, and use Graph Convolutional Networks (GCN) to propagate information between nodes to further mine the relationships between labels. Subsequently, we employ a label-text bi-attention mechanism to learn the feature relationships between labels and text. The label-to-text attention mechanism extracts label-relevant text representations, while the text-to-label attention mechanism extracts the most relevant label representations for the text. We then merge these two types of feature representations to obtain fused representations that incorporate label-text bi-directional information. Finally, the fused features are fed into a capsule network classifier to capture multi-level semantic information and match the corresponding labels. The experimental results demonstrate that LTBACN outperforms other methods in terms of classification effectiveness. Compared to state-of-the-art methods, LTBACN achieves a significant improvement of 0.41%–0.68% in Micro−F1 measure, 0.52%–3.26% in Macro−F1 measure, 0.32%–2.18% in P@k measure, and 0.01%–1.18% in nDCG@k measure on the AAPD and RCV1-v2 datasets.

Unveiling the Biomarkers: Identifying Key Signatures for Cancer Hallmarks

INTRODUCTION: Finding biomarkers that are closely associated with cancer-related traits is critical to the advancement of cancer research, especially when it comes to personalised treatment. The objective of this research is to explore multiple biomarker categories, including genetics, proteins, and chemicals, in order to better understand the complex terrain of cancer.
 OBJECTIVES: Few of the objectives include examining a variety of biomarker types, such as chemical, protein, and genetic markers and determining which important biomarker signatures correspond to each cancer hallmark.
 Also the study aims to perform a comparative analysis to show how the SVM model's features incorporating identified biomarkers improves classification performance.
 METHODS: The study includes NLP and ML techniques for the identification and classification of biomarkers for the hallmark of cancer dataset. 
 RESULTS: The discovery of important biomarker signatures connected to every cancer hallmark is one of the study's primary findings. In addition, our new SVM-based classification model performed well in the multilabel text classification of PubMed abstracts, showing a significant improvement in performance when the biomarkers were used as features.
 CONCLUSION: To sum up, this study makes a substantial contribution to the area of cancer research by identifying important biomarker signatures connected to many cancer hallmarks.

Machine learning based software effort estimation using development-centric features for crowdsourcing platform

Multi-label text classification is a method for categorizing textual data based on features extracted from the original textual information. When it comes to modelling text structural properties, Graph Convolutional Network (GCN) has demonstrated outstanding performance. However, most existing graph-based models do not model the structure of a single text unit and do not consider the sequence information in each document (e.g., word order). To resolve these issues and fully utilize the text’s structural and sequential details, a text classification model called Sequential GCN with Multi-Head Attention (SGCN-MHA) is proposed in this paper. For each text, a separate text graph is constructed in which nodes are the words of the text, and the edges between nodes corresponding to the word relations. Then the GCN is used to extract the structural feature. To enable the word nodes in the document graph to hold contextual information, the BiLSTM is also applied to learn the sequential feature for each graph. Finally, the Multi-Head Attention mechanism is adopted to interact with these two features and then aggregate them to get access to critical information in the text. The efficiency of our approach has been tested on two standard datasets, including comparative and ablation experiments.

BeNet: BERT Doc-Label Attention Network for Multi-Label Text Classification

Multi-label Text Classification (MLTC) holds significant importance and serves as a foundational aspect in Natural Language Processing (NLP), which aims at assigning multiple labels for a given document. Many real-world tasks can be viewed as MLTC, such as tag recommendation, information retrieval, etc. Nevertheless, researchers are faced with numerous challenging issues regarding the establishment of linkages between labels or the differentiation of comparable sub-labels. To address this issue, we provide a novel approach known as the BERT Doc-Label Attention Network (BeNet) in this paper, which consist of the BERTdoc layer, the label embeddings layer, the doc encoder layer, the doc-label attention layer and the prediction layer. We apply the powerful technique of BERT to pretrain documents to capture their deep semantic features and encode them via Bi-LSTM to obtain a two-directional contextual representation of uniform length. Then we create label embeddings and feed them together with encoded-pretrained-documents to the doc-label attention mechanism to obtain interactive information between documents and their corresponding labels, finally using MLP to make predictions. We carry out experiments on two real-world datasets, and the empirical results demonstrate that our proposed model outperforms all state-of-the-art MLTC benchmarks. Furthermore, we have undertaken a case study to effectively illustrate the practical implementation of our method.

On the Value of Head Labels in Multi-Label Text Classification

A formidable challenge in the multi-label text classification (MLTC) context is that the labels often exhibit a long-tailed distribution, which typically prevents deep MLTC models from obtaining satisfactory performance. To alleviate this problem, most existing solutions attempt to improve tail performance by means of sampling or introducing extra knowledge. Data-rich labels, though more trustworthy, have not received the attention they deserve. In this work, we propose a multiple-stage training framework to exploit both model- and feature-level knowledge from the head labels, to improve both the representation and generalization ability of MLTC models. Moreover, we theoretically prove the superiority of our framework design over other alternatives. Comprehensive experiments on widely used MLTC datasets clearly demonstrate that the proposed framework achieves highly superior results to state-of-the-art methods, highlighting the value of head labels in MLTC.

Harnessing the Power of Beta Scoring in Deep Active Learning for Multi-Label Text Classification

Within the scope of natural language processing, the domain of multi-label text classification is uniquely challenging due to its expansive and uneven label distribution. The complexity deepens due to the demand for an extensive set of annotated data for training an advanced deep learning model, especially in specialized fields where the labeling task can be labor-intensive and often requires domain-specific knowledge. Addressing these challenges, our study introduces a novel deep active learning strategy, capitalizing on the Beta family of proper scoring rules within the Expected Loss Reduction framework. It computes the expected increase in scores using the Beta Scoring Rules, which are then transformed into sample vector representations. These vector representations guide the diverse selection of informative sample, directly linking this process to the model's expected proper score. Comprehensive evaluations across both synthetic and real datasets reveal our method's capability to often outperform established acquisition techniques in multi-label text classification, presenting encouraging outcomes across various architectural and dataset scenarios.

Compositional Generalization for Multi-Label Text Classification: A Data-Augmentation Approach

Despite significant advancements in multi-label text classification, the ability of existing models to generalize to novel and seldom-encountered complex concepts, which are compositions of elementary ones, remains underexplored. This research addresses this gap. By creating unique data splits across three benchmarks, we assess the compositional generalization ability of existing multi-label text classification models. Our results show that these models often fail to generalize to compositional concepts encountered infrequently during training, leading to inferior performance on tests with these new combinations. To address this, we introduce a data augmentation method that leverages two innovative text generation models designed to enhance the classification models' capacity for compositional generalization. Our experiments show that this data augmentation approach significantly improves the compositional generalization capabilities of classification models on our benchmarks, with both generation models surpassing other text generation baselines. Our codes available at https://github.com/yychai74/LD-VAE.

Research and Development of a Modern Deep Learning Model for Emotional Analysis Management of Text Data

There are many ways people express their reactions in the media. Text data is one of them, for example, comments, reviews, blog posts, messages, etc. Analysis of emotions expressed there is in high demand nowadays for various purposes. This research provides a method of performing sentiment analysis of text information using machine learning. The authors trained a classifier based on the BERT encoder, which recognizes emotions in text messages in English written in chat style. To handle raw chat-style messages, authors developed an enhanced text standardization layer. The list of emotions identified includes admiration, amusement, anger, annoyance, approval, caring, confusion, curiosity, desire, disappointment, disapproval, disgust, embarrassment, excitement, fear, gratitude, grief, joy, love, nervousness, optimism, pride, realization, relief, remorse, sadness, and surprise. The model solves the problem of multiclass multilabel text classification, which means that more than one class can be predicted from one piece of text. The authors trained the model on the GoEmotions dataset, which consists of 54,263 text comments from Reddit. The model reached a macro-averaged F1-Score of 0.50704 in emotions prediction and 0.7349 in sentiments prediction on the testing dataset. The presented model increased the quality of emotions prediction by 10.2% and sentiments prediction by 6.5% in comparison to the baseline approach.

A Hybrid Model Based on Convolutional Neural Network and Long Short-Term Memory for Multi-label Text Classification

Multi-label text classification (MLTC) is a popular method for organizing electronic documents, which is crucial for accessing and processing data. As the number of classes increases, learning multi-label data will be challenging. The number of possible states for various labels increases exponentially, and learning algorithms in single-label data cannot be used to solve these problems. In the meantime, using single-label data algorithms could be very time-consuming. In MLTC, complexity costs should be reduced. Deep-learning neural networks that can learn intricate patterns are used in many real-world problems because of their high power and accuracy. This paper proposed a hybridization of the long short-term memory (LSTM) neural network and the convolutional neural network (CNN) method for MLTC. The proposed model uses LSTM to enhance CNN to improve the proposed model’s accuracy. Also, the competitive search algorithm (CSA) is used to improve the LSTM hyperparameters. The LSTM hyperparameters play an important role in increasing the detection accuracy. The CSA algorithm finds the best values for the hyperparameters by searching the problem space. It was tested on four different datasets of multi-label texts: Reuters-21578, RCV1-v2, EUR-Lex, and Bookmarks. The result showed that the proposed model performed better than CNN and LSTM-CSA in terms of accuracy percentage and that it has improved by an average of more than 10%. Also, the results show that the LSTM-CSA model has higher detection accuracy compared to LSTM—Gradient-based optimizer (GBO) and LSTM—whale optimization algorithm (WOA).

Capturing experts’ knowledge in heritage planning enhanced by AI: A case study of windcatchers in Yazd, Iran

Experts have always played an important role in heritage planning, practice, and theory. There is a wealth of literature published every year regarding heritage and its cultural significance. Experts also contribute to heritage planning and developing policy documents. Still, literature is rarely used as a source of primary research to systematically reveal and compare experts’ opinions on the cultural significance of built heritage. Analyzing them as a whole is costly and time-consuming, especially on built heritage, when much has been written about. While the automation of methods has proven to mitigate such restrictions in other fields, as digital humanities, their application in heritage planning, practice, and theory is still scarce. Hence, this paper aims to investigate the potentials of AI models (e.g., multi label text classification) in analyzing scientific documents, revealing the cultural significance of built heritage, values and attributes. This was done to better understand the similarities and differences between the experts’ opinions. Yazd, Iran, is taken a case study, with a particular focus on windcatchers, a key attribute conveying cultural significance, of outstanding universal value, due to its inscription on the UNESCO World Heritage List. This paper has three subsequent phases: 1) state of the art on the application of AI in heritage planning; 2) methodology of data collection and data analysis related to coding values and attributes of windcatchers, addressed in relevant documents; 3) preliminary findings on the experts’ opinions over values and attributes of windcatchers. Results contribute to the scientific discussion, revealing the cultural significance of windcatchers of Yazd from experts’ point of view. Besides, the potential of AI for heritage planning is revealed in terms of (de)coding and measuring the cultural significance of built heritage from the available documents, showing the opinions of experts with various backgrounds. This model can be applied to other key attributes in Yazd and other case studies and scales to support heritage planning, practice, and theory.

TLC-XML: Transformer with Label Correlation for Extreme Multi-label Text Classification

Extreme multi-label text classification (XMTC) annotates related labels for unknown text from large-scale label sets. Transformer-based methods have become the dominant approach for solving the XMTC task due to their effective text representation capabilities. However, the existing Transformer-based methods fail to effectively exploit the correlation between labels in the XMTC task. To address this shortcoming, we propose a novel model called TLC-XML, i.e., a Transformer with label correlation for extreme multi-label text classification. TLC-XML comprises three modules: Partition, Matcher and Ranker. In the Partition module, we exploit the semantic and co-occurrence information of labels to construct the label correlation graph, and further partition the strongly correlated labels into the same cluster. In the Matcher module, we propose cluster correlation learning, which uses the graph convolutional network (GCN) to extract the correlation between clusters. We then introduce these valuable correlations into the classifier to match related clusters. In the Ranker module, we propose label interaction learning, which aggregates the raw label prediction with the information of the neighboring labels. The experimental results on benchmark datasets show that TLC-XML significantly outperforms state-of-the-art XMTC methods.

A new sentence embedding framework for the education and professional training domain with application to hierarchical multi-label text classification

In recent years, Natural Language Processing (NLP) has made significant advances through advanced general language embeddings, allowing breakthroughs in NLP tasks such as semantic similarity and text classification. However, complexity increases with hierarchical multi-label classification (HMC), where a single entity can belong to several hierarchically organized classes. In such complex situations, applied on specific-domain texts, such as the Education and professional training domain, general language embedding models often inadequately represent the unique terminologies and contextual nuances of a specialized domain. To tackle this problem, we present HMCCCProbT, a novel hierarchical multi-label text classification approach. This innovative framework chains multiple classifiers, where each individual classifier is built using a novel sentence-embedding method BERTEPro based on existing Transformer models, whose pre-training has been extended on education and professional training texts, before being fine-tuned on several NLP tasks. Each individual classifier is responsible for the predictions of a given hierarchical level and propagates local probability predictions augmented with the input feature vectors to the classifier in charge of the subsequent level. HMCCCProbT tackles issues of model scalability and semantic interpretation, offering a powerful solution to the challenges of domain-specific hierarchical multi-label classification. Experiments over three domain-specific textual HMC datasets indicate the effectiveness of HMCCCProbT to compare favorably to state-of-the-art HMC algorithms in terms of classification accuracy and also the ability of BERTEPro to obtain better probability predictions, well suited to HMCCCProbT, than three other vector representation techniques.

Multi-Label Text Classification Model Based on Multi-Level Constraint Augmentation and Label Association Attention

In the multi-label text classification task, a text usually corresponds to multiple label categories, and the labels have correlation and hierarchical structure. However, when the label hierarchy is unknown, the number of various labels is not balanced, which makes it difficult for the model to classify low-frequency labels. In addition, labels have semantic similarities that make it difficult for the model to distinguish between them. In this article, we propose a multi-label text classification model based on multi-level constraint augmentation and label association attention. Compared with traditional methods, our method has two contributions: (1) In order to alleviate the problem of unbalanced number of different label categories and ensure the rationality of sample generation, we propose a data augmentation method based on multi-level constraints. In the process of sample generation, this method uses historical generation information, sample original text information, and sample topic to constrain the generated text. (2) In order to make the model recognize the associated labels accurately, we propose an interaction mechanism based on label association attention and filter gate. This method combines text information and label weight information. At the same time, our classification model considers the important weights of text sentences and effectively utilizes the co-occurrence relationship between labels. Experimental results on three benchmark datasets show that our model outperforms state-of-the-art methods on all main evaluation metrics, especially on low-frequency label prediction with sparse samples.

When graph convolution meets double attention: online privacy disclosure detection with multi-label text classification

With the rise of Web 2.0 platforms such as online social media, people’s private information, such as their location, occupation and even family information, is often inadvertently disclosed through online discussions. Therefore, it is important to detect such unwanted privacy disclosures to help alert people affected and the online platform. In this paper, privacy disclosure detection is modeled as a multi-label text classification (MLTC) problem, and a new privacy disclosure detection model is proposed to construct an MLTC classifier for detecting online privacy disclosures. This classifier takes an online post as the input and outputs multiple labels, each reflecting a possible privacy disclosure. The proposed presentation method combines three different sources of information, the input text itself, the label-to-text correlation and the label-to-label correlation. A double-attention mechanism is used to combine the first two sources of information, and a graph convolutional network is employed to extract the third source of information that is then used to help fuse features extracted from the first two sources of information. Our extensive experimental results, obtained on a public dataset of privacy-disclosing posts on Twitter, demonstrated that our proposed privacy disclosure detection method significantly and consistently outperformed other state-of-the-art methods in terms of all key performance indicators.