Multi-label Text Classification Research Articles

Multi-label text classification of cardiovascular drug attributes based on BERT and BiGRU

Extracting and digitizing drug attributes from medical literature is the first step to build a knowledge computing system for precision disease treatment. In order to build a cardiovascular drug knowledge base, this paper proposes a multi-label text classification method for cardiovascular drug attributes from the Chinese drug guideline. The drug attributes are characterized by a BERT pre-trained model, and a dual-feature extraction structure is proposed based on the BiGRU neural network to capture high-level semantic information. Label categorization of cardiovascular drug attributes, such as indications and mode of administration, is accomplished. The F1 score of 0.8431 was obtained using 5-fold cross-validation. Comparing KNN and Naïve bayes, and conducting CNN and BiGRU control experiments on the basis of Word2Vec characterization of medication guidelines, the proposed multi-label text classification method is effective and the F1 value is significantly improved. Proved by analysis of ablation and crossover experiments, the proposed method can achieve a high accuracy rate averaged at 0.8339.

Label-text bi-attention capsule networks model for multi-label text classification

Multi-label text classification (MLTC) is the process of establishing relationships between documents and their corresponding labels. Previous research has focused on mining textual information, treating labels as information-less vectors in classification. This ignores the semantic and dependency relationships of labels. In real-life scenarios, the neglect of label information contradicts the classification process, which presents significant challenges for MLTC tasks. Label embedding partially resolves label information loss. Efficiently exploring semantic and dependency relationships of labels and their text connections remains a new challenge. In this paper, we propose a Label-Text Bi-Attention Capsule Networks (LTBACN) model for in-depth exploration of the dependency relationships between labels and text. Specifically, we first incorporate label information into nodes through label embedding, construct a graph structure to represent the dependency relationships between labels, and use Graph Convolutional Networks (GCN) to propagate information between nodes to further mine the relationships between labels. Subsequently, we employ a label-text bi-attention mechanism to learn the feature relationships between labels and text. The label-to-text attention mechanism extracts label-relevant text representations, while the text-to-label attention mechanism extracts the most relevant label representations for the text. We then merge these two types of feature representations to obtain fused representations that incorporate label-text bi-directional information. Finally, the fused features are fed into a capsule network classifier to capture multi-level semantic information and match the corresponding labels. The experimental results demonstrate that LTBACN outperforms other methods in terms of classification effectiveness. Compared to state-of-the-art methods, LTBACN achieves a significant improvement of 0.41%–0.68% in Micro−F1 measure, 0.52%–3.26% in Macro−F1 measure, 0.32%–2.18% in P@k measure, and 0.01%–1.18% in nDCG@k measure on the AAPD and RCV1-v2 datasets.

Unveiling the Biomarkers: Identifying Key Signatures for Cancer Hallmarks

INTRODUCTION: Finding biomarkers that are closely associated with cancer-related traits is critical to the advancement of cancer research, especially when it comes to personalised treatment. The objective of this research is to explore multiple biomarker categories, including genetics, proteins, and chemicals, in order to better understand the complex terrain of cancer.
 OBJECTIVES: Few of the objectives include examining a variety of biomarker types, such as chemical, protein, and genetic markers and determining which important biomarker signatures correspond to each cancer hallmark.
 Also the study aims to perform a comparative analysis to show how the SVM model's features incorporating identified biomarkers improves classification performance.
 METHODS: The study includes NLP and ML techniques for the identification and classification of biomarkers for the hallmark of cancer dataset. 
 RESULTS: The discovery of important biomarker signatures connected to every cancer hallmark is one of the study's primary findings. In addition, our new SVM-based classification model performed well in the multilabel text classification of PubMed abstracts, showing a significant improvement in performance when the biomarkers were used as features.
 CONCLUSION: To sum up, this study makes a substantial contribution to the area of cancer research by identifying important biomarker signatures connected to many cancer hallmarks.

Machine learning based software effort estimation using development-centric features for crowdsourcing platform

Multi-label text classification is a method for categorizing textual data based on features extracted from the original textual information. When it comes to modelling text structural properties, Graph Convolutional Network (GCN) has demonstrated outstanding performance. However, most existing graph-based models do not model the structure of a single text unit and do not consider the sequence information in each document (e.g., word order). To resolve these issues and fully utilize the text’s structural and sequential details, a text classification model called Sequential GCN with Multi-Head Attention (SGCN-MHA) is proposed in this paper. For each text, a separate text graph is constructed in which nodes are the words of the text, and the edges between nodes corresponding to the word relations. Then the GCN is used to extract the structural feature. To enable the word nodes in the document graph to hold contextual information, the BiLSTM is also applied to learn the sequential feature for each graph. Finally, the Multi-Head Attention mechanism is adopted to interact with these two features and then aggregate them to get access to critical information in the text. The efficiency of our approach has been tested on two standard datasets, including comparative and ablation experiments.

BeNet: BERT Doc-Label Attention Network for Multi-Label Text Classification

Multi-label Text Classification (MLTC) holds significant importance and serves as a foundational aspect in Natural Language Processing (NLP), which aims at assigning multiple labels for a given document. Many real-world tasks can be viewed as MLTC, such as tag recommendation, information retrieval, etc. Nevertheless, researchers are faced with numerous challenging issues regarding the establishment of linkages between labels or the differentiation of comparable sub-labels. To address this issue, we provide a novel approach known as the BERT Doc-Label Attention Network (BeNet) in this paper, which consist of the BERTdoc layer, the label embeddings layer, the doc encoder layer, the doc-label attention layer and the prediction layer. We apply the powerful technique of BERT to pretrain documents to capture their deep semantic features and encode them via Bi-LSTM to obtain a two-directional contextual representation of uniform length. Then we create label embeddings and feed them together with encoded-pretrained-documents to the doc-label attention mechanism to obtain interactive information between documents and their corresponding labels, finally using MLP to make predictions. We carry out experiments on two real-world datasets, and the empirical results demonstrate that our proposed model outperforms all state-of-the-art MLTC benchmarks. Furthermore, we have undertaken a case study to effectively illustrate the practical implementation of our method.

On the Value of Head Labels in Multi-Label Text Classification

A formidable challenge in the multi-label text classification (MLTC) context is that the labels often exhibit a long-tailed distribution, which typically prevents deep MLTC models from obtaining satisfactory performance. To alleviate this problem, most existing solutions attempt to improve tail performance by means of sampling or introducing extra knowledge. Data-rich labels, though more trustworthy, have not received the attention they deserve. In this work, we propose a multiple-stage training framework to exploit both model- and feature-level knowledge from the head labels, to improve both the representation and generalization ability of MLTC models. Moreover, we theoretically prove the superiority of our framework design over other alternatives. Comprehensive experiments on widely used MLTC datasets clearly demonstrate that the proposed framework achieves highly superior results to state-of-the-art methods, highlighting the value of head labels in MLTC.

Harnessing the Power of Beta Scoring in Deep Active Learning for Multi-Label Text Classification

Within the scope of natural language processing, the domain of multi-label text classification is uniquely challenging due to its expansive and uneven label distribution. The complexity deepens due to the demand for an extensive set of annotated data for training an advanced deep learning model, especially in specialized fields where the labeling task can be labor-intensive and often requires domain-specific knowledge. Addressing these challenges, our study introduces a novel deep active learning strategy, capitalizing on the Beta family of proper scoring rules within the Expected Loss Reduction framework. It computes the expected increase in scores using the Beta Scoring Rules, which are then transformed into sample vector representations. These vector representations guide the diverse selection of informative sample, directly linking this process to the model's expected proper score. Comprehensive evaluations across both synthetic and real datasets reveal our method's capability to often outperform established acquisition techniques in multi-label text classification, presenting encouraging outcomes across various architectural and dataset scenarios.

Compositional Generalization for Multi-Label Text Classification: A Data-Augmentation Approach

Despite significant advancements in multi-label text classification, the ability of existing models to generalize to novel and seldom-encountered complex concepts, which are compositions of elementary ones, remains underexplored. This research addresses this gap. By creating unique data splits across three benchmarks, we assess the compositional generalization ability of existing multi-label text classification models. Our results show that these models often fail to generalize to compositional concepts encountered infrequently during training, leading to inferior performance on tests with these new combinations. To address this, we introduce a data augmentation method that leverages two innovative text generation models designed to enhance the classification models' capacity for compositional generalization. Our experiments show that this data augmentation approach significantly improves the compositional generalization capabilities of classification models on our benchmarks, with both generation models surpassing other text generation baselines. Our codes available at https://github.com/yychai74/LD-VAE.

Research and Development of a Modern Deep Learning Model for Emotional Analysis Management of Text Data

There are many ways people express their reactions in the media. Text data is one of them, for example, comments, reviews, blog posts, messages, etc. Analysis of emotions expressed there is in high demand nowadays for various purposes. This research provides a method of performing sentiment analysis of text information using machine learning. The authors trained a classifier based on the BERT encoder, which recognizes emotions in text messages in English written in chat style. To handle raw chat-style messages, authors developed an enhanced text standardization layer. The list of emotions identified includes admiration, amusement, anger, annoyance, approval, caring, confusion, curiosity, desire, disappointment, disapproval, disgust, embarrassment, excitement, fear, gratitude, grief, joy, love, nervousness, optimism, pride, realization, relief, remorse, sadness, and surprise. The model solves the problem of multiclass multilabel text classification, which means that more than one class can be predicted from one piece of text. The authors trained the model on the GoEmotions dataset, which consists of 54,263 text comments from Reddit. The model reached a macro-averaged F1-Score of 0.50704 in emotions prediction and 0.7349 in sentiments prediction on the testing dataset. The presented model increased the quality of emotions prediction by 10.2% and sentiments prediction by 6.5% in comparison to the baseline approach.

A Hybrid Model Based on Convolutional Neural Network and Long Short-Term Memory for Multi-label Text Classification

Multi-label text classification (MLTC) is a popular method for organizing electronic documents, which is crucial for accessing and processing data. As the number of classes increases, learning multi-label data will be challenging. The number of possible states for various labels increases exponentially, and learning algorithms in single-label data cannot be used to solve these problems. In the meantime, using single-label data algorithms could be very time-consuming. In MLTC, complexity costs should be reduced. Deep-learning neural networks that can learn intricate patterns are used in many real-world problems because of their high power and accuracy. This paper proposed a hybridization of the long short-term memory (LSTM) neural network and the convolutional neural network (CNN) method for MLTC. The proposed model uses LSTM to enhance CNN to improve the proposed model’s accuracy. Also, the competitive search algorithm (CSA) is used to improve the LSTM hyperparameters. The LSTM hyperparameters play an important role in increasing the detection accuracy. The CSA algorithm finds the best values for the hyperparameters by searching the problem space. It was tested on four different datasets of multi-label texts: Reuters-21578, RCV1-v2, EUR-Lex, and Bookmarks. The result showed that the proposed model performed better than CNN and LSTM-CSA in terms of accuracy percentage and that it has improved by an average of more than 10%. Also, the results show that the LSTM-CSA model has higher detection accuracy compared to LSTM—Gradient-based optimizer (GBO) and LSTM—whale optimization algorithm (WOA).

Capturing experts’ knowledge in heritage planning enhanced by AI: A case study of windcatchers in Yazd, Iran

Experts have always played an important role in heritage planning, practice, and theory. There is a wealth of literature published every year regarding heritage and its cultural significance. Experts also contribute to heritage planning and developing policy documents. Still, literature is rarely used as a source of primary research to systematically reveal and compare experts’ opinions on the cultural significance of built heritage. Analyzing them as a whole is costly and time-consuming, especially on built heritage, when much has been written about. While the automation of methods has proven to mitigate such restrictions in other fields, as digital humanities, their application in heritage planning, practice, and theory is still scarce. Hence, this paper aims to investigate the potentials of AI models (e.g., multi label text classification) in analyzing scientific documents, revealing the cultural significance of built heritage, values and attributes. This was done to better understand the similarities and differences between the experts’ opinions. Yazd, Iran, is taken a case study, with a particular focus on windcatchers, a key attribute conveying cultural significance, of outstanding universal value, due to its inscription on the UNESCO World Heritage List. This paper has three subsequent phases: 1) state of the art on the application of AI in heritage planning; 2) methodology of data collection and data analysis related to coding values and attributes of windcatchers, addressed in relevant documents; 3) preliminary findings on the experts’ opinions over values and attributes of windcatchers. Results contribute to the scientific discussion, revealing the cultural significance of windcatchers of Yazd from experts’ point of view. Besides, the potential of AI for heritage planning is revealed in terms of (de)coding and measuring the cultural significance of built heritage from the available documents, showing the opinions of experts with various backgrounds. This model can be applied to other key attributes in Yazd and other case studies and scales to support heritage planning, practice, and theory.

TLC-XML: Transformer with Label Correlation for Extreme Multi-label Text Classification

Extreme multi-label text classification (XMTC) annotates related labels for unknown text from large-scale label sets. Transformer-based methods have become the dominant approach for solving the XMTC task due to their effective text representation capabilities. However, the existing Transformer-based methods fail to effectively exploit the correlation between labels in the XMTC task. To address this shortcoming, we propose a novel model called TLC-XML, i.e., a Transformer with label correlation for extreme multi-label text classification. TLC-XML comprises three modules: Partition, Matcher and Ranker. In the Partition module, we exploit the semantic and co-occurrence information of labels to construct the label correlation graph, and further partition the strongly correlated labels into the same cluster. In the Matcher module, we propose cluster correlation learning, which uses the graph convolutional network (GCN) to extract the correlation between clusters. We then introduce these valuable correlations into the classifier to match related clusters. In the Ranker module, we propose label interaction learning, which aggregates the raw label prediction with the information of the neighboring labels. The experimental results on benchmark datasets show that TLC-XML significantly outperforms state-of-the-art XMTC methods.

A new sentence embedding framework for the education and professional training domain with application to hierarchical multi-label text classification

In recent years, Natural Language Processing (NLP) has made significant advances through advanced general language embeddings, allowing breakthroughs in NLP tasks such as semantic similarity and text classification. However, complexity increases with hierarchical multi-label classification (HMC), where a single entity can belong to several hierarchically organized classes. In such complex situations, applied on specific-domain texts, such as the Education and professional training domain, general language embedding models often inadequately represent the unique terminologies and contextual nuances of a specialized domain. To tackle this problem, we present HMCCCProbT, a novel hierarchical multi-label text classification approach. This innovative framework chains multiple classifiers, where each individual classifier is built using a novel sentence-embedding method BERTEPro based on existing Transformer models, whose pre-training has been extended on education and professional training texts, before being fine-tuned on several NLP tasks. Each individual classifier is responsible for the predictions of a given hierarchical level and propagates local probability predictions augmented with the input feature vectors to the classifier in charge of the subsequent level. HMCCCProbT tackles issues of model scalability and semantic interpretation, offering a powerful solution to the challenges of domain-specific hierarchical multi-label classification. Experiments over three domain-specific textual HMC datasets indicate the effectiveness of HMCCCProbT to compare favorably to state-of-the-art HMC algorithms in terms of classification accuracy and also the ability of BERTEPro to obtain better probability predictions, well suited to HMCCCProbT, than three other vector representation techniques.

Multi-Label Text Classification Model Based on Multi-Level Constraint Augmentation and Label Association Attention

In the multi-label text classification task, a text usually corresponds to multiple label categories, and the labels have correlation and hierarchical structure. However, when the label hierarchy is unknown, the number of various labels is not balanced, which makes it difficult for the model to classify low-frequency labels. In addition, labels have semantic similarities that make it difficult for the model to distinguish between them. In this article, we propose a multi-label text classification model based on multi-level constraint augmentation and label association attention. Compared with traditional methods, our method has two contributions: (1) In order to alleviate the problem of unbalanced number of different label categories and ensure the rationality of sample generation, we propose a data augmentation method based on multi-level constraints. In the process of sample generation, this method uses historical generation information, sample original text information, and sample topic to constrain the generated text. (2) In order to make the model recognize the associated labels accurately, we propose an interaction mechanism based on label association attention and filter gate. This method combines text information and label weight information. At the same time, our classification model considers the important weights of text sentences and effectively utilizes the co-occurrence relationship between labels. Experimental results on three benchmark datasets show that our model outperforms state-of-the-art methods on all main evaluation metrics, especially on low-frequency label prediction with sparse samples.

When graph convolution meets double attention: online privacy disclosure detection with multi-label text classification

With the rise of Web 2.0 platforms such as online social media, people’s private information, such as their location, occupation and even family information, is often inadvertently disclosed through online discussions. Therefore, it is important to detect such unwanted privacy disclosures to help alert people affected and the online platform. In this paper, privacy disclosure detection is modeled as a multi-label text classification (MLTC) problem, and a new privacy disclosure detection model is proposed to construct an MLTC classifier for detecting online privacy disclosures. This classifier takes an online post as the input and outputs multiple labels, each reflecting a possible privacy disclosure. The proposed presentation method combines three different sources of information, the input text itself, the label-to-text correlation and the label-to-label correlation. A double-attention mechanism is used to combine the first two sources of information, and a graph convolutional network is employed to extract the third source of information that is then used to help fuse features extracted from the first two sources of information. Our extensive experimental results, obtained on a public dataset of privacy-disclosing posts on Twitter, demonstrated that our proposed privacy disclosure detection method significantly and consistently outperformed other state-of-the-art methods in terms of all key performance indicators.

Weighted Asymmetric Loss for Multi-Label Text Classification on Imbalanced Data

Weighted Asymmetric Loss for Multi-Label Text Classification on Imbalanced Data

LMTCSG: Multilabel Text Classification Combining Sequence-Based and GNN-Based Features

LMTCSG: Multilabel Text Classification Combining Sequence-Based and GNN-Based Features

BoostXML: Gradient Boosting for Extreme Multilabel Text Classification With Tail Labels.

Multilabel learning involving hundreds of thousands or even millions of labels is referred to as extreme multilabel learning (XML), in which the labels often follow a power-law distribution with the majority occurring in very few data points as tail labels. Recent years have witnessed the intensive use of deep-learning methods for high-performance XML, but they are typically optimized for the head labels with abundant training instances and less consider the performance on tail labels, which, however, like the needles in haystacks, are often the focus of attention in real-life applications. In light of this, we present BoostXML, a deep learning-based XML method for extreme multilabel text classification, enhanced greatly by gradient boosting. In BoostXML, we pay more attention to tail labels in each Boosting Step by optimizing the residual mostly from unfitted training instances with tail labels. A Corrective Step is further proposed to avoid the mismatching between the text encoder and weak learners during optimization, which reduces the risk of falling into local optima and improves model performance. A Pretraining Step is also introduced in the initial stage of BoostXML to avoid exorbitant bias to tail labels. Extensive experiments on five benchmark datasets with state-of-the-art baselines demonstrate the advantage of BoostXML in tail-label prediction.

The path of management of dispute cases of legal issues of webcasting bandwagon industry in the information age

Abstract In this paper, for the common live streaming with cargo dispute cases in the context of the information technology era, the live streaming with cargo industry dispute case retrieval algorithm based on case-based reasoning calculates the attribute similarity measure and weights of the dispute cases in order to realize the live streaming with cargo industry dispute case retrieval. Three thousand judicial documents with live streaming with cargo disputes as the cause of the case were crawled from the China Judicial Documents Network as the initial data for the study and the method of BIOES was used to annotate the initial data and formulate eight basic legal elements of live streaming with cargo dispute cases. The form of extracting legal elements of live streaming with goods industry is transformed into a multi-label text classification task, and the neural network modeling method combined with the attention mechanism is used to design the extraction model of legal elements of the live streaming with goods dispute cases, and then the live streaming with goods dispute cases on the Internet are analyzed by examples. The results show that the Acc value of this paper’s model is 0.973, the Loss value is 0.051, and the F1 value is 0.981, and on the whole, this paper’s model performs better compared with the other five models, i.e., it shows that this paper’s model can be more intelligent and accurate to collect the case element information under the field of live streaming with goods disputes. This study aims to contribute to the development of a new sales model of live streaming with goods benignly.

Multi-label text classification based on semantic-sensitive graph convolutional network

Multi-Label Text Classification (MLTC) is an important but challenging task in the field of natural language processing. In this paper, we propose a novel method, Semantic-sensitive Graph Convolutional Network (S-GCN), by simultaneously considering semantic and word-global associations. More specifically, we first leverage texts, words, and labels to construct a global graph, which helps mine the relevance between similar documents. Then we design and pre-train an encoder to initialize text nodes in the graph, from which the semantic features of documents are extracted. Next, we employ a graph convolutional network to classify text nodes, which can well fuse node information. Finally, we normalize the adjacency matrix and store hidden layer representations of word nodes, tackling the issue that conventional graph-based methods cannot predict texts that did not appear during training. We conduct experiments on three public datasets, AAPD, RMSC-V2, and Reuters-21578, and demonstrate the superiority of our model over the baselines on the MLTC task. Source code is available at https://github.com/sysu18364004/SGCN.