Multi-label Classification Algorithms Research Articles

Imbalance-Robust Multi-Label Self-Adjusting kNN

In the task of multi-label classification in data streams, instances arriving in real time need to be associated with multiple labels simultaneously. Various methods based on the k Nearest Neighbors algorithm have been proposed to address this task. However, these methods face limitations when dealing with imbalanced data streams, a problem that has received limited attention in existing works. To approach this gap, this paper introduces the Imbalance-Robust Multi-Label Self-Adjusting kNN (IRMLSAkNN), designed to tackle multi-label imbalanced data streams. IRMLSAkNN’s strength relies on maintaining relevant instances with imbalance labels by using a discarding mechanism that considers the imbalance ratio per label. On the other hand, it evaluates subwindows with an imbalance-aware measure to discard older instances that are lacking performance. We conducted statistical experiments on 32 benchmark data streams, evaluating IRMLSAkNN against eight multi-label classification algorithms using common accuracy-aware and imbalance-aware measures. The obtained results demonstrate that IRMLSAkNN consistently outperforms these algorithms in terms of predictive capacity and time cost across various levels of imbalance.

Multi-Label Sampling based on Label Imbalance Rate and Neighborhood Distribution

Existing multi-label classification algorithms often assume that label distribution in the training set is balanced, but practical datasets frequently exhibit significant label imbalance. This imbalance affects the learning and generalization performance of the classifiers. To address the problem of label imbalance in multi-label classification, this paper proposes a new synthetic oversampling algorithm, named Multi-Label Synthetic Oversampling based on Label Imbalance Rate and Neighborhood Distribution (MLSIN). This algorithm synthesizes new samples by considering both the imbalance rate of labels and the distribution of samples in their neighborhood, aiming to improve the classifiers performance on minority labels. The rest of this chapter first introduces the evaluation metrics for multi-label classification effectiveness. Then it defines and computes the degree of label imbalance, describes the calculation of imbalance weights, and proposes a sample type correction penalty strategy, detailing the algorithm's process for selecting base and auxiliary samples., and validates the proposed method on public datasets and summarizes the experimental results.

Semantic features analysis for biomedical lexical answer type prediction using ensemble learning approach

AbstractLexical answer type prediction is integral to biomedical question–answering systems. LAT prediction aims to predict the expected answer’s semantic type of a factoid or list-type biomedical question. It also aids in the answer processing stage of a QA system to assign a high score to the most relevant answers. Although considerable research efforts exist for LAT prediction in diverse domains, it remains a challenging biomedical problem. LAT prediction for the biomedical field is a multi-label classification problem, as one biomedical question might have more than one expected answer type. Achieving high performance on this task is challenging as biomedical questions have limited lexical features. One biomedical question must be assigned multiple labels given these limited lexical features. In this paper, we develop a novel feature set (lexical, noun concepts, verb concepts, protein–protein interactions, and biomedical entities) from these lexical features. Using ensemble learning with bagging, we use the label power set transformation technique to classify multi-label. We evaluate the integrity of our proposed methodology on the publicly available multi-label biomedical questions dataset (MLBioMedLAT) and compare it with twelve state-of-the-art multi-label classification algorithms. Our proposed method attains a micro-F1 score of 77%, outperforming the baseline model by 25.5%.

A Hybrid Principal Label Space Transformation-Based Binary Relevance Support Vector Machine and Q-Learning Algorithm for Multi-label Classification

A Hybrid Principal Label Space Transformation-Based Binary Relevance Support Vector Machine and Q-Learning Algorithm for Multi-label Classification

Learning Label Correlations for Multi-Label Online Passive Aggressive Classification Algorithm

Label correlations are an essential technique for data mining that solves the possible correlation problem between different labels in multi-label classification. Although this technique is widely used in multi-label classification problems, batch learning deals with most issues, which consumes a lot of time and space resources. Unlike traditional batch learning methods, online learning represents a promising family of efficient and scalable machine learning algorithms for large-scale datasets. However, existing online learning research has done little to consider correlations between labels. On the basis of existing research, this paper proposes a multi-label online learning algorithm based on label correlations by maximizing the interval between related labels and unrelated labels in multi-label samples. We evaluate the performance of the proposed algorithm on several public datasets. Experiments show the effectiveness of our algorithm.

Online Passive-Aggressive Multilabel Classification Algorithms.

Most existing multilabel classification methods are batch learning methods, which may suffer from expensive retraining costs when dealing with new incoming data. In order to overcome the drawbacks of batch learning, we develop a family of online multilabel classification algorithms, which can update the model instantly and efficiently, and make a timely online prediction when new data arrive. Our algorithms all take a closed-form update, which is obtained by solving a constrained optimization problem in each round of online learning. Label correlation is explicitly modeled in our optimization problem. The label thresholding function, an important component of our online classifier, can also be learned online. Our algorithms can be easily generalized to the nonlinear prediction cases using Mercer kernels. The worst case loss bounds for our algorithms are provided. The bounds are relative to the cumulative loss suffered by the best fixed predictive model that can be attained in hindsight. Finally, we corroborate the merits of our algorithms in both linear and nonlinear predictions on nine open multilabel benchmark datasets.

Exploiting local label correlation from sample perspective for multi-label classification via three-way decision theory

In multi-label classification, the expansion of output dimension seriously interferes learning performance, and even fails to build a joint prediction model. In order to restrain the proliferation of multi-label classifier’s hypothesis space, the current works focus on the application of global positive label correlation. However, the “black or white” mechanism ignore other possible forms of label correlation, such as negative or neutral correlation. By introducing the doctrine of the mean, three-way decision (3WD) theory provides a solution for in-depth research on local label correlation, and aims to handle the uncertainty of multi-label learning tasks. In this paper, a novel learning algorithm for multi-label joint classification, namely ML-3WD, is proposed by considering the 3WD label correlation from the perspective of samples. According to the weights of different features on any label, the comprehensive loss of each sample to three action strategies can be measured. Obviously, the 3WD rules for any label variable in multi-label output space is obtained. By aggregating the cutting thresholds between different labels, the division principles of 3WD label correlation are further established. Given any multi-label sample, the local fuzzy membership to co-occurrence or mutual state for label pair is examined based on kernelized fuzzy rough sets. The 3WD local label relevance of each sample is confirmed, that is, positive, negative or neutral. The global application strategy for multi-label classification is utilized to avoid over-fitting induced by local mining strategy. Based on the integral mean of the distribution of 3WD local label relevance in multi-label sample space, two different versions of empirical label relevance are constructed. By constraining the relative position between sub-separation hyperplanes, the 3WD label correlation distribution-based model for multi-label joint classification is designed. The experiment results on fifteen real world multi-label datasets reflect that our algorithm achieves good classification ability and versatility. The impact of core parameters on learning performance is also dissected.

Improved RAkEL's Fault Diagnosis Method for High-Speed Train Traction Transformer.

The traction system is very important to ensure the safe operation of high-speed trains, and the failure of the traction transformer is the most likely fault in the traction system. Fault diagnosis in actual work relies largely on manual experience. This paper proposes an improved RAkEL (Random k-Labelsets) algorithm for the fault diagnosis of high-speed train traction transformers. Firstly, this article starts from the large amount of "sleeping" fault maintenance data accumulated by the railway department, takes a single maintenance record as an instance, uses specific monitoring values to construct an instance vector, and uses the fault phenomena corresponding to the monitoring indicators as labels. Then, this paper improves the step of selecting k-labelsets in RAkEL, and extracts associated faults using the Relief algorithm. Finally, this paper excavates and uses the association rules between data and faults to identify traction transformer faults. The results showed that the improved RAkEL diagnostic method had a significant improvement in the evaluation indicators. Compared with other multi-label classification algorithms, including BR (Binary Relevance) and CLR (Calibrated Label Ranking), this method performs well on multiple evaluation indicators. It can further help engineers perform timely maintenance work in the future.

A Novel Learning-Based PLST Algorithm for Multi-Label Classification

In Multi-Label Classification (MLC), each data sample is characterized by multiple labels. In ML, there is no restriction on the number of classes a data sample could belong to. Various methods are proposed to construct a model to map data samples to the class labels. Transforming the MLC problem into several binary classification problems is the baseline approach, also called the binary relevance technique. As an alternative solution, the label powerset approach transforms the MLC problem into a multi-class classification problem by building a binary classifier for every label composition. However, in Ensemble Methods (EMs), multiple classifiers are consolidated to form a multi-label ensemble classifier. In this case, each classifier predicts a class label. Thereupon, the predicted class labels are joined using an ensemble method. This paper presents a neoteric EM classifier using the Ridge Regression-based Principal Label Space Transformation (RR-PLST). In the proposed model, the PLST utilizes SVD to discover correlations between the labels and data samples. Dimensionality reduction is a well-known technique to enhance the model's precision and diminish execution time. Consequently, the OPLS technique is recruited to reduce data dimensions in the proposed model. Moreover, in the proposed model, a Q-Learning method is used to amend the initially predicted labels. To demonstrate the effectiveness of the proposed model, vast experiments are conducted on the various widely used multi-label datasets. The proposed algorithm results are compared with similar algorithms by the AP, EM, HS, Macro, and Micro F1 metrics. The experimental results demonstrate the superiority of the proposed model.

Multi-label borderline oversampling technique

Class imbalance problem commonly exists in multi-label classification (MLC) tasks. It has non-negligible impacts on the classifier performance and has drawn extensive attention in recent years. Borderline oversampling has been widely used in single-label learning as a competitive technique in dealing with class imbalance. Nevertheless, the borderline samples in multi-label data sets (MLDs) have not been studied. Hence, this paper deeply discussed the borderline samples in MLDs and found they have different neighboring relationships with class borders, which makes their roles different in the classifier training. For that, they are divided into two types named the self-borderline samples and the cross-borderline samples. Further, a novel MLDs resampling approach called Multi-Label Borderline Oversampling Technique (MLBOTE) is proposed for multi-label imbalanced learning. MLBOTE identifies three types of seed samples, including interior, self-borderline, and cross-borderline samples, and different oversampling mechanisms are designed for them, respectively. Meanwhile, it regards not only the minority classes but also the classes suffering from one-vs-rest imbalance as those in need of oversampling. Experiments on eight data sets with nine MLC algorithms and three base classifiers are carried out to compare MLBOTE with some state-of-art MLDs resampling techniques. The results show MLBOTE outperforms other methods in various scenarios.

A deep learning method for foot-type classification using plantar pressure images.

Background: Flat foot deformity is a prevalent and challenging condition often leading to various clinical complications. Accurate identification of abnormal foot types is essential for appropriate interventions. Method: A dataset consisting of 1573 plantar pressure images from 125 individuals was collected. The performance of the You Only Look Once v5 (YOLO-v5) model, improved YOLO-v5 model, and multi-label classification model was evaluated for foot type identification using the collected images. A new dataset was also collected to verify and compare the models. Results: The multi-label classification algorithm based on ResNet-50 outperformed other algorithms. The improved YOLO-v5 model with Squeeze-and-Excitation (SE), the improved YOLO-v5 model with Convolutional Block Attention Module (CBAM), and the multilabel classification model based on ResNet-50 achieved an accuracy of 0.652, 0.717, and 0.826, respectively, which is significantly higher than those obtained using the ordinary plantar-pressure system and the standard YOLO-v5 model. Conclusion: These results indicate that the proposed DL-based multilabel classification model based on ResNet-50 is superior in flat foot type detection and can be used to evaluate the clinical rehabilitation status of patients with abnormal foot types and various foot pathologies when more data on patients with various diseases are available for training.

Multi-kernel learning for multi-label classification with local Rademacher complexity

Multi-label classification aims to construct prediction models from input space to output space for multi-label datesets. However, the feature space of multi-label dataset and the hypothesis space of classifier are always complex. In order to obtain high performance multi-kernel learning algorithms, the core problem is to design algorithms that compress both hypothesis space and feature space. In this paper, by combining the local Rademacher complexity and Hilbert-Schmidt independence criterion (HSIC), an effective multi-kernel learning algorithm for multi-label classification is proposed to compress the feature space and hypothesis space simultaneously. Based on the tail sum of eigenvalues of the integral operator corresponding to kernels, the upper bound of local Rademacher complexity of linear functions class in the feature space is determined. The Hilbert-Schmidt independence criterion is applied to maximize the correlation between the convex combination of the input kernel matrices and the ideal kernel matrix in the label space. Finally, the Laplace regularization model is built by controlling the tail sum of eigenvalues of the integral operator corresponding to kernels of this criterion. Therefore, the above process obtains a combined kernel, simultaneously by sorting the coefficients of the combined kernel, a feature selection algorithm can be constructed. On the basis, the combined kernel can also be used to design a new binary classifier for multi-label classification. The experimental results verify that our proposed multi-kernel learning algorithms can effectively compress the hypothesis space, while the feature selection algorithms can effectively compress the feature space.

Multi-Label Classification via Adaptive Resonance Theory-Based Clustering.

This paper proposes a multi-label classification algorithm capable of continual learning by applying an Adaptive Resonance Theory (ART)-based clustering algorithm and the Bayesian approach for label probability computation. The ART-based clustering algorithm adaptively and continually generates prototype nodes corresponding to given data, and the generated nodes are used as classifiers. The label probability computation independently counts the number of label appearances for each class and calculates the Bayesian probabilities. Thus, the label probability computation can cope with an increase in the number of labels. Experimental results with synthetic and real-world multi-label datasets show that the proposed algorithm has competitive classification performance to other well-known algorithms while realizing continual learning.

Multiple voice disorders in the same individual: Investigating handcrafted features, multi-label classification algorithms, and base-learners

Non-invasive acoustic analyses of voice disorders have been at the forefront of current biomedical research. Usual strategies, essentially based on machine learning (ML) algorithms, commonly classify a subject as being either healthy or pathologically-affected. Nevertheless, the latter state is not always a result of a sole laryngeal issue, i.e., multiple disorders might exist, demanding multi-label classification procedures for effective diagnoses. Consequently, the objective of this paper is to investigate the application of five multi-label classification methods based on problem transformation to play the role of base-learners, i.e., Label Powerset, Binary Relevance, Nested Stacking, Classifier Chains, and Dependent Binary Relevance with Random Forest (RF) and Support Vector Machine (SVM), in addition to a Deep Neural Network (DNN) from an algorithm adaptation method, to detect multiple voice disorders, i.e., Dysphonia, Laryngitis, Reinke’s Edema, Vox Senilis, and Central Laryngeal Motion Disorder. Receiving as input three handcrafted features, i.e., signal energy (SE), zero-crossing rates (ZCRs), and signal entropy (SH), which allow for interpretable descriptors in terms of speech analysis, production, and perception, we observed that the DNN-based approach powered with SE-based feature vectors presented the best values of F1-score among the tested methods, i.e., 0.943, as the averaged value from all the balancing scenarios, under Saarbrücken Voice Database (SVD) and considering 20% of balancing rate with Synthetic Minority Over-sampling Technique (SMOTE). Finally, our findings of most false negatives for laryngitis may explain the reason why its detection is a serious issue in speech technology. The results we report provide an original contribution, allowing for the consistent detection of multiple speech pathologies and advancing the state-of-the-art in the field of handcrafted acoustic-based non-invasive diagnosis of voice disorders.

A Bayesian Convolutional Neural Network Model with Uncertainty for Multi-label Text Classification on Mechanisms of Action (MoA) Prediction

With the development of scientific research techniques, drug discovery has shifted from the serendipitous approach of the past to more targeted models based on an understanding of the underlying biological mechanisms of disease. However, there are hundreds or more of mechanism of action (MoA) data in the known drugs, which makes this process faced with complicated multi-label classification of text data. Traditional multi-label text classification algorithms will increase the complexity of the model and reduce the accuracy as the number of labels increases. Although deep learning algorithms can solve the problem of model complexity, they are currently only suitable for processing image format data. To overcome these problems, this study proposes a multi-label classification method based on Bayesian deep learning, which can convert non-image data format into image data, making it suitable for Convolutional neural network algorithm requirements. Then in the PyTorch environment, the Bayesian deep learning algorithm and the EfficientNet convolutional neural network are perfectly combined using the BLiTZ library to construct the Bayesian convolutional neural network model which named BCNNM. Not only improves the classification efficiency, this method also solves the problem of imbalanced classification of multi-label data, and fully considers the uncertainty in the neural network. In the process of drug development, this method has important practical significance for processing the multi-label classification of MoA data.

Predicting and characterising persuasion strategies in misinformation content over social media based on the multi-label classification approach

Persuasion aims at affecting the audience’s attitude and behaviour through a series of messages containing persuasion strategies. In the context of misinformation spread, identifying the persuasion strategies is important in order to warn people to be aware of the analogous persuasion attempts in the future. In this work, we address the prediction of persuasion strategies in micro-blogging posts through a multi-label classification approach based on a variety of lexical and semantic features. We conduct our experiments using a set of well-known multi-label classification algorithms, including multi-label decision tree, multi-label k-nearest neighbours, multi-label random forest, binary relevance and classifier chains. The results show that the model incorporating classifier chains and XGBoost algorithm achieves the best subset accuracy of 0.779 and the highest macro F1-score of 0.847. In addition, we evaluated and compared the features’ importance for different persuasion strategies and analysed the major errors of miss-out prediction. The findings of this article provide a benchmark for the multi-label classification of persuasion strategies in micro-blogging posts and lead to a better understanding of different persuasion attempts contained in social media misinformation.

Multi-Label Classification Based on Associations

Associative classification (AC) has been shown to outperform other methods of single-label classification for over 20 years. In order to create rules that are both more precise and simpler to grasp, AC combines the rules of mining associations with the task of classification. However, the current state of knowledge and the views of various specialists indicate that the issue of multi-label classification (MLC) cannot be solved by any AC method. Since this is the case, adapting or using an AC algorithm to manage multi-label datasets is one of the most pressing issues. To solve the MLC issue, this research proposes modifying the classification based on associations (msCBA) method by extending its capabilities to consider more than one class label in the consequent of its rules and modifying its rules order procedure to fit the nature of the multi-label dataset. The proposed algorithm outperforms several other MLC algorithms from various learning techniques across a variety of performance measuresand using six datasets with different domains. The main findings of this research are the significance of utilizing the local dependencies among labels compared to global dependencies, and the important rule of AC in solving the problem of MLC.

Distributed Online Multi-Label Learning with Privacy Protection in Internet of Things

With the widespread use of end devices, online multi-label learning has become popular as the data generated by users using the Internet of Things devices have become huge and rapidly updated. However, in many scenarios, the user data are often generated in a geographically distributed manner that is often inefficient and difficult to centralize for training machine learning models. At the same time, current mainstream distributed learning algorithms always require a centralized server to aggregate data from distributed nodes, which inevitably causes risks to the privacy of users. To overcome this issue, we propose a distributed approach for multi-label classification, which trains the models in distributed computing nodes without sharing the source data from each node. In our proposed method, each node trains its model with its local online data while it also learns from the neighbour nodes without transferring the training data. As a result, our proposed method achieved the online distributed approach for multi-label classification without losing performance when taking existing centralized algorithms as a reference. Experiments show that our algorithm outperforms the centralized online multi-label classification algorithm in F1 score, being 0.0776 higher in macro F1 score and 0.1471 higher for micro F1 score on average. However, for the Hamming loss, both algorithms beat each other on some datasets, and our proposed algorithm loses 0.005 compared to the centralized approach on average, which can be neglected. Furthermore, the size of the network and the degree of connectivity are not factors that affect the performance of this distributed online multi-label learning algorithm.

An Improved Multilabel k-Nearest Neighbor Algorithm Based on Value and Weight

Multilabel data share important features, including label imbalance, which has a significant influence on the performance of classifiers. Because of this problem, a widely used multilabel classification algorithm, the multilabel k-nearest neighbor (ML-kNN) algorithm, has poor performance on imbalanced multilabel data. To address this problem, this study proposes an improved ML-kNN algorithm based on value and weight. In this improved algorithm, labels are divided into minority and majority, and different strategies are adopted for different labels. By considering the label of latent information carried by the nearest neighbors, a value calculation method is proposed and used to directly classify majority labels. Additionally, to address the misclassification problem caused by a lack of nearest neighbor information for minority labels, weight calculation is proposed. The proposed weight calculation converts distance information with and without label sets in the nearest neighbors into weights. The experimental results on multilabel datasets from different benchmarks demonstrate the performance of the algorithm, especially for datasets with high imbalance. Different evaluation metrics show that the results are improved by approximately 2–10%. The verified algorithm could be applied to a multilabel classification of various fields involving label imbalance, such as drug molecule identification, building identification, and text categorization.

Robust fused hypergraph neural networks for multi-label classification

Deep neural networks have been adopted in multi-label classification for their excellent performance, however, existing methods fail to comprehensively utilize the high-order correlations between instances and the high-order correlations between labels, and these methods are difficult to deal with label noise effectively. We propose a novel end-to-end deep framework named Robust Fused Hypergraph Neural Networks for Multi-Label Classification (RFHNN), which can effectively utilize the two kinds of high-order correlations and adopt them to mitigate the impact of label noise. In RFHNN, Hypergraph Neural Networks (HNNs) are adopted to mine and utilize the high-order correlations of the instances in the feature space and the label space respectively. The high-order correlations of the instances can not only improve the accuracy of the classification and the discrimination of the proposed model, but also lay the foundation for the subsequent noise correction module. Meanwhile, a hypergraph construction method based on the Apriori algorithm is proposed to realize Hypergraph Neural Networks (HNNs), which can mine robust second-order and high-order label correlations effectively. Effective classifiers are learned based on the correlations between the labels, which will not only improve the accuracy of the model, but can also enhance the subsequent noise correction module. In addition, we have designed a noise correction module in the networks. With the help of the high-order correlations among the instances and the effective classifier, the framework can effectively correct the noise and improve the robustness of the model. Extensive experimental results on datasets demonstrate that our proposed approach is better than the state-of-the-art multi-label classification algorithms. When dealing with the multi-label training datasets with noise in the label space, our proposed method also has great performance.