Multi-label Learning Algorithms Research Articles

Multi-label Feature Selection based on Label-specific features and Manifold Learning

Each instance in multi-label data is associated with multiple labels, and there are irrelevant or redundant features in its feature space, which leads to the performance degradation of multi-label learning algorithms. Multi-label feature selection selects representative features from the feature space to improve the accuracy of the model. Due to the high cost of labels and the difficulty of data collection, there will be some missing labels in the data set, which affects the accuracy of feature selection. To solve this problem, a multi-label feature selection algorithm based on label-specific features and manifold learning is proposed. The algorithm uses the linear relationship between the features and labels in known label samples to build a linear regression model for learning label-specific features. By using the nonlinear relation between instances and the nonlinear relation between features, we can precisely learn the label-specific features. We use the Laplacian feature mapping method to construct the instance manifold model and the feature manifold model, which are also used as the regular term constraint weight matrix. The final model can not only complete the missing labels, but also select sparse and representative features. The feature selection is carried out by analyzing the weight of the feature given by the final model. Experiments were conducted to verify the effectiveness of the proposed algorithm under different label deletion rates on four evaluation indexes.

Learning Label Correlations for Multi-Label Online Passive Aggressive Classification Algorithm

Label correlations are an essential technique for data mining that solves the possible correlation problem between different labels in multi-label classification. Although this technique is widely used in multi-label classification problems, batch learning deals with most issues, which consumes a lot of time and space resources. Unlike traditional batch learning methods, online learning represents a promising family of efficient and scalable machine learning algorithms for large-scale datasets. However, existing online learning research has done little to consider correlations between labels. On the basis of existing research, this paper proposes a multi-label online learning algorithm based on label correlations by maximizing the interval between related labels and unrelated labels in multi-label samples. We evaluate the performance of the proposed algorithm on several public datasets. Experiments show the effectiveness of our algorithm.

Research on Multi-Label Semi-Supervised Learning Algorithm Based on Dual Selection Criteria

Research on Multi-Label Semi-Supervised Learning Algorithm Based on Dual Selection Criteria

CC<i>k</i>EL: Compensation-based correlated <i>k</i>-labelsets for classifying imbalanced multi-label data

<abstract> <p>Imbalanced data distribution and label correlation are two intrinsic characteristics of multi-label data. This occurs because in this type of data, instances associated with certain labels may be sparse, and some labels may be associated with others, posing a challenge for traditional machine learning techniques. To simultaneously adapt imbalanced data distribution and label correlation, this study proposed a novel algorithm called compensation-based correlated <italic>k</italic>-labelsets (CC<italic>k</italic>EL). First, for each label, the CC<italic>k</italic>EL selects the <italic>k</italic>-1 strongest correlated labels in the label space to constitute multiple correlated <italic>k</italic>-labelsets; this improves its efficiency in comparison with the random <italic>k</italic>-labelsets (RA<italic>k</italic>EL) algorithm. Then, the CC<italic>k</italic>EL transforms each <italic>k</italic>-labelset into a multiclass issue. Finally, it uses a fast decision output compensation strategy to address class imbalance in the decoded multi-label decision space. We compared the performance of the proposed CC<italic>k</italic>EL algorithm with that of multiple popular multi-label imbalance learning algorithms on 10 benchmark multi-label datasets, and the results show its effectiveness and superiority.</p> </abstract>

A partition-based problem transformation algorithm for classifying imbalanced multi-label data

Multi-label learning has garnered much research interest due to its wide range of real-world applications. Many multi-label learning methods have been proposed; however, few have addressed the class imbalance problem existing in multi-label data. Even though some studies have taken this issue into account, most of them have ignored the label correlations or only considered random correlations between them. In this study, we propose a novel partition-based imbalanced multi-label learning algorithm, named Multi-label Learning based on Hierarchical Clustering (MLHC), to tackle this problem. MLHC first carries out hierarchical clustering on the original label space to divide it into several disconnected subspaces, each of which contains several labels that are strongly correlated with each other. Then, for each label subspace, we use the problem transformation strategy to convert it into a multi-class problem by binary coding. Any multi-class imbalance learning algorithm can be applied to the transformed multi-class data. Finally, the classification results will be decoded to retrieve the corresponding label subspace, and all label subspace results are combined to show the predicted label vector in the original label space. We conducted experiments not only on thirteen benchmark multi-label datasets but also carried out them on XJTU-SY which is a multi-label engineering application dataset, and the results indicated that our proposed MLHC learning algorithm outperforms several state-of-the-art class imbalance multi-label learning algorithms, demonstrating the effectiveness and necessity of discovering label correlations and transforming the original imbalanced multi-label learning problem into multiple strongly correlated multi-class imbalanced learning problems.

A multi-instance multi-label learning algorithm based on radial basis functions and multi-objective particle swarm optimization

Radial basis function (RBF) neural networks for Multi-Instance Multi-Label (MIML) directly can exploit the connections between instances and labels so that they can preserve useful prior information, but they only adopt Gaussian radial basis function as their RBF whose parameters are difficult to determine. In this paper, parameters can be obtained by multi-objective optimization methods with multi performance measures treated as objectives, specifically, parameter estimation of different RBFs by an improved multi-objective particle swarm optimization (MOPSO) is proposed where Recall rate and Precision rate are chosen to obtain the most desirable Pareto optimal solution set. Furthermore, share-learning factor is proposed to modify the particle velocity in standard MOPSO to improve the global search ability and group cooperative ability. It is experimentally demonstrated that the proposed method can estimate the reliable parameters of different RBFs, and it is also very competitive with the state of art MIML methods.

PLVI-CE: a multi-label active learning algorithm with simultaneously considering uncertainty and diversity

PLVI-CE: a multi-label active learning algorithm with simultaneously considering uncertainty and diversity

ECC + +: An algorithm family based on ensemble of classifier chains for classifying imbalanced multi-label data

Multi-label learning has attracted a great deal of research interests as it has a wide range of real-world applications. Although many multi-label learning methods have been proposed, very few of them have addressed the problem of class imbalance distribution in multi-label data. Moreover, most of the existing class imbalance multi-label learning algorithms only focus on solving the class imbalance problem, without taking into account the correlations among labels. To address these issues simultaneously, we propose to combine the well-known ensemble of classifier chain (ECC) algorithm with various binary-class imbalance learning techniques such as sampling, cost-sensitive learning, and threshold moving. This approach creates a new algorithm family called ECC++, designed specifically for class imbalance multi-label learning. ECC is already an excellent ensemble high-order binary relevance multi-label learning algorithm that is well-suited to exploiting correlations among labels. Combining it with binary-class imbalance learning techniques enables each link in a classifier chain (CC) to overcome the negative effect of skewed data distribution. ECC++ is a dynamic algorithm family that can be extended arbitrarily by applying any new binary-class imbalance learning techniques. To demonstrate the effectiveness and superiority of the proposed ECC++ algorithm family, we developed several ECC++ family members using some popular binary-class imbalance learning techniques. We then compared them with several state-of-the-art class imbalance multi-label learning algorithms on twelve benchmark and four real-world multi-label datasets. Our experimental results showed the effectiveness and superiority of the proposed ECC++ algorithm family over existing class imbalance multi-label learning algorithms. In conclusion, the proposed ECC++ algorithm family combines the strengths of the well-established ECC algorithm and binary-class imbalance learning techniques, resulting in a superior methodology for class imbalance multi-label learning.

Extending version-space theory to multi-label active learning with imbalanced data

Version space, defined as the subset of the hypothesis space consistent with the training samples, is an important concept in supervised learning. It has been successfully applied for evaluating the informativeness of unlabeled samples in traditional single-label active learning. Specifically, the most inconsistent samples among the version space members can reduce the size of the version space as fast as possible, these samples are given high priority for domain expert annotation, thereby the learner can construct a high-performance classifier by labeling as few samples as possible. We point out that the concept of version space has not been extended to multi-label environments yet, which hinders its application in multi-label active learning. This paper makes an attempt to extend the version space theory from single-label scenario to multi-label scenario, builds up a spatial structure for the multi-label version space, generalizes it from finite case to infinite case, puts forward a simplified representation for it and accordingly proposes a new multi-label active learning algorithm. Moreover, considering the imbalance issue in multi-label data, the algorithm is further improved by allocating different annotation numbers to the labels. Experimental comparisons verify the feasibility and effectiveness of the proposed methods.

Novelty detection for multi-label stream classification under extreme verification latency

Multi-Label Stream Classification (MLSC) is the classification streaming examples into multiple classes simultaneously. Since new classes may emerge during the streaming process (concept evolution) and known classes may change over time (concept drift) it is challenging task. In real situations, concept drift and concept evolution occur in scenarios where the actual labels of arriving examples are never available; hence it is impractical to update decision models in a supervised fashion. This is known as Extreme Verification Latency, a topic that has not been well investigated in MLSC literature. This paper proposes a new method called MultI-label learNing Algorithm for Data Streams with Binary Relevance transformation (MINAS-BR), integrated with a Novelty Detection (ND) procedure for detecting concept evolution and concept drift, updating the model in an unsupervised fashion. Furthermore, since the label space is not static, we propose a new evaluation methodology for MLSC under extreme verification latency. Experiments over synthetic and real-world data sets with different concept drift and concept evolution scenarios confirmed the strategies employed in the MINAS-BR and presented relevant advances for handling streaming multi-label data.

Distributed Online Multi-Label Learning with Privacy Protection in Internet of Things

With the widespread use of end devices, online multi-label learning has become popular as the data generated by users using the Internet of Things devices have become huge and rapidly updated. However, in many scenarios, the user data are often generated in a geographically distributed manner that is often inefficient and difficult to centralize for training machine learning models. At the same time, current mainstream distributed learning algorithms always require a centralized server to aggregate data from distributed nodes, which inevitably causes risks to the privacy of users. To overcome this issue, we propose a distributed approach for multi-label classification, which trains the models in distributed computing nodes without sharing the source data from each node. In our proposed method, each node trains its model with its local online data while it also learns from the neighbour nodes without transferring the training data. As a result, our proposed method achieved the online distributed approach for multi-label classification without losing performance when taking existing centralized algorithms as a reference. Experiments show that our algorithm outperforms the centralized online multi-label classification algorithm in F1 score, being 0.0776 higher in macro F1 score and 0.1471 higher for micro F1 score on average. However, for the Hamming loss, both algorithms beat each other on some datasets, and our proposed algorithm loses 0.005 compared to the centralized approach on average, which can be neglected. Furthermore, the size of the network and the degree of connectivity are not factors that affect the performance of this distributed online multi-label learning algorithm.

MIML-GAN: A GAN-Based Algorithm for Multi-Instance Multi-Label Learning on Overlapping Signal Waveform Recognition

Existing studies for automatic waveform recognition of overlapping signals have mostly been conducted in a supervised manner. Although demonstrating superior performance in recent years, supervised methods rely heavily on sufficient labeled samples, rely but the acquisition of annotated data is expensive, time-consuming, and sometimes rely infeasible. This shortage drives the need for semi-supervised learning methods, where the unlabeled samples can be fully exploited in the training stage. In addition, multi-instance multi-label (MIML) learning is essentially another weakly supervised learning protocol, and precisely fits the form of the time-frequency images TFIs obtained from the transformation of overlapping signals. In this paper, delving into the MIML learning problem, we leverage the advantage of adversarial training to formulate an effective algorithm MIML-GAN, which is tailored to the MIML problem of overlapping signal waveform recognition. After feeding the TFIs into the network, MIML-GAN approximates the distribution of the training data using the adversarial learning principle. Subsequently, the baglevel prediction can be derived from the instance-level prediction upon the MIML discriminator through adaptive threshold calibration. Specifically, we elaborately studied the global optimality of the MIML-GAN objective function, and extensive simulations are carried out with overlapping signal dataset, validating the ascendancy of the proposed method. Comparative experiments demonstrate that the proposed algorithm possesses promising feature representation capability, and outperforms the existing semi-supervised and supervised signal waveform recognition approaches.

Technique recommendation to recover missing labels by performance analysis of Multi-Label Machine Learning Algorithms

With today’s technological growth Machine learning is revalued as a tool that provides the ability in automating systems. The machines learn from experience based on the available datasets with no intervention from humans. Here in this research, we consider a variant of the classification, multi-label learning where each instance belongs to more than one label simultaneously. In contrast to other classification tasks, there are a number of challenges, the most significant is to figure out the class when the labels are absent. We compared various multi-label methods from algorithm adaptation and problem transformation perspectives. We applied the algorithms to eight datasets. Problem transformation algorithms are applied to analyze label correlation in a positive sense and then both in a positive and negative sense. We contribute by recommending the technique to address the issue of missing labels by either relying solely on label correlation or by combining it with data locality. Experimental results on multiple benchmark data sets demonstrate that MLLCRS-ML outperforms other cutting-edge techniques.

Fuzzy MLKNN in Credit User Portrait

Aiming at the problems of subjective enhancement caused by the discretization of credit data and the lack of a multi-dimensional portrait of credit users in the current credit data research, this paper proposes an improved Fuzzy MLKNN multi-label learning algorithm based on MLKNN. On the one hand, the subjectivity of credit data after discretization is weakened by introducing intuitionistic fuzzy numbers. On the other hand, the algorithm is improved by using the corresponding fuzzy Euclidean distance to realize the multi-label portrait of credit users. The experimental results show that Fuzzy MLKNN performs significantly better than MLKNN on credit data and has the most significant improvement on One Error.

Weight matrix sharing for multi-label learning

Multi-label learning on real-world data is a challenging task due to sparse labels, missing labels, and sparse structures. Some existing approaches are effective in addressing the former two issues. In this paper, we propose a shared weight matrix with low-rank and sparse regularization for multi-label learning (2SML) algorithm to address the issues simultaneously. First, two explicit correlation matrices are constructed from the feature matrix and label matrix. Second, we select informative labels by instance representativeness to learn implicit correlations. Third, a feature manifold and label manifold are employed to guide the shared weight learning process. Extensive experiments are undertaken on multiple benchmark datasets with and without missing labels. The results show that the proposed method outperforms the state-of-the-art methods.

A Novel Probabilistic Label Enhancement Algorithm for Multi-Label Distribution Learning

We propose a novel probabilistic label enhancement algorithm, called PLEA, to solve challenging label distribution learning (LDL) for multi-label classification problems. We adopt the well-known maximum entropy model based label distribution learner. However, unlike the existing LDL algorithms based on the maximum entropy model, we propose to use manifold learning to enhance the label distribution learner. Specifically, the supervised information in the label manifold is utilized in the feature manifold space construction to improve the accuracy of feature extraction, while dramatically reducing the feature dimension. Then the robust linear regression is employed to estimate the label distributions associated with the extracted reduced-dimension features. Using the enhanced reduced-dimension features and their associated estimated label distributions in the maximum entropy model, the unknown true label distributions can be estimated more accurately, while imposing considerably lower computational complexity. We evaluate the proposed PLEA method on a wide-range artificial and high-dimensional real-world datasets. Experimental results obtained demonstrate that our proposed PLEA method has advantages in LDL accuracy and runtime performance, compared to the latest multi-label LDL approaches. The results also show that our PLEA compares favourably with the state-of-the-arts multi-label learning algorithms for classification tasks.

LAMB: A novel algorithm of label collaboration based multi-label learning

Exploiting label correlation is crucially important in multi-label learning, where each instance is associated with multiple labels simultaneously. Multi-label learning is more complex than single-label learning for that the labels tend to be correlated. Traditional multi-label learning algorithms learn independent classifiers for each label and employ ranking or threshold on the classification results. Most existing methods take label correlation as prior knowledge, which have worked well, but they failed to make full use of label dependency. As a result, the real relationship among labels may not be correctly characterized and the final prediction is not explicitly correlated. To address these problems, we propose a novel high-order multi-label learning algorithm of Label collAboration based Multi-laBel learning (LAMB). With regard to each label, LAMB utilizes collaboration between its own prediction and the prediction of other labels. Extensive experiments on various datasets demonstrate that our proposed LAMB algorithm achieves superior performance over existing state-of-the-art algorithms. In addition, one real-world dataset of channelrhodopsins chimeras is assessed, which would be of great value as pre-screen for membrane proteins function.

Automated Multi Label Classification on Fertilizer Themed Patent Documents in Indonesia


 Patent literature research has a high scientific value for the industrial, commercial, legal, and policymaking communities. Therefore, patent analysis has become crucial. Patent topic classification is an important process in patent topic modeling analysis. However, the classification process is time-consuming and expensive, as it is usually carried out manually by an expert. Moreover, a patent document may be categorised in more than one category or label, further complicating the task. As the number of patent documents submitted increases, creating an automated patent classification system that yields accurate results becomes increasingly critical. Therefore, in this paper, we analyse the performance of two algorithms with regard to multi-label classification in patent documents: multi-label k-nearest neighbor (ML-KNN) and classifier chain k-nearest neighbor (CC-KNN), combined with latent Dirichlet allocation (LDA). These two methods have a considerable advantage in handling the continuously updated dataset; they also exhibit superior performance compared to other multi-label learning algorithms. This study also compares these two algorithms with the term frequency (TF)-weighting measure. The optimal value obtained is based on the following evaluation parameters: micro F1, accuracy, Hamming loss, and one error. The result shows that the ML-KNN method is better than the CC-KNN method and that the multi-label classification based on topics (patent LDA) is better than the TF-weighting technique.

In-Depth Research and Analysis of Multilabel Learning Algorithm

Multilabel learning (MLL), as a hot topic in the field of machine learning, has attracted wide attention from many scholars due to its ability to express output space polysemy. In recent years, a large number of achievements about MLL have emerged. Among these achievements, there are several typical issues worthy of attention. Firstly, the correlation among labels plays a key role in improving MLL model training process. Many MLL algorithms try to fully and effectively use the correlation among labels to improve the performance. Secondly, existing MLL evaluation metrics, which is different from those in binary classification, often reflects the generalization performance of MLL classifiers in some aspects. How to choose metrics in algorithms to improve their generalization performance and fairness is another issue that should be concerned. Thirdly, in many practical MLL applications, there are many unlabeled instances due to their labeling cost in training datasets. How to use the wealth information contained in the correlation among unlabeled instances may contribute to reducing of the labeling cost in MLL and improving performance. Fourthly, labels assigned to instances may not be equally descriptive in many applications. How to describe the importance of each label in output space to an instance has become one of research points that many scholars have paid attention to in recent years. This paper reviews the MLL-related research results of correlation among labels, evaluation metric, multilabel semisupervised learning, and label distribution learning (LDL) from a theoretical and algorithmic perspective. Finally, the related research work on MLL is summarized and discussed.

Stable Label-Specific Features Generation for Multi-Label Learning via Mixture-Based Clustering Ensemble

Multi-label learning deals with objects associated with multiple class labels, and aims to induce a predictive model which can assign a set of relevant class labels for an unseen instance. Since each class might possess its own characteristics, the strategy of extracting label-specific features has been widely employed to improve the discrimination process in multi-label learning, where the predictive model is induced based on tailored features specific to each class label instead of the identical instance representations. As a representative approach, LIFT generates label-specific features by conducting clustering analysis. However, its performance may be degraded due to the inherent instability of the single clustering algorithm. To improve this, a novel multi-label learning approach named SENCE (stable label-Specific features gENeration for multi-label learning via mixture-based Clustering Ensemble) is proposed, which stabilizes the generation process of label-specific features via clustering ensemble techniques. Specifically, more stable clustering results are obtained by firstly augmenting the original instance representation with cluster assignments from base clusters and then fitting a mixture model via the expectation-maximization (EM) algorithm. Extensive experiments on eighteen benchmark data sets show that SENCE performs better than LIFT and other well-established multi-label learning algorithms.