Multi-label Learning Methods Research Articles

Learning from multi-label data with interactivity constraints: An extensive experimental study

Interactive classification aims at introducing user preferences in the learning process to produce individualized outcomes more adapted to each user’s behavior than the fully automatic approaches. The current interactive classification systems generally adopt a single-label classification paradigm that constrains items to span one label at a time and consequently limit the user’s expressiveness while he/she interacts with data that are inherently multi-label. Moreover, the experimental evaluations are mainly subjective and closely depend on the targeted use cases and the interface characteristics. This paper presents the first extensive study of the impact of the interactivity constraints on the performances of a large set of twelve well-established multi-label learning methods. We restrict ourselves to the evaluation of the classifier predictive and time-computation performances while the number of training examples regularly increases and we focus on the beginning of the classification task where few examples are available. The classifier performances are evaluated with an experimental protocol independent of any implementation environment on a set of twelve multi-label benchmarks of various sizes from different domains. Our comparison shows that four classifiers can be distinguished for the prediction quality: RF-PCT (Random Forest of Predictive Clustering Trees, (Kocev, 2011)), EBR (Ensemble of Binary Relevance, (Read et al., 2011)), CLR (Calibrated Label Ranking, (Fürnkranz et al., 2008)) and MLkNN (Multi-label kNN, (Zhang and Zhou, 2007)) with an advantage for the first two ensemble classifiers. Moreover, only RF-PCT competes with the fastest classifiers and is therefore considered as the most promising classifier for an interactive multi-label learning system.

Experimental Comparison of Methods for Multi-label Classification in different Application Domains

world applications have begun to adopt the multi-label paradigm. The multi-label classification implies an extra dimension because each example might be associated with multiple labels (different possible classes), as opposed to a single class or label (binary, multi-class) classification. And with increasing number of possible multi-label applications in most ecosystems, there is little effort in comparing the different multi-label methods in different domains. Hence, there is need for a comprehensive overview of methods and metrics. In this study, we experimentally evaluate 11 methods for multi-label learning using 6 evaluation measures over seven benchmark datasets. The results of the experimental comparison revealed that the best performing method for both the example- based evaluation measures and the label-based evaluation measures are ECC on all measures when using C4.5 tree classifier as a single-label base learner. Keywords-Label classification, Multi-Label learning, Data Mining

Complex video event detection via pairwise fusion of trajectory and multi-label hypergraphs

The video data is rich in motion event information. Detecting complex events and analyzing the inherent high-level semantics information have been a hot topic in video analysis and understanding. Detecting complex events in the video involves detecting multiple semantic concepts, describing features of multiple moving targets and discovering the relationship between low-level features and high-level semantic concepts. It can extract semantic concept patterns from various video features and original video data, thus bridging the semantic gap. Based on the hypergraph theory, this paper proposes to construct trajectory and multi-label hypergraphs considering the features of moving targets. The two hypergraphs are fused to detect complex events. The experimental results show that in comparison with other methods including ordinary graph based method and hypergraph based multi-label semi-supervised learning method, our method achieves better average precision and average recall when detecting complex events.

Multi-label learning with missing labels for image annotation and facial action unit recognition

Many problems in computer vision, such as image annotation, can be formulated as multi-label learning problems. It is typically assumed that the complete label assignment for each training image is available. However, this is often not the case in practice, as many training images may only be annotated with a partial set of labels, either due to the intensive effort to obtain the fully labeled training set or the intrinsic ambiguities among the classes. In this work, we propose a method for multi-label learning that explicitly handles missing labels. We train classifiers with the multi-label with missing labels (MLML) learning framework by enforcing the consistency between the predicted labels and the provided labels as well as the local smoothness among the label assignments. Experiments on three benchmark data sets in image annotation and one benchmark data set in facial action unit recognition demonstrate the improved performance of our method in comparison of several state-of-the-art methods.

Predicting Protein Function Using Multiple Kernels.

High-throughput experimental techniques provide a wide variety of heterogeneous proteomic data sources. To exploit the information spread across multiple sources for protein function prediction, these data sources are transformed into kernels and then integrated into a composite kernel. Several methods first optimize the weights on these kernels to produce a composite kernel, and then train a classifier on the composite kernel. As such, these approaches result in an optimal composite kernel, but not necessarily in an optimal classifier. On the other hand, some approaches optimize the loss of binary classifiers and learn weights for the different kernels iteratively. For multi-class or multi-label data, these methods have to solve the problem of optimizing weights on these kernels for each of the labels, which are computationally expensive and ignore the correlation among labels. In this paper, we propose a method called Predicting Protein Function using Multiple Kernels (ProMK). ProMK iteratively optimizes the phases of learning optimal weights and reduces the empirical loss of multi-label classifier for each of the labels simultaneously. ProMK can integrate kernels selectively and downgrade the weights on noisy kernels. We investigate the performance of ProMK on several publicly available protein function prediction benchmarks and synthetic datasets. We show that the proposed approach performs better than previously proposed protein function prediction approaches that integrate multiple data sources and multi-label multiple kernel learning methods. The codes of our proposed method are available at https://sites.google.com/site/guoxian85/promk.

Analysis of the diagnostic consistency of Chinese medicine specialists in cardiovascular disease cases and syndrome identification based on the relevant feature for each label learning method.

To analyze the diagnostic consistency of Chinese medicine (CM) specialists in patients with cardiovascular disease and to study syndrome classification and identification based on the multi-label learning method. Using self-developed CM clinical scales to collect cases, inquiry information, complexity, tongue manifestation and pulse manifestation were assessed. The number of cases collected was 2,218. Firstly, each case was differentiated by two CM specialists according to the same diagnostic criteria. The consistency of the diagnosis based on Cohen's Kappa coefficient was analyzed. Secondly, take the same diagnosis syndromes of two specialists as the results of the cases. According to injury information in the CM scale "yes" or "no" was assigned "1" or "0", and according to the syndrome type in each case "yes" or "no" was assigned "1" or "0". CM information data on cardiovascular disease cases were established. We studied CM syndrome classification and identification based on the relevant feature for each label (REAL) learning method, and the diagnostic rate of the syndrome was studied using the REAL method when the number of features selected was 5, 10, 15, 20, 30, 50, 70, and 100, respectively. The syndromes with good diagnostic consistency were Heart (Xin)-qi deficiency, Heart-yang deficiency, Heart-yin deficiency, phlegm, stagnation of blood and stagnation of qi. Syndromes with poor diagnostic consistency were heart-blood deficiency and blood deficiency of Heart and Liver (Gan). The highest diagnostic rates using the REAL method were Heart-yang deficiency followed by Heart-qi deficiency. A different number of features, such as 5, 10, 15, 20, 30, 40, 50, 70, and 100, respectively, were selected and the diagnostic accuracy based on five features showed the highest diagnostic accuracy. The top five features which had a strong correlation with the syndromes were in accordance with the CM theory. CM syndrome differentiation is strongly subjective and it is difficult to obtain good diagnostic consistency. The REAL method fully considers the relationship between syndrome types and injury symptoms, and is suitable for the establishment of models for CM syndrome classification and identification. This method can probably provide the prerequisite for objectivity and standardization of CM differentiation.

Generalized <italic>k</italic>-Labelsets Ensemble for Multi-Label and Cost-Sensitive Classification

Label powerset (LP) method is one category of multi-label learning algorithm. This paper presents a basis expansions model for multi-label classification, where a basis function is an LP classifier trained on a random k-labelset. The expansion coefficients are learned to minimize the global error between the prediction and the ground truth. We derive an analytic solution to learn the coefficients efficiently. We further extend this model to handle the cost-sensitive multi-label classification problem, and apply it in social tagging to handle the issue of the noisy training set by treating the tag counts as the misclassification costs. We have conducted experiments on several benchmark datasets and compared our method with other state-of-the-art multi-label learning methods. Experimental results on both multi-label classification and cost-sensitive social tagging demonstrate that our method has better performance than other methods.

Boosted MIML method for weakly-supervised image semantic segmentation

Weakly-supervised image semantic segmentation aims to segment images into semantically consistent regions with only image-level labels are available, and is of great significance for fine-grained image analysis, retrieval and other possible applications. In this paper, we propose a Boosted Multi-Instance Multi-Label (BMIML) learning method to address this problem, the approach is built upon the following principles. We formulate the image semantic segmentation task as a MIML problem under the boosting framework, where the goal is to simultaneously split the superpixels obtained from over-segmented images into groups and train one classifier for each group. In the method, a loss function which uses the image-level labels as weakly-supervised constraints, is employed to suitable semantic labels to these classifiers. At the same time a contextual loss term is also combined to reduce the ambiguities existing in the training data. In each boosting round, we introduce an "objectness" measure to jointly reweigh the instances, in order to overcome the disturbance from highly frequent background superpixels. We demonstrate that BMIML outperforms the state-of-the-arts for weakly-supervised semantic segmentation on two widely used datasets, i.e., MSRC and LabelMe.

Protein Function Prediction with Incomplete Annotations.

Automated protein function prediction is one of the grand challenges in computational biology. Multi-label learning is widely used to predict functions of proteins. Most of multi-label learning methods make prediction for unlabeled proteins under the assumption that the labeled proteins are completely annotated, i.e., without any missing functions. However, in practice, we may have a subset of the ground-truth functions for a protein, and whether the protein has other functions is unknown. To predict protein functions with incomplete annotations, we propose a Protein Function Prediction method with Weak-label Learning (ProWL) and its variant ProWL-IF. Both ProWL and ProWL-IF can replenish the missing functions of proteins. In addition, ProWL-IF makes use of the knowledge that a protein cannot have certain functions, which can further boost the performance of protein function prediction. Our experimental results on protein-protein interaction networks and gene expression benchmarks validate the effectiveness of both ProWL and ProWL-IF.

단백질의 세포내 위치 예측을 위한 다중레이블 분류 방법의 성능 비교

단백질이 존재하는 세포내의 다중 위치를 정확하게 예측하기 위하여 다중레이블 학습 방법을 광범위하게 비교한다. 이를 위하여 다중레이블 분류의 접근 방법인 알고리즘 적응, 문제 변환, 메타 학습의 여러 방법을 비교 평가한다. 다양한 관점에서 다중레이블 분류 방법의 특성을 평가하기 위하여 12가지 평가 척도를 사용하였고, 최적의 성능을 보이는 방법을 찾기 위하여 새로운 요약 척도를 사용하였다. 비교 실험 결과, 흔하지 않은 다중레이블 집합을 가지치기 하는 멱집합 방법과, 관련 레이블들을 추가된 특징으로 나타내는 분류기-체인 방법의 성능이 높았다. 또한, 이들 방법들로 구성된 여러 개의 분류기를 조합하면 더욱 성능이 향상되었다. 즉, 세포내 위치간의 연관관계를 사용하는 것이 예측에 효과적인데, 특정 생물학적 기능을 수행하는 단백질의 세포내 위치들의 관계는 독립적이지 않고 서로 관련되어 있기 때문이라 판단된다. This paper presents an extensive experimental comparison of a variety of multi-label learning methods for the accurate prediction of subcellular localization of proteins which simultaneously exist at multiple subcellular locations. We compared several methods from three categories of multi-label classification algorithms: algorithm adaptation, problem transformation, and meta learning. Experimental results are analyzed using 12 multi-label evaluation measures to assess the behavior of the methods from a variety of view-points. We also use a new summarization measure to find the best performing method. Experimental results show that the best performing methods are power-set method pruning a infrequently occurring subsets of labels and classifier chains modeling relevant labels with an additional feature. futhermore, ensembles of many classifiers of these methods enhance the performance further. The recommendation from this study is that the correlation of subcellular locations is an effective clue for classification, this is because the subcellular locations of proteins performing certain biological function are not independent but correlated.

Enhancing multi-label classification by modeling dependencies among labels

In this paper, we propose a novel framework for multi-label classification, which directly models the dependencies among labels using a Bayesian network. Each node of the Bayesian network represents a label, and the links and conditional probabilities capture the probabilistic dependencies among multiple labels. We employ our Bayesian network structure learning method, which guarantees to find the global optimum structure, independent of the initial structure. After structure learning, maximum likelihood estimation is used to learn the conditional probabilities among nodes. Any current multi-label classifier can be employed to obtain the measurements of labels. Then, using the learned Bayesian network, the true labels are inferred by combining the relationship among labels with the labels׳ estimates obtained from a current multi-labeling method. We further extend the proposed multi-label classification method to deal with incomplete label assignments. Structural Expectation-Maximization algorithm is adopted for both structure and parameter learning. Experimental results on two benchmark multi-label databases show that our approach can effectively capture the co-occurrent and the mutual exclusive relation among labels. The relation modeled by our approach is more flexible than the pairwise or fixed subset labels captured by current multi-label learning methods. Thus, our approach improves the performance over current multi-label classifiers. Furthermore, our approach demonstrates its robustness to incomplete multi-label classification.

Cardinality and Density Measures and Their Influence to Multi-Label Learning Methods

Two main characteristics of multi-label dataset are cardinality and density, related to the number of labels of (each instance of) a multi-label dataset. The relation between these characteristics and multi-label learning performance has been observed with different datasets. However, the difference in domain dataset attributes also interfere on multi-label learning performance. In this work, we use a real dataset, named The Million Song Dataset, available in the internet, which presents the property of having too many labels associated to their instances (songs), as well as so many instances. In this work we present the processed datasets used to conduct our experiments, and we also describe our experiments results.

Multiple emotional tagging of multimedia data by exploiting dependencies among emotions

Digital multimedia may elicit a mixture of human emotions. Most current emotional tagging research typically tags the multimedia data with a single emotion, ignoring the phenomenon of multi-emotion coexistence. To address this problem, we propose a novel multi-emotion tagging approach by explicitly modeling the dependencies among emotions. First, several audio or visual features are extracted from the multimedia data. Second, four traditional multi-label learning methods: Binary Relevance, Random k label sets, Binary Relevance k Nearest Neighbours and Multi-Label k Nearest Neighbours, are used as the classifiers to obtain the measurements of emotional tags. Then, a Bayesian network is automatically constructed to capture the relationships among emotional tags. Finally, the Bayesian network is used to infer the data's multi-emotion tags by combining the measurements obtained from those traditional methods with the dependencies among emotions. Experiments on two multi-label media data sets demonstrate the superiority of our approach to the existing methods.

Multilabel relationship learning

Multilabel learning problems are commonly found in many applications. A characteristic shared by many multilabel learning problems is that some labels have significant correlations between them. In this article, we propose a novel multilabel learning method, called MultiLabel Relationship Learning (MLRL), which extends the conventional support vector machine by explicitly learning and utilizing the relationships between labels. Specifically, we model the label relationships using a label covariance matrix and use it to define a new regularization term for the optimization problem. MLRL learns the model parameters and the label covariance matrix simultaneously based on a unified convex formulation. To solve the convex optimization problem, we use an alternating method in which each subproblem can be solved efficiently. The relationship between MLRL and two widely used maximum margin methods for multilabel learning is investigated. Moreover, we also propose a semisupervised extension of MLRL, called SSMLRL, to demonstrate how to make use of unlabeled data to help learn the label covariance matrix. Through experiments conducted on some multilabel applications, we find that MLRL not only gives higher classification accuracy but also has better interpretability as revealed by the label covariance matrix.

Protein function prediction using multilabel ensemble classification.

High-throughput experimental techniques produce several kinds of heterogeneous proteomic and genomic data sets. To computationally annotate proteins, it is necessary and promising to integrate these heterogeneous data sources. Some methods transform these data sources into different kernels or feature representations. Next, these kernels are linearly (or nonlinearly) combined into a composite kernel. The composite kernel is utilized to develop a predictive model to infer the function of proteins. A protein can have multiple roles and functions (or labels). Therefore, multilabel learning methods are also adapted for protein function prediction. We develop a transductive multilabel classifier (TMC) to predict multiple functions of proteins using several unlabeled proteins. We also propose a method called transductive multilabel ensemble classifier (TMEC) for integrating the different data sources using an ensemble approach. The TMEC trains a graph-based multilabel classifier on each single data source, and then combines the predictions of the individual classifiers. We use a directed birelational graph to capture the relationships between pairs of proteins, between pairs of functions, and between proteins and functions. We evaluate the effectiveness of the TMC and TMEC to predict the functions of proteins on three benchmarks. We show that our approaches perform better than recently proposed protein function prediction methods on composite and multiple kernels. The code, data sets used in this paper and supplemental material are available at https://sites.google.com/site/guoxian85/tmec.

Multi-Label Learning with PRO Loss

Multi-label learning methods assign multiple labels to one object. In practice, in addition to differentiating relevant labels from irrelevant ones, it is often desired to rank the relevant labels for an object, whereas the rankings of irrelevant labels are not important. Such a requirement, however, cannot be met because most existing methods were designed to optimize existing criteria, yet there is no criterion which encodes the aforementioned requirement. In this paper, we present a new criterion, Pro Loss, concerning the prediction on all labels as well as the rankings of only relevant labels. We then propose ProSVM which optimizes Pro Lossefficiently using alternating direction method of multipliers. We further improve its efficiency with an upper approximation that reduces the number of constraints from O(T,2) to O(T), where T is the number of labels. Experiments show that our proposals are not only superior on Pro Loss, but also highly competitive on existing evaluation criteria.

Beam search algorithms for multilabel learning

Multilabel learning is a machine learning task that is important for applications, but challenging. A recent method for multilabel learning called probabilistic classifier chains (PCCs) has several appealing properties. However, PCCs suffer from the computational issue that inference (i.e., predicting the label of an example) requires time exponential in the number of tags. Also, PCC accuracy is sensitive to the ordering of the tags while training. In this paper, we show how to use the classical technique of beam search to solve both these problems. Specifically, we show how to apply beam search to make inference tractable, and how to integrate beam search with training to determine a suitable tag ordering. Experimental results on a range of datasets show that the proposed improvements yield a state-of-the-art method for multilabel learning.

On the consistency of multi-label learning

Multi-label learning has attracted much attention during the past few years. Many multi-label approaches have been developed, mostly working with surrogate loss functions because multi-label loss functions are usually difficult to optimize directly owing to their non-convexity and discontinuity. These approaches are effective empirically, however, little effort has been devoted to the understanding of their consistency, i.e., the convergence of the risk of learned functions to the Bayes risk. In this paper, we present a theoretical analysis on this important issue. We first prove a necessary and sufficient condition for the consistency of multi-label learning based on surrogate loss functions. Then, we study the consistency of two well-known multi-label loss functions, i.e., ranking loss and hamming loss. For ranking loss, our results disclose that, surprisingly, none of convex surrogate loss is consistent; we present the partial ranking loss, with which some surrogate losses are proven to be consistent. We also discuss on the consistency of univariate surrogate losses. For hamming loss, we show that two multi-label learning methods, i.e., one-vs-all and pairwise comparison, which can be regarded as direct extensions from multi-class learning, are inconsistent in general cases yet consistent under the dominating setting, and similar results also hold for some recent multi-label approaches that are variations of one-vs-all. In addition, we discuss on the consistency of learning approaches that address multi-label learning by decomposing into a set of binary classification problems.

Transductive Multilabel Learning via Label Set Propagation

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image annotations and gene function analysis. Current research on multilabel classification focuses on supervised settings which assume existence of large amounts of labeled training data. However, in many applications, the labeling of multilabeled data is extremely expensive and time consuming, while there are often abundant unlabeled data available. In this paper, we study the problem of transductive multilabel learning and propose a novel solution, called Trasductive Multilabel Classification (TraM), to effectively assign a set of multiple labels to each instance. Different from supervised multilabel learning methods, we estimate the label sets of the unlabeled instances effectively by utilizing the information from both labeled and unlabeled data. We first formulate the transductive multilabel learning as an optimization problem of estimating label concept compositions. Then, we derive a closed-form solution to this optimization problem and propose an effective algorithm to assign label sets to the unlabeled instances. Empirical studies on several real-world multilabel learning tasks demonstrate that our TraM method can effectively boost the performance of multilabel classification by using both labeled and unlabeled data.

RB: A New Method for Constructing Multi-Label Classifiers Based on Random Selection and Bagging

In many real world prediction problems, a classifier must, or should, assign more than one label to an instance, e.g. prediction of machine failures, musical genre classification, etc. For this kind of problem, multi-label classification methods are needed. One approach frequently used to learn multi-label predictors divides the problem into one or more multi-class classification problems, and combines the models constructed for each sub-problem to classify new instances with multiple labels. Although there are many multi-label learning methods, there is a need for exploring methods that can lead to improvement in prediction power. In this work, we propose and evaluate a new method, called RB (Random-Bagging), based on dataset transformation and combination of classifiers. Six real-world datasets were used to evaluate our method, which was compared to three existing methods. Results were considered promising.