Bipartite Ranking Research Articles

Analysis of k-partite ranking algorithm in area under the receiver operating characteristic curve criterion

ABSTRACTThe k-partite ranking, as an extension of bipartite ranking, is widely used in information retrieval and other computer applications. Such implement aims to obtain an optimal ranking function which assigns a score to each instance. The AUC (Area Under the ROC Curve) measure is a criterion which can be used to judge the superiority of the given k-partite ranking function. In this paper, we study the k-partite ranking algorithm in AUC criterion from a theoretical perspective. The generalization bounds for the k-partite ranking algorithm are presented, and the deviation bounds for a ranking function chosen from a finite function class are also considered. The uniform convergence bound is expressed in terms of a new set of combinatorial parameters which we define specially for the k-partite ranking setting. Finally, the generally margin-based bound for k-partite ranking algorithm is derived.

Online Pairwise Learning Algorithms.

Pairwise learning usually refers to a learning task that involves a loss function depending on pairs of examples, among which the most notable ones are bipartite ranking, metric learning, and AUC maximization. In this letter we study an online algorithm for pairwise learning with a least-square loss function in an unconstrained setting of a reproducing kernel Hilbert space (RKHS) that we refer to as the Online Pairwise lEaRning Algorithm (OPERA). In contrast to existing works (Kar, Sriperumbudur, Jain, & Karnick, 2013 ; Wang, Khardon, Pechyony, & Jones, 2012 ), which require that the iterates are restricted to a bounded domain or the loss function is strongly convex, OPERA is associated with a non-strongly convex objective function and learns the target function in an unconstrained RKHS. Specifically, we establish a general theorem that guarantees the almost sure convergence for the last iterate of OPERA without any assumptions on the underlying distribution. Explicit convergence rates are derived under the condition of polynomially decaying step sizes. We also establish an interesting property for a family of widely used kernels in the setting of pairwise learning and illustrate the convergence results using such kernels. Our methodology mainly depends on the characterization of RKHSs using its associated integral operators and probability inequalities for random variables with values in a Hilbert space.

Fundamental Limits of Data Analytics in Sociotechnical Systems

In the Big Data era, informational systems involving humans and machines are being deployed in multifarious societal settings. Many use data analytics as subcomponents for descriptive, predictive, and prescriptive tasks, often trained using machine learning. Yet when analytics components are placed in large-scale sociotechnical systems, it is often difficult to characterize how well the systems will act, measured with criteria relevant in the world. Here, we propose a system modeling technique that treats data analytics components as `noisy black boxes' or stochastic kernels, which together with elementary stochastic analysis provides insight into fundamental performance limits. An example application is helping prioritize people's limited attention, where learning algorithms rank tasks using noisy features and people sequentially select from the ranked list. This paper demonstrates the general technique by developing a stochastic model of analytics-enabled sequential selection, derives fundamental limits using concomitants of order statistics, and assesses limits in terms of system-wide performance metrics like screening cost and value of objects selected. Connections to sample complexity for bipartite ranking are also made.

An Efficient Method for Rare Spectra Retrieval in Astronomical Databases

One of the most important aims of astronomical data mining is to systematically search for specific rare objects in a massive spectral data set, given a small fraction of identified samples with the same type. Most existing methods are mainly based on binary classification, which usually suffers from incompleteness when there are too few known samples. Rank-based methods could provide good solutions for such cases. After investigating several algorithms, a method combining a bipartite ranking model with bootstrap aggregating techniques was developed in this paper. The method was applied while searching for carbon stars in the spectral data of Sloan Digital Sky Survey Data Release 10 and compared with several other popular methods used for data mining. Experimental results validate that the proposed method is not only the most effective but also the least time-consuming technique among its competitors when searching for rare spectra in a large but unlabeled data set.

A Bipartite Graph-Based Ranking Approach to Query Subtopics Diversification Focused on Word Embedding Features

A Bipartite Graph-Based Ranking Approach to Query Subtopics Diversification Focused on Word Embedding Features

Uncooperative gait recognition: Re-ranking based on sparse coding and multi-view hypergraph learning

Gait is an important biometric which can operate from a distance without subject cooperation. However, it is easily affected by changes in covariate conditions (carrying, clothing, view angle, walking speed, random noise etc.). It is hard for training set to cover all conditions. Bipartite ranking model has achieved success in gait recognition without assumption of subject cooperation. We propose a multi-view hypergraph learning re-ranking (MHLRR) method by integrating multi-view hypergraph learning (MHL) with hypergraph-based re-ranking framework. Sparse coding re-ranking (SCRR) and MHLRR are integrated under the graph-based framework to get a model. We define it as the sparse coding multi-view hypergraph learning re-ranking (SCMHLRR) method, which makes our approach achieve higher recognition accuracy under a genuine uncooperative setting. Extensive experiments demonstrate that our approach drastically outperforms existing ranking based methods, achieving good increase in recognition rate under the most difficult uncooperative settings.

Statistical Optimality in Multipartite Ranking and Ordinal Regression.

Statistical optimality in multipartite ranking is investigated as an extension of bipartite ranking. We consider the optimality of ranking algorithms through minimization of the theoretical risk which combines pairwise ranking errors of ordinal categories with differential ranking costs. The extension shows that for a certain class of convex loss functions including exponential loss, the optimal ranking function can be represented as a ratio of weighted conditional probability of upper categories to lower categories, where the weights are given by the misranking costs. This result also bridges traditional ranking methods such as proportional odds model in statistics with various ranking algorithms in machine learning. Further, the analysis of multipartite ranking with different costs provides a new perspective on non-smooth list-wise ranking measures such as the discounted cumulative gain and preference learning. We illustrate our findings with simulation study and real data analysis.

Swiss-System Based Cascade Ranking for Gait-Based Person Re-Identification

Human gait has been shown to be an efficient biometric measure for person identification at a distance. However, it often needs different gait features to handle various covariate conditions including viewing angles, walking speed, carrying an object and wearing different types of shoes. In order to improve the robustness of gait-based person re-identification on such multi-covariate conditions, a novel Swiss-system based cascade ranking model is proposed in this paper. Since the ranking model is able to learn a subspace where the potential true match is given the highest ranking, we formulate the gait-based person re-identification as a bipartite ranking problem and utilize it as an effective way for multi-feature ensemble learning. Then a Swiss multi-round competition system is developed for the cascade ranking model to optimize its effectiveness and efficiency. Extensive experiments on three indoor and outdoor public datasets demonstrate that our model outperforms several state-of-the-art methods remarkably.

The TreeRank Tournament algorithm for multipartite ranking

Whereas various efficient learning algorithms have been recently proposed to perform bipartite ranking tasks, cast as receiver operating characteristic (ROC) curve optimisation, no method fully tailored to K-partite ranking when K≥3 has been documented in the statistical learning literature yet. The goal is to optimise the ROC manifold, or summary criteria such as its volume, the gold standard for assessing performance in K-partite ranking. It is the main purpose of this paper to describe at length an efficient approach to recursive maximisation of the ROC surface, extending the TreeRank methodology originally tailored for the bipartite situation (i.e. when K=2). The main barrier arises from the fact that, in contrast to the bipartite case, the volume under the ROC surface criterion of any scoring rule taking K≥3 values cannot be interpreted as a cost-sensitive misclassification error and no method is readily available to perform the recursive optimisation stage. The learning algorithm we propose, called TreeRank Tournament (referred to as ‘TRT’ in the tables), breaks it and builds recursively an ordered partition of the feature space. It defines a piecewise scoring function whose ROC manifold can be remarkably interpreted as a statistical version of an adaptive piecewise linear approximant of the optimal ROC manifold. Rate bounds in sup norm describing the generalisation ability of the scoring rule thus built are established and numerical results illustrating the performance of the TRT approach, compared to that of natural competitors such as aggregation methods, are also displayed.

Combination of multiple bipartite ranking for multipartite web content quality evaluation

Web content quality evaluation is crucial to various web content processing applications. Bagging has a powerful classification capacity by combining multiple classifiers. In this study, similar to Bagging, multiple pairwise bipartite ranking learners are combined to solve the multipartite ranking problems for web content quality evaluation. Both encoding and decoding mechanisms are used to combine bipartite rankers to form a multipartite ranker and, hence, the multipartite ranker is called MultiRank.ED. Both binary encoding and ternary encoding extend each rank value to an L −1 dimensional vector for a ranking problem with L different rank values. Predefined weighting and adaptive weighting decoding mechanisms are used to combine the ranking results of bipartite rankers to obtain the final ranking results. In addition, some theoretical analyses of the encoding and the decoding strategies in the MultiRank.ED algorithm are provided. Computational experiments using the DC2010 datasets show that the combination of binary encoding and predefined weighting decoding yields the best performance in all four combinations. Furthermore, this combination performs better than the best winning method of the DC2010 competition.

Uncooperative gait recognition by learning to rank

Gait is a useful biometric because it can operate from a distance and without subject cooperation. However, it is affected by changes in covariate conditions (carrying, clothing, view angle, etc.). Existing methods suffer from lack of training samples, can only cope with changes in a subset of conditions with limited success, and implicitly assume subject cooperation. We propose a novel approach which casts gait recognition as a bipartite ranking problem and leverages training samples from different people and even from different datasets. By exploiting learning to rank, the problem of model over-fitting caused by under-sampled training data is effectively addressed. This makes our approach suitable under a genuine uncooperative setting and robust against changes in any covariate conditions. Extensive experiments demonstrate that our approach drastically outperforms existing methods, achieving up to 14-fold increase in recognition rate under the most difficult uncooperative settings.

Generalization performance of bipartite ranking algorithms with convex losses

Previous works describing the generalization performance of bipartite ranking algorithms are usually based on the assumption of (0–1) loss or the area under the receiver operating characteristic (ROC) curve. In this paper we go far beyond this classical framework by investigating the generalization performance of bipartite ranking algorithms with convex losses over reproducing kernel Hilbert spaces. Based on the McDiarmid inequality and Rademacher complexity, we establish the upper bound on the generalization error for a bipartite ranking algorithm. The theoretical analysis is different from the previous results on error analysis and shows the attractive uniform convergence property of regularized bipartite ranking algorithms.

Upper bounds and aggregation in bipartite ranking

One main focus of learning theory is to find optimal rates of convergence. In classification, it is possible to obtain optimal fast rates (faster than $n^{-1/2}$) in a minimax sense. Moreover, using an aggregation procedure, the algorithms are adaptive to the parameters of the class of distributions. Here, we investigate this issue in the bipartite ranking framework. We design a ranking rule by aggregating estimators of the regression function. We use exponential weights based on the empirical ranking risk. Under several assumptions on the class of distribution, we show that this procedure is adaptive to the margin parameter and smoothness parameter and achieves the same rates as in the classification framework. Moreover, we state a minimax lower bound that establishes the optimality of the aggregation procedure in a specific case.

Ranking data with ordinal labels: optimality and pairwise aggregation

The paper describes key insights in order to grasp the nature of K-partite ranking. From the theoretical side, the various characterizations of optimal elements are fully described, as well as the likelihood ratio monotonicity condition on the underlying distribution which guarantees that such elements do exist. Then, a pairwise aggregation procedure based on Kendall tau is introduced to relate learning rules dedicated to bipartite ranking and solutions of the K-partite ranking problem. Criteria reflecting ranking performance under these conditions such as the ROC surface and its natural summary, the volume under the ROC surface (VUS), are then considered as targets for empirical optimization. The consistency of pairwise aggregation strategies are studied under these criteria and shown to be efficient under reasonable assumptions. Eventually, numerical results illustrate the relevance of the methodology proposed.

An empirical comparison of learning algorithms for nonparametric scoring: the TreeRank algorithm and other methods

The TreeRank algorithm was recently proposed in [1] and [2] as a scoring-based method based on recursive partitioning of the input space. This tree induction algorithm builds orderings by recursively optimizing the Receiver Operating Characteristic curve through a one-step optimization procedure called LeafRank. One of the aim of this paper is the in-depth analysis of the empirical performance of the variants of TreeRank/LeafRank method. Numerical experiments based on both artificial and real data sets are provided. Further experiments using resampling and randomization, in the spirit of bagging and random forests are developed [3, 4] and we show how they increase both stability and accuracy in bipartite ranking. Moreover, an empirical comparison with other efficient scoring algorithms such as RankBoost and RankSVM is presented on UCI benchmark data sets.

Approximation Analysis of Gradient Descent Algorithm for Bipartite Ranking

We introduce a gradient descent algorithm for bipartite ranking with general convex losses. The implementation of this algorithm is simple, and its generalization performance is investigated. Explicit learning rates are presented in terms of the suitable choices of the regularization parameter and the step size. The result fills the theoretical gap in learning rates for ranking problem with general convex losses.

On Equivalence Relationships Between Classification and Ranking Algorithms

We demonstrate that there are machine learning algorithms that can achieve success for two separate tasks simultaneously, namely the tasks of classification and bipartite ranking. This means that a...

Batch and online learning algorithms for nonconvex neyman-pearson classification

We describe and evaluate two algorithms for Neyman-Pearson (NP) classification problem which has been recently shown to be of a particular importance for bipartite ranking problems. NP classification is a nonconvex problem involving a constraint on false negatives rate. We investigated batch algorithm based on DC programming and stochastic gradient method well suited for large-scale datasets. Empirical evidences illustrate the potential of the proposed methods.

On the ERA ranking representability of pairwise bipartite ranking functions

In domains like decision theory and social choice theory it is known for a long time that stochastic transitivity properties yield necessary and sufficient conditions for the ranking or utility representability of reciprocal preference relations. In this article we extend these results for reciprocal preference relations originating from the pairwise comparison of random vectors in a machine learning context. More specifically, the expected ranking accuracy (ERA) is such a reciprocal relation that occurs in multi-class classification problems, when ranking or utility functions are fitted to the data in a pairwise manner. We establish necessary and sufficient conditions for which these pairwise bipartite ranking functions can be simplified to a single ranking function such that the pairwise expected ranking accuracies of both models coincide. Similarly as for more common reciprocal preference relations, cycle transitivity plays a crucial role in this new setting. We first consider the finite sample case, for which expected ranking accuracy can be estimated by means of the area under the ROC curve (AUC), and subsequently, we further generalize these results to the underlying distributions. It turns out that the ranking representability of pairwisely compared random vectors can be expressed elegantly in a distribution-independent way by means of a specific type of cycle transitivity, defined by a conjunctor that is closely related to the algebraic product.

Incorporating Relevance and Importance for Dynamic Ranking in Folksonomies

The rapidly increasing popularity of social tagging systems and growing amount of users and resources make it a difficult task to find expert users and relevant resources in folksonomies. In this paper, we propose a bipartite graph-based dynamic ranking algorithm, RicoRank (Relevance and Importance inCOrporated RANK), for improving search performance in folksonomies. We combine both the query relevance and importance effectively to generate the final ranking score. We derive the query relevance from a smoothed probabilistic generative model that demonstrates user interest and resource content. We characterize the importance with the mutual reinforcement between users and resources. We assign each mutual reinforcing relation with a weight corresponding to the coherence between the associated tags, the user interest and the resource content. Finally, we employ an iterative procedure, which incorporates well with both query relevance and importance, to simultaneously compute the ranking scores of users and resource. We conduct experiments on a dataset collected from a real-world system. Experimental results on both user expertise and resource quality ranking show a convincing performance of the proposed algorithm.