Bipartite Ranking Research Articles

Learning to rank anomalies: scalar performance criteria and maximization of rank statistics

AbstractThe ability to collect and store ever more massive data, unlabeled in many cases, has been accompanied by the need to process them efficiently in order to extract relevant information and possibly design solutions based on the latter. In various situations, the vast majority of the observations exhibit the same behavior, while a small proportion deviates from it. Detecting these outlier observations (or equivalently defined as anomalies) is now one of the major challenges for machine learning applications (e.g. fraud detection or predictive maintenance). We propose here a novel methodology for outlier/anomaly detection, by learning a scoring function defined on the feature space allowing for ranking the observations by degree of abnormality. The scoring function is built through maximization of an empirical performance criterion taking the form of a (two-sample) linear rank statistic. We show that bipartite ranking algorithms can thus be used to learn nearly optimal scoring function with provable theoretical guarantees. We illustrate our methodology with numerical experiments based on open access online code.

Bipartite Ranking Fairness Through a Model Agnostic Ordering Adjustment.

Recently, with the applications of algorithms in various risky scenarios, algorithmic fairness has been a serious concern and received lots of interest in machine learning community. In this article, we focus on the bipartite ranking scenario, where the instances come from either the positive or negative class and the goal is to learn a ranking function that ranks positive instances higher than negative ones. We are interested in whether the learned ranking function can cause systematic disparity across different protected groups defined by sensitive attributes. While there could be a trade-off between fairness and performance, we propose a model agnostic post-processing framework xOrder for achieving fairness in bipartite ranking and maintaining the algorithm classification performance. In particular, we optimize a weighted sum of the utility as identifying an optimal warping path across different protected groups and solve it through a dynamic programming process. xOrder is compatible with various classification models and ranking fairness metrics, including supervised and unsupervised fairness metrics. In addition to binary groups, xOrder can be applied to multiple protected groups. We evaluate our proposed algorithm on four benchmark data sets and two real-world patient electronic health record repositories. xOrder consistently achieves a better balance between the algorithm utility and ranking fairness on a variety of datasets with different metrics. From the visualization of the calibrated ranking scores, xOrder mitigates the score distribution shifts of different groups compared with baselines. Moreover, additional analytical results verify that xOrder achieves a robust performance when faced with fewer samples and a bigger difference between training and testing ranking score distributions.

Distributed spectral pairwise ranking algorithms

This paper considers spectral pairwise ranking algorithms in a reproducing kernel Hilbert space. The concerned algorithms include a large family of regularized pairwise learning algorithms. Motivated by regularization methods, spectral algorithms are proposed to solve ill-posed linear inverse problems, then developed in learning theory and inverse problems. Recently, pairwise learning tasks such as bipartite ranking, similarity metric learning, Minimum Error Entropy principle, and AUC maximization have received increasing attention due to their wide applications. However, the spectral algorithm acts on the spectrum of the empirical integral operator or kernel matrix, involving the singular value decomposition or the inverse of the matrix, which is time-consuming when the sample size is immense. Our contribution is twofold. First, under some general source conditions and capacity assumptions, we establish the first-ever mini-max optimal convergence rates for spectral pairwise ranking algorithms. Second, we consider the distributed version of the algorithms based on a divide-and-conquer approach and show that, as long as the partition of the data set is not too large, the distributed learning algorithm enjoys both computational efficiency and statistical optimality.

Pairwise learning problems with regularization networks and Nyström subsampling approach

Pairwise learning usually refers to the learning problem that works with pairs of training samples, such as ranking, similarity and metric learning, and AUC maximization. To overcome the challenge of pairwise learning in the large scale computation, this paper introduces Nyström sampling approach to the coefficient-based regularized pairwise algorithm in the context of kernel networks. Our theorems establish that the obtained Nyström estimator achieves the minimax error over all estimators using the whole data provided that the subsampling level is not too small. We derive the function relation between the subsampling level and regularization parameter that guarantees computation cost reduction and asymptotic behaviors’ optimality simultaneously. The Nyström coefficient-based pairwise learning method does not require the kernel to be symmetric or positive semi-definite, which provides more flexibility and adaptivity in the learning process. We apply the method to the bipartite ranking problem, which improves the state-of-the-art theoretical results in previous works. By developing probability inequalities for U-statistics on Hilbert–Schmidt operators, we provide new mathematical tools for handling pairs of examples involved in pairwise learning.

Error bounds of adversarial bipartite ranking

Generalization analysis for learning models with adversarial examples has attracted increasing attentions recently. However, the previous theoretical results are usually limited to learning models with the pointwise loss. In this paper, we beyond this restriction by investigating generalization ability of bipartite ranking associated with adversarial perturbations. The upper bounds of adversarial ranking risk are established by formulating the pairwise learning in a minimax framework and introducing the transfer mapping to relate data distributions. In particular, our results are suitable to general loss satisfying Lipschitz conditions, e.g, the logistic loss and the least squared loss.

End-to-End Speaker Verification via Curriculum Bipartite Ranking Weighted Binary Cross-Entropy

End-to-end speaker verification achieves the verification through estimating directly the similarity score between a pair of utterances, which is formulated as a binary (i.e., target versus non-target) classification problem. Unlike the stage-wise method, an end-to-end verification approach optimizes the evaluation metrics directly and its output layer is parameter-free, which can save great computing and memory resources. However, there are two important issues that need to be meticulously handled in training an end-to-end speaker verification model. The first one is how to deal with severely imbalanced trials, i.e., the number of target trials is much smaller than that of nontarget trials, and the other is about how to handle easy trials that do not help improve the model in training. To circumvent these two issues, we propose in this paper a binary cross-entropy (BCE) type of loss function and present a method to train the deep neural network (DNN) models based on the proposed loss function for end-to-end speaker verification. The training process employs a bipartite ranking method to deal with the trial imbalance problem and a curriculum learning method to help improve both the training stability and performance of the model by selecting non-target trials from easy to hard ones gradually along the convergence process. Since the training process employs bipartite ranking and curriculum learning and the loss function is of the generalized BCE form, we name the new approach \textit{curriculum bipartite ranking weighted binary cross-entropy} (CBRW-BCE). Experimental results show that the model trained with CBRW-BCE not only achieves the state-of-the-art performance but is also well calibrated.

Fast Manifold Ranking With Local Bipartite Graph

During the past decades, manifold ranking has been widely applied to content-based image retrieval and shown excellent performance. However, manifold ranking is computationally expensive in both graph construction and ranking learning. Much effort has been devoted to improve its performance by introducing approximating techniques. In this paper, we propose a fast manifold ranking method, namely Local Bipartite Manifold Ranking (LBMR). Given a set of images, we first extract multiple regions from each image to form a large image descriptor matrix, and then use the anchor-based strategy to construct a local bipartite graph in which a regional k -means (RKM) is proposed to obtain high quality anchors. We propose an iterative method to directly solve the manifold ranking problem from the local bipartite graph, which monotonically decreases the objective function value in each iteration until the algorithm converges. Experimental results on several real-world image datasets demonstrate the effectiveness and efficiency of our proposed method.

Concentration inequalities for two-sample rank processes with application to bipartite ranking

The ROC curve is the gold standard for measuring the performance of a test/scoring statistic regarding its capacity to discriminate between two statistical populations in a wide variety of applications, ranging from anomaly detection in signal processing to information retrieval, through medical diagnosis. Most practical performance measures used in scoring/ranking applications such as the AUC, the local AUC, the p-norm push, the DCG and others, can be viewed as summaries of the ROC curve. In this paper, the fact that most of these empirical criteria can be expressed as two-sample linear rank statistics is highlighted and concentration inequalities for collections of such random variables, referred to as two-sample rank processes here, are proved, when indexed by VC classes of scoring functions. Based on these nonasymptotic bounds, the generalization capacity of empirical maximizers of a wide class of ranking performance criteria is next investigated from a theoretical perspective. It is also supported by empirical evidence through convincing numerical experiments.

M-BiRank: co-ranking developers and projects using multiple developer-project interactions in open source software community

Social collaborative coding is a popular trend in software development, and such platforms as GitHub provide rich social and technical functionalities for developers to collaborate on open source projects through multiple interactions. Developers often follow popular developers and projects for learning, technical selection, and collaboration. Thus, identifying popular developers and projects is very meaningful. In this paper, we propose a multiplex bipartite network ranking model, M-BiRank, to co-rank developers and projects using multiple developer-project interactions. Firstly, multiple developer-project interactions such as commit, issue, and watch are extracted and a multiplex developer-project bipartite network is constructed. Secondly, a random layer is selected from this multiplex bipartite network and initial ranking scores are calculated for developers and projects using BiRank. Finally, initial ranking scores diffuse to other layers and mutual reinforcement is taken into consideration to iteratively calculate ranking scores of developers and projects in different layers. Experiments on real-world GitHub dataset show that M-BiRank outperforms degree centrality, traditional single layer ranking methods, and multiplex ranking method.

A Smoothing Optimization Approach Applied to the Supervised MDS Method

This paper presents an efficient approach to the Supervised MDS method. This method handles the problems of data visualization, supervised classification and bipartite ranking. In order to overcome the non-differentiable nature of the Supervised MDS method, the mathematical formulation proposed in this work is based on the hyperbolic smoothing technique. The performance of the algorithm is evaluated by computational experiments. The results show that the proposed methodology presented, in most cases, better results than the results available in the literature. Furthermore, the methodology presents a good performance in relation to the methods Logistic regression, Naive Bayes and Support Vector Machine.

Addressing time bias in bipartite graph ranking for important node identification

For online service platforms such as Netflix, it is important to propose a list of high quality movies to their users. This type of problem can be regarded as a ranking problem in a bipartite network. This is a well-known problem, that can be solved by a ranking algorithm. However, many classical ranking algorithms share a common drawback: they tend to rank higher older movies rather than newer ones, though some new movies may be of higher quality. In the study, we develop a ranking method using a rebalance approach to decrease the time bias of the rankings in bipartite graphs. We then conduct experiments on three real datasets with ground truth benchmark. The results show that our proposed method not only reduces the time bias of the ranking scores, but also improves the prediction accuracy by at least 20%, and up to 80%.

Click-aware purchase prediction with push at the top

Eliciting user preferences from purchase records for performing purchase prediction is challenging because negative feedback is not explicitly observed, and because treating all non-purchased items equally as negative feedback is unrealistic. Therefore, in this study, we present a framework that leverages the past click records of users to compensate for the missing user–item interactions of purchase records, i.e., non-purchased items. We begin by formulating various model assumptions, each one assuming a different order of user preferences among purchased, clicked-but- not-purchased, and non-clicked items, to study the usefulness of leveraging click records. We implement the model assumptions using the Bayesian personalized ranking model, which maximizes the area under the curve for bipartite ranking. However, we argue that using click records for bipartite ranking needs a meticulously designed model because of the relative unreliableness of click records compared with that of purchase records. Therefore, we ultimately propose a novel learning-to-rank method, called P3Stop, for performing purchase prediction. The proposed model is customized to be robust to relatively unreliable click records by particularly focusing on the accuracy of top-ranked items. Experimental results on two real-world e-commerce datasets demonstrate that P3STop considerably outperforms the state-of-the-art implicit-feedback-based recommendation methods, especially for top-ranked items.

A new multi-view learning machine with incomplete data

Multi-view learning with incomplete views (MVL-IV) is a reliable algorithm to process incomplete datasets which consist of instances with missing views or features. In MVL-IV, it exploits the connections among multiple views and suggests that different views are generated from a shared subspace such that it can recover the missing views or features well while MVL-IV neglects two facts. One is that different views should always be generated from different subspaces. The other is that the information of view-based classifiers is useful to the design of MVL-IV. Thus, on the base of MVL-IV, we consider these two facts and develop a new multi-view learning with incomplete data (NMVL-IV). Related experiments on clustering, regression, classification, bipartite ranking, and image retrieval have validated that the proposed NMVL-IV can recover the incomplete data much better.

Scalable Collaborative Ranking for Personalized Prediction

Personalized prediction presents an important yet challenging task, which predicts user-specific preferences on a large number of items given limited information. It is often modeled as certain recommender systems focusing on ordinal or continuous ratings, as in collaborative filtering and content-based filtering. In this article, we propose a new collaborative ranking system to predict most-preferred items for each user given search queries. Particularly, we propose a ψ-ranker based on ranking functions incorporating information on users, items, and search queries through latent factor models. Moreover, we show that the proposed nonconvex surrogate pairwise ψ-loss performs well under four popular bipartite ranking losses, such as the sum loss, pairwise zero-one loss, discounted cumulative gain, and mean average precision. We develop a parallel computing strategy to optimize the intractable loss of two levels of nonconvex components through difference of convex programming and block successive upper-bound minimization. Theoretically, we establish a probabilistic error bound for the ψ-ranker and show that its ranking error has a sharp rate of convergence in the general framework of bipartite ranking, even when the dimension of the model parameters diverges with the sample size. Consequently, this result also indicates that the ψ-ranker performs better than two major approaches in bipartite ranking: pairwise ranking and scoring. Finally, we demonstrate the utility of the ψ-ranker by comparing it with some strong competitors in the literature through simulated examples as well as Expedia booking data. Supplementary materials for this article are available online.

Weight-and-Universum-based semi-supervised multi-view learning machine

Semi-supervised multi-view learning machine is developed to process the corresponding semi-supervised multi-view data sets which consist of labeled and unlabeled instances. But in real-world applications, for a multi-view data set, only few instances are labeled with the limitation of manpower and cost. As a result, few prior knowledge which is necessary for the designing of a learning machine is provided. Moreover, in practice, different views and features play diverse discriminant roles while traditional learning machines treat these roles equally and assign the same weight just for convenience. In order to solve these problems, we introduce Universum learning to obtain more prior knowledge and assign different weights for views and features to reflect their diverse discriminant roles. The proposed learning machine is named as weight-and-Universum-based semi-supervised multi-view learning machine (WUSM). In WUSM, we first obtain weights of views and features. Then, we construct Universum set to obtain more prior knowledge on the basis of these weights. Different from traditional construction ways, the used construction way makes full use of the information of all labeled and unlabeled instances rather than only a pair of positive and negative training instances. Finally, we design the machine with the usage of the Universum set along with original data set. Our contributions are given as follows. (1) With the usage of all (labeled, unlabeled) instances of the data set, the Universum set provides more useful prior knowledge. (2) WUSM considers the diversities of views and features. (3) WUSM advances the development of semi-supervised multi-view learning machines. Experiments on bipartite ranking, feature selection, dimensionality reduction, classification, clustering, etc. validate the advantages of WUSM and draw a conclusion that with the introduction of Universum learning, view weights, and feature weights, the performance of a semi-supervised multi-view learning machine is boosted.

Stability and optimization error of stochastic gradient descent for pairwise learning

In this paper, we study the stability and its trade-off with optimization error for stochastic gradient descent (SGD) algorithms in the pairwise learning setting. Pairwise learning refers to a learning task which involves a loss function depending on pairs of instances among which notable examples are bipartite ranking, metric learning, area under ROC curve (AUC) maximization and minimum error entropy (MEE) principle. Our contribution is twofolded. Firstly, we establish the stability results for SGD for pairwise learning in the convex, strongly convex and non-convex settings, from which generalization errors can be naturally derived. Secondly, we establish the trade-off between stability and optimization error of SGD algorithms for pairwise learning. This is achieved by lower-bounding the sum of stability and optimization error by the minimax statistical error over a prescribed class of pairwise loss functions. From this fundamental trade-off, we obtain lower bounds for the optimization error of SGD algorithms and the excess expected risk over a class of pairwise losses. In addition, we illustrate our stability results by giving some specific examples of AUC maximization, metric learning and MEE.

Weight-based label-unknown multi-view data set generation approach

Multi-view learning aims to solve multi-view data set which consists of multiple instances with different views. Traditional multi-view learning approaches always encounter small-scale label-known multi-view instances problem and insufficient discriminant information problem. Although some label-unknown multi-view data set generation approaches are developed to enhance useful discriminant information, the weights of views and features are not considered. This paper proposes a weight-based label-unknown multi-view data set generation approach (WLM) to overcome such a disadvantage. The procedure of WLM consists of three main steps. First, get the weights of views and features. Second, get similar instances of each label-known instance. Third, generate and select feasible label-unknown instances which are applied to multi-view learning approaches along with the original label-known multi-view instances. Further, comparisons and analysis about classification performance, clustering performance, bipartite ranking performance, image retrieval performance, significance on some multi-view data sets with some multi-view learning approaches validate the usefulness of WLM.

The risk of trivial solutions in bipartite top ranking

Given a sample of instances with binary labels, the bipartite top ranking problem is to produce a ranked list of instances whose head is dominated by positives. One popular existing approach to this problem is based on constructing surrogates to a performance measure known as the fraction of positives of the top (PTop). In this paper, we theoretically show that the measure and its surrogates have an undesirable property: for certain noisy distributions, it is optimal to trivially predict the same score for all instances. We propose a simple rectification which avoids such trivial solutions, while still focussing on the head of the ranked list and being as easy to optimise.

PAC-Bayesian high dimensional bipartite ranking

This paper is devoted to the bipartite ranking problem, a classical statistical learning task, in a high dimensional setting. We propose a scoring and ranking strategy based on the PAC-Bayesian approach. We consider nonlinear additive scoring functions, and we derive non-asymptotic risk bounds under a sparsity assumption. In particular, oracle inequalities in probability holding under a margin condition assess the performance of our procedure, and prove its minimax optimality. An MCMC-flavored algorithm is proposed to implement our method, along with its behavior on synthetic and real-life datasets.

On Theoretically Optimal Ranking Functions in Bipartite Ranking

ABSTRACTThis article investigates the theoretical relation between loss criteria and the optimal ranking functions driven by the criteria in bipartite ranking. In particular, the relation between area under the ROC curve (AUC) maximization and minimization of ranking risk under a convex loss is examined. We characterize general conditions for ranking-calibrated loss functions in a pairwise approach, and show that the best ranking functions under convex ranking-calibrated loss criteria produce the same ordering as the likelihood ratio of the positive category to the negative category over the instance space. The result illuminates the parallel between ranking and classification in general, and suggests the notion of consistency in ranking when convex ranking risk is minimized as in the RankBoost algorithm for instance. For a certain class of loss functions including the exponential loss and the binomial deviance, we specify the optimal ranking function explicitly in relation to the underlying probability distribution. In addition, we present an in-depth analysis of hinge loss optimization for ranking and point out that the RankSVM may produce potentially many ties or granularity in ranking scores due to the singularity of the hinge loss, which could result in ranking inconsistency. The theoretical findings are illustrated with numerical examples. Supplementary materials for this article are available online.