Learning-to-rank Algorithms Research Articles

Revisiting Code Smell Severity Prioritization using learning to rank techniques

Code Smell Severity Prioritization (CSSP) is crucial in helping software developers minimize software maintenance costs and enhance software quality, particularly when faced with limited refactoring resources. Traditional code smell prioritization methods rely heavily on manual and semi-automatic approaches based on developer experience, often demanding considerable time and effort from experienced experts. Leveraging automated machine learning techniques can effectively overcome these limitations. However, most existing machine learning-based CSSP works have only considered limited pointwise Learning To Rank (LTR) algorithms and have used inappropriate metrics (e.g., Accuracy, Spearman, and MAE) to assess the performance of models. To address these limitations, we make a comprehensive comparison of 41 pointwise, 4 pairwise, and 4 listwise LTR algorithms for CSSP on four code smell severity datasets. Furthermore, we propose the adoption of Severity@20% and Cumulative Lift Chart (CLC) as the primary evaluation metrics to assess CSSP models more effectively. The results show that: (1) The ordinal Bagging (O-Bagging) algorithm demonstrates the highest performance for CSSP, achieving superior results in Severity@20% and CLC. (2) The ordinal classification method can help the top-performing base classification algorithms Bagging and XGBoost achieve better performance for CSSP tasks. (3) A higher (lower) Accuracy, higher (lower) Spearman, and lower (higher) MAE do not reliably indicate better (worse) performance for CSSP. This further underscores that Accuracy, Spearman, and MAE are unsuitable metrics for evaluating CSSP models’ effectiveness. To summarize, our study suggest that developers employ the O-Bagging algorithm for CSSP, with Severity@20% and CLC serving as the primary evaluation metrics.

Learning To Rank for E commerce Cart Optimization

In the competitive landscape of e-commerce, optimizing the shopping cart experience is crucial for enhancing customer satisfaction and boosting conversion rates. This paper explores the application of Learning to Rank (LTR) techniques to optimize e-commerce cart functionality. Learning to Rank, a subset of machine learning, involves training models to order items or features in a way that maximizes relevance based on user preferences. By employing LTR algorithms, e-commerce platforms can improve the relevance of product recommendations, streamline the checkout process, and tailor user interactions based on historical data and user behaviour. The study outlines the integration of LTR models into e-commerce systems, focusing on how these models can predict and prioritize products that users are more likely to purchase. This involves analysing user interactions, purchase history, and product attributes to develop ranking models that enhance the cart's efficiency and effectiveness. The research also addresses challenges such as data sparsity, dynamic user preferences, and the need for real-time updates, offering solutions and best practices for implementing LTR in practical settings. The results demonstrate that applying LTR techniques can significantly improve cart optimization by presenting users with more relevant product options, reducing cart abandonment rates, and increasing overall sales. This paper provides a comprehensive framework for e-commerce businesses seeking to leverage LTR for cart optimization, highlighting both theoretical insights and practical implications.

LtRFT: Mitigate the Low-Rate Data Plane DDoS Attack With Learning-To-Rank Enabled Flow Tables

Software-Defined Networking (SDN) switches typically have limited ternary content addressable memory (TCAM) that caches the flow entries on the data plane. The scarcity and strong resource competitiveness of TCAM space put the flow tables at the risk of malicious Distributed Denial-of-Service (DDoS) attacks. In this paper, we propose LtRFT, a Learning-To-Rank (LtR) based scheme for mitigating the low-rate DDoS attacks targeted at flow tables. LtRFT consists of three modules: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">monitor</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ranker</i> , and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mitigator</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Monitor</i> manages the flow table status and sends alerts to other modules after detecting attacks. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ranker</i> models the attack mitigation problem as a flow entry ranking task, and ranks malicious flows with a high eviction priority using a pairwise-based LtR algorithm. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mitigator</i> frees up the flow table space by deleting malicious flow entries according to the flow entry ranking sequence generated by <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ranker</i> . We introduce LtR to network attack detection innovatively and use both classification and information retrieval metrics to describe and evaluate LtRFT. Extensive experiments were conducted to validate the effectiveness and robustness of LtRFT in detecting and mitigating the low-rate data plane DDoS attacks. LtRFT can detect malicious attack flows with an accuracy of over 96%, and can reduce the attack flow duration by 97.7% with an average extra latency of 0.5 seconds, which proves that LtRFT is practicable in SDN deployments.

Understanding Bias and Variance of Learning-to-Rank Algorithms: An Empirical Framework

ABSTRACT Learning-to-rank (LtR) algorithms are at the heart of modern day information retrieval systems. While a good number of LtR algorithms have been developed and scrutinized over the past decade, theoretical underpinnings of these algorithms are not thoroughly investigated so far. Amongst the theoretical aspects of a supervised learning algorithm, the bias-variance profiles are of utmost importance. In this article we aim to better understand the bias-variance profiles of rank-learning algorithms. Firstly, we formalize the bias and variance from a pointwise perspective where each query-document pair is treated as an individual training example. Secondly, we develop a framework to analyze the variability and systematic error of a rank-learner in terms of its ranking error, i.e., we analyze the bias-variance from a listwise perspective where a query and all of its associated documents are treated as a single training example. After developing the theoretical framework, we move on to test its applicability in practice. In particular, we choose a promising algorithm, namely random forest-based rank-learning algorithms for our investigation. We study the effect of varying an important parameter of the algorithm, namely the sub-sample size used to learn each tree, on bias and variance. Our hypothesis is that as the sub-sample size (per tree) increases, classical bias-variance tradeoff should be observed. On two large LtR benchmark datasets, experimental results show that our hypothesis holds true. We also explain the relative performance of two of the top performing LtR algorithms using their bias and variance profiles. To the best of our knowledge, this article presents the first thorough investigation into bias and variance analysis of rank-learning algorithms.

New Application of Life Rank Algorithm: A Case Study

The past few years have seen the emergence of learning-to-rank (LTR) in the field of machine learning. In information acquiring the size of data is very large and empowering a learning-to-rank model on it will be a costly and time taking process. High dimension data leads to irrelevant and redundant data which results in overfitting. “Dimensionality reduction” methods are used to manage this issue. There are two-dimensionality reduction techniques namely feature selection and feature reduction. There is extensive research available on the algorithm for learning-to-rank but this not the case for dimensionality reduction approaches in LTR, despite its importance. Feature selection techniques for classification are directly used for ranking. To the best of our understanding, feature extraction techniques in the context of ranking problems are not explored much to date. So, we make an effort to fill this void and explore feature extraction in the context of LTR problems. The LifeRank algorithm is a linear feature extraction algorithm for ranking. Its performance is analyzed on RankSVM and Linear regression. It is not applied to other learning-to-rank algorithms. So, in this task, an attempt is made to study the effect of the application of the LifeRank algorithm on other LTR algorithms. LifeRank algorithm is applied on RankNet and RankBoost. Then, the performance of several LTR algorithms on the LETOR dataset is analyzed before and after feature extraction.

Application of learning to rank in bioinformatics tasks.

Over the past decades, learning to rank (LTR) algorithms have been gradually applied to bioinformatics. Such methods have shown significant advantages in multiple research tasks in this field. Therefore, it is necessary to summarize and discuss the application of these algorithms so that these algorithms are convenient and contribute to bioinformatics. In this paper, the characteristics of LTR algorithms and their strengths over other types of algorithms are analyzed based on the application of multiple perspectives in bioinformatics. Finally, the paper further discusses the shortcomings of the LTR algorithms, the methods and means to better use the algorithms and some open problems that currently exist.

Boosting learning to rank with user dynamics and continuation methods

Learning to rank (LtR) techniques leverage assessed samples of query-document relevance to learn effective ranking functions able to exploit the noisy signals hidden in the features used to represent queries and documents. In this paper we explore how to enhance the state-of-the-art LambdaMart LtR algorithm by integrating in the training process an explicit knowledge of the underlying user-interaction model and the possibility of targeting different objective functions that can effectively drive the algorithm towards promising areas of the search space. We enrich the iterative process followed by the learning algorithm in two ways: (1) by considering complex query-based user dynamics instead than simply discounting the gain by the rank position; (2) by designing a learning path across different loss functions that can capture different signals in the training data. Our extensive experiments, conducted on publicly available datasets, show that the proposed solution permits to improve various ranking quality measures by statistically significant margins.

An empirical comparison of random forest-based and other learning-to-rank algorithms

Random forest (RF)-based pointwise learning-to-rank (LtR) algorithms use surrogate loss functions to minimize the ranking error. In spite of their competitive performance to other state-of-the-art LtR algorithms, these algorithms, unlike other frameworks such as boosting and neural network, have not been thoroughly investigated in the literature so far. In the first part of this study, we aim to better understand and improve the RF-based pointwise LtR algorithms. When working with such an algorithm, currently we need to choose a setting from a number of available options such as (1) classification versus regression setting, (2) using absolute relevance judgements versus mapped labels, (3) the number of features using which a split-point for data is chosen, and (4) using weighted versus un-weighted average of the predictions of multiple base learners (i.e., trees). We conduct a thorough study on these four aspects as well as on a pairwise objective function for RF-based rank-learners. Experimental results on several benchmark LtR datasets demonstrate that performance can be significantly improved by exploring these aspects. In the second part of this paper, we, guided by our investigations performed into RF-based rank-learners, conduct extensive comparison between these and state-of-the-art rank-learning algorithms. This comparison reveals some interesting and insightful findings about LtR algorithms including the finding that RF-based LtR algorithms are among the most robust techniques across datasets with diverse properties.

Improving the pull requests review process using learning-to-rank algorithms

Collaborative software development platforms (such as GitHub and GitLab) have become increasingly popular as they have attracted thousands of external contributors to contribute to open source projects. The external contributors may submit their contributions via pull requests, which must be reviewed before being integrated into the central repository. During the review process, reviewers provide feedback to contributors, conduct tests and request further modifications before finally accepting or rejecting the contributions. The role of reviewers is key to maintain the effective review process of the project. However, the number of decisions that reviewers can make is far superseded by the increasing number of pull requests submissions. To help reviewers to perform more decisions on pull requests within their limited working time, we propose a learning-to-rank (LtR) approach to recommend pull requests that can be quickly reviewed by reviewers. Different from a binary model for predicting the decisions of pull requests, our ranking approach complements the existing list of pull requests based on their likelihood of being quickly merged or rejected. We use 18 metrics to build LtR models and we use six different LtR algorithms, such as ListNet, RankNet, MART and random forest. We conduct empirical studies on 74 Java projects to compare the performances of the six LtR algorithms. We compare the best performing algorithm against two baselines obtained from previous research regarding pull requests prioritization: the first-in-and-first-out (FIFO) baseline and the small-size-first baseline. We then conduct a survey with GitHub reviewers to understand the perception of code reviewers regarding the usefulness of our approach. We observe that: (1) The random forest LtR algorithm outperforms other five well adapted LtR algorithms to rank quickly merged pull requests. (2) The random forest LtR algorithm performs better than both the FIFO and the small-size-first baselines, which means our LtR approach can help reviewers make more decisions and improve their productivity. (3) The contributor’s social connections and contributor’s experience are the most influential metrics to rank pull requests that can be quickly merged. (4) The GitHub reviewers that participated in our survey acknowledge that our approach complements existing prioritization baselines to help them to prioritize and to review more pull requests.

An Improved LambdaMART Algorithm Based on the Matthew Effect

Matthew effect is a desirable phenomenon for a ranking model in search engines and recommendation systems. However, most of algorithms of learning to rank (LTR) do not pay attention to Matthew effect. LambdaMART is a well-known LTR algorithm that can be further optimized based on Matthew effect. Inspired by Matthew effect, we distinguish queries with different effectiveness and then assign a higher weight to a query with higher effectiveness. We improve the gradient in the LambdaMART algorithm to optimize the queries with high effectiveness, that is, to highlight the Matthew effect of the produced ranking models. In addition, we propose strategies of evaluating a ranking model and dynamically decreasing the learning rate to both strengthen the Matthew effect of ranking models and improve the effectiveness of ranking models. We use Gini coefficient, mean-variance, quantity statistics, and winning number to measure the performances of the ranking models. Experimental results on multiple benchmark datasets show that the ranking models produced by our improved LambdaMART algorithm can exhibit a stronger Matthew effect and achieve higher effectiveness compared to the original one and other state-of-the-art LTR algorithms.

X-CLE a VER

Learning-to-Rank (LtR) solutions are commonly used in large-scale information retrieval systems such as Web search engines, which have to return highly relevant documents in response to user query within fractions of seconds. The most effective LtR algorithms adopt a gradient boosting approach to build additive ensembles of weighted regression trees. Since the required ranking effectiveness is achieved with very large ensembles, the impact on response time and query throughput of these solutions is not negligible. In this article, we propose X-CLE a VER, an iterative meta-algorithm able to build more efficient and effective ranking ensembles. X-CLE a VER interleaves the iterations of a given gradient boosting learning algorithm with pruning and re-weighting phases. First, redundant trees are removed from the given ensemble, then the weights of the remaining trees are fine-tuned by optimizing the desired ranking quality metric. We propose and analyze several pruning strategies and we assess their benefits showing that interleaving pruning and re-weighting phases during learning is more effective than applying a single post-learning optimization step. Experiments conducted using two publicly available LtR datasets show that X-CLE a VER can be successfully exploited on top of several LtR algorithms as it is effective in optimizing the effectiveness of the learnt ensembles, thus obtaining more compact forests that hence are much more efficient at scoring time.

Query-dependent learning to rank for cross-lingual information retrieval

Learning to rank (LTR), as a machine learning technique for ranking tasks, has become one of the most popular research topics in the area of information retrieval (IR). Cross-lingual information retrieval (CLIR), in which the language of the query is different from the language of the documents, is one of the important IR tasks that can potentially benefit from LTR. Our focus in this paper is the use of LTR for CLIR. To rank the documents in the target language in response to the query in the source language, we propose a local query-dependent approach based on LTR for CLIR, which is called LQ-DLTR for CLIR. The core idea of LQ-DLTR for CLIR is the use of the local characteristics of similar queries to construct the LTR model, instead of using a single global ranking model for all queries. Since the query and the documents are in different languages, the traditional features that are used in LTR cannot be used directly for CLIR. Thus, defining appropriate features is a major step in the use of LTR for CLIR. In this paper, three categories of cross-lingual features are defined: query–document features, document features, and query features. To define the cross-lingual features, translation resources are used to fill the gap between the documents and the queries. Then, in LQ-DLTR for CLIR, a neighborhood of similar queries based on cross-lingual query features is used to create a local ranking function by the LTR algorithm for a given query. The LTR algorithm uses two cross-lingual feature sets, namely document features and query–document features, to learn the model. The query features that are used to identify the neighbors are not involved in the learning phase. Experimental results indicate that the CLIR performance improves with the use of cross-lingual features that use several translations and their probabilities to compute the features, compared to the use of monolingual features in traditional LTR, which translate a query according to the best translation and ignore the probabilities. Moreover, experimental results show that LQ-DLTR for CLIR outperforms the baseline information retrieval methods and other LTR ranking models in terms of the MAP and NDCG measures.

Scalability and Performance of Random Forest based Learning-to-Rank for Information Retrieval

For a query submitted by a user, the goal of an information retrieval system is to return a list of documents which are highly relevant with respect to that query. Traditionally different scoring methods, ranging from simple heuristic models to probabilistic models, have been used for this task. Recently researchers have started to use supervised machine learning techniques for solving this problem which is then called the learning-to-rank (LtR) problem. Many supervised learning methods have been tested so far with empirical success over conventional methods [4] The random forest is a relatively simple but effective and efficient learning algorithm which aggregates the predictions of a large number of independent and variant base learners, namely decision trees. Its major benefits over other state-of-the-art methods include inherent parallelizability, ease of tuning and competitive performance. These benefits attract researchers across various disciplines where a random forest is a very popular choice. However, for LtR task, the random forest has not been thoroughly investigated. In this research, we investigate the random forest based LtR algorithms. We aim at improving the efficiency, effectiveness, and understanding of these algorithms. With respect to the first goal, we employ undersampling techniques and leverage the inherent structure of a random forest to achieve better scalability, especially for highly imbalanced datasets [2]. We also reduce the correlation among the trees to reduce learning time and to improve performance [3]. With respect to the second goal, we investigate various objective functions ranging from completely randomized splitting criterion to so-called listwise splitting [1]. We also conduct a thorough study on random forest based pointwise algorithms. With respect to the third goal, we develop methods for estimating the bias and variance of rank-learning algorithms, and examine their empirical behavior against parameters of the learning algorithm

Comparing Pointwise and Listwise Objective Functions for Random-Forest-Based Learning-to-Rank

Current random-forest (RF)-based learning-to-rank (LtR) algorithms use a classification or regression framework to solve the ranking problem in a pointwise manner. The success of this simple yet effective approach coupled with the inherent parallelizability of the learning algorithm makes it a strong candidate for widespread adoption. In this article, we aim to better understand the effectiveness of RF-based rank-learning algorithms with a focus on the comparison between pointwise and listwise approaches. We introduce what we believe to be the first listwise version of an RF-based LtR algorithm. The algorithm directly optimizes an information retrieval metric of choice (in our case, NDCG) in a greedy manner. Direct optimization of the listwise objective functions is computationally prohibitive for most learning algorithms, but possible in RF since each tree maximizes the objective in a coordinate-wise fashion. Computational complexity of the listwise approach is higher than the pointwise counterpart; hence for larger datasets, we design a hybrid algorithm that combines a listwise objective in the early stages of tree construction and a pointwise objective in the latter stages. We also study the effect of the discount function of NDCG on the listwise algorithm. Experimental results on several publicly available LtR datasets reveal that the listwise/hybrid algorithm outperforms the pointwise approach on the majority (but not all) of the datasets. We then investigate several aspects of the two algorithms to better understand the inevitable performance tradeoffs. The aspects include examining an RF-based unsupervised LtR algorithm and comparing individual tree strength. Finally, we compare the the investigated RF-based algorithms with several other LtR algorithms.

Quality versus efficiency in document scoring with learning-to-rank models

Learning-to-Rank (LtR) techniques leverage machine learning algorithms and large amounts of training data to induce high-quality ranking functions. Given a set of documents and a user query, these functions are able to precisely predict a score for each of the documents, in turn exploited to effectively rank them. Although the scoring efficiency of LtR models is critical in several applications – e.g., it directly impacts on response time and throughput of Web query processing – it has received relatively little attention so far.The goal of this work is to experimentally investigate the scoring efficiency of LtR models along with their ranking quality. Specifically, we show that machine-learned ranking models exhibit a quality versus efficiency trade-off. For example, each family of LtR algorithms has tuning parameters that can influence both effectiveness and efficiency, where higher ranking quality is generally obtained with more complex and expensive models. Moreover, LtR algorithms that learn complex models, such as those based on forests of regression trees, are generally more expensive and more effective than other algorithms that induce simpler models like linear combination of features.We extensively analyze the quality versus efficiency trade-off of a wide spectrum of state-of-the-art LtR, and we propose a sound methodology to devise the most effective ranker given a time budget. To guarantee reproducibility, we used publicly available datasets and we contribute an open source C++ framework providing optimized, multi-threaded implementations of the most effective tree-based learners: Gradient Boosted Regression Trees (GBRT), Lambda-Mart (λ-MART), and the first public-domain implementation of Oblivious Lambda-Mart (Ωλ-MART), an algorithm that induces forests of oblivious regression trees.We investigate how the different training parameters impact on the quality versus efficiency trade-off, and provide a thorough comparison of several algorithms in the quality-cost space. The experiments conducted show that there is not an overall best algorithm, but the optimal choice depends on the time budget.