Learning To Rank Approach Research Articles

SABER: Stochastic-Aware Bootstrap Ensemble Ranking for portfolio management

Portfolio management (PM) is a central subject of finance that presents many unresolved challenges. Although solutions have been proposed since the 1950’s, recent advances in machine learning are revolutionizing investments. Whereas supervised paradigms such as regression and classification are established in this direction, emerging research suggests that the learning-to-rank (LtR) paradigm can be more effective. Yet, these works are not many and overlook some important aspects. These aspects include the risk inherent in stocks, feature representation, portfolio cardinality and allocation. Addressing these gaps, this paper presents Stochastic-Aware Bootstrap Ensemble Ranking (SABER), an LtR method that directly minimizes the uncertainty and improves ranking performance. In addition, we propose Merged Bootstrap Selection for dynamic portfolio cardinality optimization. Trained on macroeconomic, fundamental and market indicators, the proposed methods outperform a state-of-the-art LtR approach, reaching annual returns and volatility of 83.48% and 29.89% across 6 years. The former also considers renowned weight allocation methods and achieves Sharpe and Sortino ratios of 2.79 and 5.28 respectively. Finally, the computational complexity of the proposed method is proven comparable to that of current methods.

Software defect prediction using learning to rank approach

Software defect prediction (SDP) plays a significant role in detecting the most likely defective software modules and optimizing the allocation of testing resources. In practice, though, project managers must not only identify defective modules, but also rank them in a specific order to optimize the resource allocation and minimize testing costs, especially for projects with limited budgets. This vital task can be accomplished using Learning to Rank (LTR) algorithm. This algorithm is a type of machine learning methodology that pursues two important tasks: prediction and learning. Although this algorithm is commonly used in information retrieval, it also presents high efficiency for other problems, like SDP. The LTR approach is mainly used in defect prediction to predict and rank the most likely buggy modules based on their bug count or bug density. This research paper conducts a comprehensive comparison study on the behavior of eight selected LTR models using two target variables: bug count and bug density. It also studies the effect of using imbalance learning and feature selection on the employed LTR models. The models are empirically evaluated using Fault Percentile Average. Our results show that using bug count as ranking criteria produces higher scores and more stable results across multiple experiment settings. Moreover, using imbalance learning has a positive impact for bug density, but on the other hand it leads to a negative impact for bug count. Lastly, using the feature selection does not show significant improvement for bug density, while there is no impact when bug count is used. Therefore, we conclude that using feature selection and imbalance learning with LTR does not come up with superior or significant results.

An Attention-Based Interactive Learning-to-Rank Model for Document Retrieval

The core issue of learning-to-rank (LTR) for document retrieval lies in finding an optimal ranking policy to meet the search intent of the user. The majority of proposed LTR approaches treat the ranking as a static process, employing a fixed ranking policy to immediately assign scores to documents. By contrast, ranking is not a static but an interactive process where the user continues interacting with the document retrieval system through information exchange such as search intent (e.g., rating or clicking for the retrieved items). We model the interactive ranking process (IRP), and propose an Attention-Based Interactive LTR model (AIRank) to constitute an intent-aware flexible ranking policy to gratify the user’s need. To enhance the ranking quality, the inherent relations among documents are procured by the self-attention method to contribute to an enriched user intent representation. Furthermore, we mend the policy gradient learning method to train the AIRank in the IRP. Experiments demonstrate the effectiveness of AIRank compared to the state-of-the-art methods in terms of normalized discounted cumulative gain and expected reciprocal rank.

Company Ranking Prediction Based on Network Big Data

With the advent of the era of big data, it has become one of the important tasks of data mining to utilize the massive info on the internet to realize the forecast of relevant aspects. In recent years, some scholars have studied how to use company’s information for the revenue or stock forecasts, but there is very little research on inter-company ranking predictions. So from the perspective, this paper brings in the Learning-to-Rank (LtR) method into the corporation ranking prognostication, and proposes the Company Ranking Prediction Model (CRPM) grounded on LtR. First, obtain the information on the Fortune 500 official website and other correlative sites. Through data analysis, data cleaning, ranking processing, label division and data formatting, a brand-new corporation data-set containing four major categories of features (10 characteristics in total) is constructed. Then bring in four commonly utilized LtR approaches, employing the previously created data-set to train different models. Finally, each model is evaluated by the two information retrieval metrics, normalized discounted cumulative gain and mean average precision. The experimental results show that the assessment best-performing model CRPM-LambdaMART and CRPM-RandomForests can tellingly forecast the ranking of Fortune 500 companies in the next year, which has certain practical value for financiers to reasonably planning their investments and corporation managements to legitimately programming their strategic deployments.

Learning to Rank for Educational Search Engines

In this digital age, there is an abundance of online educational materials in public and proprietary platforms. To allow effective retrieval of educational resources, it is a necessity to build keyword-based search engines over these collections. In modern Web search engines, high-quality rankings are obtained by applying machine learning techniques, known as learning to rank (LTR). In this article, our focus is on constructing machine-learned ranking models to be employed in a search engine in the education domain. Our contributions are threefold. First, we identify and analyze a rich set of features (including click-based and domain-specific ones) to be employed in educational search. LTR models trained on these features outperform various baselines based on ad-hoc retrieval functions and two neural models. As our second contribution, we utilize domain knowledge to build query-dependent ranking models specialized for certain courses or education levels. Our experiments reveal that query-dependent models outperform both the general ranking model and other baselines. Finally, given well-known importance of user clicks in LTR, our third contribution is for handling singleton queries without any click information. To this end, we propose a new strategy to “propagate” click information from the other, similar, queries to the singleton queries. The proposed click propagation approach yields a better ranking performance than the general ranking model and another baseline from the literature. Overall, these findings reveal that both the general and query-dependent ranking models, trained using LTR approaches, yield high effectiveness in educational search, which may ultimately lead to a better learning experience.

Learning to rank for multi-label text classification: Combining different sources of information

AbstractEfficiently exploiting all sources of information such as labeled instances, classes’ representation, and relations of them has a high impact on the performance of Multi-Label Text Classification (MLTC) systems. Most of the current approaches use labeled documents as the primary source of information for MLTC. We investigate the effectiveness of different sources of information— such as the labeled training data, textual labels of classes, and taxonomy relations of classes— for MLTC. More specifically, first, for each document–class pair, different features are extracted using different sources of information. The features reflect the similarity of classes and documents. Then, MLTC is considered to be a ranking problem, and a learning to rank (LTR) approach is used for ranking classes regarding documents and selecting labels of documents. An important characteristic of many MLTC instances is that documents can belong to multiple classes and there are implicit relations between classes. We apply score propagation on top of LTR to incorporate co-occurrence patterns of classes in labeled documents. Our main findings are the following. First, using an LTR approach integrating all features, we observe significantly better performance than previous systems for MLTC. Specifically, we show that simple classification approaches fail when there is a high number of classes. Second, the analysis of feature weights reveals the relative importance of various sources of evidence, also giving insight into the underlying classification problem. Interestingly, the results indicate that the titles of documents are more informative than all other sources of information. Third, a lean-and-mean system using only four features is able to perform at 96% of the large LTR model that we propose in this paper. Fourth, using the co-occurrence information of classes helps in classifying documents more accurately. Our results show that the co-occurrence information is more helpful when the underlying classifier has a poor performance.

Improving the pull requests review process using learning-to-rank algorithms

Collaborative software development platforms (such as GitHub and GitLab) have become increasingly popular as they have attracted thousands of external contributors to contribute to open source projects. The external contributors may submit their contributions via pull requests, which must be reviewed before being integrated into the central repository. During the review process, reviewers provide feedback to contributors, conduct tests and request further modifications before finally accepting or rejecting the contributions. The role of reviewers is key to maintain the effective review process of the project. However, the number of decisions that reviewers can make is far superseded by the increasing number of pull requests submissions. To help reviewers to perform more decisions on pull requests within their limited working time, we propose a learning-to-rank (LtR) approach to recommend pull requests that can be quickly reviewed by reviewers. Different from a binary model for predicting the decisions of pull requests, our ranking approach complements the existing list of pull requests based on their likelihood of being quickly merged or rejected. We use 18 metrics to build LtR models and we use six different LtR algorithms, such as ListNet, RankNet, MART and random forest. We conduct empirical studies on 74 Java projects to compare the performances of the six LtR algorithms. We compare the best performing algorithm against two baselines obtained from previous research regarding pull requests prioritization: the first-in-and-first-out (FIFO) baseline and the small-size-first baseline. We then conduct a survey with GitHub reviewers to understand the perception of code reviewers regarding the usefulness of our approach. We observe that: (1) The random forest LtR algorithm outperforms other five well adapted LtR algorithms to rank quickly merged pull requests. (2) The random forest LtR algorithm performs better than both the FIFO and the small-size-first baselines, which means our LtR approach can help reviewers make more decisions and improve their productivity. (3) The contributor’s social connections and contributor’s experience are the most influential metrics to rank pull requests that can be quickly merged. (4) The GitHub reviewers that participated in our survey acknowledge that our approach complements existing prioritization baselines to help them to prioritize and to review more pull requests.

An evolutionary strategy with machine learning for learning to rank in information retrieval

Learning to rank (LTR) is one of the problems attracting researchers in information retrieval (IR). The LTR problem refers to ranking the retrieved documents for users in search engines, question answering and product recommendation systems. There is a number of LTR approaches based on machine learning and computational intelligence techniques. Most existing LTR methods have limitations, such as being too slow or not being very effective or requiring a huge computer memory to operate. This paper proposes a LTR method that combines a $$(1+1)$$ -evolutionary strategy with machine learning. Three variants of the method are investigated: ES-Rank, IESR-Rank and IESVM-Rank. They differ on the chromosome initialisation mechanism for the evolutionary process. ES-Rank simply sets all genes in the initial chromosome to the same value. IESR-Rank uses linear regression, and IESVM-Rank uses support vector machine for the initialisation process. Experimental results from comparing the proposed method to fourteen other approaches from the literature show that IESR-Rank achieves the overall highest performance. Ten problem instances are used here, obtained from four datasets: MSLR-WEB10K, LETOR 3 and LETOR 4. Performance is measured at the top-10 query–document pairs retrieved, using five metrics: mean average precision (MAP), root-mean-square error (RMSE), precision (P@10), reciprocal rank (RR@10) and normalized discounted cumulative gain (NDCG@10). The contribution of this paper is proposing an effective and efficient LTR method combining a list-wise evolutionary technique with point-wise and pair-wise machine learning techniques.

Robust Learning to Rank Based on Portfolio Theory and AMOSA Algorithm

Effectiveness is the most important factor considered in the ranking models yielded by algorithms of learning to rank (LTR). Most of the related ranking models only focus on improving the average effectiveness but ignore robustness. When a ranking model ignores robustness, the effectiveness for many queries is possibly very poor although the average effectiveness for all queries is relatively high. Therefore, Wang et al. first consider robustness in their ranking models. However, the robustness formula defined by Wang et al. cannot characterize those queries whose effectiveness are hurt seriously in comparison with the baseline model. In order to overcome this shortcoming, we propose a novel formula of characterizing robustness based on portfolio theory, and construct a multiobjective optimization model of the robust LTR in which the formula is used. Based on this model, we propose an approach of risk-sensitive and robust LTR, named as R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Rank, which is based on the framework of archived multiobjective simulated annealing algorithm and the idea of preference ranking organization method for enrichment evaluation. The experimental results show that the ranking models produced by our proposed R <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Rank approach are better in both effectiveness and robustness than those produced by three state-of-the-art LTR approaches.