Input Rankings Research Articles

Measuring robustness in rank aggregation based on the error-effectiveness curve

Rank aggregation is an obligatory operation for many tasks of democratic elections, product recommendation, and gene identification. While the awareness of imperfect information in input rankings would lead to unreliable aggregation results has promoted the robust rank aggregation to become an increasingly active research topic recently. In this study, we focus on the problem of robustness measuring of rank aggregation methods. We first provide an analytical framework of rank aggregation robustness, in which we generate rankings with adjustable imperfect information and observe the response of rank aggregation methods. In particular, to quantify the robustness of rank aggregation methods, we introduce the concept of error-effectiveness curve, which presents the aggregation effectiveness under different imperfect information scenarios, and describes the ability of each method in giving stable aggregation results. By doing so, a robustness measure of rank aggregation with random error is developed. Comprehensive experimental evaluations were conducted considering synthetic datasets with various levels of random error to demonstrate the validity of the proposed measure. Exact robustness was quantified for each evaluated rank aggregation methods, and significant robustness distinctions were achieved among them.

Weight set decomposition for weighted rank and rating aggregation: An interpretable and visual decision support tool

The problem of interpreting or aggregating multiple rankings is common to many real-world applications. Perhaps the simplest and most common approach is a weighted rank aggregation, wherein a (convex) weight is applied to each input ranking and then ordered. This paper describes a new tool for visualizing and displaying ranking information for the weighted rank aggregation method. Traditionally, the aim of rank aggregation is to summarize the information from the input rankings and provide one final ranking that hopefully represents a more accurate or truthful result than any one input ranking. While such an aggregated ranking is, and clearly has been, useful to many applications, it also obscures information. In this paper, we show the wealth of information that is available for the weighted rank aggregation problem due to its structure. We apply weight set decomposition to the set of convex multipliers, study the properties useful for understanding this decomposition, and visualize the indifference regions. This methodology reveals information–that is otherwise collapsed by the aggregated ranking–into a useful, interpretable, and intuitive decision support tool. Included are multiple illustrative examples, along with heuristic and exact algorithms for computing the weight set decomposition.

Методы определения приоритета входных параметров функционально-математических библиотек с закрытым исходным кодом

This work mainly aimed to identify the most suitable calculation approaches and methods for the effect of inputs upon the outputs of a functionally-mathematical library with inaccessible source code, and further use these methods and approaches to develop an input ranking tool. These methods and approaches were studied using a generalized regressive model representing an arbitrary functionally-mathematical library (an executable module). The paper studied two approaches to the determination of input effects upon the outputs: 1) correlation analysis; 2) neural network method. The first method is analytical calculation (couple by couple) of Pierson correlation coefficients for all input and output parameters. As an alternative, the study also investigated rank-based Spierman and Kendall correlations. The second approach basically meant neural network learning with required accuracy using the variation set of executable module inputs and their respective outputs with subsequent calculation of contribution from each parameter to the overall result for the neural network after learning. These approaches have also been compared in terms of several major criteria, i.e. application peculiarities, speed and output quality. The analysis has shown that neural-network learning method, despite its certain drawbacks, is more suitable for the task of this study. The paper also outlined possible ways for further improvement of this method and complexity increase of the suggested functionally-mathematical library model.

Beyond pairwise comparisons in social choice: A setwise Kemeny aggregation problem

In this paper, we advocate the use of setwise contests for aggregating a set of input rankings into an output ranking. We propose a generalization of the Kemeny rule where one minimizes the number of k-wise disagreements instead of pairwise disagreements (one counts 1 disagreement each time the top choice in a subset of alternatives of cardinality at most k differs between an input ranking and the output ranking). After an algorithmic study of this k-wise Kemeny aggregation problem, we introduce a k-wise counterpart of the majority graph. This graph reveals useful to divide the aggregation problem into several sub-problems, which enables to speed up the exact computation of a consensus ranking. By introducing a k-wise counterpart of the Spearman distance, we also provide a 2-approximation algorithm for the k-wise Kemeny aggregation problem. We conclude with numerical tests.

Efficient, robust and effective rank aggregation for massive biological datasets

Massive biological datasets are available in various sources. To answer a biological question (e.g., “which are the genes involved in a given disease?”), life scientists query and mine such datasets using various techniques. Each technique provides a list of results usually ranked by importance (e.g., a list of ranked genes). Combining the results obtained by various techniques, that is, combining ranked lists of elements into one list of elements is of paramount importance to help life scientists make the most of various results and prioritize further investigations. Rank aggregation techniques are particularly well-fitted with this context as they take in a set of rankings and provide a consensus, that is, a single ranking which is the “closest” to the input rankings. However, (i) the problem of rank aggregation is NP-hard in most cases (using an exact algorithm is currently not possible for more than a few dozens of elements) and (ii) several (possibly very different) exact solutions can be obtained. As answer to (i), many heuristics and approximation algorithms have been proposed. However, heuristics cannot guarantee how far from an exact solution the consensus ranking will be, and the approximation ratio of approximation algorithms dedicated to the problem is fairly high (not less than 3/2). No solution has yet been proposed to help true-users dealing with the problem encountered in point (ii).In this paper we present a complete system able to perform rank aggregation of massive biological datasets. Our solution is efficient as it is based on an original partitioning method making it possible to compute a high-quality consensus using an exact algorithm in a large number of cases. Our solution is robust as it clearly identifies for the user groups of elements whose relative order is the same in any optimal solution. These features provide answers to points (i) and (ii) and lie in mathematical bases offering guarantees on the computed result. Also, our solution is effective as it has been implemented into a real tool, ConquR-BioV2 is used for the life science community, and evaluated at large-scale using a very large number of datasets.

A highly scalable algorithm for weak rankings aggregation

• Fast and scalable algorithms to deal with optimal bucket order problem. • First ordering then append (FOTA) approach. • Comparison with counterpart approach (first clustering then ordering). • Sound experimental evaluation including benchmarks with very large datasets. The Optimal Bucket Order Problem (OBOP) is a rank aggregation problem which consists in finding a consensus ranking (with ties) that generalizes a set of input rankings. In this paper, with the aim of solving the OBOP in an efficient and scalable way, we propose several greedy algorithms based on different sort-first and cluster-second strategies. More specifically, the sorting step is based on the Borda method, whereas in the cluster step, pairs of adjacent buckets are suitably joined. The proposed methods are experimentally compared with the state-of-the-art greedy algorithms for solving the OBOP by using a large benchmark of real-world databases. Furthermore, we provide a complete statistical analysis of the experimental study, which shows that several of the proposed algorithms outperform the current state-of-the-art greedy algorithms. We also analyze the trade-off between accuracy and execution time of the algorithms to guide the users regarding the selection of the best option for each particular case. The study carried out shows that our proposal is not only competitive in terms of accuracy with the state-of-the-art evolutionary strategy for dealing with the OBOP, but is also fast and scalable.

Heuristic algorithms for aggregation of incomplete rankings in multiple criteria group decision making

We propose heuristics for constructing a compromise incomplete ranking based on partial rankings admitting incomparability. We consider the utilitarian and egalitarian perspectives oriented toward minimizing an average or a maximal distance from any input ranking. The proposed algorithms incorporate genetic algorithms, simulated annealing, tabu search, local search, and intuitive, dedicated procedures. We demonstrate their efficiency in a real-world case study concerning the ranking of insulating materials based on the conflicting, incomplete preferences of a few tens of Decision Makers (DMs). For each DM, we consider a single representative ranking consistent with his/her preferences or thousands of such rankings following incorporation of the robustness concern. The experimental comparison is generalized to artificially generated problems that differ in terms of the numbers of alternatives and input rankings, and diversity levels. The results are quantified with the quality of obtained rankings and computation time.

Towards a Framework for Evaluating the Research Performance of Chinese Double First-Class Universities

The study focuses on the research performance of Double First-Class (DFC) universities in China. A theoretical framework based on economic theory and the Analytic Hierarchy Process (AHP) method for a novel evaluation model is proposed. The model is performed on a sample of 41 participant DFC universities in China. Using data collected from these universities, it was found that there is no consistency between performance ranking, input ranking, and output ranking, with the best ranked universities far from the most efficient. These findings provide empirical evidence of DFC universities’ research performance situation and suggest strategies that the government can use to propel their sustainable development.

Beyond Pairwise Comparisons in Social Choice: A Setwise Kemeny Aggregation Problem

In this paper, we advocate the use of setwise contests for aggregating a set of input rankings into an output ranking. We propose a generalization of the Kemeny rule where one minimizes the number of k-wise disagreements instead of pairwise disagreements (one counts 1 disagreement each time the top choice in a subset of alternatives of cardinality at most k differs between an input ranking and the output ranking). After an algorithmic study of this k-wise Kemeny aggregation problem, we introduce a k-wise counterpart of the majority graph. It reveals useful to divide the aggregation problem into several sub-problems. We conclude with numerical tests.

Sparse hierarchical regression with polynomials

We present a novel method for sparse polynomial regression. We are interested in that degree r polynomial which depends on at most k inputs, counting at most $$\ell$$ monomial terms, and minimizes the sum of the squares of its prediction errors. Such highly structured sparse regression was denoted by Bach (Advances in neural information processing systems, pp 105–112, 2009) as sparse hierarchical regression in the context of kernel learning. Hierarchical sparse specification aligns well with modern big data settings where many inputs are not relevant for prediction purposes and the functional complexity of the regressor needs to be controlled as to avoid overfitting. We propose an efficient two-step approach to this hierarchical sparse regression problem. First, we discard irrelevant inputs using an extremely fast input ranking heuristic. Secondly, we take advantage of modern cutting plane methods for integer optimization to solve the remaining reduced hierarchical $$(k, \ell )$$-sparse problem exactly. The ability of our method to identify all k relevant inputs and all $$\ell$$ monomial terms is shown empirically to experience a phase transition. Crucially, the same transition also presents itself in our ability to reject all irrelevant features and monomials as well. In the regime where our method is statistically powerful, its computational complexity is interestingly on par with Lasso based heuristics. Hierarchical sparsity can retain the flexibility of general nonparametric methods such as nearest neighbors or regression trees (CART), without sacrificing much statistical power. The presented work hence fills a void in terms of a lack of powerful disciplined nonlinear sparse regression methods in high-dimensional settings. Our method is shown empirically to scale to regression problems with $$n\approx 10{,}000$$ observations for input dimension $$p\approx 1000$$.

Is Rank Aggregation Effective in Recommender Systems? An Experimental Analysis

Recommender Systems are tools designed to help users find relevant information from the myriad of content available online. They work by actively suggesting items that are relevant to users according to their historical preferences or observed actions. Among recommender systems, top- N recommenders work by suggesting a ranking of N items that can be of interest to a user. Although a significant number of top- N recommenders have been proposed in the literature, they often disagree in their returned rankings, offering an opportunity for improving the final recommendation ranking by aggregating the outputs of different algorithms. Rank aggregation was successfully used in a significant number of areas, but only a few rank aggregation methods have been proposed in the recommender systems literature. Furthermore, there is a lack of studies regarding rankings’ characteristics and their possible impacts on the improvements achieved through rank aggregation. This work presents an extensive two-phase experimental analysis of rank aggregation in recommender systems. In the first phase, we investigate the characteristics of rankings recommended by 15 different top- N recommender algorithms regarding agreement and diversity. In the second phase, we look at the results of 19 rank aggregation methods and identify different scenarios where they perform best or worst according to the input rankings’ characteristics. Our results show that supervised rank aggregation methods provide improvements in the results of the recommended rankings in six out of seven datasets. These methods provide robustness even in the presence of a big set of weak recommendation rankings. However, in cases where there was a set of non-diverse high-quality input rankings, supervised and unsupervised algorithms produced similar results. In these cases, we can avoid the cost of the former in favor of the latter.

A comprehensive comparison of four input variable selection methods for artificial neural network flow forecasting models

Artificial neural networks (ANNs) are increasingly used for flood forecasting. The performance of these models relies on the selection of appropriate inputs. However, Input Variable Selection (IVS) is typically performed using expert knowledge or simple linear methods. This research compares and evaluates four IVS methods including two model-free methods: partial correlation (PC), partial mutual information (PMI), and two novel model-based methods: an improved input omission (IO), and improved combined neural pathway strength (CNPS). A comprehensive comparison of performance efficacy for multiple IVS methods has not been published in literature before. Each method is used for daily and hourly lead times in the Bow and Don Rivers (both in Canada), respectively. These watersheds represent different hydrological systems and were selected to highlight the performance of the IVS methods under differing conditions. This research determines that the proposed CNPS produces the strongest performing ANNs based on the robustness of the inputs selected, comparison to other IVS methods, and models developed without IVS. Additionally, this research demonstrates that standard termination criteria do not reliably identify the optimum number of inputs for the ANNs and using a model-based optimization of inputs is recommended. As a result, it is recommended that the number of inputs be determined using a systematic approach, where each input selection is informed by an IVS-based input ranking, rather than a predefined termination criterion. Lastly, this research demonstrates that input usefulness is not binary concept; the correct number of selected inputs is dependant on the desired model complexity, instead of an arbitrarily selected IVS termination criteria.

Shapley effect application for variance-based sensitivity analysis of the few-group cross-sections

Analyzing the variance of complex computer models is an essential practice to assess and improve that model by identifying the influential parameters that cause the output variance. Variance-based sensitivity analysis is the process of decomposing the output variance into components associated with each input parameter. In this study, we applied a new concept of variance-based sensitivity analysis inspired by the game theory proposed by Shapley. The technique is called the Shapley effect, and it investigates the contribution of each input parameter as well as its interactions with every other parameter in the system by exploring all possible permutations between them. The Shapley effect is compared to the common Sobol indices technique (first order and total effects) to investigate their performance under correlated and uncorrelated parameters. The Shapley effect demonstrated superior performance when compared to the Sobol indices for correlated input parameters. Shapley effect captured the correlation between the input parameters, expressing the variance contribution in a single index instead of two indices, and normalization of the fractional indices is preserved without over or underestimation. On the other hand, the two algorithms we selected to calculate Sobol indices under correlated inputs experienced different issues including: over/underestimating the output variance, first order effect could be larger than the total effect, possibility of negative indices, unnormalized fractional indices, and difficulty to interpret the results. However, Sobol showed satisfactory performance when the inputs are uncorrelated as the numerical values and input ranking were in good agreement with Shapley effect. The main disadvantage of Shapley effect is its large computational cost especially for high dimensional problems where the number of possible input subsets becomes very large. The results of our tests showed that the thermal fission cross-section carried most of the uncertainty at BOL, and its contribution declines after fuel burnup, which is replaced by the uncertainty contribution of the fast cross-section parameters.

A New Method For Personnel Selection Based On Ranking Aggregation Using A Reinforcement Learning Approach

In this paper propose a new approach to the problem of aggregating rankings for obtaining an overall ranking. This is also referred to as the aggregation ranking in the personnel selection problem. Our approach is based on a distance measure between the individual and the overall ranking, and looks for the solution that minimizes the disagreement between the input rankings and the resulting aggregation. The method uses a reinforcement learning approach to build the aggregation and its performance and comparison with other approaches shows promising results.

Iterative ranking aggregation using quality improvement of subgroup ranking

Ranking aggregation concerns the combination of rankings to obtain a consensus ranking that best represents all the input rankings according to a specific criterion. However, finding the optimal aggregated ranking, i.e., the one with the highest quality, is usually NP-hard. To reduce the computational cost, many heuristic aggregation methods have been proposed. They cannot ensure the optimality and the qualities of aggregated rankings obtained actually can be further improved. To find a better aggregated ranking, a novel iterative ranking aggregation method (IRAM) is proposed in this paper using the quality improvement of the subgroup ranking. IRAM starts from an aggregated ranking generated by a traditional heuristic ranking aggregation method. In each iteration step, IRAM attempts to improve the ranking’s quality of a subgroup of items. If the ranking’s quality of a subgroup of items is improved, the full aggregated ranking’s quality is consequently improved. Moreover, a window iterative ranking aggregation method (W-IRAM) is designed, which is simpler than the IRAM. We prove that IRAM and W-IRAM bring better (or at least the same) aggregation qualities compared with the traditional heuristic ranking aggregation methods. Simulation results show that our iterative ranking aggregation approaches perform well.

Global sensitivity metrics from active subspaces

Predictions from science and engineering models depend on several input parameters. Global sensitivity analysis quantifies the importance of each input parameter, which can lead to insight into the model and reduced computational cost; commonly used sensitivity metrics include Sobol' total sensitivity indices and derivative-based global sensitivity measures. Active subspaces are part of an emerging set of tools for identifying important directions in a model's input parameter space; these directions can be exploited to reduce the model's dimension enabling otherwise infeasible parameter studies. In this paper, we develop global sensitivity metrics called activity scores from the active subspace, which yield insight into the important model parameters. We mathematically relate the activity scores to established sensitivity metrics, and we discuss computational methods to estimate the activity scores. We show two numerical examples with algebraic functions taken from simplified engineering models. For each model, we analyze the active subspace and discuss how to exploit the low-dimensional structure. We then show that input rankings produced by the activity scores are consistent with rankings produced by the standard metrics.

Context-aware Service Input Ranking by Learning from Historical Information

Users visit on-line services and compose them to accomplish on-line tasks, such as shopping on-line. Quite often, users enter the same information into various on-line services to finish on-line tasks. However, repetitively typing the same information into web forms is a tedious job for users. In this paper, we propose a context-aware ranking framework to rank values for input parameters. We propose 6 categories of ranking features constructed from various types of information, such as user contexts and patterns of user inputs. Our framework adopts learning-to-rank (LtR) algorithms that consist of a set of machine learned models to automatically construct ranking models by integrating the ranking features. When a user enters a value to an input parameter, an interaction between the user input and the input parameter is established. Our framework learns information relevant to such interactions and ranks user inputs in different contexts. Through empirical studies on the real-world on-line services, we obtain the following main results: (1) Among the 8 state-of-the-art learning-to-rank models, RankBoost can outperform other LtR models on ranking user inputs for input parameters; (2) Our framework using IRSVM that performs the worst among the LtR models outperforms the two baseline conventional ranking models and Google Chrome Autofilling, an industrial tool, on ranking user inputs to input parameters; and (3) We observe that the textual information of user inputs and input parameters is the most influential factor on ranking user inputs. Among the various types of contextual data, user locations and time matter the most to the ranking of user inputs.

Multiobjective PSO-based rank aggregation: Application in gene ranking from microarray data

Rank aggregation refers to the problem of aggregating a number of individual preference rankings of some items to yield a consensus ranking. This problem is usually posed as an optimization problem in which the average distance of the aggregated ranking from the input rank lists is minimized in order to obtain a consensus. However, it may be noted that minimizing the average distance does not always ensure an impartial aggregation. This is because the aggregated ranking may be very similar to one input ranking, whereas at the same time, it may be distant from another. This may reduce the overall average distance while producing a biased aggregated ranking. This possibility motivates us to pose the problem of rank aggregation as a multiobjective optimization problem. Here the objective is not only to minimize the average distance of the aggregated ranking from the input rank lists, but also to minimize the standard deviation of the distance values simultaneously to avoid any bias towards any particular input ranking. In order to minimize both the objective functions simultaneously, we propose a multiobjective particle swarm optimization (PSO) based rank aggregation algorithm in this article. The performance of the proposed technique is demonstrated for some artificial datasets. Moreover, its application in gene ranking from microarray gene expression data is studied and the performance is compared with that of some state-of-art techniques to establish its superiority.

LCA of emerging technologies: addressing high uncertainty on inputs' variability when performing global sensitivity analysis

In the life cycle assessment (LCA) context, global sensitivity analysis (GSA) has been identified by several authors as a relevant practice to enhance the understanding of the model's structure and ensure reliability and credibility of the LCA results. GSA allows establishing a ranking among the input parameters, according to their influence on the variability of the output. Such feature is of high interest in particular when aiming at defining parameterized LCA models.When performing a GSA, the description of the variability of each input parameter may affect the results. This aspect is critical when studying new products or emerging technologies, where data regarding the model inputs are very uncertain and may cause misleading GSA outcomes, such as inappropriate input rankings. A systematic assessment of this sensitivity issue is now proposed.We develop a methodology to analyze the sensitivity of the GSA results (i.e. the stability of the ranking of the inputs) with respect to the description of such inputs of the model (i.e. the definition of their inherent variability). With this research, we aim at enriching the debate on the application of GSA to LCAs affected by high uncertainties.We illustrate its application with a case study, aiming at the elaboration of a simple model expressing the life cycle greenhouse gas emissions of enhanced geothermal systems (EGS) as a function of few key parameters. Our methodology allows identifying the key inputs of the LCA model, taking into account the uncertainty related to their description.

Optimal Aggregation of Uncertain Preferences

A paradigmatic problem in social choice theory deals with the aggregation of subjective preferences of individuals --- represented as rankings of alternatives --- into a social ranking. We are interested in settings where individuals are uncertain about their own preferences, and represent their uncertainty as distributions over rankings. Under the classic objective of minimizing the (expected) sum of Kendall tau distances between the input rankings and the output ranking, we establish that preference elicitation is surprisingly straightforward and near-optimal solutions can be obtained in polynomial time. We show, both in theory and using real data, that ignoring uncertainty altogether can lead to suboptimal outcomes.