Paired Comparison Model Research Articles

Receiver operating characteristic analysis for paired comparison data

Abstract Paired comparison models are used for analysing data that involves pairwise comparisons among a set of objects. When the outcomes of the pairwise comparisons have no ties, the paired comparison models can be generalized as a class of binary response models. Receiver operating characteristic (ROC) curves and their corresponding areas under the curves are commonly used as performance metrics to evaluate the discriminating ability of binary response models. Despite their individual wide range of usage and their close connection to binary response models, ROC analysis to our knowledge has never been extended to paired comparison models since the problem of using different objects as the reference in paired comparison models prevents traditional ROC approach from generating unambiguous and interpretable curves. We focus on addressing this problem by proposing two novel methods to construct ROC curves for paired comparison data which provide interpretable statistics and maintain desired asymptotic properties. The methods are then applied and analysed on head-to-head professional sports competition data.

Comparative Analysis of the Existence and Uniqueness Conditions of Parameter Estimation in Paired Comparison Models

In this paper, paired comparison models with stochastic background are investigated. We focus on the models that allow three options for choice and the parameters are estimated by maximum likelihood method. The existence and uniqueness of the estimator are key issues of the evaluation. In the case of two options, a necessary and sufficient condition is given by Ford in the Bradley–Terry model. We generalize this statement for the set of strictly log-concave distribution. Although in the case of three options the necessary and sufficient condition is not known, there are two different sufficient conditions that are formulated in the literature. In this paper, we generalize them; moreover, we compare these conditions. Their capacities to indicate the existence of the maximum were analyzed using a large number of computer simulations. These simulations support that the new condition indicates the existence of the maximum much more frequently than the previously known ones.

Assisting the decision making-A generalization of choice models to handle the binary choices

<abstract> <p>This research fundamentally aims at providing a generalized framework to assist the launch of paired comparison models while dealing with discrete binary choices. The purpose is served by exploiting the fundaments of the exponential family of distributions. The proposed generalization is proved to cater to seven paired comparison models as members of this newly developed mechanism. The legitimacy of the devised scheme is demonstrated through rigorous simulation-based investigation as well as keenly persuaded empirical evaluations. A detailed analysis, covering a wide range of parametric settings, through the launch of Gibbs Sampler—a notable extension of Markov Chain Monte Carlo methods, is conducted under the Bayesian paradigm. The outcomes of this research substantiate the legitimacy of the devised general structure by not only successfully retaining the preference ordering but also by staying consistent with the established theoretical framework of comparative models.</p> </abstract>

Uncovering the differences between the violin and erhu musical instruments by statistical analysis of multiple musical pieces

The violin and erhu are two of Western and Chinese music’s main bowed string instruments. Recent work has compared the different emotional characteristics between the violin and erhu on the Butterfly Lovers Concerto. In our study, we examine several hypotheses to investigate whether the previous studies’ results hold generally. Four musical excerpts were extracted from four famous Chinese and Western classical pieces, and the excerpts were divided into four emotional categories: Happy, Sad, Agitated, and Calm. Based on the Butterfly Lovers results, we expected that: (1) the violin has a more Happy emotional characteristic than the erhu, while the erhu is comparatively more Sad, and (2) the violin is better at conveying high-Arousal excerpts. We used the Bradley–Terry–Luce (BTL) paired-comparison model to obtain the ranking scores and identify statistically significant differences between the two instruments. The erhu was consistently perceived as sadder than the violin for all Sad excerpts, while the violin was generally calmer and more agitated for those categories. Further study with more listeners and excerpts is needed to verify whether these results generally hold and at a statistically significant level.

On ranking climate factors affecting the living organisms based on paired comparison model, a Bayesian approach

The method of paired comparisons (PC) endeavors to rank treatments presented in pairs to panelists (or respondents, judges, jurists, etc.) and they have to select the better one based on sensory evaluations. Sometimes the situations may occur when the panelists cannot discriminate between the treatments and declare a tie. In this study, an effort is made to extend the Weibull PC model to accommodate ties. The extended Weibull PC model is analyzed using Bayesian paradigm. Four different loss functions are used under noninformative (Uniform and Jeffreys) priors. The posterior and marginal posterior distributions are derived. The posterior estimates, posterior risks, preference probabilities, posterior probabilities and predictive probabilities are evaluated to know the ranking of ecological factor. The goodness of the proposed model is assessed. The entire analysis is carried out using a real data set based on the preference for the ecological factors.

Accelerated MM Algorithms for Inference of Ranking Scores from Comparison Data

Accelerated Algorithms for RankingAssigning ranking scores to items based on observed comparison data (e.g., paired comparisons, choice, and full ranking outcomes) has been of continued interest in a wide range of applications, including information search, aggregation of social opinions, electronic commerce, online gaming platforms, and more recently, evaluation of machine learning algorithms. The key problem is to compute ranking scores, which are of interest for quantifying the strength of skills, relevancies, or preferences, and prediction of ranking outcomes. One of the most popular statistical models of ranking outcomes is the Bradley–Terry model for paired comparisons and its extensions to choice and full ranking outcomes. In “Accelerated MM Algorithms for Inference of Ranking Scores from Comparison Data,” M. Vojnovic, S.-Y. Yun, and K. Zhou show that a popular MM algorithm for inference of ranking scores for generalized Bradley–Terry ranking models suffers a slow convergence issue, and they propose a new accelerated algorithm that resolves this shortcoming and can yield substantial convergence speedups.

Bayesian paired comparison with the bpcs package

This article introduces the bpcs R package (Bayesian Paired Comparison in Stan) and the statistical models implemented in the package. This package aims to facilitate the use of Bayesian models for paired comparison data in behavioral research. Bayesian analysis of paired comparison data allows parameter estimation even in conditions where the maximum likelihood does not exist, allows easy extension of paired comparison models, provides straightforward interpretation of the results with credible intervals, has better control of type I error, has more robust evidence towards the null hypothesis, allows propagation of uncertainties, includes prior information, and performs well when handling models with many parameters and latent variables. The bpcs package provides a consistent interface for R users and several functions to evaluate the posterior distribution of all parameters to estimate the posterior distribution of any contest between items and to obtain the posterior distribution of the ranks. Three reanalyses of recent studies that used the frequentist Bradley–Terry model are presented. These reanalyses are conducted with the Bayesian models of the bpcs package, and all the code used to fit the models, generate the figures, and the tables are available in the online appendix.

Estimation of final standings in football competitions with a premature ending: the case of COVID-19

We study an alternative approach to determine the final league table in football competitions with a premature ending. For several countries, a premature ending of the 2019/2020 football season has occurred due to the COVID-19 pandemic. We propose a model-based method as a possible alternative to the use of the incomplete standings to determine the final table. This method measures the performance of the teams in the matches of the season that have been played and predicts the remaining non-played matches through a paired-comparison model. The main advantage of the method compared to the incomplete standings is that it takes account of the bias in the performance measure due to the schedule of the matches in a season. Therefore, the resulting ranking of the teams based on our proposed method can be regarded as more fair in this respect. A forecasting study based on historical data of seven of the main European competitions is used to validate the method. The empirical results suggest that the model-based approach produces more accurate predictions of the true final standings than those based on the incomplete standings.

The semialgebraic geometry of saturated optimal designs for the Bradley–Terry model

Optimal design theory for nonlinear regression studies local optimality on a given design space. We identify designs for the Bradley--Terry paired comparison model with small undirected graphs and prove that every saturated D-optimal design is represented by a path. We discuss the case of four alternatives in detail and derive explicit polynomial inequality descriptions for optimality regions in parameter space. Using these regions, for each point in parameter space we can prescribe a D-optimal design.

Modelling changes over time in a multivariate paired comparison: An application to window display design

This article proposes an alternative method of making comparative judgements in multivariate paired comparisons (PCs) where judgements about change are made directly by comparing an object at two time points for each of a series of attributes. The application deals with the design of shop window displays where products should be arranged by teams of vocational students according to aesthetic principles (attributes). The photos of the students’ window displays at time 1 (before feedback) and at time 2 (after feedback) were compared by judging each attribute as to whether it was fulfilled better at time 1 or at time 2. An advantage of this PC approach over an alternative of a scoring system is the possibility to assess even subtle changes of various aspects of attractiveness, which cannot easily be measured using a score. To analyse these data, we used earlier work which developed both a multivariate PC pattern model for multi-attribute data and a PC model over time and defined a multivariate PC model of changes (MPCC). The model can be fitted as a non-standard Poisson log-linear model and provides estimates of change for the three attributes for time 2 and we were able to check for possible interaction effects between these attributes.

Detecting Heterogeneity of Intervention Effects in Comparative Judgments.

Comparative measures such as paired comparisons and rankings are frequently used to evaluate health states and quality of life. The present article introduces log-linear Bradley-Terry (LLBT) models to evaluate intervention effectiveness when outcomes are measured as paired comparisons or rankings and presents a combination of the LLBT model and model-based recursive partitioning (MOB) to detect treatment effect heterogeneity. The MOB LLBT approach enables researchers to identify subgroups that differ in the preference order and in the effect an intervention has on choice behavior. Applicability of MOB LLBT models is demonstrated using an artificial data example with known data-generating mechanism and a real-world data example focusing on drug-harm perception among music festival visitors. In the artificial data example, the MOB LLBT model is able to adequately recover the "true" (population) model. In the real-world data example, the standard LLBT model confirms the existence of a situational willingness among festival visitors to trivialize drug harm when peer consumption behavior is made cognitively accessible. In addition, MOB LLBT results suggest that this trivialization effect is highly context-dependent and most pronounced for participants with low-to-moderate alcohol intoxication who also proactively contacted a substance counselor at the festival venue. Both data examples suggest that MOB LLBT models allow for more nuanced statements about the effectiveness of interventions. We provide R code examples to implement MOB LLBT models for paired comparisons, rankings, and rating (Likert-type) data.

Bayesian Analysis of the Weibull Paired Comparison Model Using Numerical Approximation

The method of paired comparisons (PC) is widely used to rank items using sensory evaluations. The PC models are developed to provide basis for such comparisons. In this study, the Weibull PC model is analyzed under the Bayesian paradigm using noninformative priors and different loss functions, namely, Squared Error Loss Function (SELF), Quadratic Loss Function (QLF), DeGroot Loss Function (DLF), and Precautionary Loss Function (PLF). Numerical approximation is used to illustrate the entire estimation procedure. A real dataset showing usage preferences for different cellphone brands, Huawei (HW), Samsung (SS), Oppo (OP), QMobile (QM), and Nokia (NK), is used. Quadrature method is used to evaluate the Bayes estimates, their posterior risks, preference probabilities, predictive probabilities, and posterior probabilities to establish and verify ranking order of the competing cellphone brands under study. The results show that the paired comparison model under the study using Bayesian approach involving various loss functions can offer mathematical approach to evaluate cellphone brand preferences. The ranking provided by the model is justifiable according to the usage preference for these cellphone brands. The ranking given by the model indicates that cellphone brand Samsung is preferred the most and QMobile is the least preferred. The plausibility of the model is also assessed using the Chi square test of goodness of fit.

Hierarchical paired comparison modeling, a cultural consensus theory approach

We introduce a set of models designed to analyze datasets involving responses from multiple subjects on pairwise comparisons from a fixed discrete set of alternatives. These models are part of a greater body of work known as Cultural Consensus Theory (CCT). Like other CCT models, these simultaneously infer each individual’s tendency toward aligning with the group consensus, level of agreement on each item, and also a latent consensus value of each alternative. Two primary models are discussed, referred to as the Strong and Weak Consensus Paired-Comparison Models (SCPCM and WCPCM respectively). The SCPCM works under the assumption that all individuals are answering in accordance to the latent consensus values but with varying degrees of accuracy, while the WCPCM relaxes this assumption and assumes minor deviations from latent consensus values in people’s average valuation of alternatives. The WCPCM also includes inferences on participants’ individual tendencies toward self-consistency (related to their tendencies toward committing violations of transitivity) as well as inferences on the tendency of each item to be evaluated consistently by individuals. The Case III Thurstonian model is used as the backbone for both CPCMs, and inference is conducted under a hierarchical Bayesian framework. Model checks along with applications to both simulated and real data are overviewed.

Forecasting the Final Ranking in Competitions with a Premature Ending due to COVID-19

We study an alternative approach to determine the final league table in football competitions with a premature ending. For several countries, a premature ending of the 2019/2020 football season has occurred due to the COVID-19 pandemic. We propose a model-based method as a possible alternative to the use of the incomplete standings to determine the final table. This method measures the performance of the teams in the matches of the season that have been played and predicts the remaining non-played matches through a paired-comparison model. The main advantage of the method compared to the incomplete standings is that it takes account of the bias in the performance measure due to the schedule of the matches in a season. Therefore, the resulting ranking of the teams based on our proposed method can be regarded as more fair in this respect. A forecasting study based on historical data of seven of the main European competitions is used to validate the method. The empirical results suggest that the model-based approach produces more accurate predictions of the true final standings than those based on the incomplete standings.

The Maxwell paired comparison model under Bayesian paradigm using informative priors

In paired comparison (PC) experiments, items are judged on subjective criterion in order to decide which of the two objects is preferable. A new PC model is developed and analyzed under Bayesian framework using two informative priors – conjugate prior and Dirichlet prior. The hyperparameters are elicited through the prior predictive distribution approach. The proposed model is employed to analyze the preference data of brands of drinking water. We consider two data sets with the sample sizes of 5 and 35 to explore the small and large sample behaviors of the proposed model. It is observed that the proposed model is capable of retaining the underlying preference ordering in both small and large sample scenarios. The conclusions are based on elicited values of worth parameters, estimated preference probabilities, and Bayes factor.

Bayesian analysis of the Weibull paired comparison model using informative prior

Paired comparison (PC) scheme is very helpful in ranking objects. In this paper, a new paired comparison model is developed using the Weibull distribution and analyzed in Bayesian framework using informative prior under different loss functions. Summary of posterior estimates is evaluated to assess the ranking of the social media applications, i.e., Facebook, Whatsapp, Instagram, Skype and Twitter. The ranking presented by the model is justifiable according to the prevalence of the usage of the social media applications. The developed model is found to be plausible by the goodness of fit criteria.

Learning the distribution of latent variables in paired comparison models with round-robin scheduling

Paired comparison data considered in this paper originate from the comparison of a large number $N$ of individuals in couples. The dataset is a collection of results of contests between two individuals when each of them has faced $n$ opponents, where $n\ll N$. Individuals are represented by independent and identically distributed random parameters characterizing their abilities. The paper studies the maximum likelihood estimator of the parameters distribution. The analysis relies on the construction of a graphical model encoding conditional dependencies of the observations which are the outcomes of the first $n$ contests each individual is involved in. This graphical model allows to prove geometric loss of memory properties and deduce the asymptotic behavior of the likelihood function. This paper sets the focus on graphical models obtained from round-robin scheduling of these contests. Following a classical construction in learning theory, the asymptotic likelihood is used to measure performance of the maximum likelihood estimator. Risk bounds for this estimator are finally obtained by sub-Gaussian deviation results for Markov chains applied to the graphical model.

Masking data: a solution to social desirability bias in paired comparison experiments

This paper deals with the reduction of evasive answering bias in paired comparisons studies by providing a masking mechanism to the judges. Specifically, the Warner (1965) masking design is applied in the application of Bradley and Terry (1952) paired comparison model. For estimating the worth parameters and preference probabilities, Bayesian method of estimation is applied. To study the behavior of Bayes estimates and the effect of masking parameter, simulation study is performed. It is observed that the preference ordering is not disturbed when the number of judges is moderate to large. Also, the preference ordering is observed to be robust with respect to the extent of masking, when the number of judges is large. These findings are also supported by a numerical study using real data.

Quantifying Visual Abstraction Quality for Computer-Generated Illustrations

We investigate how the perceived abstraction quality of computer-generated illustrations is related to the number of primitives (points and small lines) used to create them. Since it is difficult to find objective functions that quantify the visual quality of such illustrations, we propose an approach to derive perceptual models from a user study. By gathering comparative data in a crowdsourcing user study and employing a paired comparison model, we can reconstruct absolute quality values. Based on an exemplary study for stippling, we show that it is possible to model the perceived quality of stippled representations based on the properties of an input image. The generalizability of our approach is demonstrated by comparing models for different stippling methods. By showing that our proposed approach also works for small lines, we demonstrate its applicability toward quantifying different representational drawing elements. Our results can be related to Weber--Fechner’s law from psychophysics and indicate a logarithmic relationship between number of rendering primitives in an illustration and the perceived abstraction quality thereof.

BTLLasso: A Common Framework and Software Package for the Inclusion and Selection of Covariates in Bradley-Terry Models

In paired comparison models, the inclusion of covariates is a tool to account for the heterogeneity of preferences and to investigate which characteristics determine the preferences. Although methods for the selection of variables have been proposed no coherent framework that combines all possible types of covariates is available. There are three different types of covariates that can occur in paired comparisons, the covariates can either vary over the subjects, the objects or both the subjects and the objects of the paired comparisons. This paper gives an overview over all possible types of covariates in paired comparisons and introduces a general framework to include covariate effects into BradleyTerry models. For each type of covariate, appropriate penalty terms that allow for sparser models and, therefore, easier interpretation are proposed. The whole framework is implemented in the R package BTLLasso. The main functionality and the visualization tools of the package are introduced and illustrated by real data sets.