Paired Comparison Model Research Articles

Dynamic paired comparison models with stochastic variances

In paired comparison experiments, the worth or merit of a unit is measured through comparisons against other units. When paired comparison outcomes are collected over time and the merits of the units may be changing, it is often convenient to assume the data follow a non-linear state-space model. Typical paired comparison state-space models that assume a fixed (unknown) autoregressive variance do not account for the possibility of sudden changes in the merits. This is a particular concern, for example, in modeling cognitive ability in human development; cognitive ability not only changes over time, but also can change abruptly. We explore a particular extension of conventional state-space models for paired comparison data that allows the state variance to vary stochastically. Models of this type have recently been developed and applied to modeling financial data, but can be seen to have applicability in modeling paired comparison data. A filtering algorithm is also derived that can be used in place of likelihood-based computations when the number of objects being compared is large. Applications to National Football League game outcomes and chess game outcomes are presented.

Hierarchical modeling of paired comparison data.

The method of paired comparisons belongs to a small group of techniques that provide explicit information about the consistency of individual and aggregated choices. This article investigates the link between the individual- and group-level judgments by extending R. D. Luce's (1959) model, which was originally developed for individual choice behavior, to a mixed-effects paired comparison model. It is shown that standard multilevel software for binary data can be used to estimate the model. The interpretation of the paired comparison parameters and statistical model tests are discussed in detail. An extensive analysis of an experimental study illustrates the usefulness of a hierarchical approach in modeling multiple pairwise judgments.

Parameter Estimation in Large Dynamic Paired Comparison Experiments

SUMMARY Paired comparison data in which the abilities or merits of the objects being compared may be changing over time can be modelled as a non-linear state space model. When the population of objects being compared is large, likelihood-based analyses can be too computationally cumbersome to carry out regularly. This presents a problem for rating populations of chess players and other large groups which often consist of tens of thousands of competitors. This problem is overcome through a computationally simple non-iterative algorithm for fitting a particular dynamic paired comparison model. The algorithm, which improves over the commonly used algorithm of Elo by incorporating the variability in parameter estimates, can be performed regularly even for large populations of competitors. The method is evaluated on simulated data and is applied to ranking the best chess players of all time, and to ranking the top current tennis-players.

Paired comparison models applied to the design of the Major League baseball play-offs

This paper presents an analysis of the eff ect of various baseball play-off configurations on the probability of advancing to the World Series. Play-off games are assumed to be independent. Several paired comparisons models are considered for modeling the probability of a home team winning a single game as a function of the winning percentages of the contestants over the course of the season. The uniform and logistic regression models are both adequate, whereas the Bradley-Terry model (modified for within-pair order eff ects, i.e. the home field advantage) is not. The single-game probabilities are then used to compute the probability of winning the play-off s under various structures. The extra round of play-off s, instituted in 1994, significantly lowers the probability of the team with the best record advancing to the World Series, whereas home field advantage and the diff erent possible play-offdraws have a minimal eff ect.

A Class of Probability Distributions Characterized Through Paired Comparison Models

We examine the paired comparison model and obtain a class of probability distributions for which the ratio of intrinsic worths of two products remains unaltered irrespective of the lower time truncation point applied to items of both products. The class of distributions so obtained has been shown to have many nice properties both from the reliability point of view and paired comparison point of view. We also present a characterization result using its within class properties.

Modeling within-subject dependencies in ordinal paired comparison data

This paper presents two probabilistic models based on the logistic and the normal distribution for the analysis of dependencies in individual paired comparison judgments. It is argued that a core assumption of latent class choice models, independence of individual decisions, may not be well-suited for the analysis of paired comparison data. Instead, the analysis and interpretation of paired comparison data may be much simplified by allowing for within-person dependencies that result from repeated evaluations of the same options in different pairs. Moreover, by relating dependencies among the individual-level responses to (in)consistencies in the judgmental process, we show that the proposed graded paired comparison models reduce to ranking models under certain conditions. Three applications are presented to illustrate the approach.

Modelling Paired Comparison Data with Large Numbers of Draws and Large Variability of Draw Percentages Among Players

A linear paired comparison model is proposed that allows a large number of draws and large variability of draw percentages among the players as is the case for chess or soccer matches. The model can also be extended to allow home ground advantage. When only summary results are available, maximum likelihood estimation is not feasible and we recommend a method based on matching the numbers of home wins, home draws, away wins and away draws for each team with their expected values. We also discuss the problem of obtaining estimated standard errors. As an illustration, the suggested model is fitted to the 1993-94 English Premier Soccer League results.

Dynamic Stochastic Models for Time-Dependent Ordered Paired Comparison Systems

Abstract When paired comparisons are made sequentially over time as for example in chess competitions, it is natural to assume that the underlying abilities do change with time. Previous approaches are based on fixed updating schemes where the increments and decrements are fixed functions of the underlying abilities. The parameters that determine the functions have to be specified a priori and are based on rational reasoning. We suggest an alternative scheme for keeping track with the underlying abilities. Our approach is based on two components: a response model that specifies the connection between the observations and the underlying abilities and a transition model that specifies the variation of abilities over time. The response model is a very general paired comparison model allowing for ties and ordered responses. The transition model incorporates random walk models and local linear trend models. Taken together, these two components form a non-Gaussian state-space model. Based on recent results, recursive posterior mode estimation algorithms are given and the relation to previous approaches is worked out. The performance of the method is illustrated by simulation results and an application to soccer data of the German Bundesliga.

Dynamic Stochastic Models for Time-Dependent Ordered Paired Comparison Systems

Abstract When paired comparisons are made sequentially over time as for example in chess competitions, it is natural to assume that the underlying abilities do change with time. Previous approaches are based on fixed updating schemes where the increments and decrements are fixed functions of the underlying abilities. The parameters that determine the functions have to be specified a priori and are based on rational reasoning. We suggest an alternative scheme for keeping track with the underlying abilities. Our approach is based on two components: a response model that specifies the connection between the observations and the underlying abilities and a transition model that specifies the variation of abilities over time. The response model is a very general paired comparison model allowing for ties and ordered responses. The transition model incorporates random walk models and local linear trend models. Taken together, these two components form a non-Gaussian state-space model. Based on recent results, recursive posterior mode estimation algorithms are given and the relation to previous approaches is worked out. The performance of the method is illustrated by simulation results and an application to soccer data of the German Bundesliga.

Numerical solution of likelihood equations for a first-order response surface model for mixture paired comparisons

A simple procedure for numerical solution of the likelihood equations for estimating the regression parameters of a first-order response surface model for the treatment parameters of mixture paired comparison experiments is developed. It is demonstrated that, for defined rotatable designs, those regression parameters are simple functions of the main effect parameters of a corresponding factorial model with no interactions. The maximum likelihood estimators of those main effect parameters, and hence of their corresponding regression parameters, are obtained through using procedures of treatment contrasts, factorial and iterations. A numerical example is given to illustrate applications of the procedures developed in this paper.

Aggregation of preferences by the generalized row sum method

The generalized row sum method proposed by the present writer generalizes the row sum method (and thus those of Borda and Copeland) to the case of paired comparisons with missing values. The method is constructed axiomatically, but also can be obtained by determining the ridge estimators of the Scheffé paired comparison model parameters. To find the object scores, one has to solve a system of linear equations. The method is applicable to incomplete numerical estimates of objects. The properties of the method are presented.

Gaussian paired comparison models

Abstract Bayesian paired comparison models based on the Gaussian additive error models for the use of expert judgements are formulated. Some limiting properties of the models are investigated. It is observed that the posterior variance cannot be made arbitrarily small by increasing the number of experts. The posterior distributions are determined by using the Gibbs sampler, and numerical examples are discussed. The value of information lost in the paired comparison process in analysed from the Bayesian point of view.

Bayesian analysis of consistent paired comparisons

A general Bayesian model for paired comparisons is formulated and applied in some special cases, in which the comparisons are assumed to be consistent. Further, a class of expert models having interesting properties connected with the calibration and bias of the expert and the independence of irrelevant variables is constructed. The asymptotic behaviour of the two-dimensional model is studied: it is observed that the posterior variance cannot be made arbitrarily small by increasing the number of experts. In addition to exactly stated expert models, the case with unknown parameters is also described. The use of models is illustrated with numerical examples, where the posterior distributions are generated by a sampling procedure.

Orthogonally invariant estimation of the skew-symmetric normal mean matrix

The unbiased estimator of risk of the orthogonally invariant estimator of the skew-symmetric normal mean matrix is obtained, and a class of minimax estimators and their order-preserving modification are proposed. The estimators have applications in paired comparisons model. A Monte Carlo study to compare the risks of the estimators is given.

Statistical theory for the ratio model of paired comparisons

This paper investigates the ratio model for paired comparisons. This model arises in Saaty's (1977, Journal of Mathematical Psychology, 5, 234–281) scaling method for priorities in hierarchical structures and it also arises in the constant sum paired comparison models presented by Hauser and Shugan (1980, Operations Research, 28, 278–320). This paper characterizes some experimental designs that yield an estimable ratio scale and it develops a uniformly minimum variance, unbiased estimator of the scale parameters. An estimator is developed when only a subject of all p(p−1) 2 comparisons are made and when the comparisons are correlated. Moreover an intercept version of the standard model is investigated and tests of consistency and reliability are presented.

Linear Logistic Latent Class Analysis for Polytomous Data

Abstract For latent class analysis, a widely known statistical method for the unmixing of an observed frequency table into several unobservable ones, a flexible model is presented in order to restrain the unknown class sizes (mixing weights) and the unknown latent response probabilities. Two systems of basic equations are stated such that they simultaneously allow parameter fixations, the equality of certain parameters as well as linear logistic constraints of each of the original parameters. The maximum likelihood equations for the parameters of this “linear logistic latent class analysis” are given, and their estimation by means of the EM algorithm is described. Further, the criteria for their local identifiability and statistical tests (Pearson- and likelihood-ratio-χ 2) for goodness of fit are outlined. The practical applicability of linear logistic latent class analysis is demonstrated by three examples: mixed logistic regression, a mixed Bradley-Terry model for paired comparisons with ties, and a local dependence latent class model in which the departure from stochastic independence is covered by a single additional parameter per class.

Linear Logistic Latent Class Analysis for Polytomous Data

Abstract For latent class analysis, a widely known statistical method for the unmixing of an observed frequency table into several unobservable ones, a flexible model is presented in order to restrain the unknown class sizes (mixing weights) and the unknown latent response probabilities. Two systems of basic equations are stated such that they simultaneously allow parameter fixations, the equality of certain parameters as well as linear logistic constraints of each of the original parameters. The maximum likelihood equations for the parameters of this “linear logistic latent class analysis” are given, and their estimation by means of the EM algorithm is described. Further, the criteria for their local identifiability and statistical tests (Pearson- and likelihood-ratio-χ 2) for goodness of fit are outlined. The practical applicability of linear logistic latent class analysis is demonstrated by three examples: mixed logistic regression, a mixed Bradley-Terry model for paired comparisons with ties, and a lo...

Probabilistic psychophysics with noisy stimuli

Abstract Mental representations of physical objects may vary from moment to moment. This occurs because of stimulus variation prior to the initiation of a neural signal and also because of neural variation in the neural mechanism. A psychophysical transformation function formalizes the connection between stimulus measures and mental representations, and this function will be assumed to be a one-to-one function (monotonic in one dimension). A very general equation is derived for the probability density function (pdf) of the momentary psychological magnitudes given any particular positive stimulus pdf, any one-to-one psychophysical transformation function and any neural pdf. Special cases of this general model are applied to the method of paired comparisons. These models for paired comparisons are derived on the basis of assumed lognormally distributed stimulus values on a stimulus continuum, a log or power psychophysical transformation and added neural normally distributed noise. The parameters of sample problems are estimated using maximum likelihood, nonlinear least squares and minimum chi-square criteria. These methods are compared with respect to bias in and the variance of the estimators.

Loglinear representation for multivariate choice data

Abstract This paper focuses on multivariate logistic representations of paired and multiple comparisons models. Bock's (1975) logistic model is chosen as a common framework for extending the Bradley-Terry-Luce model, Luce's model of first choices, the Babington–Smith and the Pendergrass-Bradley ranking models to accommodate multiple responses. Relationships between the responses are constrained to vary as an additive function of the objects being compared. The methodology is exemplified by analyzing bivariate choice data from a questionnaire study.

Are all linear paired comparison models empirically equivalent?

Abstract Previous authors (Jackson and Fleckenstein, 1957; Mosteller, 1958; Noether, 1960) have found that different models of paired comparisons data lead to similar fits. This phenomenon is examined by means of a set of paired comparison models, based on gamma random variables, that includes the frequently applied Bradley-Terry and Thurstone-Mosteller models. A theoretical result provides a natural ordering of the models in the gamma family on the basis of their composition rules. Analysis of several sports data sets indicates that all of the paired comparison models in the family provide adequate, and almost identical fits to the data. Simulations are used to further explore this result. Although not all approaches to paired comparisons experiments are covered by this discussion, the evidence is strong that for samples of the size usually encountered in practice all linear paired comparison models are virtually equivalent.