Analysis Of Paired Comparison Data Research Articles

Bayesian paired comparison with the bpcs package

This article introduces the bpcs R package (Bayesian Paired Comparison in Stan) and the statistical models implemented in the package. This package aims to facilitate the use of Bayesian models for paired comparison data in behavioral research. Bayesian analysis of paired comparison data allows parameter estimation even in conditions where the maximum likelihood does not exist, allows easy extension of paired comparison models, provides straightforward interpretation of the results with credible intervals, has better control of type I error, has more robust evidence towards the null hypothesis, allows propagation of uncertainties, includes prior information, and performs well when handling models with many parameters and latent variables. The bpcs package provides a consistent interface for R users and several functions to evaluate the posterior distribution of all parameters to estimate the posterior distribution of any contest between items and to obtain the posterior distribution of the ranks. Three reanalyses of recent studies that used the frequentist Bradley–Terry model are presented. These reanalyses are conducted with the Bayesian models of the bpcs package, and all the code used to fit the models, generate the figures, and the tables are available in the online appendix.

Models for Paired Comparison Data: A Review with Emphasis on Dependent Data

Thurstonian and Bradley-Terry models are the most commonly applied models in the analysis of paired comparison data. Since their introduction, numerous developments have been proposed in different areas. This paper provides an updated overview of these extensions, including how to account for object- and subject-specific covariates and how to deal with ordinal paired comparison data. Special emphasis is given to models for dependent comparisons. Although these models are more realistic, their use is complicated by numerical difficulties. We therefore concentrate on implementation issues. In particular, a pairwise likelihood approach is explored for models for dependent paired comparison data, and a simulation study is carried out to compare the performance of maximum pairwise likelihood with other limited information estimation methods. The methodology is illustrated throughout using a real data set about university paired comparisons performed by students.

On the importance of distinguishing between within- and between-subject effects in intransitive intertemporal choice

Most probabilistic paired comparison models treat inconsistent choices as caused by independent and random errors in the pairwise judgments. In this paper, we argue that this assumption is too restrictive for the analysis of paired comparison data obtained from multiple judges when transitivity violations are systematic. We present a new framework that contains the random error assumption as a special case but also allows for systematic changes in an option's utility assessments over the pairwise comparisons. Accounting for both between- and within-judge sources of variability, we demonstrate in an application on intertemporal choice that the proposed framework can capture systematic transitivity violations as well as individual taste differences.

Modelling intransitive preferences: A random-effects approach

Current models for individual-level paired comparison data are based on the three assumptions that (1) pairwise judgments are independent, (2) the utility of an item remains invariant across trials, and (3) pair-specific variability can account for intransitive choice behaviour. All three assumptions seem strong and likely to be violated in empirical applications. This paper introduces a new framework for the analysis of paired comparison data which relaxes these three assumptions and considers the utilities associated with the same item across trials to be neither independent nor identical, but related. The proposed approach provides new insights about the reliability and consistency of paired comparison judgments and can account for systematic violations of transitivity. An application, based on a replication of Tversky's [(1969). Intransitivity of preference. Psychological Review, 76, 31–48] gamble study, illustrates the usefulness of the new approach in modelling both transitive and intransitive preferences.

Thresholds and Intransitivities in Pairwise Judgments: A Multilevel Analysis

The method of paired comparisons belongs to a small class of models that simultaneously allow for consistency checks of the response behavior of a person and the estimation of item parameters. As a result, the paired comparison technique is of much importance when there are reasons to expect that respondents have difficulties appraising their values or preferences about an issue of interest. This article presents a hierarchical framework for the analysis of paired comparison data with three response categories that allow judges to be indifferent or undecided. The approach can be viewed as a stochastic representation of Luce’s (1956) semiorder. It facilitates statistical analyses of different types of inconsistent responses and yields new tests of the underlying judgmental process. An extensive analysis of a survey study illustrates the usefulness of a multilevel approach for modeling multiple trinary judgments.

New models for the analysis of paired comparison data; segmentation of the panel

Paired comparison experiments provide an efficient tool for the assessment of the preferences. We propose two hierarchically organised models for the analysis of paired comparison data. The first model is easy to interpret and allows straightforward ranking of the products according to the scores derived from the model. The second model is likely to lead to a better fit of the data and allows a graphical display of the products on a plane. A procedure of segmentation of the panel performing the paired comparison test is also discussed.

Modelling the Effect of Subject-Specific Covariates in Paired Comparison Studies with an Application to University Rankings

SUMMARY Preference decisions will usually depend on the characteristics of both the judges and the objects being judged. In the analysis of paired comparison data concerning European universities and students’ characteristics, it is demonstrated how to incorporate subject-specific information into Bradley–Terry-type models. Using this information it is shown that preferences for universities and therefore university rankings are dramatically different for different groups of students. A log-linear representation of a generalized Bradley–Terry model is specified which allows simultaneous modelling of subject- and object-specific covariates and interactions between them. A further advantage of this approach is that standard software for fitting log-linear models, such as GLIM, can be used.

Modeling within-subject dependencies in ordinal paired comparison data

This paper presents two probabilistic models based on the logistic and the normal distribution for the analysis of dependencies in individual paired comparison judgments. It is argued that a core assumption of latent class choice models, independence of individual decisions, may not be well-suited for the analysis of paired comparison data. Instead, the analysis and interpretation of paired comparison data may be much simplified by allowing for within-person dependencies that result from repeated evaluations of the same options in different pairs. Moreover, by relating dependencies among the individual-level responses to (in)consistencies in the judgmental process, we show that the proposed graded paired comparison models reduce to ranking models under certain conditions. Three applications are presented to illustrate the approach.

Response Surface Fitting Using a Generalization of the Bradley-Terry Paired Comparison Model

SUMMARY A paired comparison technique is presented for fitting response surfaces. This is especially useful when subjective responses are involved, where it is often difficult to justify the basic assumptions of the classical procedure. This report discusses aspects of the estimation of parameters and their properties, tests of relevant hypotheses and the selection of experimental designs. The method is applied to an example in food testing. PAIRED comparison techniques have been used for many years for the subjective comparison of several treatments (David, 1963), and extensions are available for the comparison of factorial treatment combinations. This report presents a further extension of the paired comparison method, the analogue of multiple regression analysis. Although it is possible to formulate a general paired comparison regression model this report will concentrate on the analogue of multiple linear regression mainly because of its application to response surface fitting. In the field of sensory assessment, it is frequently required to determine the effect of varying product composition. The classical response surface approach is useful here, despite many of its basic assumptions being invalid in the sensory situation, but occasionally it becomes virtually impossible to implement the classical method, particularly when carry-over and sensory organ fatigue effects are present. The degree to which these affect the results depends on the sense used, taste and smell being usually the more prone. Classical response surface designs, even with blocking, frequently require more units than may be reliably assessed in one sensory assessment session. Further, the assessors may not, due to the above effects-and others-be able reliably to score their impressions on a continuous scale. If any of these factors is present, the method of paired comparisons becomes extremely useful. A number of models have been proposed for the analysis of paired comparison data in the analogue of one-way analysis of variance. The specific analogue one-way ANOVA paired comparison model which this paper adapts to analogue multiple regression is that due to Rao and Kupper (1967), itself a generalization of the Bradley-Terry model (1952). A parallel generalization of the Bradley-Terry model, due to Davidson (1970), could also be extended for the present purpose in an exactly similar manner.

An individual difference model for the multidimensional analysis of preference data

A model for the analysis of paired comparison data is presented which combines features of the BTL-model with features of the Unfolding model. The model is metric, mathematically tractable, and has an exact algebraic solution. Since it is multidimensional and allows for individual differences, it is thought to be more realistic for some choice situations than either the Thurstone model or the BTL-model. No claim is made that the present model will be appropriate for all conceivable choice situations. Rather, it is argued that the fact that it is explicitly falsifiable is a point in its favor.

Factor Analysis Applied to Paired Preferences Among Four Grape Juices

SUMMARYFour grape juices differing in amount of added citric acid were examined by paired comparison by 72 tasters. Some judges preferred sweet and some preferred acid juices, causing pronounced and progressive changes in the frequency distributions of the preference ratings. The correlation coefficient between variances and acid differences was 0.98. The direction and degree of preference was much more consistent for larger differences in acid. Mathematical models developed by other authors for analysis of paired comparison data are of doubtful validity or awkward in application to the present data.