Anchored paired comparisons

Abstract

The method of paired comparisons is often used in image quality evaluations. Psychometric scale values for quality judgments are modeled using Thurstone's Law of Comparative Judgment in which distance in a psychometric scale space is a function of the probability of preference. The transformation from psychometric space to probability is a cumulative probability distribution. The major drawback of a complete paired comparison experiment is that every treatment is compared to every other, thus the number of comparisons grows quadratically. We ameliorate this difficulty by performing paired comparisons in two stages, by precisely estimating anchors in the psychometric scale space which are spaced apart to cover the range of scale values and comparing treatments against those anchors. In this model, we employ a generalized linear model where the regression equation has a constant offset vector determined by the anchors. The result of this formulation is a straightforward statistical model easily analyzed using any modern statistics package. This enables model fitting and diagnostics. This method was applied to overall preference evaluations of color pictorial hardcopy images. The results were found to be compatible with complete paired comparison experiments, but with significantly less effort.