Position Bias in Paired Product Tests

Abstract

Paired comparison tests appear to be the most scientific form of product testing. This technique originated in the laboratory procedures of classical psychophysicists [7]. It closely approximates the laboratory experiment since the two test items are presented under controlled circumstances and the compared experiences are placed as close in time as possible. Like any controlled experiment, the paired comparison product test can be criticized for the artificiality of the circumstances under which the test items are used [1, 2]. This makes it imperative that paired product tests be carefully designed and that inferences about real-world behavior be made cautiously. When properly used, however, paired product tests are a powerful tool of the product researcher and continue to be one of the most popular approaches to preference measurement [3, 5, 6]. A phenomenon that has annoyed researchers who have used paired comparison tests is position bias or testing order effects. Many researchers have reported systematic biases in indicated preferences that can be related to the testing order of the two items. In some cases the bias favors the first item tested [4, 5]; in other cases it favors the second item [4]. An example of paired product test results with position bias was reported by Greenberg [5]. His results are shown in Table 1. A majority of participants who used Item A first indi ated a preference for Item A. Among those who tested Item B first, a majority reported a preference for B. A chi-square test indicated that the influence of order was significant at the .02 level [5]. When the two groups are combined, the effects of testing order are balanced, and there is no significant difference in indicated preferences across the sample. Obviously, misleading results would have occurred if all tests had been conducted with the same testing order. Because of the frequent occurrence of results such as these, it has become standard practice in conducting paired product tests to randomize the testing order.