Equally vs. unequally keyed blocks in forced-choice questionnaires: Implications on validity and reliability

Abstract

Forced-choice (FC) questionnaires have gained scientific interest over the last decades. However, the inclusion of unequally keyed item pairs in FC questionnaires remains a subject of debate, as there is evidence supporting both their usage and avoidance. Designing unequally keyed pairs may be more difficult when considering social desirability, as they might allow the identification of ideal responses. Nevertheless, they may enhance the reliability and the potential for normative interpretation of scores. To empirically investigate this topic, data were collected from 1,125 undergraduate Psychology students who completed a personality item pool measuring the Big Five personality traits in Likert-type format and two FC questionnaires (with and without unequally keyed pairs). These questionnaires were compared in terms of reliability, convergent and criterion validity, and ipsativity of the scores, along with insights on the construction process. While constructing questionnaires with unequally keyed blocks presented challenges in matching items on their social desirability, the differences observed in terms of reliability, validity, or ipsativity were sporadic and lacked systematic patterns. This suggests that neither questionnaire format exhibited a clear superiority. Given these results, it is recommended using only equally keyed blocks to minimize potential validity issues associated with response biases.