A novel two-step rating-based ‘double-faced applicability’ test. Part 1: Its performance in sample discrimination in comparison to simple one-step applicability rating

Abstract

Due to its simplicity and efficiency for evaluating product attributes, check-all-that-apply (CATA) questions have been widely used. Yet, CATA questions have been reported to lack discriminability of subtle product differences and suffer from the problems of response bias from satisficing and acquiescence behaviors. In the present paper, a novel two-step rating-based ‘double-faced applicability’ test was developed as an extended response format of CATA, to improve its sensitivity for product discrimination and for stabilizing subjects’ evaluative criteria. In the ‘double-faced applicability’ test, each attribute was ‘double-faced’, meaning that two descriptors (a pair of semantic-differential descriptors) are separately presented in the questionnaires representing both sides of each attribute. For performing the two-step rating on each attribute in the questionnaire, subjects are instructed to first respond ‘Yes (does apply)’ or ‘No (does not apply)’ and then to answer a 3-point sureness rating (how sure they were about their Yes or No response). The performance of the two-step ratings in this new method was compared to a simple one-step applicability rating test method as well as the forced-choice Yes/No questions without the sureness rating in terms of sensitivity in sample discrimination. The results showed that the ‘double-faced applicability’ test provided better product discrimination and showed the potential to reduce acquiescence response bias when using existing variants of CATA response formats.