Sequential dependencies of responses in a questionnaire survey and their effects on the reliability and validity of measurement.

Abstract

The reliability and validity of measurements in questionnaire surveys are affected by the survey design. Previous studies have discussed whether the items on a multidimensional scale should be grouped by subscale or randomized to increase the reliability and validity of the measurement. However, it is not clear what effect item order has on the true reliability and validity and whether the order causes bias in these estimates. Furthermore, it is not clear whether there are sequential dependencies of responses, which refers to the effect of past responses on current responses and, if so, what effects that they have on reliability, coefficient α, and validity. Study 1 empirically revealed that each respondent's previous response had a positive effect on his or her current response. In Study 2, we conducted a theoretical investigation and found that the first-order positive sequential dependencies of responses reduced the reliability, coefficient α, and validity of the subscales. In addition, sequential dependencies were found to render these indicators higher when the items were grouped than when they were randomized and caused an overestimation of reliability by coefficient α. Study 3 confirmed the effects of sequential dependencies and item order on coefficient α using real data. Thus, this study showed the potential impact of sequential dependencies of responses on psychological measurements.