A change-point analysis procedure based on weighted residuals to detect back random responding.

Abstract

Careless or inattentive responding is frequently observed in questionnaire or survey data, which jeopardizes test validity and the generalizability of research findings. It is therefore very important to detect such response behavior. The most frequently encountered type of careless response behavior is back random responding (BRR). Literature suggests that BRR is challenging to detect, with reported power of detection around .5 or lower. Change point analysis (CPA), which is a widely used statistical process control method, can be applied to item response data to detect if aberrant behavior exists in a response pattern. Existing CPA methods, however, may not be suitable for detecting BRR, because the change may not be directional. In this article we propose a weighted-residual-based CPA procedure to detect BRR behavior. The performance of the proposed method was evaluated in a comprehensive simulation study and compared against 3 existing CPA methods. Results indicated that the proposed residual-based CPA procedure can detect BRR behavior with high power for tests of 20 items or longer, while keeping the Type-I error rate well under control. Compared with the 3 existing CPA methods, it leads to comparable empirical Type-I error but a gain in power of 17%-42%. An empirical study further illustrated the utility of the proposed method to detect BRR with a real dataset. Implications of the proposed method, its limitations, as well as future directions are provided at the end. (PsycINFO Database Record (c) 2019 APA, all rights reserved).