Identifying Dynamic Shifts to Careless and Insufficient Effort Behavior in Questionnaire Responses; a Novel Approach and Experimental Validation

Abstract

Careless and insufficient effort responding (C/IER) is a situation where participants respond to survey instruments without considering the item content. This phenomena adds noise to data leading to erroneous inference. There are multiple approaches to identifying and accounting for C/IER in survey settings, of these approaches the best performing are model based classification techniques. Classic approaches to accounting for C/IER treat it as a person level phenomena. They first use some method to identify participants who exhibit C/IER, then use list-wise deletion to remove them prior to analysis. We argue that C/IER is actually a state that participants may exhibit for a portion of a survey instrument. In other words, participants start a survey with the intention to follow instructions, but at some point transition to a C/IER state and no longer respond in line with item contents. Accounting for C/IER at the item level, as opposed to the person level preserves data resulting in increased power. In this article we present a Bayesian Dynamic Latent Class Structural Equation Modeling (DLCSEM) approach for simultaneously accounting for C/IER at the item level and estimating a model of interest. We use a simulation study to establish the approaches performance under empirically relevant conditions and to compare it to other methods. We then conducted an experimental validation in which we induce C/IER like responses from human subjects and investigate the approaches ability to identify the point in which participants transition to a C/IER state. We also compare the model to existing approaches. In both the simulation and experimental validation the DLCSEM outperforms the alternative approaches. We conclude that the approach should be used by applied researchers for the pragmatic benefits of the method. Conclusions, limitations, and future directions are discussed.