Abstract
Some respondents make careless responses due to the “satisficing,” which is an attempt to complete a questionnaire as quickly and easily as possible. To obtain results that reflect a fact, detecting satisficing and excluding the responses with satisficing from the analysis targets are required. One of the devised methods detects satisficing by adding questions that check violations of instructions and inconsistencies. However, this approach may cause respondents to lose their motivation and prompt them to satisficing. Additionally, a deep learning model that automatically answers these questions was reported. This threatens the reliability of the conventional method. To detect careless responses without inserting such screening questions, machine learning (ML) detection using data obtained from answer results was attempted in a previous study, with a detection rate of 55.6%, which is not sufficient from the viewpoint of practicality. Therefore, we hypothesized that a supervised ML model with a higher detection rate could be constructed by using on-screen answering behavior as features. However, (1) no existing questionnaire system can record on-screen answering behavior and (2) even if the answering behavior can be recorded, it is unclear which answering behavior features are associated with satisficing. We developed an answering behavior recording plug-in for LimeSurvey, an online questionnaire system used all over the world, and collected a large amount of data (from 5,692 people) in Japan. Then, a variety of features were examined and generated from answering behavior, and we constructed ML models to detect careless responses. We call this detection method the ML-ABS (ML-based answering behavior scale). Evaluation by cross-validation demonstrated that the detection rate for careless responses was 85.9%, which is much higher than the previous ML method. Among the various features we proposed, we found that reselecting the Likert scale and scrolling particularly contributed to the detection of careless responses.
Highlights
Questionnaire results have the problem of low reliability due to the practice of ‘‘satisficing,’’ which means to complete a task as quickly and as possible [1]
In this paper, we aimed to detect careless responses, which are an attempt to complete questionnaires as and quickly as possible, with high accuracy in an environment that does not place a psychological burden on the respondent due to screening questions
We developed the Operation Logger plug-in to record the answering behavior and collected the desired data in a large-scale experiment using this plug-in, so we can say that Challenge 1, the recording of answer behavior, was accomplished
Summary
Questionnaire results have the problem of low reliability due to the practice of ‘‘satisficing,’’ which means to complete a task as quickly and as possible [1]. In response to this problem, some methods have been devised to detect. Satisficing by inserting screening questions, such as questions that determine instruction violations or detect inconsistency [2], [3]. It is desirable to avoid such questions because they would undermine the intrinsic motivation to cooperate with the survey of respondents who are answering carefully, and the questions themselves may induce satisficing.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
