Comparing Chatbots and Online Surveys for (Longitudinal) Data Collection: An Investigation of Response Characteristics, Data Quality, and User Evaluation

Abstract

ABSTRACT As chatbots are gaining more popularity than ever, they have recently been considered as interesting tools for survey administration in social science research. To explore this idea, we investigated the extent to which there are differences in response characteristics and data quality between a traditional, web-based survey and a conversational, chatbot-based survey (which we integrated in an instant messaging app). In addition, we zoomed into how respondents evaluate both survey modes. Using a longitudinal design, we also explored how response characteristics evolved over a period of two weeks. Overall, we did not find evidence that chatbots might be better survey administration tools than web surveys. On the contrary, the web survey often seemed to generate more favorable response characteristics and data quality. Finally, when it comes to user perceptions, we found that the chatbot survey was evaluated less favorably in terms of perceived enjoyment, usefulness, and security. Based on these results, we draw conclusions about whether chatbots can be considered as valid alternatives for traditional web survey methods.