I Need a CAVAA: How Conversational Agent Voting Advice Applications (CAVAAs) Affect Users' Political Knowledge and Tool Experience.

Abstract

In election times, millions of voters consult Voting Advice Applications (VAAs) to learn more about political parties and their standpoints. While VAAs have been shown to enhance political knowledge and increase electoral turnout, research also demonstrates that voters frequently experience comprehension problems when responding to the political attitude statements in a VAA. We describe two studies in which we test a new type of VAA, called Conversational Agent VAA (CAVAA), in which users can easily access relevant information about the political issues in the VAA statements by asking questions to a chatbot. Study 1 reports about an online experiment (N = 229) with a 2 (Type: traditional VAA/CAVAA) x 2 (Political sophistication: low/high) design. Results show that CAVAA users report higher perceived political knowledge scores and also answer more factual knowledge questions correctly than users of a regular VAA. Also, participants' CAVAA experience was evaluated better. In Study 2 (N = 180), we compared three CAVAA designs (a structured design with buttons, a non-structured design with an open text field, and a semi-structured design with both buttons and an open text field), again for higher and lower politically sophisticated users. While the three designs score equally high on factual and perceived knowledge indicators, the experience of the structured CAVAA was evaluated more positively than the non-structured version. To explore the possible cause for these results, we conducted an additional qualitative content analysis on 90 chatbot-conversations (30 per chatbot version). This analysis shows that users more frequently access additional information in a structured design than in a non-structured design, whereas the number of break-offs is the same. This suggests that the structured design delivers the best experience, because it provides the best trigger to ask questions to the chatbot.