Evaluating Audience Engagement as a Measure of Digital Diplomacy Effectiveness

Abstract

This article presents relevant tools for analyzing engagement in digital diplomacy, where ‘engagement’ is defined as the two-way interaction between a digital diplomacy channel on social media and its users. Existing methods for evaluating engagement often fail to fully utilize the available data or lack the flexibility needed for specific diplomatic purposes. Using the case study of US digital diplomacy in Russia, this paper addresses two research questions: (1) To what extent are the automated evaluation metrics developed by data aggregator platforms applicable for analyzing engagement in digital diplomacy? (2) Do these metrics help identify significant patterns in digital diplomacy?The findings of our pilot study indicate that the automated engagement metrics provided by the Popsters and TGStat data aggregator services effectively measure engagement in terms of audience reach and expansion, geographic distribution of traffic, overall engagement, and engagement based on content type (text, photo, video, or link) or text length. The analysis of selected metrics reveals two key trends in US digital diplomacy in Russia: since 2022 there has been a notable shift in focus toward Telegram as the primary platform, and while the content partially reaches the existing Russian-speaking audience, it struggles to expand its overall reach.The authors provide six recommendations for diplomatic agencies and consulting firms to enhance the tools for analyzing digital diplomacy engagement: (1) adopt automated engagement metrics that account for the interface differences among social media platforms; (2) incorporate built-in topic modeling of posts; (3) integrate sentiment analysis of comments; (4) develop mechanisms to detect traffic manipulation; (5) track multi-level reposts, the timing and spread of hashtag diplomacy, and audience growth considering these processes; and (6) evaluate digital diplomacy activity concerning responses to comments. The authors advocate for closer collaboration between the academic community and diplomatic practitioners to improve engagement metrics and thereby enhance the effectiveness of digital diplomacy.