Measuring Social Solidarity During Crisis: The Role of Design Choices

Abstract

Building on our previous work, we assess how social solidarity towards migrants and refugees has changed before and after the onset of the COVID-19 pandemic, by collecting and analyzing a large, novel, and longitudinal dataset of migration-related tweets. To this end, we first annotate above 2000 tweets for (anti-) solidarity expressions towards immigrants, utilizing two annotation approaches (experts vs. crowds). On these annotations, we train a BERT model with multiple data augmentation strategies, which performs close to the human upper bound. We use this high-quality model to automatically label over 240 000 tweets between September 2019 and June 2021. We then assess the automatically labeled data for how statements related to migrant (anti-)solidarity developed over time, before and during the COVID-19 crisis. Our findings show that migrant solidarity became increasingly salient and contested during the early stages of the pandemic but declined in importance since late 2020, with tweet numbers falling slightly below pre- pandemic levels in summer 2021. During the same period, the share of anti-solidarity tweets increased in a sub- sample of COVID-19-related tweets. These findings highlight the importance of long-term observation, pre- and post-crisis comparison, and sampling in research interested in crisis related effects. As one of our main contributions, we outline potential pitfalls of an analysis of social solidarity trends: for example, the ratio of solidarity and anti-solidarity statements depends on the sampling design, i.e., tweet language, Twitter-user accounts' national identification (country known or unknown) and selection of relevant tweets. In our sample, the share of anti-solidarity tweets is higher in native (German) language tweets and among “anonymous” Twitter users writing in German compared to English-language tweets of users located in Germany.