Moderation Of Content Research Articles

An Exploratory Study of Fact-Checking Practices in Conflict and Authoritarian Contexts

This study explores fact-checking practices in Ethiopia and Mali in times of conflict and in a context marked by increasing restrictions to press freedom. The objective is to understand how, in this hostile environment, fact-checkers in these two countries manage to carry out their activities. Our findings reveal that fact-checkers are often victims of online bullying and harassment and fear reprisal from governments. This pushes them to self-censor, avoiding working on sensitive topics, such as military issues in Mali. In addition, fact-checking organizations in both countries highlight the difficulty of accessing reliable sources. Consequently, they focus more on debunking viral social media content, thus effectively becoming content moderators who have turned away from the mission of holding leaders accountable, one of the primary functions of fact-checking. Regarding their role conception, fact-checkers in Ethiopia and Mali see themselves more as guides helping navigate the information disorder than “guardians of truth” or “truth keepers.”

Content Moderator Coping Strategies

Abstract: Content moderators (CMs) apply policies set by platforms to protect users from harmful content. It is a stressful job, associated with reduced mental health and well-being. In this study, an anonymous survey was used to demonstrate most CMs cope by seeking support from colleagues and this is associated with lower psychological distress and secondary trauma and higher well-being whereas increased smoking and alcohol consumption is associated with increased symptomology. Well-being services were not related to a reduction in psychological distress or trauma. We argue these results fit within a framework of trauma-informed working and provide evidence for its utility in the trust and safety sector. They also highlight the need for continued research into “what works” to support the resilience of frontline staff.

Mapping interpretations of the law in online content moderation in Germany

Content moderation is a vital condition that online platforms must facilitate, according to the law, to create suitable online environments for their users. By the law, we mean national or European laws that require the removal of content by online platforms, such as EU Regulation 2021/784, which addresses the dissemination of terrorist content online. Content moderation required by these national or European laws, summarised here as ‘the law’, is different from the moderation of pieces of content that is not directly required by law but instead is conducted voluntarily by the platforms. New regulatory requests create an additional layer of complexity of legal grounds for the moderation of content and are relevant to platforms’ daily decisions. The decisions made are either grounded in reasons stemming from different sources of law, such as international or national provisions, or can be based on contractual grounds, such as the platform's Terms of Service and Community Standards. However, how to empirically measure these essential aspects of content moderation remains unclear. Therefore, we ask the following research question: How do online platforms interpret the law when they moderate online content?To understand this complex interplay and empirically test the quality of a platform's content moderation claims, this article develops a methodology that facilitates empirical evidence of the individual decisions taken per piece of content while highlighting the subjective element of content classification by human moderators. We then apply this methodology to a single empirical case, an anonymous medium-sized German platform that provided us access to their content moderation decisions. With more knowledge of how platforms interpret the law, we can better understand the complex nature of content moderation, its regulation and compliance practices, and to what degree legal moderation might differ from moderation due to contractual reasons in dimensions such as the need for context, information, and time.Our results show considerable divergence between the platform's interpretation of the law and ours. We believe that a significant number of platform legal interpretations are incorrect due to divergent interpretations of the law and that platforms are removing legal content that they falsely believe to be illegal (‘overblocking’) while simultaneously not moderating illegal content (‘underblocking’). In conclusion, we provide recommendations for content moderation system design that takes (legal) human content moderation into account and creates new methodological ways to test its quality and effect on speech in online platforms.

Social Evidence Tampering and the Epistemology of Content Moderation

AbstractSocial media misinformation is widely thought to pose a host of threats to the acquisition of knowledge. One response to these threats is to remove misleading information from social media and to de-platform those who spread it. While content moderation of this sort has been criticized on various grounds—including potential incompatibility with free expression—the epistemic case for the removal of misinformation from social media has received little scrutiny. Here, I provide an overview of some costs and benefits of the removal of misinformation from social media. On the one hand, removing misinformation from social media can promote knowledge acquisition by removing misleading evidence from online social epistemic environments. On the other hand, such removals require the exercise of power over evidence by content moderators. As I argue, such exercises of power can encourage suspicions on the part of social media users and can compromise the force of the evidence possessed by such users. For these reasons, the removal of misinformation from social media poses its own threats to knowledge.

Algorithmic Subjectivities

This paper considers how subjectivities are enlivened in algorithmic systems. We first review related literature to clarify how we see “subjectivities” as emerging through a tangled web of processes and actors. We then offer two case studies exemplifying the emergence of algorithmic subjectivities: one involving computational topic modeling of blogs written by parents with children on the autism spectrum, and one involving algorithmic moderation of social media content. Drawing on these case studies, we then articulate a series of qualities that characterizes algorithmic subjectivities. We also compare and contrast these qualities with a number of related concepts from prior literature to articulate how algorithmic subjectivities constitutes a novel theoretical contribution, as well as how it offers a focal lens for future empirical investigation and for design. In short, this paper points out how certain worlds are being made and/or being made possible via algorithmic systems, and it asks HCI to consider what other worlds might be possible.

Secondary Trauma by Internet Content Moderation: a Case Report

Introduction In recent years, a global debate has emerged regarding the protection of Internet users from exposure to harmful content. Content moderation is defined as the organized practice of filtering user-generated content posted on internet, social networks, and media to determine the appropriateness of the content for a site, locality, or jurisdiction. The growing volume of this content along with the psychological impact of this activity have promoted the application of automated approaches based on artificial intelligence and machine-learning. However, the changing characteristics of content, as well as the cultural differences that influence its appropriateness, mean that human moderation of Internet content currently continues to exist. Psychological effects of this activity such as symptoms of post-traumatic stress disorder (PTSD) could represent an example of secondary trauma.Objectives Our aim is to describe a clinical case of post-traumatic stress disorder presenting with specific traumatic exposure idiosyncrasy that could lead to a better consequence characterization of a recent social phenomena such as internet content moderation.MethodsWe expose the clinical case of a woman with emotional distress who was reffered to our outpatient psychiatric unit in Barcelona in 2022 after five years working as an internet content moderator.ResultsWe describe the case of a 35-year-old woman without relevant medical, toxicologic or psychiatric record that presents to our out-patient psychiatric clinic with post-traumatic stress disorder after five years of working as an internet content moderator and being exposed to visual traumatic content such as sexual assault and paedophilia. The clinical presentation consisted with one year of recurrent daily panic attacks, intrusive images about the traumatic exposure, intrusive thoughts, insomnia, vivid nightmares, avoidance of exposure to her son, distrust of the environment and intense fear for her son security. The disorder interfered in her capacity to work. The patient received psychological treatment and ISRS (Sertraline) was prescribed, however only partial response was reached with persistence of the majority of symptoms.ConclusionsThe presented case suggests a temporal and symptom content relationship between the described work exposure and the appearance of emotional distress in a patient without PTSD history. Although previous evidence of secondary trauma in people exposed to indirect traumatic experiences has been reported, for example in healthcare professionals, the exposure to alien trauma through digital exposure as a workactivity is yet to be specifically examined. It is necessary to expand knowledge on the clinical expression of this phenomenon due to the observed recurrence of anxious and depressive symptomatology related to repeated exposure to traumatic content.Disclosure of InterestNone Declared

Graph convolution networks for social media trolls detection use deep feature extraction

This study presents a novel approach to identifying trolls and toxic content on social media using deep learning. We developed a machine-learning model capable of detecting toxic images through their embedded text content. Our approach leverages GloVe word embeddings to enhance the model's predictive accuracy. We also utilized Graph Convolutional Networks (GCNs) to effectively analyze the intricate relationships inherent in social media data. The practical implications of our work are significant, despite some limitations in the model's performance. While the model accurately identifies toxic content more than half of the time, it struggles with precision, correctly identifying positive instances less than 50% of the time. Additionally, its ability to detect all positive cases (recall) is limited, capturing only 40% of them. The F1-score, which is a measure of the model's balance between precision and recall, stands at around 0.4, indicating a need for further refinement to enhance its effectiveness. This research offers a promising step towards more effective monitoring and moderation of toxic content on social platforms.

Supporting digital key workers: addressing the challenges faced by content moderators during and after the COVID-19 pandemic

Supporting digital key workers: addressing the challenges faced by content moderators during and after the COVID-19 pandemic

Supporting Digital Key Workers: Addressing the Challenges Faced by Content Moderators during and after the COVID-19 Pandemic

Supporting Digital Key Workers: Addressing the Challenges Faced by Content Moderators during and after the COVID-19 Pandemic

How to co-create content moderation policies: the case of the AutSPACEs project

Abstract The moderation of user-generated content on online platforms remains a key solution to protecting people online, but also remains a perpetual challenge as the appropriateness of content moderation guidelines depends on the online community that they aim to govern. This challenge affects marginalized groups in particular, as they more frequently experience online abuse but also end up falsely being the target of content-moderation guidelines. While there have been calls for democratic, community-moderation, there has so far been little research into how to implement such approaches. Here, we present the co-creation of content moderation strategies with the users of an online platform to address some of these challenges. Within the context of AutSPACEs—an online citizen science platform that aims to allow autistic people to share their own sensory processing experiences publicly—we used a community-based and participatory approach to co-design a content moderation solution that would fit the preferences, priorities, and needs of its autistic user community. We outline how this approach helped us discover context-specific moderation dilemmas around participant safety and well-being and how we addressed those. These trade-offs have resulted in a moderation design that differs from more general social networks in aspects such as how to contribute, when to moderate, and what to moderate. While these dilemmas, processes, and solutions are specific to the context of AutSPACEs, we highlight how the co-design approach itself could be applied and useful for other communities to uncover challenges and help other online spaces to embed safety and empowerment.

ALGORITHMIC FOLK THEORIES OF ONLINE HARASSMENT: HOW SOCIAL MEDIA ALGORITHMS ENABLE ONLINE HARASSMENT AND PREVENT INTERVENTION

Online harassment is a public health concern, and social media algorithms are often proposed as a solution by social media companies. As online harassment grows, there are concerns that algorithms as content moderators fail to achieve their desired effect because of inabilities to contextualize social issues. This research contributes to the intersection of algorithms and online harassment by investigating the algorithmic folk theories of the victims, perpetrators, and bystanders of online harassment. Strategically sampling the experiences of marginalized identity categories who experienced harassment, we conducted grounded theory interviews and found that people theorize that algorithmic failures fuel online harassment and isolate victims. We describe four folk theories that victims, perpetrators, and witnesses utilize to make sense of their experiences of online harassment. The $2 asserts algorithms only pay attention to harassment incidents with a large number of flags. $2 describes perceptions of how algorithms amplify harassment content to increase engagement. $2 refers to perceptions that algorithms seek to form new audiences for content, which networks harassers together. $2 finds victims perceive that algorithms fail to contextualize the harassment of marginalized communities. Victims, bystanders, and perpetrators each described using their folk theories to instigate, push back, or succumb to the culture of online harassment. Understanding these algorithmic online harassment folk theories highlights how social media algorithms perpetuate harassment and fail to support victims. $2 : algorithmic folk theories, online harassment, networked harassment, social media, content moderation.

MENTAL HEALTH AND THE DIGITAL CARE ASSEMBLAGE: MODERATION PRACTICES & USER EXPERIENCES

This paper examines the socio-technical ecosystems that shape the moderation of mental health content. To explore how care is formulated across and between different actors and automated systems, I focus on the experiences of moderators and users of three peer-based mental health support platforms. The analysis is framed by the notion of the 'digital care assemblage' to delineate the interactions between goal-oriented moderation policies, automated systems, human content moderators or platform managers, and users seeking or giving help in relation to mental ill-health. Each of these actors contribute to the supportive capacity of the platforms for addressing mental health issues in the community.

Content Moderator Mental Health, Secondary Trauma, and Well-being: A Cross-Sectional Study.

Content moderators (CMs) analyze and remove offensive or harmful user generated content that has been uploaded to the internet. Jobs which involve exposure to other people's suffering are associated with raised rates of secondary traumatic stress and mental health problems. However, research establishing psychological baseline symptoms in CMs is lacking. This study used an online survey to explore rates of psychological distress, secondary trauma, and well-being in a sample of CMs. Regression analysis explored how various features of the work affected mental health. There was a dose-response effect between frequency of exposure to distressing content and psychological distress and secondary trauma, but not well-being. The results suggested supportive colleagues and feedback about the importance of their role ameliorated this relationship. Implications for CM working conditions are discussed.

Health and toxicity in content moderation: the discursive work of justification

ABSTRACT Within academia, industry, and government, the terms ‘health’ and ‘toxicity’ are widely used to describe and justify decisions around online content and its removal. However, the meanings of these terms are assumed to be self-evident and therefore are rarely examined. This article turns a critical eye to the health and toxicity metaphor to unpack its hidden political work. We trace the metaphor through three different discourses: the historical political economy of the term, the usage by cultural elites in the last two decades, and finally through its contemporary instrumental usage by volunteer content moderators on Facebook. By linking these discourses together, we argue that the metaphor of health and toxicity serves as a means for justification and legitimacy under contemporary neoliberalized orders that typically chafe at modes of public intervention and the language of democratic statecraft. Rather than elucidating the challenges of online content, we find that the metaphor often serves to obfuscate or sidestep the hardest problems in democratic governance. This analysis therefore has practical significance for researchers, policymakers, journalists, and other speakers that publicly traffic in this discourse at large.

The WEIRD governance of fact-checking and the politics of content moderation

In this article, we chart the conflicting standards of fact-checking outside Western, Educated, Industrialized, Rich, and Democratic (WEIRD) countries that shifted their focus from holding politicians to account to acting as content moderators. We apply reflexive thematic analysis to a set of interviews with 37 fact-checking experts from 35 organizations in 27 countries to catalog the pressures they face and their struggle with tasks that are increasingly different from the journalistic values underpinning the practice. We find that fact-checkers have to balance the number of checks across each side of the partisan divide, an exercise in “bothsidesism” to manage the expectations of partisan social media users; that they increasingly prioritize the checking of viral content; and that Meta’s third-party fact-checking program prevents them from holding local politicians to account. We conclude with a discussion of our findings and recommendations for content moderation outside WEIRD countries.

Moderating borderline content while respecting fundamental values

AbstractAs efforts to identify and remove online terrorist and violent extremist content have intensified, concern has also grown about so‐called lawful but awful content. Various options have been touted for reducing the visibility of this borderline content, including removing it from search and recommendation algorithms, downranking it and redirecting those who search for it. This article contributes to this discussion by considering the moderation of such content, in terms of three sets of values. First, definitional clarity. This is necessary to provide users with fair warning of what content is liable to moderation and to place limits on the discretion of content moderators. Yet, at present, definitions of borderline content are vague and imprecise. Second, necessity and proportionality. While downranking and removal from search and recommender algorithms should be distinguished from deplatforming, tech companies' efforts to deamplify borderline content give rise to many of the same concerns as content removal and account shutdowns. Third, transparency. While a number of platforms now publish their content moderation policies and transparency data reports, these largely focus on violative, not borderline content. Moreover, there remain questions around access to data for independent researchers and transparency at the level of the individual user.

An investigation of social media labeling decisions preceding the 2020 U.S. election.

Since it is difficult to determine whether social media content moderators have assessed particular content, it is hard to evaluate the consistency of their decisions within platforms. We study a dataset of 1,035 posts on Facebook and Twitter to investigate this question. The posts in our sample made 78 misleading claims related to the U.S. 2020 presidential election. These posts were identified by the Election Integrity Partnership, a coalition of civil society groups, and sent to the relevant platforms, where employees confirmed receipt. The platforms labeled some (but not all) of these posts as misleading. For 69% of the misleading claims, Facebook consistently labeled each post that included one of those claims-either always or never adding a label. It inconsistently labeled the remaining 31% of misleading claims. The findings for Twitter are nearly identical: 70% of the claims were labeled consistently, and 30% inconsistently. We investigated these inconsistencies and found that based on publicly available information, most of the platforms' decisions were arbitrary. However, in about a third of the cases we found plausible reasons that could explain the inconsistent labeling, although these reasons may not be aligned with the platforms' stated policies. Our strongest finding is that Twitter was more likely to label posts from verified users, and less likely to label identical content from non-verified users. This study demonstrates how academic-industry collaborations can provide insights into typically opaque content moderation practices.

Countering online terrorist content: A social regulation approach

AbstractAfter a period of self‐regulation, countries around the world began to implement regulations for the removal of terrorist content from tech platforms. However, much of this regulation has been criticised for a variety of reasons, most prominently for concerns of infringing free speech and creating unfair burdens for smaller platforms. In addition to this, regulation is heavily centred around content moderation, however, fails to consider or address the psychosocial risks it poses to human content moderators. This paper argues that where regulation has been heavily criticised yet continues to inspire similar regulation a new regulatory approach is required. The aim of this paper is to undertake an introductory examination of the use of a social regulation approach in three other industries (environmental protection, consumer protection and occupational health and safety) to learn and investigate new regulatory avenues that could be applied to the development of new regulation that seeks to counter terrorist content on tech platforms and is concerned with the safety of content moderators.

The cognitive adaptability and resiliency employment screener (CARES): tool development and testing.

To decrease psychological risk for content moderators, the study initiated the first steps of developing a robust employment screening tool, namely, the Cognitive Adaptability and Resiliency Employment Screener. The study consisted of three phases with 4,839 total participants. In Phase 1, a set of 76 items were developed and tested via exploratory factor analysis, yielding three factors (i.e., Psychological Perseverance & Agility, Rumination & Emotional Lingering, and Expressiveness & Sociability) and also reducing the scale to 68 items. In Phase 2 through confirmatory factor analysis, the three-factor structure showed good fit (CFI = 0.92, RMSEA = 0.05) and demonstrated sufficient overall reliability. In Phase 3, the convergent validity and divergent validity of the tool were established relative to constructs such as resilience, cognitive control and flexibility, emotion regulation, and optimism. Altogether, the findings revealed that the scale demonstrated good psychometric properties that, pending future studies, may serve as a promising employment screener for content moderators.

The psychological impacts of content moderation on content moderators: A qualitative study

Content moderation generally refers to the practice of monitoring user generated content and applying a set of rules defining what is acceptable. As a result, Content Moderators (CMs) are responsible for reviewing and removing harmful online material, which has the potential to cause psychological harm. However, little is known about the perceived impacts of the job. Semi-structured interviews were conducted with 11 commercial CMs who were exposed to child sexual abuse material (CSAM) as part of their job. Interviews were analysed using framework analysis to investigate any psychological impacts. Results demonstrated that CMs manifested with a range of symptoms consistent with experiencing repeated trauma. This included experiencing intrusive thoughts of CSAM, triggered by situations with similar contexts to those encountered at work, avoidance of children and negative cognitive and emotional effects such as cynicism, anxiety and detachment. These symptoms fit into a framework of post traumatic and secondary traumatic stress and suggest CMs wellbeing and exposure to stressful stimuli may be comparable to professionals working in the emergency services or caring professions, such as social workers. These personnel are at heightened risk of various problems including compassion fatigue, vicarious trauma and burnout. Companies which employ moderators should learn from these comparable professions and provide psychoeducation and trauma-informed care to CMs which have been demonstrated to help mitigate these challenges.