Extremist Content Research Articles

Social media, extremism, and radicalization.

Fears that YouTube recommendations radicalize users are overblown, but social media still host and profit from dubious and extremist content.

YouTube's recommendation algorithm is left-leaning in the United States.

With over two billion monthly active users, YouTube currently shapes the landscape of online political video consumption, with 25% of adults in the United States regularly consuming political content via the platform. Considering that nearly three-quarters of the videos watched on YouTube are delivered via its recommendation algorithm, the propensity of this algorithm to create echo chambers and deliver extremist content has been an active area of research. However, it is unclear whether the algorithm may exhibit political leanings toward either the Left or Right. To fill this gap, we constructed archetypal users across six personas in the US political context, ranging from Far Left to Far Right. Utilizing these users, we performed a controlled experiment in which they consumed over eight months worth of videos and were recommended over 120,000 unique videos. We find that while the algorithm pulls users away from political extremes, this pull is asymmetric, with users being pulled away from Far Right content stronger than from Far Left. Furthermore, we show that the recommendations made by the algorithm skew left even when the user does not have a watch history. Our results raise questions on whether the recommendation algorithms of social media platforms in general, and YouTube, in particular, should exhibit political biases, and the wide-reaching societal and political implications that such biases could entail.

COMPARATIVE ANALYSIS OF MACHINE LEARNING ALGORTMS TO IDENTIFY EXTREMIST TEXTS IN THE KAZAKH LANGUAGE

The article explores various models and methods employed in classifying text content with the aim of identifying destructive information within social networks. The study focuses on utilizing machine learning techniques, such as support vector machines, naive Bayes classifiers, random tree methods, decision tree, k-Nearest Neighbors algorithm, logistic regression, gradient boosting to identify extremist texts. The research findings showcase the effectiveness of these methodologies in the identification process. The article also offers an overview of existing research, methodologies, and software products in the analysis of extremist texts, emphasizing the importance of case-based learning, deductive learning models, and automated data collection and analysis. Additionally, the article provides an overview of existing research, methods, and software products within the field of analyzing extremist texts. It highlights the significance of case-based learning and the use of deductive learning models, as well as automated data collection and analysis techniques. These approaches contribute to the overall understanding and detection of extremist content. The article further discusses the relevance and future prospects of the presented research. It emphasizes the need to expand the corpus of documents studied, enabling a more comprehensive analysis of texts, including those in photo, audio, and video formats. The development of complex models for recognizing hidden extremist propaganda is also identified as a key direction for future work. By addressing these areas of focus, the research presented in the article aims to advance the field of identifying and combating extremist content within social networks. The incorporation of advanced techniques and technologies is crucial to effectively detect and address the presence of such content in various forms and formats.

Digital Streets of Rage: Identifying Rhizomatic Extremist Messages During a Hybrid Media Event using Natural Language Processing

This research explores how to identify extreme messages during a hybrid media event happening in a small language area by utilizing natural language processing (NLP), a type of artificial intelligence (AI). A hybrid media event gathers attention all sides of the media environment: mainstream media, social media, instant messaging apps and fringe communities. Hybrid media events call attention for participation and activities both in the physical world and online. On the darker side of media events, the media landscape can act as a channel for all kinds of disinformation, hate speech and conspiracy theories. In addition, fringe communities such as 4chan also spread hate speech and duplicated content during hybrid media events. From theoretical point of view, this connection between the physical world and information networks can be seen as rhizomatic in nature, because information spreads without regard to a traditional hierarchy. The result is that when individuals participate in a big media event, there is a viral awareness of different viewpoints and all kind of topics may be posted online for discussion. In addition, in rhizomatic context different kind of arguments can twist each other, “copy and paste”, and create very diversity meanings of new comments. The role of extremist speech in online spaces can have effects in physical world.
 The focus of this paper is to present the findings of a case study on messages posted online by three different actor groups who participated in demonstrations organized on Finnish Independence Day. In this research, two data sets were collected from Twitter and Telegram and Natural Language Processing (NLP) was used to classify messages using extremist media index labels. Three actor groups were identified as participating in the demonstrations, and they were labelled as: far-right, antifascists and conspiracists. Computational analysis was done by using NLP to categorize the messages based upon the definitions provided by the extremist media index. The analysis shows how AI technology can help identifying messages which include extremist content and approve the use of violence in a small language area. The model of rhizome was valid in making the connections between fringe, extremist content and moderate discussion visible. This article is part of larger project related to extremist networks and criminality in online darknet environments.

The role of information skewness in shaping extremist content: A look at four extremists

Introduction. Extremism—distinct from activism—poses a serious threat to the healthy functioning of a society. In the contemporary world, the ability of extremists to spread their narratives using digital information environments has increased tremendously. Despite a substantial body of research on extremism, our understanding of the role of information and its properties in shaping extremist content is sketchy. Method. To fill this gap, the current research has used ‘content analysis’ and ‘affective lexicon’ to identify and categorise terms from the publicly available online content of four extremists – two groups and two individuals. The property of information skewness provided the deciphering lens through which the categorised content was assessed. Analysis. Contextual categories of information relevant to all the extremists were developed to analyse the content meaningfully. Six categories of religion, ideology, politics-history, cognition, affection, and conation provided the framework used to analyse and deductively categorise the data using content analysis. The affective lexicon developed by Ortony et al. (1987) was used to identify words belonging to the categories of cognition, affection (emotions and feelings), and conation (behaviour/actions). Results. The findings reveal that the property of information skewness plays a significant role in shaping extremist content and two aspects of this property (a) intensity and (b) positivity or negativity can be used to (1) classify extremists into meaningful categories and (2) identify generalisable information strategies of extremists. Conclusions. It is hoped that the findings of this research will inform future enquiries into the role of information and its properties in shaping extremist content and help security agencies to effectively engage in information warfare with extremists.

TamTam: The Online Drums of Hate

Recognizing the correlation between the growing spread of violent extremist and terrorist content on the internet and online platforms and the significant increase in attacks inspired by these posts prompted governments, security agencies, and private companies to launch various countermeasures to reduce the spread and impact of such material. Struggling to adjust to these countermeasures and maintain their online presence, terrorists and violent extremists have migrated from mainstream online platforms to alternative online channels like 4chan, 8chan, Telegram, TikTok, and most recently to the new platform TamTam. This paper describes the migration to and within social media generally, with a specific focus on the newly discovered migration to the platform TamTam and presents the results of our extensive content analysis of hundreds of posts and shares containing terrorist and violent extremist content on this platform. These findings are discussed in terms of ethical and practical implications.

Social Strategies of Terrorism

Terrorism poses a significant challenge to societies worldwide, employing various social strategies to achieve political objectives, disseminate ideologies, and recruit new members. This article explores the social strategies utilized by terrorist organizations, focusing on radicalization, propaganda dissemination, support network formation, exploitation of social cleavages, the role of charismatic leaders, and the use of intimidation to create fear and uncertainty. By examining the process of radicalization, the article highlights the significance of personal grievances, psychological vulnerabilities, and exposure to extremist ideologies, as well as the importance of social networks in facilitating connections and access to extremist content. The article also investigates how terrorist organizations use propaganda to advance their objectives, justify violence, promote shared identity, and vilify perceived enemies. The establishment of support networks that provide financial and logistical assistance, embedded within local communities, and the exploitation of social divisions and grievances to mobilize support and recruit new members are discussed. The role of charismatic leaders in inspiring followers, cultivating shared identity, and projecting an image of strength and defiance is examined. The article emphasizes the need for counterterrorism efforts to address these social strategies, including initiatives aimed at countering extremist ideologies, disrupting support networks, and promoting community engagement and resilience-building measures. Through an analysis of scientific research and expert insights, this article provides a comprehensive understanding of the social dynamics underpinning terrorism and informs the development of effective counterterrorism measures.

GIFCT Tech Trials: Combining Behavioural Signals to Surface Terrorist and Violent Extremist Content Online

Technical approaches for surfacing, reviewing, and removing terrorist and violent extremist content online have evolved in recent years. Larger companies have added metrics and insights in transparency reports, disclosing that most content removed for terrorist and violent extremist offences is proactive, using hybrid models deploying tooling in combination with human oversight. However, less is known about the algorithmic tools or hybrid models deployed by tech platforms to ensure greater accuracy in surfacing terrorist threats. This paper reviews existing tools deployed by platforms to counter terrorism and violent extremism online, including ethical concerns and oversight needed for algorithmic deployment, before analysing initial results from a GIFCT technical trial. The Global Internet Forum to Counter Terrorism (GIFCT) Technical Trial discussed in this paper presents the results from testing a methodology using behavioural and linguistic signals to more accurately and proactively surface terrorist and violent extremist content relating to potential real-world attacks. As governments, tech companies, and networks like GIFCT develop crisis and incident response protocols, the ability to quickly identify perpetrator content associated with attacks is crucial, whether that relates to the live streaming of an attack, or an attacker manifesto launched in parallel to the real-world violence. Building on previous academic research, while the deployment of layered signals shows promise in proactive detection and the reduction of false/positives, it also highlights the complexities of user speech, online behaviours, and cultural nuances.

Policy vs reality: comparing the policies of social media sites and users’ experiences, in the context of exposure to extremist content

ABSTRACT Social media platforms have become a prominent feature in modern-day communication, allowing users to express opinion and communicate with friends and likeminded individuals. However, with this revolutionary form of communication comes risks of exploitation and utilisation of these platforms for potentially illegal and harmful means. This article aims to explore the community guidelines and policies of prominent social media sites regarding extremist material, comparing the platform’s policies with the user experiences. To measure social media user experience and user exposure to extremist material, this article pilots the use of a new scale: The Online Extremism Exposure Scale (OECE), which measures both the user’s exposure to extremist communication and hate speech online. Users reported varied levels of exposure to both hate speech and extremist communication, with. the results indicating that users in the sample are being exposed to extremist material approximately 48.44% of the time they spend on social media daily. The results of this pilot study highlight potential failings by prominent social media platforms in their efforts to reduce users being exposed to extremist material. Limitations, future research, and implications are discussed in detail.

Identification of Online Extremism in Text Content: a Systematic Review of the Literature

Social networks such as Twitter, Facebook, YouTube are becoming increasingly powerful tools that are used to spread radical ideologies and propaganda. The three main social networks of today – Facebook, Twitter and YouTube – have a total of 3.2 billion users, which is a staggering figure of about 42% of the world's population. Identifying extremist content is a top priority for anti-terrorism agencies, technology companies and governments. The main purpose of the article is to understand the definition of extremism and provide a detailed overview of current research on extremism on the Internet. This study offers a critical analysis of an emerging area of research. This study offers a critical analysis of an emerging area of research

“Please. Do. Not. Share. Videos. Share. Cats.”: Counteracting Terrorist and Violent Extremist Content on Twitter during Terrorist Attacks

Obtaining accurate information from social media during a crisis can be difficult, but should all information really be disseminated? Social media platforms actively filter out terrorist and violent extremist content (TVEC), but how are users themselves counteracting its spread? This paper aims to connect the research on media events with studies currently being conducted in information science and digital media research through a case study of tweets during the Vienna terror attack in late 2020. These tweets were manually coded in accordance with Braun and Clarke’s reflexive thematic analysis. This study shows that during the 2020 Vienna attack, GIFs shared on Twitter served three functions: amplification, personalisation and ethical practice. The paper ends with a discussion on the ways cats may function as a countermeasure against the prevalence of TVEC on social media during terrorist attacks and the implications of such countermeasures.

YouTube radical content recommendations: a meta-analysis

YouTube's recommendation system algorithms have been the subject of controversy and debate, as there have been incidents of the recommendation system offering hazardous, most often extremist content to users. The purpose of this study is to test the hypothesis that YouTube recommendations can contain and promote radical content. The scientific and practical significance of this work is that YouTube has a huge popularity, and knowledge about the algorithms of its recommendation system would help to improve information literacy of users and thus prevent the potential danger that can be caused by exposure to malicious information. This study seeks to popularize the idea that all information on the Internet requires a thorough verification. Improving information literacy of users, this research contributes globally to the prevention of terrorist attacks, acts of self-mutilation, suicide, pedophilia, etc. The sources for the present investigation with description at least one type of radical content were published in the last 5 years. The material of interest was searched through authoritative scientific and metric databases, namely: Google Scholar, Scopus, Web of Science, and PubMed by extracting suitable records. A total of 22 studies were pooled according to the eligibility and exclusion criteria, and a meta-analysis was performed which found that in 13 of them YouTube recommendations contained and promoted hazardous content, in 7 studies the scientists presented ambiguous results, and only in 2 studies the authors found that the recommendations did not contain or promote any radical content. Thus, according to the results of this research, it was found that YouTube recommendations can contain and disseminate prohibited hazardous content, and therefore the authors strongly recommend making adjustments to the algorithms of the recommendations in order to protect users from prohibited information, including the promotion of extremism and violence. Keywords: YouTube, recommendations, recommendation system, hazardous content, pseudoscientific content, radical content, extremism, pedophilia, meta-analysis.

COUNTERING THE THREAT FROM RADICALISING INFLUENCES AND FOCUSING ON PROTECTING VULNERABLE COMMUNITIES

On November 23, 2022, Andrew Hayes, Chief Inspector (CI) of the South Wales territorial police force presented on Countering the Threat from Radicalizing Influences and Focusing on Protecting Vulnerable Communities. The presentation was followed by a question-and-answer period with questions from the audience and CASIS-Vancouver executives. The key points discussed were current trends in extremist recruitment and policy enactment compared to the preceding year, and counteracting extremist content showcased over decentralised platforms containing end-to-end encryption. Received: 2022-12-25Revised: 2023-01-05

ОРГАНІЗАЦІЙНО-ПРАВОВІ ЗАСАДИ ПРОТИДІЇ ЕКСТРЕМІЗМУ ТА ШЛЯХИ ВДОСКОНАЛЕННЯ

The article examines the meaning of the concept of "extremism", reveals its illegal essence, signs and causes of its occurrence, and also outlines the lack of a legal definition of the concept of "extremism" in the legislation of Ukraine. The author emphasizes the organizational and legal principles of combating extremism and substantiates the opinion that the main source of extremism is the crisis state of society. A description of the types of extremism and its main forms and consequences for society is given. Thus, among the main forms of extremism we distinguish: production, storage or distribution of extremist materials (inscriptions, leaflets, audiovisual and other materials), use of Internet resources for extremist activities; public statements of extremist content; public appeals or incitement to commit extremist actsfinancing of extremist actions or other assistance in their organization, preparation and implementation, including by providing financial funds, real estate, educational, printing and material-technical base, telephone, fax and other types of communication, as well as providing information services. Derivatives of extremism are singled out, in particular: terrorism, collaborationism, separatism. The characteristics of subjects fighting extremism are presented separately (state authorities and local self-government bodies that participate within their competence; the Security Service of Ukraine and the State Bureau of Investigation, if representatives of authorities participate in extremist activities), as well as the content of the main directions of combating it. Prospective directions for further scientific research are the study of the experience of the countries of the European Union in countering extremism, the current state of countering extremism in Ukraine, and ways to improve the domestic legislation.

ЦИФРОВИЗАЦИЯ КАК ФАКТОР РИСКА РАЗВИТИЯ СОЦИАЛЬНЫХ ИНСТИТУТОВ И КОММУНИКАТИВНЫХ ПРАКТИК

The article focuses on the risks of destabilization of social institutions due to the expansion of the digital environment and its penetration into all spheres of society. The main categories of threats arising from the redundancy of information flows, as well as the widespread distribution of content that has disinforming, aggressive, and sometimes clearly extremist content are highlighted. Possible mechanisms of the influence of such Internet practices on the processes of transformation and self-organization in the social system are discussed.

Anxious activism: The role of behavioral inhibition system in the radicalization process

Many individuals that engage with “terroristic content” do so in online settings. Most research on online exposure to dangerous materials focuses on the content and who is posting it, and thus there is a lack of understanding on what individual traits motivate individuals to engage with extreme online content. The Behavioral Inhibition System (BIS) controls motivations to withdraw from a stimulus and can coordinate anxiety responses to acts of terrorism (Carver, 2004). However, anxiety (and therefore the BIS) can also motivate individuals to engage further with terroristic material and/or groups via the “quest for significance” (Kruglanski et al., 2014). As such, individual differences in BIS functioning may play a role in governing the relationship between exposure to extremist content and motivations to engage in extremist behavior. Using an experimental survey design in which individuals are exposed to terroristic content in an online setting, this study explores the role of BIS sensitivity in the willingness to engage with terroristic content and individual differences in ensuing activism and radicalism intentions. In support of our hypotheses, BIS sensitivity predicted increased intentions towards radicalism and activism in support of a terrorist group after exposure to terroristic content online. These findings support the quest for significance model, which posits that existential anxiety can play a role in motivating individuals toward radicalization and violent extremism.

An Investigation of the Effectiveness of Facebook and Twitter Algorithm and Policies on Misinformation and User Decision Making

Prominent social media sites such as Facebook and Twitter use content and filter algorithms that play a significant role in creating filter bubbles that may captivate many users. These bubbles can be defined as content that reinforces existing beliefs and exposes users to content they might have otherwise not seen. Filter bubbles are created when a social media website feeds user interactions into an algorithm that then exposes the user to more content similar to that which they have previously interacted. By continually exposing users to like-minded content, this can create what is called a feedback loop where the more the user interacts with certain types of content, the more they are algorithmically bombarded with similar viewpoints. This can expose users to dangerous or extremist content as seen with QAnon rhetoric, leading to the January 6, 2021 attack on the U.S. Capitol, and the unprecedented propaganda surrounding COVID-19 vaccinations. This paper hypothesizes that the secrecy around content algorithms and their ability to perpetuate filter bubbles creates an environment where dangerous false information is pervasive and not easily mitigated with the existing algorithms designed to provide false information warning messages. In our research, we focused on disinformation regarding the COVID-19 pandemic. Both Facebook and Twitter provide various forms of false information warning messages which sometimes include fact-checked research to provide a counter viewpoint to the information presented. Controversially, social media sites do not remove false information outright, in most cases, but instead promote these false information warning messages as a solution to extremist or false content. The results of a survey administered by the authors indicate that users would spend less time on Facebook or Twitter once they understood how their data is used to influence their behavior on the sites and the information that is fed to them via algorithmic recommendations. Further analysis revealed that only 23% of respondents who had seen a Facebook or Twitter false information warning message changed their opinion "Always" or "Frequently" with 77% reporting the warning messages changed their opinion only "Sometimes" or "Never" suggesting the messages may not be effective. Similarly, users who did not conduct independent research to verify information were likely to accept false information as factual and less likely to be vaccinated against COVID-19. Conversely, our research indicates a possible correlation between having seen a false information warning message and COVID-19 vaccination status.

К вопросу о проблеме нарративов и контрнарративов в деятельности международного исламистского движения Хизб ут-Tахрир

The dissemination of extremist narratives as well as the propaganda of violence and terrorist ideology summon experts to take urgent measures in order to nullify their effects in the contempo-rary society. Purposefully created counter-narratives and the forms of their implementation are pro-posed as a way to weaken extremist content. The author analyzes the problems in assessing the ex-tremist potential of Hizb ut-Tahrir movement and suggests a counter-narrative analysis of their vid-eos as a methodological tool for determining the capabilities of Hizb ut-Tahrir’s radicalization. The videos call for the abolition of the existing world order and replacing it with the organizational prin-ciples adopted in Islam. The author concludes that the propaganda of Hizb ut-Tahrir, despite the statements about the peaceful nature of its activities, poses a threat to the modern world. The analy-sis of discussion materials on the effectiveness of using the counter-narrative as a tool to counter terrorist ideology is also presented.

Combining artificial intelligence and expert content analysis to explore radical views on twitter: Case study on far-right discourse

Public opinion is among the critical types of information that often inform policy changes and political strategies. Extreme public opinion is of particular importance due to the potential it has in leading to radical behaviors and violent actions. The monitoring and analysis of extreme public opinion are therefore of great interest to government officials seeking to maintain public order and preempt violent actions. Radicalization is the process employed to influence individuals to develop extreme views and behaviors. Due to their global reach and popularity, social media platforms are used tools for recruitment and radicalization. Recently, data mining and natural language processing techniques have been employed for the detection of extremism and radicalization online. However, the existing approaches focus on the classification of individuals as extremists or non-extremists, thus simplifying the concept of radicalization to a binary problem. Such approaches fail to capture the nuances and the complexity of the radicalization process that is influenced by many aspects such as peer pressure, social interactions, and the impact of prominent figures. Aiming at addressing those limitations, this work proposed a sophisticated approach for the analysis of extreme views expressed on social media. The proposed approach combines the power of artificial intelligence and natural language processing techniques with expert content analysis to achieve a fine-grained and detailed analysis of Twitter extremist content related to the Far-right ideology as a case study. A dataset of over 259,000 tweets collected over five years was used to test our approach, leading to sophisticated analytics and insights about Far-right extremism. The proposed approach can serve as a powerful decision support tool for governments for the analysis of extreme public opinion that is expressed online, and open the door for more effective and responsive decision making.

A semi-supervised algorithm for detecting extremism propaganda diffusion on social media

Abstract Extremist online networks reportedly tend to use Twitter and other Social Networking Sites (SNS) in order to issue propaganda and recruitment statements. Traditional machine learning models may encounter problems when used in such a context, due to the peculiarities of microblogging sites and the manner in which these networks interact (both between themselves and with other networks). Moreover, state-of-the-art approaches have focused on non-transparent techniques that cannot be audited; so, despite the fact that they are top performing techniques, it is impossible to check if the models are actually fair. In this paper, we present a semi-supervised methodology that uses our Discriminatory Expressions algorithm for feature selection to detect expressions that are biased towards extremist content (Francisco and Castro 2020). With the help of human experts, the relevant expressions are filtered and used to retrieve further extremist content in order to iteratively provide a set of relevant and accurate expressions. These discriminatory expressions have been proved to produce less complex models that are easier to comprehend, and thus improve model transparency. In the following, we present close to 70 expressions that were discovered by using this method alongside the validation test of the algorithm in several different contexts.