Offensive Content Research Articles

The Adoption of Artificial Intelligence Technologies in Arab Newsrooms: Potentials and Challenges

Arab newsrooms and journalists are still attempting to process how to benefit from using artificial intelligence (AI) in their daily news production. Many have started experimenting with generative AI. However, in countries with strong economies, or in newsrooms funded by strong economies like in the UAE, Saudi Arabia, and Qatar, news organizations have endorsed AI technologies beyond ChatGPT and similar tools. The majority of Arab journalists, however, are yet to understand what is meant by AI. Very few news organizations in the region have strategically approached the question of AI. Arab newsrooms need to examine when and why they need to use AI technologies, whether they have the infrastructure or can afford the resources. Implementing AI in Arab newsrooms presents an additional set of challenges, including language complexities, cultural sensitivities, potential biases in algorithms, and the need for tailored solutions that resonate with local audiences. The ethical implications present its own set of challenges. ChatGPT has proven to generate inaccurate data in Arabic. Resources are to be put in place to ensure the model is trained on a diverse and comprehensive dataset of Arabic text. Implementing bias detection and mitigation techniques to ensure the models are free from biases and offensive content is crucial. Newsrooms operating in authoritarian regimes have to be aware of governments' attempts to use generative AI to whitewash and promote state-centered actions and policies. Maintaining editorial integrity remains an important consideration.

Sentiment Analysis of Arabic Tweets: Detecting Revilement

Social media systems play an necessary function in shaping public opinion and reflecting societal sentiments. This study focuses on sentiment analysis in Arabic tweets with a particular focus on offensive or offensive content. The aim of this research is to boost a dependable sentiment evaluation model that can accurately classify Arabic tweets as positive or negative, with a particular focus on identifying offensive language. A multinomial Naive Bayes classifier is trained on pre-processed data to perform sentiment classification. The classifier is fine-tuned to differentiate between positive and negative emotions, with a particular focus on identifying offensive or swearing language. The model is evaluated the usage of a complete set of metrics along with precision, precision, recall, and F1score. Experimental consequences point out promising overall performance of the developed sentiment evaluation model. The model achieved an accuracy of 93%, effectively classifying Arabic tweets into effective and bad sentiments. The precision, recall, and F1-score metrics similarly validate the model's capacity to precisely become aware of revilement and offensive language. These outcomes spotlight the conceivable of the proposed strategy in successfully examining Arabic tweets for sentiment and offensive content, contributing to higher grasp on line behaviors and sentiments in the context of revilement.

Technology-Facilitated Sexual Violence: Victimization and Risk Factors

Technology-Facilitated Sexual Violence (TFSV) has been a permanent concern in contemporary society. This study aims to provide a global understanding of the TFSV phenomenon in Portuguese context. Using quantitative analyses, the rate and prevalence of victimization, victim characteristics, and technology use were examined. An online questionnaire was completed by 500 people (75.8% female) aged 18–70, mostly cisgender (96.2%), and heterosexual (85.8%). The main results point to a high victimization global rate (72%) as well as in the last 12 months (70.8%). There were no significant differences between the sexes except in sexual aggression/coercion, with more females suffering the more severe type of TFSV. However, the gendering of TFSV emerges with specific behaviors. Females tend to be more sexually harassed, only females report non-consensual sexual experiences with someone they met online, and males tend to report receiving offensive content about their gender/sexuality. Younger people tend to report a higher rate of victimization. The regression model with only victim characteristics is more predictive of TFSV victimization, with younger and heterosexual victims as significant predictors. This study argues that the culture of harassment is deeply rooted and finds an easy way to be disseminated in the digital world.

OLF-ML: An Offensive Language Framework for Detection, Categorization, and Offense Target Identification Using Text Processing and Machine Learning Algorithms

The pervasiveness of offensive language on social media emphasizes the necessity of automated systems for identifying and categorizing content. To ensure a more secure online environment and improve communication, effective identification and categorization of this content is essential. However, existing research encounters challenges such as limited datasets and biased model performance, hindering progress in this domain. To address these challenges, this research presents a comprehensive framework that simplifies the utilization of support vector machines (SVM), random forest (RF) and artificial neural networks (ANN). The proposed methodology yields notable gains in offensive language detection, automatic categorization of offensiveness, and offense target identification tasks by utilizing the Offensive Language Identification Dataset (OLID). The simulation results indicate that SVM performs exceptionally well, exhibiting excellent accuracy scores (77%, 88%, and 68%), precision scores (76%, 87%, and 67%), F1 scores (57%, 88%, and 68%), and recall rates (45%, 88%, and 68%), proving to be practically successful in identifying and moderating offensive content on social media. By applying sophisticated preprocessing and meticulous hyperparameter tuning, our model outperforms some earlier research in detecting and categorizing offensive language tasks.

Detecting Offensive Language on Malay Social Media: A Zero-Shot, Cross-Language Transfer Approach Using Dual-Branch mBERT

Social media serves as a platform for netizens to stay informed and express their opinions through the Internet. Currently, the social media discourse environment faces a significant security threat—offensive comments. A group of users posts comments that are provocative, discriminatory, and objectionable, intending to disrupt online discussions, provoke others, and incite intergroup conflict. These comments undermine citizens’ legitimate rights, disrupt social order, and may even lead to real-world violent incidents. However, current automatic detection of offensive language primarily focuses on a few high-resource languages, leaving low-resource languages, such as Malay, with insufficient annotated corpora for effective detection. To address this, we propose a zero-shot, cross-language unsupervised offensive language detection (OLD) method using a dual-branch mBERT transfer approach. Firstly, using the multi-language BERT (mBERT) model as the foundational language model, the first network branch automatically extracts features from both source and target domain data. Subsequently, Sinkhorn distance is employed to measure the discrepancy between the source and target language feature representations. By estimating the Sinkhorn distance between the labeled source language (e.g., English) and the unlabeled target language (e.g., Malay) feature representations, the method minimizes the Sinkhorn distance adversarially to provide more stable gradients, thereby extracting effective domain-shared features. Finally, offensive pivot words from the source and target language training sets are identified. These pivot words are then removed from the training data in a second network branch, which employs the same architecture. This process constructs an auxiliary OLD task. By concealing offensive pivot words in the training data, the model reduces overfitting and enhances robustness to the target language. In the end-to-end framework training, the combination of cross-lingual shared features and independent features culminates in unsupervised detection of offensive speech in the target language. The experimental results demonstrate that employing cross-language model transfer learning can achieve unsupervised detection of offensive content in low-resource languages. The number of labeled samples in the source language is positively correlated with transfer performance, and a greater similarity between the source and target languages leads to better transfer effects. The proposed method achieves the best performance in OLD on the Malay dataset, achieving an F1 score of 80.7%. It accurately identifies features of offensive speech, such as sarcasm, mockery, and implicit expressions, and showcases strong generalization and excellent stability across different target languages.

Misogynistic attitude detection in YouTube comments and replies: A high-quality dataset and algorithmic models

Social media platforms are now not only a medium for expressing users views, feelings, emotions and sentiments but are also being abused by people to propagate unpleasant and hateful content. Consequently, research efforts have been made to develop techniques and models for automatically detecting and identifying hateful, abusive, vulgar, and offensive content on different platforms. Although significant progress has been made on the task, the research on design of methods to detect misogynistic attitude of people in non-English and code-mixed languages is not very well-developed. Non-availability of suitable datasets and resources is one main reason for this. Therefore, this paper attempts to bridge this research gap by presenting a high-quality curated dataset in the Hindi-English code-mixed language. The dataset includes 12,698 YouTube comments and replies, with each comment annotated under two-level categories, first as optimistic and pessimistic, and then into different types at second level based on the content. The inter-annotator agreement in the dataset is found to be 0.84 for the first subtask, and 0.79 for the second subtask, indicating the reasonably high quality of annotations. Different algorithmic models are explored for the task of automatic detection of the misogynistic attitude expressed in the comments, with the mBERT model giving best performance on both subtasks (reported macro average F1 scores of 0.59 and 0.52, and weighted average F1 scores of 0.66 and 0.65, respectively). The analysis and results suggest that the dataset can be used for further research on the topic and that the developed algorithmic models can be applied for automatic detection of misogynistic attitude in social media conversations and posts.

Hate Speech Detection by Using Rationales for Judging Sarcasm

The growing number of social media users has impacted the rise in hate comments and posts. While extensive research in hate speech detection attempts to combat this phenomenon by developing new datasets and detection models, reconciling classification accuracy with broader decision-making metrics like plausibility and faithfulness remains challenging. As restrictions on social media tighten to stop the spread of hate and offensive content, users have adapted by finding new approaches, often camouflaged in the form of sarcasm. Therefore, dealing with new trends such as the increased use of emoticons (negative emoticons in positive sentences) and sarcastic comments is necessary. This paper introduces sarcasm-based rationale (emoticons or portions of text that indicate sarcasm) combined with hate/offensive rationale for better detection of hidden hate comments/posts. A dataset was created by labeling texts and selecting rationale based on sarcasm from the existing benchmark hate dataset, HateXplain. The newly formed dataset was then applied in the existing state-of-the-art model. The model’s F1-score increased by 0.01 when using sarcasm rationale with hate/offensive rationale in a newly formed attention proposed in the data’s preprocessing step. Also, with the new data, a significant improvement was observed in explainability metrics such as plausibility and faithfulness.

'Getting it write' in a new era of patient online access to primary care health records.

To explore how to help make online consultation notes easier for patient audiences to understand. Most patients in England will soon be able to access all new prospective entries (including free-text) within their online primary care health record via the NHS App or other online services. We conducted interviews and focus group discussions with 26 patients from underserved communities. Participants responded to vignettes about fictional patient consultation scenarios and assessed the clarity of corresponding simulated records. Participants were then asked to identify potential comprehension issues, offensive content, or anxiety triggers. Most participants struggled to understand a large proportion of simulated notes, particularly medical acronyms, clinician shorthand, and non-clinical abbreviations. Participants also identified issues that may cause unintended offence or additional anxiety. Participants considered that most patients will struggle to fully understand the content of their consultation notes in their current format. They made a number of suggestions about how this service may be improved to meet the needs of patient audiences and maintain positive patient-clinician relationships. Opening up online record access to include patient audiences necessitates a significant cultural shift in the way that consultation notes are written and used. To maximise NHS investment in this policy and avoid worsening health inequalities, it is essential to ensure that all patients can understand and access the benefits of online access to their notes. To do this, clinicians need to be supported to manage the challenges of writing for patient audiences, while continuing to maintain effective clinical care.

Tone Tracker

Abstract: Tone Tracker's integration of BERT and LSTM technologies allows it to predict and flag offensive language in both Tamil and English social media comments. BERT, a transformer-based model, enables the tool to understand the semantic context of text in multiple languages, ensuring accurate detection of inappropriate content regardless of language. The incorporation of LSTM further enhances this capability by capturing nuanced contextual information, refining the tool's proficiency in content moderation for both Tamil and English content. With its user-friendly features, Tone Tracker becomes accessible to a diverse user base, empowering them to swiftly remove offensive content in both languages and contribute to fostering a secure digital environment. This groundbreaking innovation not only boosts content moderation efficiency but also ensures scalability across various digital platforms, making it adaptable to the linguistic diversity of online communities. Ultimately, Tone Tracker, powered by LSTM and BERT, plays a pivotal role in cultivating positive online spaces where users can engage confidently and respectfully in both Tamil and English.

Advanced Social Media Toxic Comments Detection System Using AI

Abstract: In the contemporary digital landscape, social media platforms have evolved into essential components of our daily lives, fostering connections, idea sharing, and meaningful conversations. Nevertheless, the escalating volume of online interactions has ushered in a concerning rise in toxic comments and cyberbullying, casting a shadow over the potential for a healthy onlineenvironment. Toxic comments, spanning hate speech, harassment, and offensive content, not only inflict harm on individuals but also detrimentally affect the overall user experience. This project seeks to address this pressing issue through the development of a cutting-edge Toxic Comment Detection system, leveraging the power of Natural Language Processing (NLP) and Machine Learning (ML) techniques. The primary objective of this endeavor is to create an automated system capable of identifying and flagging toxic comments in realtime across various social media platforms. By employing advanced NLP algorithms and ML models, the system aims to analyze textual content swiftly and accurately, pinpointing instances of toxicity. Once identified, the system will promptly notify moderators, enabling swift intervention and potential removal of the harmful content. By implementing this technological solution, the project aspires to contribute significantly to fostering a safer and more inclusive online environment where users can engage without fear of encountering toxic behavior. Through the fusion of NLP and ML, this endeavor aims to exemplify the transformative potential of technology in mitigating the challenges posed by toxic comments in the digital age.

User Attitudes to Content Moderation in Web Search

Internet users highly rely on and trust web search engines, such as Google, to find relevant information online. However, scholars have documented numerous biases and inaccuracies in search outputs. To improve the quality of search results, search engines employ various content moderation practices such as interface elements informing users about potentially dangerous websites and algorithmic mechanisms for downgrading or removing low-quality search results. While the reliance of the public on web search engines and their use of moderation practices is well-established, user attitudes towards these practices have not yet been explored in detail. To address this gap, we first conducted an overview of content moderation practices used by search engines, and then surveyed a representative sample of the US adult population (N=398) to examine the levels of support for different moderation practices applied to potentially misleading and/or potentially offensive content in web search. We also analyzed the relationship between user characteristics and their support for specific moderation practices. We find that the most supported practice is informing users about potentially misleading or offensive content, and the least supported one is the complete removal of search results. More conservative users and users with lower levels of trust in web search results are more likely to be against content moderation in web search.

Semantic Conversational Content Moderation

In today's digital environment, social media platforms have become an integral part of our daily lives, facilitating global communication, knowledge sharing and community building. However, these platforms are increasingly vulnerable to the spread of offensive and toxic content, including misinformation, harassment and hate speech. Such content poses a significant threat to the safety and well-being of Internet users. In response to this immediate problem, we set out to develop an AI-based conversation moderation service aimed at effectively detecting and removing offensive or semantically toxic information in real time. Our solutions strive to make the internet safer and more user-friendly, thereby promoting a more positive and inclusive online environment. Key Words: Natural Language Processing(NLP), Moderation, Semantic Analysis, Mistral.

MIMIC: Misogyny Identification in Multimodal Internet Content in Hindi-English Code-Mixed Language

Over the years, social media has emerged as one of the most popular platforms where people express their views and share thoughts about various aspects. The social media content now includes a variety of components such as text, images, videos etc. One type of interest is memes, which often combine text and images. It is relevant to mention here that, social media being an unregulated platform, sometimes also has instances of discriminatory, offensive and hateful content being posted. Such content adversely affects the online well-being of the users. Therefore, it is very important to develop computational models to automatically detect such content so that appropriate corrective action can be taken. Accordingly, there have been research efforts on automatic detection of such content focused mainly on the texts. However, the fusion of multimodal data (as in memes) creates various challenges in developing computational models that can handle such data, more so in the case of low-resource languages. Among such challenges, the lack of suitable datasets for developing computational models for handling memes in low-resource languages is a major problem. This work attempts to bridge the research gap by providing a large-sized curated dataset comprising 5,054 memes in Hindi-English code-mixed language, which are manually annotated by three independent annotators. It comprises two subtasks: (i) Subtask-1 (Binary classification involving tagging a meme as misogynous or non-misogynous), and (ii) Subtask-2 (multi-label classification of memes into different categories). The data quality is evaluated by computing Krippendorff's alpha. Different computational models are then applied on the data in three settings: text-only, image-only, and multimodal models using fusion techniques. The results show that the proposed multimodal method using the fusion technique may be the preferred choice for the identification of misogyny in multimodal Internet content and that the dataset is suitable for advancing research and development in the area.

A survey on multi-lingual offensive language detection.

The prevalence of offensive content on online communication and social media platforms is growing more and more common, which makes its detection difficult, especially in multilingual settings. The term "Offensive Language" encompasses a wide range of expressions, including various forms of hate speech and aggressive content. Therefore, exploring multilingual offensive content, that goes beyond a single language, focus and represents more linguistic diversities and cultural factors. By exploring multilingual offensive content, we can broaden our understanding and effectively combat the widespread global impact of offensive language. This survey examines the existing state of multilingual offensive language detection, including a comprehensive analysis on previous multilingual approaches, and existing datasets, as well as provides resources in the field. We also explore the related community challenges on this task, which include technical, cultural, and linguistic ones, as well as their limitations. Furthermore, in this survey we propose several potential future directions toward more efficient solutions for multilingual offensive language detection, enabling safer digital communication environment worldwide.

Syntax-aware Offensive Content Detection in Low-resourced Code-mixed Languages with Continual Pre-training

Social media is a widely used platform that includes a vast amount of user-generated content, allowing the extraction of information about users’ thoughts from texts. Individuals freely express their thoughts on these platforms, often without constraints, even if the content is offensive or contains hate speech. The identification and removal of offensive content from social media are imperative to prevent individuals or groups from becoming targets of harmful language. Despite extensive research on offensive content detection, addressing this challenge in code-mixed languages remains unsolved, characterised by issues such as imbalanced datasets and limited data sources. Most previous studies on detecting offensive content in these languages focus on creating datasets and applying deep neural networks, such as Recurrent Neural Networks (RNNs), or pre-trained language models (PLMs) such as BERT and its variations. Given the low-resource nature and imbalanced dataset issues inherent in these languages, this study delves into the efficacy of the syntax-aware BERT model with continual pre-training for the accurate identification of offensive content and proposes a framework called Cont-Syntax-BERT by combining continual learning with continual pre-training. Comprehensive experimental results demonstrate that the proposed Cont-Syntax-BERT framework outperforms state-of-the-art approaches. Notably, this framework addresses the challenges posed by code-mixed languages, as evidenced by its proficiency on the DravidianCodeMix [10,19] and HASOC 2109 [37] datasets. These results demonstrate the adaptability of the proposed framework in effectively addressing the challenges of code-mixed languages.

Dataset Augmentation for Counteracting Bias in Toxic Comment Classification


 Toxic comments are a prevalent issue on online social media and networking platforms. These comments contain offensive, malicious, hate speech, or other harmful content that negatively impacts audiences and communities. Effectively detecting and categorizing toxic comments is essential for maintaining order in the online environ ment, protecting user safety, and enhancing user experience. This is despite the fact that researchers and companies have developed various models to recognize toxicity in online chats and comments, achieving some success. However, many of the currently used models incorrectly classify non-toxic comments that contain certain identity terms as potentially toxic. This misclassification hinders the ability to accurately identify categorized comments. In this paper, the detection and classification of toxic comments were implemented using Term Frequency-Inverse Document Frequency (TF-IDF) and machine learning techniques. Additionally, two dataset-specific optimizations were proposed to mitigate the impact of bias on text classification by expanding the number of datasets. Comparative analysis of bias evaluation metrics demonstrates that this approach can effectively mitigate bias while maintaining the accuracy of the original model as much as possible.

Insightsphere: Social Analytics Command Center

The Social Media Command Center is a complex program designed to monitor and analyze user activity on various social media platforms. The innovative tool includes sentiment analysis to measure user sentiment and sentiment. One of its main features is the ability to disable unwanted features, allowing you to access content and communications. The system uses sentiment analysis to identify and correct harmful or counterproductive user behavior to create a better and more manageable online environment. This allows the system to distinguish between positive, negative, and neutral emotions. The plug-in feature is a special feature of the Command Center that allows administrators to disable or block user actions based on arbitrary diagnostic results. For example, if the analysis finds that there is no offensive or harmful content, the system can automatically take steps to reduce the impact by removing or reducing the scope of that content. Command Center also offers dashboards and reports that provide deep insights into social media trends, user behavior, and sentiment patterns. This helps organizations make decisions, improve community management, and maintain a better online environment.

“HOT” ChatGPT: The Promise of ChatGPT in Detecting and Discriminating Hateful, Offensive, and Toxic Comments on Social Media

Harmful textual content is pervasive on social media, poisoning online communities and negatively impacting participation. A common approach to this issue is developing detection models that rely on human annotations. However, the tasks required to build such models expose annotators to harmful and offensive content and may require significant time and cost to complete. Generative AI models have the potential to understand and detect harmful textual content. We used ChatGPT to investigate this potential and compared its performance with MTurker annotations for three frequently discussed concepts related to harmful textual content on social media: Hateful, Offensive, and Toxic (HOT). We designed five prompts to interact with ChatGPT and conducted four experiments eliciting HOT classifications. Our results show that ChatGPT can achieve an accuracy of approximately 80% when compared to MTurker annotations. Specifically, the model displays a more consistent classification for non-HOT comments than HOT comments compared to human annotations. Our findings also suggest that ChatGPT classifications align with the provided HOT definitions. However, ChatGPT classifies “hateful” and “offensive” as subsets of “toxic.” Moreover, the choice of prompts used to interact with ChatGPT impacts its performance. Based on these insights, our study provides several meaningful implications for employing ChatGPT to detect HOT content, particularly regarding the reliability and consistency of its performance, its understanding and reasoning of the HOT concept, and the impact of prompts on its performance. Overall, our study provides guidance on the potential of using generative AI models for moderating large volumes of user-generated textual content on social media.

ALGORITHM FOR DETECTION OF ABUSIVE CONTENT IN AUDIO CONTENT FOR IMPLEMENTATION IN OBJECT-ORIENTED INFORMATION SYSTEM

The paper proposes the basic principles of developing an object-oriented information system for detecting abusive content in Ukrainian-language audio content based on a new algorithm that uses statistical and neural network approaches to detect abusive content. Detecting offensive content in text and audio content is an urgent task, as it helps to create a safe and healthy environment for communication, especially in online platforms. Offensive content can harm the people who hear or read it and violate their rights. It can also have a negative impact on society, contributing to the spread of hatred and violence. To detect abusive speech in audio content, the proposed approach uses two key components: the use of dictionary methods and the analysis of the emotional tonality of utterances. A set of reviews was used as a dataset to determine the abusive component of the content, which was expanded by the authors by adding words of abuse. An object-oriented information system architecture written in the Python programming language in the PyCharm programming environment is proposed. The information system consists of a software module for training recurrent neural network models and further saving trained instances, and a software module for detecting abusive content in Ukrainian-language audio content using trained RNN models. Since the recurrent neural network is trained on a short text data set, the system is less efficient at identifying texts that have a larger number of words. In the example of the proposed approach, the accuracy of detecting offensive content is more than 90%. This means that the algorithm works correctly in the absence of offending highlights in the test data set. The results of the analysis of the effectiveness of the proposed approach show that in the vast majority of cases the conclusions regarding the acceptability of audio content based on the level of abuse are correct.

Filtering offensive language from multilingual social media contents: A deep learning approach

In the face of uncontrolled offensive content on social media, automated detection emerges as a critical need. This paper tackles this challenge by proposing a novel approach for identifying offensive language in multilingual, code-mixed, and script-mixed settings. The study presents a novel multilingual hybrid dataset constructed by merging diverse monolingual and bilingual resources. Further, we systematically evaluate the impact of input representations (Word2Vec, Global Vectors for Word Representation (or GloVe), Bidirectional Encoder Representations from Transformers (or BERT), and uniform initialization) and deep learning models (Convolutional Neural Network (or CNN), Bidirectional Long Short Term Memory (or Bi-LSTM), Bi-LSTM-Attention, and fine-tuned BERT) on detection accuracy. Our comprehensive experiments on a dataset of 42,560 social media comments from five languages (English, Hindi, German, Tamil, and Malayalam) reveal the superiority of fine-tuned BERT. Notably, it achieves a macro average F1-score of 0.79 for monolingual tasks and an impressive 0.86 for code-mixed and script-mixed tasks. These findings significantly advance offensive language detection methodologies and shed light on the complex dynamics of multilingual social media, paving the way for more inclusive and safer online communities.