Training Multilingual and Adversarial Attack-Robust Models for Hate Detection on Social Media

Abstract

Social media provide plenty of textual information in various languages. This information can contain or provoke hatred towards different social or religious groups. In this paper, we study methods to process short text messages in English, Hindi, and Russian and identify such intolerance with cross-lingual Transformer models. Moreover, these models can be easily adapted to analyze other languages. We fine-tuned these models with several training techniques to build accurate hate speech detectors that are robust to adversarial attacks. Additional preprocessing was carried out for all datasets to improve the quality of model training. Also, for one of the training datasets, we applied the text attack algorithm that replaces some words with synonyms. For some languages, such an attack can greatly reduce the quality of the model. Experiment results show that mixing adversarial examples to a training dataset and combining deep models to randomized ensembles allows not only to reduce test error on attacked data for languages from the dataset (Hindi, Russian) but also to achieve better accuracy in other languages.