Diacritics generation and application in hate speech detection on Vietnamese social networks

Abstract

One of the challenging problems in text processing is diacritics generation where one needs to generate diacritic marks for non-accented text. With an ever increasing amount of informal text without accents such as short text messages, emails or blog posts on social media, a software system which is capable of generating diacritic marks accurately is very useful and necessary in many situations. This paper presents an approach to improve the accuracy of diacritics generation for Vietnamese text. We propose two novel deep learning models which leverage a plausible conceptual representation for the phonetic structure of Vietnamese syllables. Experimental results on real-world datasets show that our models achieve a significant improvement as compared to the state-of-the-art methods for diacritics generation. We also demonstrate that the proposed models can be applied efficiently to improve the accuracy of hate speech detection on Vietnamese social networks.