Code-mixed Text Research Articles

Hatred and trolling detection transliteration framework using hierarchical LSTM in code-mixed social media text

The paper describes the usage of self-learning Hierarchical LSTM technique for classifying hatred and trolling contents in social media code-mixed data. The Hierarchical LSTM-based learning is a novel learning architecture inspired from the neural learning models. The proposed HLSTM model is trained to identify the hatred and trolling words available in social media contents. The proposed HLSTM systems model is equipped with self-learning and predicting mechanism for annotating hatred words in transliteration domain. The Hindi–English data are ordered into Hindi, English, and hatred labels for classification. The mechanism of word embedding and character-embedding features are used here for word representation in the sentence to detect hatred words. The method developed based on HLSTM model helps in recognizing the hatred word context by mining the intention of the user for using that word in the sentence. Wide experiments suggests that the HLSTM-based classification model gives the accuracy of 97.49% when evaluated against the standard parameters like BLSTM, CRF, LR, SVM, Random Forest and Decision Tree models especially when there are some hatred and trolling words in the social media data.

Characterization and mechanical properties of offensive language taxonomy and detection techniques

Machine learning is the sub field of Artificial Intelligence which is responsible for developing a system that has the capability to learn and improve from past experience. There is an increasing demand for the offensive language detection because there is a large code-mixed text. Sometimes there is a possibility of code-mixed texts which are written using native scripts. If the system is getting trained on the data which is created using single language then it will leads to failure because of complexity. Using online communities and social media platforms has become increasing now. Recently many of the researchers have been investigating different ways of identifying the abusive content and developing systems to detect its different types like aggression identification, cyber bullying detection, hate speech identification, offensive language identification and toxic comments identification. Developing mechanisms to detect this is a challenging task. In the last few years, machine learning techniques and deep learning concept has been playing a significant role in the development of solutions for offensive language detection. So, considerable efforts have also been devoted to the development of machine learning techniques and deep learning models which is used to identify toxic content in recent years. First, this survey provides offensive language taxonomy and detection approaches. Then, the article focuses on the offensive language identification and toxic comments identification approaches. This article reviews various models used, type of learning model, languages for which the developed model is applied, other methods or approaches that were compared against this model, dataset used, offensive language detection type and performance metrics. The findings from the review are as well summarized and they urge that further effort is required to get improvement in the current state-of-the art approaches. This article also focuses on future work to improve the offensive language classification and toxic comments identification.

Zera-Shot Sentiment Analysis for Code-Mixed Data

Code-mixing is the practice of alternating between two or more languages. A major part of sentiment analysis research has been monolingual and they perform poorly on the code-mixed text. We introduce methods that use multilingual and cross-lingual embeddings to transfer knowledge from monolingual text to code-mixed text for code-mixed sentiment analysis. Our methods handle code-mixed text through zero-shot learning and beat state-of-the-art English-Spanish code-mixed sentiment analysis by an absolute 3% F1-score. We are able to achieve 0.58 F1-score (without a parallel corpus) and 0.62 F1-score (with the parallel corpus) on the same benchmark in a zero-shot way as compared to 0.68 F1-score in supervised settings. Our code is publicly available on github.com/sedflix/unsacmt.

A Pun Identification Framework for Retrieving Equivocation Terms based on HLSTM Learning Model

The artificial intelligence system is made linguistically intelligent through HLSTM model to identify pun expressions from code mixed text. The text available on social media domain is written in mixed script format and from these content puns word identification is a challenging task in this scenario. The retrieval of pun and its corresponding equivocation terms is very hard to retrieve from the transliterated text. The pun retrieval and its equivocation representation are widely used to present the opinions over the network applications. The work described in the paper gives the comparative view of different learning models and techniques applied in the area of transliteration for pun word retrieval. The rule framed approach is presented which accepts the roman form text as input and as per the defined rules the system is developed to give the equivocation words available in the sentence. The evaluation measures used here to validate the hypothesis is based on statistical measures along with HLSTM learning model. Further the result is validated using the voting technique that can choose appropriate equivocation label which are not identifies by the learning model. The use of voting technique here is to provide an extra edge when the proposed approaches suggest incorrect tag against the pun word. The voting approach enhances the overall result accuracy with high precision value.

Neural Network Pun Material Identifiction Framework based on Artificial Intelligence Learning

The Neural based artificial intelligence system is made linguistically intelligent through simulation model to identify pun material expressions from code mixed text. The text available on social media consist contents which are written in mixed script format and from these content puns word identification is a challenging task in this scenario. The retrieval of pun and its corresponding equivocation terms is very hard to retrieve from the transliterated text. The pun retrieval and its equivocation representation are widely used to present the opinions over the network applications. The work described in the paper gives the comparative view of different neural network and its corresponding learning and simulation techniques applied in the area of transliteration. The rule framed approach is presented which accepts the roman form text as input and as per the defined rules the system is developed to give the equivocation words available in the sentence. The evaluation measures used here to validate the hypothesis is based on statistical measures along with HLSTM learning model. Further the result is validated using the voting technique that can choose appropriate equivocation label which are not identifies by the learning model. The use of voting technique here is to provide an extra edge when the proposed approaches suggest incorrect tag against the pun word. The voting approach enhances the overall result accuracy with high precision value.

Named Entity Recognition for Code Mixed Social Media Sentences

With the increase of internet applications and social media platforms there has been an increase in the informal way of text communication. People belonging to different regions tend to mix their regional language with English on social media text. This has been the trend with many multilingual nations now and is commonly known as code mixing. In code mixing, multiple languages are used within a statement. The problem of named entity recognition (NER) is a well-researched topic in natural language processing (NLP), but the present NER systems tend to perform inefficiently on code-mixed text. This paper proposes three approaches to improve named entity recognizers for handling code-mixing. The first approach is based on machine learning techniques such as support vector machines and other tree-based classifiers. The second approach is based on neural networks and the third approach uses long short-term memory (LSTM) architecture to solve the problem.

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Usage of code-mixed text has increased in re-cent years among Indonesian internet users, who often mix Indonesian-language with English-language text. Normalisation of this code-mixed text into Indonesian needs to be performed to capture the meaning of English parts of the text and process them effectively. We improve a state-of-the-art code-mixed Indonesian-English normalisation system by modifying its pipeline modules. We further analyse the effect of code-mixed normalisation on emotion classification tasks. Our approach significantly improved on a state-of-the-art Indonesian-English code-mixed text normal-isation system in both the individual pipeline modules and the overall system. The new feature set in the language identification module showed an improvement of 4.26% in terms of F1 score. The combination of machine translation and ruleset in the lexical normalisation module improved BLEU score by 25.22% and lowered WER by 62.49%. The use of context in the translation module improved BLEU score by 2.5% and lowered WER by 8.84%. The effectiveness of the overall pipeline normalisation system increased by 32.11% and 33.82%, in terms of BLEU score and WER, respectively. Code-mixed normalisation also improved the accuracy of emotion classification by up to 37.74% in terms of F1 score.

Word Level Language Identification on Code-Mixed English-Bodo Text

Since social media has become an active part of one’s life, people express their views freely in mixed informal languages on such platforms. So, in a multi-lingual country like India, it becomes really difficult for conventional language detectors to identify such languages. This paper mainly aims to detect the language at word level where the code mixed text can be in English-Bodo-Assamese. The data for the same is collected from some related Facebook pages and various classification algorithms are used to predict and compare the accuracy with which the detection is done.

Transformer Based Language Identification for Malayalam-English Code-Mixed Text

Social media users have the proclivity to write majority of the data for under resourced languages in code-mixed format. Code-mixing is defined as mixing of two or more languages in a single sentence. Research in code-mixed text helps apprehend security threats, prevalent on social media platforms. In such instances, language identification is an imperative task of code-mixed text. The focus of this paper is to carry out a word-level language identification (WLLI) of Malayalam-English code-mixed data, from social media platforms like YouTube. This study was centered around BERT, a transformer model, along with its variants - CamemBERT, DistilBERT - for intuitive perception of the language at the word-level. The propounded approach entails tagging Malayalam-English code-mixed data set with six labels: Malayalam (mal), English (eng), acronyms (acr), universal (univ), mixed (mix) and undefined (undef). Newly developed corpus of Malayalam-English was deployed for appraisal of the effectiveness of state-of-the-art models like BERT. Evaluation of the proffered approach, accomplished with other code-mixed language such as Hindi-English, notched a 9% increase in the F1-score.

Distributional Word Representations for Code-mixed Text in Moroccan Darija

In this paper, we build different word embedding representations for code-mixed Moroccan Darija-French-English text. We begin by collecting a large corpus based on social media posts and song lyrics. We then experiment with four algorithms: namely, Word2Vec using both Continuous Bag of Words (CBOW) and SkipGram flavors, FastText-CBOW and FastText-SkipGram. Finally, we evaluate these models on two tasks: Word Analogies and Word-Level Language Detection. The use of character n-grams in the FastText models allowed them to perform much better than their word2vec counterparts on the word analogy task, while Word2Vec-SkipGram performed best on the language identification. To the knowledge of the authors, our embeddings are the first set of distributed representations for Arabizi Moroccan Darija text. In the context of this work, we also managed to build the first ever code-switched English – French - Moroccan Darija word analogy dataset and the largest Arabizi code-switched English – French - Moroccan Darija.

An Effective Bi-LSTM Word Embedding System for Analysis and Identification of Language in Code-Mixed social Media Text in English and Roman Hindi

The paper describes the application of the code mixed index in Indian social media texts and comparing the complexity to identify language at word level using BLSTM neural model. In Natural Language Processing one of the imperative and relatively less mature areas is a transliteration. During transliteration, issues like language identification, script specification, missing sounds arise in code mixed data. Social media platforms are now widely used by people to express their opinion or interest. The language used by the users in social media nowadays is Code-mixed text, i.e., mixing of two or more languages. In code-mixed data, one language will be written using another language script. So to process such code-mixed text, identification of language used in each word is important for language processing. The major contribution of the work is to propose a technique for identifying the language of Hindi-English code-mixed data used in three social media platforms namely, Facebook, Twitter, and WhatsApp. We propose a deep learning framework based on cBoW and Skip gram model for language identification in code mixed data. Popular word embedding features were used for the representation of each word. Many researches have been recently done in the field of language identification, but word level language identification in the transliterated environment is a current research issue in code mixed data. We have implemented a deep learning model based on BLSTM that predicts the origin of the word from language perspective in the sequence based on the specific words that have come before it in the sequence. The multichannel neural networks combining CNN and BLSTM for word level language identification of code-mixed data where English and Hindi roman transliteration has been used. Combining this with a cBoW and Skip gram for evaluation. The proposed system BLSTM context capture module gives better accuracy for word embedding model as compared to character embedding evaluated on our two testing sets. The problem is modeled collectively with the deep-learning design. We tend to gift an in-depth empirical analysis of the proposed methodology against standard approaches for language identification.

Annotated corpus creation for sentiment analysis in code-mixed Hindi-English (Hinglish) social network data

Background: Evaluating the sentiments of tweets, blogs, comments and posts have become a crucial part of many applications. Sentiment analysis of social network data is very helpful for decision-making in application areas like movie reviews, product feedback and impact of the speech of a politician etc. The users often comment in their native languages or in slang languages or more often they use abbreviations and do not even stick to grammatical rules of the language. The bilingual and multilingual community often mixes two or more than two languages in their comments. Unavailability of annotated code-mixed data for native language adds to the difficulty in performing sentiment analysis. Objectives: The motive of this article is to present the process of creating annotated corpus for code mixed social media text in Hindi, English and Hinglish which is collected from twitter. Ambiguous meaning words and inconsistent spellings of both the languages have also been included in the study to provide wide spread canvas. Method: This study will provide significant elements that should be considered while developing the annotated corpus of Hindi, Hinglish & English dataset. The annotation is calculated on the basis of polarity of words in three categories as positive, negative and neutral. There are words which have mixed feeling i.e. these words have positive as well as negative sentiments. To consider these words, inner agreement among the polarities has been considered. The words used for sarcasm or slangs have also been taken into account. The study has included ambiguous meaning and inconsistent spelling words of both languages as well. Findings: The proposed work provides a standard annotated corpus for code switched social media text in Hindi-English (Hinglish). The process of developing the corpus and calculating the polarity has been shown. It is found that if one considers the code-mixed text, the accuracy can be enhanced. Application: The proposed corpus can be utilized in the area of market analysis, customer behavior, polling analysis, brand monitoring, etc. The corpus serves as dataset which can further be extended according to the problem definition.Keywords: Machine learning; sentiment analysis; data preprocessing; data cleaning; code-switch; linguistic-switching; multilingual

Roman to Gurmukhi Social Media Text Normalization

PurposeNormalization is an important step in all the natural language processing applications that are handling social media text. The text from social media poses a different kind of problems that are not present in regular text. Recently, a considerable amount of work has been done in this direction, but mostly in the English language. People who do not speak English code mixed the text with their native language and posted text on social media using the Roman script. This kind of text further aggravates the problem of normalizing. This paper aims to discuss the concept of normalization with respect to code-mixed social media text, and a model has been proposed to normalize such text.Design/methodology/approachThe system is divided into two phases – candidate generation and most probable sentence selection. Candidate generation task is treated as machine translation task where the Roman text is treated as source language and Gurmukhi text is treated as the target language. Character-based translation system has been proposed to generate candidate tokens. Once candidates are generated, the second phase uses the beam search method for selecting the most probable sentence based on hidden Markov model.FindingsCharacter error rate (CER) and bilingual evaluation understudy (BLEU) score are reported. The proposed system has been compared with Akhar software and RB\_R2G system, which are also capable of transliterating Roman text to Gurmukhi. The performance of the system outperforms Akhar software. The CER and BLEU scores are 0.268121 and 0.6807939, respectively, for ill-formed text.Research limitations/implicationsIt was observed that the system produces dialectical variations of a word or the word with minor errors like diacritic missing. Spell checker can improve the output of the system by correcting these minor errors. Extensive experimentation is needed for optimizing language identifier, which will further help in improving the output. The language model also seeks further exploration. Inclusion of wider context, particularly from social media text, is an important area that deserves further investigation.Practical implicationsThe practical implications of this study are: (1) development of parallel dataset containing Roman and Gurmukhi text; (2) development of dataset annotated with language tag; (3) development of the normalizing system, which is first of its kind and proposes translation based solution for normalizing noisy social media text from Roman to Gurmukhi. It can be extended for any pair of scripts. (4) The proposed system can be used for better analysis of social media text. Theoretically, our study helps in better understanding of text normalization in social media context and opens the doors for further research in multilingual social media text normalization.Originality/valueExisting research work focus on normalizing monolingual text. This study contributes towards the development of a normalization system for multilingual text.

Deep Learning Based Sentiment Analysis in a Code-Mixed English-Hindi and English-Bengali Social Media Corpus

Sentiment analysis is a circumstantial analysis of text, identifying the social sentiment to better understand the source material. The article addresses sentiment analysis of an English-Hindi and English-Bengali code-mixed textual corpus collected from social media. Code-mixing is an amalgamation of multiple languages, which previously mainly was associated with spoken language. However, social media users also deploy it to communicate in ways that tend to be somewhat casual. The coarse nature of social media text poses challenges for many language processing applications. Here, the focus is on the low predictive nature of traditional machine learners when compared to Deep Learning counterparts, including the contextual language representation model BERT (Bidirectional Encoder Representations from Transformers), on the task of extracting user sentiment from code-mixed texts. Three deep learners (a BiLSTM CNN, a Double BiLSTM and an Attention-based model) attained accuracy 20–60% greater than traditional approaches on code-mixed data, and were for comparison also tested on monolingual English data.

Improving Code-mixed POS Tagging Using Code-mixed Embeddings

Social media data has become invaluable component of business analytics. A multitude of nuances of social media text make the job of conventional text analytical tools difficult. Code-mixing of text is a phenomenon prevalent among social media users, wherein words used are borrowed from multiple languages, though written in the commonly understood roman script. All the existing supervised learning methods for tasks such as Parts Of Speech (POS) tagging for code-mixed social media (CMSM) text typically depend on a large amount of training data. Preparation of such large training data is resource-intensive, requiring expertise in multiple languages. Though the preparation of small dataset is possible, the out of vocabulary (OOV) words pose major difficulty, while learning models from CMSM text as the number of different ways of writing non-native words in roman script is huge. POS tagging for code-mixed text is non-trivial, as tagging should deal with syntactic rules of multiple languages. The important research question addressed by this article is whether abundantly available unlabeled data can help in resolving the difficulties posed by code-mixed text for POS tagging. We develop an approach for scraping and building word embeddings for code-mixed text illustrating it for Bengali-English, Hindi-English, and Telugu-English code-mixing scenarios. We used a hierarchical deep recurrent neural network with linear-chain CRF layer on top of it to improve the performance of POS tagging in CMSM text by capturing contextual word features and character-sequence–based information. We prepared a labeled resource for POS tagging of CMSM text by correcting 19% of labels from an existing resource. A detailed analysis of the performance of our approach with varying levels of code-mixing is provided. The results indicate that the F1-score of our approach with custom embeddings is better than the CRF-based baseline by 5.81%, 5.69%, and 6.3% in Bengali, Hindi , and Telugu languages, respectively.

Language identification framework in code-mixed social media text based on quantum LSTM — the word belongs to which language?

Machine learning (ML) architectures based on neural model have garnered considerable attention in the field of language classification. Code-mixing is a common phenomenon on social networking sites for exhibiting opinion on a topic. The code-mixed text is the approach of mixing two or more languages. This paper describes the application of the code-mixed index in Indian social media texts and compares the complexity to identify language at the word level using Bi-directional Long Short-Term Memory model. The major contribution of the work is to propose a technique for identifying the language of Hindi–English code-mixed data used in three social media platforms namely, Facebook, Twitter and WhatsApp. Here, we demonstrate that a special class of quantum LSTM network model is capable of learning and accurately predicting the languages used in social media texts. Our work paves the way for future applications of machine learning methods in quantum dynamics without relying on the explicit form of the Hamiltonian.

Current State of Hinglish Text Sentiment Analysis

The paper reviews about “sentiment analysis of Hinglish text”. Sentiment analysis is one of the important areas in the modern technical world. Research related to extracting sentiments, emotions from real world data comes under sentiment analysis, an extension of text mining. Sentiment analysis of text can be useful for various decision-making processes. However, social media comments do not follow strict grammatical rules, and they do a mixture of many languages, often written in non-original scripts. In India, lots of people use Hinglish (combination of Hindi and English) as their colloquial language in writings. Aspect based Sentiment analysis from Hinglish text is a challenging problem in the field of artificial intelligence. In this paper, the recent developments on sentiment analysis from English text as well as code mixed text and different problems related to the same have been presented. The main challenge of the Hinglish text model is the classification of aspect based sentiment .So to choose an appropriate classification model is vital. Different types of features of emotion text data and extraction techniques concerned with them are described in this paper along with the previous work review. The validity of various classification procedures was also analyzed. The analysis was also performed on text and textual properties of different ML techniques for real word analysis.

Detection of Hate Speech Text in Hindi-English Code-mixed Data

Abstract Social media sites like Twitter, Facebook, being user-friendly and a free source, provide opportunities to people to air their voice. People, irrespective of the age group, use these sites to share every moment of their life making these sites flooded with data. Apart from these commendable features, these sites have down side as well. Due to lack of restrictions set by these sites for its users to express their views as they like, anybody can make adverse and unrealistic comments in abusive language against anybody with an ulterior motive to tarnish one’s image and status in the society. So it became a huge responsibility for the Government and these sites to identify this hate content before it disseminates to mass. Automatic hate speech detection faces quite a lot of challenges due to the non-standard variations in spelling and grammar. Especially for a country like India with huge multilingual and bilingual population, this hate content would be in code-mixed form which makes the task demanding. So our paper projects a machine learning model to detect hate speech in Hindi-English code-mixed social media text. The methodology makes use of Facebook’s pre-trained word embedding library, fastText to represent 10000 data samples collected from different sources as hate and non-hate. The performance of the proposed methodology is compared with word2vec and doc2vec features and it is observed that fastText features gave better feature representation with Support Vector Machine (SVM)-Radial Basis Funcrion (RBF) classifier. The paper also provides an insight to the researchers working in the field of code-mixed data that character level features provide best result for code-mixed data.

Emotion Detection in Hinglish(Hindi+English) Code-Mixed Social Media Text

Abstract Human communication is often embedded with emotion and it can be expressed via different mediums like vocal interaction, texts, non-verbal communication like facial expressions and gestures. Even though textual communication is a more common way of interaction the rapid utilization of social media has taken it to another level. Social media open an easy way for people to express their emotions. So people around the world utilize this opportunity and express themselves in social media platforms through texts over various subjects. People make use of these platforms to exhibit their like or dislike towards something, how they felt about a situation, their reaction to a government decision and so on. Hence, understanding the emotion expressed in such social media texts has a significant number of applications emphasizing the need to detect it. The human brain is quite intelligent to sense such kind of emotion associated with a text but for a machine to gain such perception is quite difficult. In Natural Language Processing, emotion recognition and classification is a commonly researched task where a model can detect these type of emotions. It is quite challenging when it comes to Indian Languages due to the lack of data, as well as being a multilingual society people tend to use code-mixed pattern in social media. The lack of annotated corpus in the Hindi-English code-mixed domain and unavailability of the standard model to classify, left this area of research still an exploring region. In this paper, to analyze such data, we created a dataset of 12000 Hindi-English code-mixed texts collected from various sources and annotated them with emotions Happy, Sad and Anger. In our work, a pretrained bilingual model is used to generate feature vectors and deep neural networks are employed as classification models. It is observed that with the selected features, CNN-BiLSTM gave better performance compared to other experimented models with 83.21% classification accuracy.

An effective cybernated word embedding system for analysis and language identification in code-mixed social media text

An effective cybernated word embedding system for analysis and language identification in code-mixed social media text