Code-switched Text Research Articles

A Comparative Study of Transformer-based Models for Hate-Speech Detection in English-Kiswahili Code-Switched Social Media Text

The transformer architecture, first introduced in 2017 by researchers at Google, has revolutionized natural language processing in various tasks, including text classification. This architecture formed the basis of future models such as those used in hate speech detection in code-switched text. In this research, we conduct a comparative study of transformer-based models for hate speech detection in English-Kiswahili code-switched text. First, the models were compared as feature extractors using a traditional classifier and then as end-to-end classifiers. The three multilingual transformer-based models compared include mBERT, mDistilBERT and XLM-RoBERTa, using SVM as the traditional classifier for the extracted features. The HateSpeech_Kenya dataset, sourced from Kaggle, was utilized in this study. As a feature extractor, mBERT’s hidden states trained the highest-performing SVM with an accuracy of 0.5461 and a macro f1 score of 0.40. Among the three models evaluated, XLM-RoBERTa achieved the highest accuracy of 0.6069 and a macro f1 score of 0.49 on a balanced dataset. In contrast, mBERT achieved the highest accuracy of 0.7820 and a macro f1 score of 0.53 on an imbalanced dataset. The comparative study establishes that using transformer-based models as end-to-end classifiers generally performs better than using them as feature extractors with traditional classifiers. This is because directly training the models allows them to learn more task-specific features. Furthermore, the varying performance across balanced and imbalanced datasets highlights the need for careful model selection based on the dataset characteristics and specific task requirements.

Use of prompt-based learning for code-mixed and code-switched text classification

Code-mixing and code-switching (CMCS) are prevalent phenomena observed in social media conversations and various other modes of communication. When developing applications such as sentiment analysers and hate-speech detectors that operate on this social media data, CMCS text poses challenges. Recent studies have demonstrated that prompt-based learning of pre-trained language models outperforms full fine-tuning across various tasks. Despite the growing interest in classifying CMCS text, the effectiveness of prompt-based learning for the task remains unexplored. This paper presents an extensive exploration of prompt-based learning for CMCS text classification and the first comprehensive analysis of the impact of the script on classifying CMCS text. Our study reveals that the performance in classifying CMCS text is significantly influenced by the inclusion of multiple scripts and the intensity of code-mixing. In response, we introduce a novel method, Dynamic+AdapterPrompt, which employs distinct models for each script, integrated with adapters. While DynamicPrompt captures the script-specific representation of the text, AdapterPrompt emphasizes capturing the task-oriented functionality. Our experiments on Sinhala-English, Kannada-English, and Hindi-English datasets for sentiment classification, hate-speech detection, and humour detection tasks show that our method outperforms strong fine-tuning baselines and basic prompting strategies.

A novel socio-pragmatic framework for sentiment analysis in Dravidian–English code-switched texts

The advent of global digital communication has accentuated the complexity of multilingual code-switching, presenting formidable challenges for sentiment analysis by necessitating nuanced interpretation of linguistic and socio-pragmatic cues. This research introduces MultiSwitchNet, a pioneering sentiment analysis model tailored for the intricate dynamics of Dravidian languages mixed with English code-switched texts. MultiSwitchNet leverages the linguistic prowess of XLM-RoBERTa, combined with socio-pragmatic insights through a novel Socio-Pragmatic Embedding Layer (SPEL) utilizing DistilBERT. Evaluated on the Dravidian Code-Mixed Dataset, MultiSwitchNet achieved superior performance metrics, including an accuracy of 72.3%, precision of 72.9%, recall of 72.1%, and an F1-score of 72.5%. Its effectiveness extends across varying degrees of code-switching complexity and demonstrates remarkable consistency in sentiment prediction across semantically similar yet syntactically diverse sentences. Statistical analyses, including ANOVA and post hoc tests, underscore the model’s significant outperformance compared to both baseline and state-of-the-art models, highlighting the indispensability of integrating socio-pragmatic awareness for accurate sentiment analysis. MultiSwitchNet’s innovative approach sets a new benchmark for sentiment analysis in multilingual contexts and opens avenues for future research into models capable of discerning the rich socio-pragmatic fabric of global digital communication.

Share What You Already Know: Cross-Language-Script Transfer and Alignment for Sentiment Detection in Code-Mixed Data

Code-switching entails mixing multiple languages. It is an increasingly occurring phenomenon in social media texts. Usually, code-mixed texts are written in a single script, even though the languages involved have different scripts. Pre-trained multilingual models primarily utilize the data in the native script of the language. In existing studies, the code-switched texts are utilized as they are. However, using the native script for each language can generate better representations of the text owing to the pre-trained knowledge. Therefore, a cross-language-script knowledge sharing architecture utilizing the cross attention and alignment of the representations of text in individual language scripts was proposed in this study. Experimental results on two different datasets containing Nepali-English and Hindi-English code-switched texts, demonstrate the effectiveness of the proposed method. The interpretation of the model using model explainability technique illustrates the sharing of language-specific knowledge between language-specific representations.

AdapterFusion-based multi-task learning for code-mixed and code-switched text classification

Social media text can be classified in different ways, viz sentiment analysis, humour detection, hate speech detection and hope speech detection. Multitask learning (MTL) models built on Large Language Models (LLMs) eliminate the need to build separate models for each of these tasks. However, building MTL models by fully fine-tuning the LLM has limitations such as catastrophic forgetting and requiring complete retraining to add a new task. AdapterFusion was introduced to address these limitations. However, existing AdapterFusion techniques have not been experimented with code-mixed or code-switched text. Moreover, they only considered task-based AdapterFusion on monolingual LLMs. However, using monolingual LLMs is sub-optimal in classifying code-mixed or code-switched text. A better alternative is multilingual LLMs. In this paper, we present an MTL model that combines task AdapterFusion with language adapters on top of a multilingual LLM. We combine language adapters sequentially, in parallel, and as a fusion with task adapters to capture cross-lingual knowledge in code-mixed and code-switched text. We believe that this is the first research to introduce language-based AdapterFusion.

The Analysis of the Sepedi-English Code-switched Radio News Corpus

Code-switching is a phenomenon that occurs mostly in multilingual countries where multilingual speakers often switch between languages in their conversations. The unavailability of large-scale code-switched corpora hampers the development and training of language models for the generation of code-switched text. In this study, we explore the initial phase of collecting and creating Sepedi-English code-switched corpus for generating synthetic news. Radio news and the frequency of code-switching on read news were considered and analysed. We developed and trained a Transformer-based language model using the collected code-switched dataset. We observed that the frequency of code-switched data in the dataset was very low at 1.1 %. We complemented our dataset with the news headlines dataset to create a new dataset. Although the frequency was still low, the model obtained the optimal loss rate of 2,361 with an accuracy of 66 %.

Tran-Switch: A transfer learning approach for sentence level cross-genre author profiling on code-switched English–RomanUrdu Text

Cross-genre author profiling aims to build generalized models for predicting profile traits of authors that can be helpful across different text genres for computer forensics, marketing, and other applications. The cross-genre author profiling task becomes challenging when dealing with low-resourced languages due to the lack of availability of standard corpora and methods. The task becomes even more challenging when the data is code-switched, which is informal and unstructured. In previous studies, the problem of cross-genre author profiling has been mainly explored for mono-lingual texts in highly resourced languages (English, Spanish, etc.). However, it has not been thoroughly explored for the code-switched text which is widely used for communication over social media. To fulfill this gap, we propose a transfer learning-based solution for the cross-genre author profiling task on code-switched (English–RomanUrdu) text using three widely known genres, Facebook comments/posts, Tweets, and SMS messages. In this article, firstly, we experimented with the traditional machine learning, deep learning and pre-trained transfer learning models (MBERT, XLMRoBERTa, ULMFiT, and XLNET) for the same-genre and cross-genre gender identification task. We then propose a novel Trans-Switch approach that focuses on the code-switching nature of the text and trains on specialized language models. In addition, we developed three RomanUrdu to English translated corpora to study the impact of translation on author profiling tasks. The results show that the proposed Trans-Switch model outperforms the baseline deep learning and pre-trained transfer learning models for cross-genre author profiling task on code-switched text. Further, the experimentation also shows that the translation of RomanUrdu text does not improve results.

Adapter-based fine-tuning of pre-trained multilingual language models for code-mixed and code-switched text classification

Code-mixing and code-switching are frequent features in online conversations. Classification of such text is challenging if one of the languages is low-resourced. Fine-tuning pre-trained multilingual language models is a promising avenue for code-mixed text classification. In this paper, we explore adapter-based fine-tuning of PMLMs for CMCS text classification. We introduce sequential and parallel stacking of adapters, continuous fine-tuning of adapters, and training adapters without freezing the original model as novel techniques with respect to single-task CMCS text classification. We also present a newly annotated dataset for the classification of Sinhala–English code-mixed and code-switched text data, where Sinhala is a low-resourced language. Our dataset of 10000 user comments has been manually annotated for five classification tasks: sentiment analysis, humor detection, hate speech detection, language identification, and aspect identification, thus making it the first publicly available Sinhala–English CMCS dataset with the largest number of task annotation types. In addition to this dataset, we also tested our proposed techniques on Kannada–English and Hindi–English datasets. These experiments confirm that our adapter-based PMLM fine-tuning techniques outperform or are on par with the basic fine-tuning of PMLM models.

Code-switched end-to-end Marathi speech recognition for especially abled people

There is no need for eyes or arms in speech communication. Instead, both humans and machines communicate by speaking and listening. Therefore, speech communication not only helps physically disabled but also help people with hearing loss and/or language disorders. It is also used by people with vision or motor disabilities. There are various types of devices that are used for people with hearing loss. They can be classified as audio-visual stimulation devices, tactile devices, and speech processing devices. These devices consider the characteristics of speech and/or deploy automatic speech recognition inside it. This article focuses on such ASR system based on E2E model which can be useful in such devices. This research is carried out on the language used by the families of India’s over 80 million speakers i.e., Marathi. These assessments provide valuable insights into the development of SR models for low-resourced Indian languages. We present E2E ASR model trained with code-switched text of Marathi language and decoded similarly to a normal E2E ASR system. This model achieves a better word error rate of 35.09%, according to experimental results on the code-switched Marathi corpus. We compared our model to Facebook’s wav2vec2 model, which produces a WER of 23.72 percent on code-switched Marathi datasets without any language model and 12.10 percent on open ASR Marathi datasets.

COVID-19 and cyberbullying: deep ensemble model to identify cyberbullying from code-switched languages during the pandemic.

It has been declared by the World Health Organization (WHO) the novel coronavirus a global pandemic due to an exponential spread in COVID-19 in the past months reaching over 100 million cases and resulting in approximately 3 million deaths worldwide. Amid this pandemic, identification of cyberbullying has become a more evolving area of research over posts or comments in social media platforms. In multilingual societies like India, code-switched texts comprise the majority of the Internet. Identifying the online bullying of the code-switched user is bit challenging than monolingual cases. As a first step towards enabling the development of approaches for cyberbullying detection, we developed a new code-switched dataset, collected from Twitter utterances annotated with binary labels. To demonstrate the utility of the proposed dataset, we build different machine learning (Support Vector Machine & Logistic Regression) and deep learning (Multilayer Perceptron, Convolution Neural Network, BiLSTM, BERT) algorithms to detect cyberbullying of English-Hindi (En-Hi) code-switched text. Our proposed model integrates different hand-crafted features and is enriched by sequential and semantic patterns generated by different state-of-the-art deep neural network models. Initial experimental results of the proposed deep ensemble model on our code-switched data reveal that our approach yields state-of-the-art results, i.e., 0.93 in terms of macro-averaged F1 score. The dataset and codes of the present study will be made publicly available on the paper’s companion repository [https://github.com/95sayanta/COVID-19-and-Cyberbullying].

Psychosocial Features for Hate Speech Detection in Code-switched Texts

This study examines the problem of hate speech identification in codeswitched text from social media using a natural language processing approach. It explores different features in training nine models and empirically evaluates their predictiveness in identifying hate speech in a ~50k human-annotated dataset. The study espouses a novel approach to handle this challenge by introducing a hierarchical approach that employs Latent Dirichlet Analysis to generate topic models that help build a high-level Psychosocial feature set that we acronym PDC. PDC groups similar meaning words in word families, which is significant in capturing codeswitching during the preprocessing stage for supervised learning models. The high-level PDC features generated are based on a hate speech annotation framework [1] that is largely informed by the duplex theory of hate [2]. Results obtained from frequency-based models using the PDC feature on the dataset comprising of tweets generated during the 2012 and 2017 presidential elections in Kenya indicate an f-score of 83% (precision: 81%, recall: 85%) in identifying hate speech. The study is significant in that it publicly shares a unique codeswitched dataset for hate speech that is valuable for comparative studies. Secondly, it provides a methodology for building a novel PDC feature set to identify nuanced forms of hate speech, camouflaged in codeswitched data, which conventional methods could not adequately identify.

Leveraging bilingual-view parallel translation for code-switched emotion detection with adversarial dual-channel encoder

Code-switched emotion detection, a task analyzing the emotion in code-switched texts, has gain increasing research attention within recent years. Prior works utilize various neural models with sophisticated features to pursuit high performance of the task, while they still overlook some crucial characteristics of the code-switched texts. In this work, we present an innovative approach for improving code-switched emotion detection. We first consider a bilingual-view parallel translation for code-switched text enhancement, i.e., translating the code-switched texts into two languages. Then we propose an adversarial dual-channel encoder architecture, where two private encoders take as inputs the parallel texts in two languages, respectively. The private encoders and the shared encoder work collaboratively, and effectively retrieve the features from monolingual and bilingual perspectives under adversarial training. We conduct extensive experiments on five code-switched benchmark datasets. Results show that our model outperforms the strongly-performing baselines that leverage external code-switched or bilingual word embedding with over 1.5% F1 score on the Chinese–English, Spanish–English and Hindi–English code-mixed data, becoming the new state-of-the-art system. Further analyses including ablation, qualitative and error studies, demonstrate the effectiveness of our proposed encoder for code-switched texts, as well as the bilingual-view parallel translation strategy.

Novel textual features for language modeling of intra-sentential code-switching data

Code-switching refers to the frequent use of non-native language words/phrases by speakers while conversating in their native languages. Traditionally, for training a language model (LM) for code-switching data, one is required to tediously collect a large amount of text corpus in the respective code-switching domain. Alternately, we recently proposed a more viable approach that adapts an existing native LM to handle the code-switching data. In this work, we present our efforts for language modeling of code-switching data following both the traditional and the proposed approaches. The salient contributions of this paper includes: (i) creation of the Hindi-English code-switching text corpus, (ii) an improved parts-of-speech (POS) labeling scheme for accurate tagging of non-native words embedded in the code-switching data, and (iii) the proposal of a novel textual feature referred to as the code-switching location (CSL) feature, that allows LMs to predict the code-switching instances. The evaluation of the proposed features has been done on two code-switching datasets: Hindi-English and Mandarin-English. On experimental evaluation, a substantial reduction in the perplexity is achieved with the use of the improvised POS features. It is also observed that the proposed CSL features provide an independent and additive improvement over the POS features in terms of perplexity.

Machine Normalization

User-generated text in social media communication (SMC) is mainly characterized by non-standard form. It may contain code switching (CS) text, a widespread phenomenon in SMC, in addition to noisy elements used, especially in written conversations (use of abbreviations, symbols, emoticons) or misspelled words. All of these factors constitute a wall in front of text mining applications. Common text mining tools are dedicated to standard use of standard languages but cannot deal with other forms, especially written text in social media. To overcome these problems, in this work we present our solution for the normalization of non-standard use of standard and non-standard languages (dialects) in SMC text with the use of existent resources and tools. The main processing in our solution consists of CS normalization from multiple to one language by the use of a machine translation--like approach. This processing relies on a linguistic approach of CS, which aims at identifying automatically the translation source and target languages (without human intervention). The remaining processing operations concern the normalization of SMC special expressions and spelling correction of out-of-vocabulary words. To preserve the coded-switched sentence meaning across translation, we adopt a knowledge-based approach for word sense translation disambiguation reinforced with a multi-lingual vertical context. All of these processes are embedded in what we refer to as the machine normalization system. Our solution can be used as a front-end of text mining processing, enabling the analysis of SMC noisy text. The conducted experiments show that our system performs better than considered baselines.

Fine-Tuning BERT for Multi-Label Sentiment Analysis in Unbalanced Code-Switching Text

Previous research on sentiment analysis mainly focuses on binary or ternary sentiment analysis in monolingual texts. However, in today's social media such as micro-blogs, emotions are often expressed in bilingual or multilingual text called code-switching text, and people's emotions are complex, including happiness, sadness, angry, afraid, surprise, etc. Different emotions may exist together, and the proportion of each emotion in the code-switching text is often unbalanced. Inspired by the recently proposed BERT model, we investigate how to fine-tune BERT for multi-label sentiment analysis in code-switching text in this paper. Our investigation includes the selection of pre-trained models and the fine-tuning methods of BERT on this task. To deal with the problem of the unbalanced distribution of emotions, a method based on data augmentation, undersampling and ensemble learning is proposed to get balanced samples and train different multi-label BERT classifiers. Our model combines the prediction of each classifier to get the final outputs. The experiment on the dataset of NLPCC 2018 shared task 1 shows the effectiveness of our model for the unbalanced code-switching text. The F1-Score of our model is higher than many previous models.

Predicting the emergence of content words in L2 diary entries during study abroad over a year

This case study examined a diary written by a Japanese senior high school student while he was studying abroad in the US for a year. His writings exhibited two types of texts throughout the year: code-mixed texts, in which a mixture of Japanese and English is used, and code-switched texts, in which only English is used, in a single sentence. The diary was examined to identify to what extent the emergence of content words under spontaneous written production (n = 1,716) could be predicted by their word frequency, degree of cognate similarity, and word length while comparing the two types of texts. According to the results of multiple regressions, word frequency was found to be a significant predictor for both texts, indicative of its robust effects. The effects of the degree of cognate similarity appeared only for code-mixed texts. The effects of word length were, if any, limited. Furthermore, code-mixed texts were found to yield longer words with lower word frequencies and higher degrees of cognate similarity than code-switched texts. Considering that the results here may be influenced by the duration of study abroad, this study reconfirms the significance of tracking the micro-development of vocabulary for second language learners.

GIRNet: Interleaved Multi-Task Recurrent State Sequence Models

In several natural language tasks, labeled sequences are available in separate domains (say, languages), but the goal is to label sequences with mixed domain (such as code-switched text). Or, we may have available models for labeling whole passages (say, with sentiments), which we would like to exploit toward better position-specific label inference (say, target-dependent sentiment annotation). A key characteristic shared across such tasks is that different positions in a primary instance can benefit from different ‘experts’ trained from auxiliary data, but labeled primary instances are scarce, and labeling the best expert for each position entails unacceptable cognitive burden. We propose GIRNet, a unified position-sensitive multi-task recurrent neural network (RNN) architecture for such applications. Auxiliary and primary tasks need not share training instances. Auxiliary RNNs are trained over auxiliary instances. A primary instance is also submitted to each auxiliary RNN, but their state sequences are gated and merged into a novel composite state sequence tailored to the primary inference task. Our approach is in sharp contrast to recent multi-task networks like the crossstitch and sluice networks, which do not control state transfer at such fine granularity. We demonstrate the superiority of GIRNet using three applications: sentiment classification of code-switched passages, part-of-speech tagging of codeswitched text, and target position-sensitive annotation of sentiment in monolingual passages. In all cases, we establish new state-of-the-art performance beyond recent competitive baselines.

IITG-HingCoS corpus: A Hinglish code-switching database for automatic speech recognition

Code-switching is a phenomenon in linguistics which refers to the use of two or more languages, especially within the same discourse. This phenomenon has been observed in many multilingual communities across the globe. In the recent past, there have been increasing demand for automatic speech recognition (ASR) systems to deal with code-switching. However, for training such systems, very limited code-switching resources are available as yet. Thus, the development of code-switching resources is highly desirable. In this work, we describe the collection of a Hinglish (Hindi-English) code-switching database at the Indian Institute of Technology Guwahati (IITG) which is referred to as the IITG-HingCoS corpus. This corpus consists of code-switching text data having 25,988 sentences with a total of 0.58 million words. In addition to that, the corpus also contains 25 h of matching speech data corresponding to 9251 code-switching sentences covering a vocabulary of 6542 words. This paper elaborates the sources and the protocol used for collecting the corpus. The baseline experimental results on the collected corpus for language modeling and ASR tasks are also presented.

Emotion Analysis in Code-Switching Text With Joint Factor Graph Model

Previous research on emotions analysis has placed much emphasis in monolingual instead of bilingual text. However, emotions on social media platforms are often found in bilingual or code-switching posts. Different from monolingual text, emotions in code-switching text can be expressed in both monolingual and bilingual forms. Moreover, more than one emotion can be expressed within a single post; yet they tend to be related in some ways which offers some implications. It is thus necessary to consider the correlation between different emotions. In this paper, a joint factor graph model is proposed to address this issue. In particular, attribute functions of the factor graph model are utilized to learn both monolingual and bilingual information from each post, factor functions are used to explore the relationship among different emotions, and a belief propagation algorithm is employed to learn and predict the model. Empirical studies demonstrate the importance of emotion analysis in code-switching text and the effectiveness of our proposed joint learning model.

Syntactic and Semantic Features For Code-Switching Factored Language Models

This paper presents our latest investigations on different features for factored language models for Code-Switching speech and their effect on automatic speech recognition (ASR) performance. We focus on syntactic and semantic features which can be extracted from Code-Switching text data and integrate them into factored language models. Different possible factors, such as words, part-of-speech tags, Brown word clusters, open class words and clusters of open class word embeddings are explored. The experimental results reveal that Brown word clusters, part-of-speech tags and open-class words are the most effective at reducing the perplexity of factored language models on the Mandarin-English Code-Switching corpus SEAME. In ASR experiments, the model containing Brown word clusters and part-of-speech tags and the model also including clusters of open class word embeddings yield the best mixed error rate results. In summary, the best language model can significantly reduce the perplexity on the SEAME evaluation set by up to 10.8% relative and the mixed error rate by up to 3.4% relative.