Monolingual Models Research Articles

Robust cross-lingual knowledge base question answering via knowledge distillation

PurposePrevious knowledge base question answering (KBQA) models only consider the monolingual scenario and cannot be directly extended to the cross-lingual scenario, in which the language of questions and that of knowledge base (KB) are different. Although a machine translation (MT) model can bridge the gap through translating questions to the language of KB, the noises of translated questions could accumulate and further sharply impair the final performance. Therefore, the authors propose a method to improve the robustness of KBQA models in the cross-lingual scenario.Design/methodology/approachThe authors propose a knowledge distillation-based robustness enhancement (KDRE) method. Specifically, first a monolingual model (teacher) is trained by ground truth (GT) data. Then to imitate the practical noises, a noise-generating model is designed to inject two types of noise into questions: general noise and translation-aware noise. Finally, the noisy questions are input into the student model. Meanwhile, the student model is jointly trained by GT data and distilled data, which are derived from the teacher when feeding GT questions.FindingsThe experimental results demonstrate that KDRE can improve the performance of models in the cross-lingual scenario. The performance of each module in KBQA model is improved by KDRE. The knowledge distillation (KD) and noise-generating model in the method can complementarily boost the robustness of models.Originality/valueThe authors first extend KBQA models from monolingual to cross-lingual scenario. Also, the authors first implement KD for KBQA to develop robust cross-lingual models.

A word embedding-based approach to cross-lingual topic modeling

The cross-lingual topic analysis aims at extracting latent topics from corpora of different languages. Early approaches rely on high-cost multilingual resources (e.g., a parallel corpus), which is hard to come by in many real cases. Some works only require a translation dictionary as a linkage between languages; however, when given an inappropriate dictionary (e.g., small coverage of dictionary), the cross-lingual topic model would shrink to a monolingual topic model and generate less diversified topics. Therefore, it is imperative to investigate a cross-lingual topic model requiring fewer bilingual resources. Recently, some space-mapping techniques have been proposed to help align multiple word embedding of different languages into a quality cross-lingual word embedding by referring to a small number of translation pairs. This work proposes a cross-lingual topic model, called Cb-CLTM, which incorporates with cross-lingual word embedding. To leverage the power of word semantics and the linkage between languages from the cross-lingual word embedding, the Cb-CLTM considers each word as a continuous embedding vector rather than a discrete word type. The experiments demonstrate that, when cross-lingual word space exhibits strong isomorphism, Cb-CLTM can generate more coherent topics with higher diversity and induce better representations of documents across languages for further tasks such as cross-lingual document clustering and classification. When the cross-lingual word space is less isomorphic, Cb-CLTM generates less coherent topics yet still prevails in topic diversity and document classification.

A Toxic Comment Classification Model Based on Ensemble

Accurate classification of toxic comments in different languages is an important task in today’s international social networking platforms. In order to improve the classification of toxic comments in different languages, this paper combines the advantages of high accuracy of monolingual model and strong generalization ability of multi-language model, and adopts the ensemble of multilingual model and monolingual model to classify toxic comments. For the monolingual model, the monolingual pre-training model is fine-tuned with labeled task data; for the multilingual model, before fine-tuning the model with labeled task data, a further pre-training is applied to train model on unlabeled data, which aims to make full use of the large amount of unlabeled data and reduce the dependency of amount of labeled comment data while improving the classification effect. Comparative experiments on Conversasion AI’s multilingual toxic comment dataset show that the model in this paper has improved results on different evaluation metrics compared to the XLM-RoBERTa multilingual fine-tuning model, illustrating the validity of the model.

ParsNER-Social: A Corpus for Named Entity Recognition in Persian Social Media Texts

Named Entity Recognition (NER) is one of the essential prerequisites for many natural language processing tasks. All public corpora for Persian named entity recognition, such as ParsNERCorp and ArmanPersoNERCorpus, are based on the Bijankhan corpus, which is originated from the Hamshahri newspaper in 2004. Correspondingly, most of the published named entity recognition models in Persian are specially tuned for the news data and are not flexible enough to be applied in different text categories, such as social media texts. This study introduces ParsNER-Social, a corpus for training named entity recognition models in the Persian language built from social media sources. This corpus consists of 205,373 tokens and their NER tags, crawled from social media contents, including 10 Telegram channels in 10 different categories. Furthermore, three supervised methods are introduced and trained based on the ParsNER-Social corpus: Two conditional random field models as baseline models and one state-of-the-art deep learning model with six different configurations are evaluated on the proposed dataset. The experiments show that the Mono-Lingual Persian models based on Bidirectional Encoder Representations from Transformers (MLBERT) outperform the other approaches on the ParsNER-Social corpus. Among different Configurations of MLBERT models, the ParsBERT+BERT-TokenClass model obtained an F1-score of 89.65%.

Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

Language model pretraining is an effective method for improving the performance of downstream natural language processing tasks. Even though language modeling is unsupervised and thus collecting data for it is relatively less expensive, it is still a challenging process for languages with limited resources. This results in great technological disparity between high- and low-resource languages for numerous downstream natural language processing tasks. In this paper, we aim to make this technology more accessible by enabling data efficient training of pretrained language models. It is achieved by formulating language modeling of low-resource languages as a domain adaptation task using transformer-based language models pretrained on corpora of high-resource languages. Our novel cross-lingual post-training approach selectively reuses parameters of the language model trained on a high-resource language and post-trains them while learning language-specific parameters in the low-resource language. We also propose implicit translation layers that can learn linguistic differences between languages at a sequence level. To evaluate our method, we post-train a RoBERTa model pretrained in English and conduct a case study for the Korean language. Quantitative results from intrinsic and extrinsic evaluations show that our method outperforms several massively multilingual and monolingual pretrained language models in most settings and improves the data efficiency by a factor of up to 32 compared to monolingual training.

How Successful Is Transfer Learning for Detecting Anorexia on Social Media?

Anorexia is a mental disorder that involves serious abnormalities in nutritional intake behavior. This behavior leads to significant weight loss, which can lead to severe malnutrition. Specifically, eating disorders exhibit the highest mortality rate of any mental illness. Early identification of anorexia, along with appropriate treatment, improves the speed of recovery in patients. Presently there is a strong and consistent association between social media use and eating concerns. Natural Language Processing, a branch of artificial intelligence, has the potential to contribute towards early anorexia detection in textual data. Currently, there is still a long way to go in the identification of anorexia on social media due to the low number of texts available and in fact, most of these are focused on the treatment of English texts. The main contribution of this paper is the application of transfer learning techniques using Transformer-based models for detecting anorexia in tweets written in Spanish. In particular, we compare the performance between already available multilingual and monolingual models, and we conduct an error analysis to understand the capabilities of these models for Spanish.

End-to-End Speech Synthesis for Tibetan Multidialect

The research on Tibetan speech synthesis technology has been mainly focusing on single dialect, and thus there is a lack of research on Tibetan multidialect speech synthesis technology. This paper presents an end-to-end Tibetan multidialect speech synthesis model to realize a speech synthesis system which can be used to synthesize different Tibetan dialects. Firstly, Wylie transliteration scheme is used to convert the Tibetan text into the corresponding Latin letters, which effectively reduces the size of training corpus and the workload of front-end text processing. Secondly, a shared feature prediction network with a cyclic sequence-to-sequence structure is built, which maps the Latin transliteration vector of Tibetan character to Mel spectrograms and learns the relevant features of multidialect speech data. Thirdly, two dialect-specific WaveNet vocoders are combined with the feature prediction network, which synthesizes the Mel spectrum of Lhasa-Ü-Tsang and Amdo pastoral dialect into time-domain waveform, respectively. The model avoids using a large number of Tibetan dialect expertise for processing some time-consuming tasks, such as phonetic analysis and phonological annotation. Additionally, it can directly synthesize Lhasa-Ü-Tsang and Amdo pastoral speech on the existing text annotation. The experimental results show that the synthesized speech of Lhasa-Ü-Tsang and Amdo pastoral dialect based on our proposed method has better clarity and naturalness than the Tibetan monolingual model.

Language Socialization at the Intersection of the Local and the Global: The Contested Trajectories of Input and Communicative Competence

This article provides a critical review of the theoretical underpinnings of two core concepts in language socialization research: input and communicative competence. We organize our discussion along two major lines of inquiry: ( a) the historical-local and ( b) the language contact–globalization bodies of work. The first part of the article contests the persistent view that input reduces to vocabulary and grammatical structures. To this end, it provides evidence for a more multifaceted approach to input that involves multiparty participant frameworks and multimodality in culturally diverse language socialization ecologies. In this vein, it problematizes language gap studies that are based on middle-class language acquisition models of mother–child dyadic verbal input. The second part of the article challenges monolingual, developmental, and speaker-based models of communicative competence that assume a linear evolution from lesser to greater communicative competence and from more peripheral to more central community membership. It also offers evidence for how communicative competence is socioculturally constructed and, sometimes, interactionally distributed.

Abstractive text summarization for Hungarian

In our research we have created a text summarization software tool for Hungarian using multilingual and Hungarian BERT-based models. Two types of text summarization method exist: abstractive and extractive. The abstractive summarization is more similar to human generated summarization. Target summaries may include phrases that the original text does not necessarily contain. This method generates the summarized text by applying keywords that were extracted from the original text. The extractive method summarizes the text by using the most important extracted phrases or sentences from the original text. In our research we have built both abstractive and extractive models for Hungarian. For abstractive models, we have used a multilingual BERT model and Hungarian monolingual BERT models. For extractive summarization, in addition to the BERT models, we have also made experiments with ELECTRA models. We find that the Hungarian monolingual models outperformed the multilingual BERT model in all cases. Furthermore, the ELECTRA small models achieved higher results than some of the BERT models. This result is important because the ELECTRA small models have much fewer parameters and were trained on only 1 GPU within a couple of days. Another important consideration is that the ELECTRA models are much smaller than the BERT models, which is important for the end users. To our best knowledge the first extractive and abstractive summarization systems reported in the present paper are the first such systems for Hungarian.

Intelligent Recognition and Teaching of English Fuzzy Texts Based on Fuzzy Computing and Big Data

In this paper, we conduct in‐depth research and analysis on the intelligent recognition and teaching of English fuzzy text through parallel projection and region expansion. Multisense Soft Cluster Vector (MSCVec), a multisense word vector model based on nonnegative matrix decomposition and sparse soft clustering, is constructed. The MSCVec model is a monolingual word vector model, which uses nonnegative matrix decomposition of positive point mutual information between words and contexts to extract low‐rank expressions of mixed semantics of multisense words and then uses sparse. It uses the nonnegative matrix decomposition of the positive pointwise mutual information between words and contexts to extract the low‐rank expressions of the mixed semantics of the polysemous words and then uses the sparse soft clustering algorithm to partition the multiple word senses of the polysemous words and also obtains the global sense of the polysemous word affiliation distribution; the specific polysemous word cluster classes are determined based on the negative mean log‐likelihood of the global affiliation between the contextual semantics and the polysemous words, and finally, the polysemous word vectors are learned using the Fast text model under the extended dictionary word set. The advantage of the MSCVec model is that it is an unsupervised learning process without any knowledge base, and the substring representation in the model ensures the generation of unregistered word vectors; in addition, the global affiliation of the MSCVec model can also expect polysemantic word vectors to single word vectors. Compared with the traditional static word vectors, MSCVec shows excellent results in both word similarity and downstream text classification task experiments. The two sets of features are then fused and extended into new semantic features, and similarity classification experiments and stack generalization experiments are designed for comparison. In the cross‐lingual sentence‐level similarity detection task, SCLVec cross‐lingual word vector lexical‐level features outperform MSCVec multisense word vector features as the input embedding layer; deep semantic sentence‐level features trained by twin recurrent neural networks outperform the semantic features of twin convolutional neural networks; extensions of traditional statistical features can effectively improve cross‐lingual similarity detection performance, especially cross‐lingual topic model (BL‐LDA); the stack generalization integration approach maximizes the error rate of the underlying classifier and improves the detection accuracy.

ABioNER: A BERT‐Based Model for Arabic Biomedical Named‐Entity Recognition

The web is being loaded daily with a huge volume of data, mainly unstructured textual data, which increases the need for information extraction and NLP systems significantly. Named‐entity recognition task is a key step towards efficiently understanding text data and saving time and effort. Being a widely used language globally, English is taking over most of the research conducted in this field, especially in the biomedical domain. Unlike other languages, Arabic suffers from lack of resources. This work presents a BERT‐based model to identify biomedical named entities in the Arabic text data (specifically disease and treatment named entities) that investigates the effectiveness of pretraining a monolingual BERT model with a small‐scale biomedical dataset on enhancing the model understanding of Arabic biomedical text. The model performance was compared with two state‐of‐the‐art models (namely, AraBERT and multilingual BERT cased), and it outperformed both models with 85% F1‐score.

Stronger Baselines for Grammatical Error Correction Using a Pretrained Encoder-Decoder Model

Studies on grammatical error correction (GEC) have reported the effectiveness of pretraining a Seq2Seq model with a large amount of pseudodata. However, this approach requires time-consuming pretraining for GEC because of the size of the pseudodata. In this study, we explore the utility of bidirectional and auto-regressive transformers (BART) as a generic pretrained encoder-decoder model for GEC. With the use of this generic pretrained model for GEC, the time-consuming pretraining can be eliminated. We find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC. Our implementations are publicly available at GitHub (https://github.com/Katsumata420/generic-pretrained-GEC).

Expanding the JHU Bible Corpus for Machine Translation of the Indigenous Languages of North America

We present an extension to the JHU Bible corpus, collecting and normalizing more than thirty Bible translations in thirty Indigenous languages of North America. These exhibit a wide variety of interesting syntactic and morphological phenomena that are understudied in the computational community. Neural translation experiments demonstrate significant gains obtained through cross-lingual, many-to-many translation, with improvements of up to 8.4 BLEU over monolingual models for extremely low-resource languages.

Translanguaging for intercultural communication in international higher education: transcending English as a lingua franca

ABSTRACT This article seeks to explore intercultural communication among a group of students with diverse linguistic and sociocultural backgrounds in an English-medium transnational university in China, focusing on multilingual students’ practices and perceptions of English as a lingua franca (ELF). Informed by an expanded understanding of translanguaging from a spatial perspective, this study explores multilingualism, multimodality, and spatiality in linguistic practices related to ELF in an international education context. Drawing upon multiple data sources including ethnographic observations, interviews facilitated by language portraits, recordings of interactions, and policy documents, findings of the study show that ELF communication was understood and practiced by multilingual students as a translanguaging act based on (partially) shared repertoires and beliefs about language and communication. It is also found that the monolithic and monolingual model of language policy in English-medium higher education ran counter to the students’ daily communicative practices and language development needs. Implications are suggested for how international universities can account for translanguaging in fostering linguistic diversity and preparing students for the practical realities of a multilingual world.

Online Multilingual Hate Speech Detection: Experimenting with Hindi and English Social Media

The last two decades have seen an exponential increase in the use of the Internet and social media, which has changed basic human interaction. This has led to many positive outcomes. At the same time, it has brought risks and harms. The volume of harmful content online, such as hate speech, is not manageable by humans. The interest in the academic community to investigate automated means for hate speech detection has increased. In this study, we analyse six publicly available datasets by combining them into a single homogeneous dataset. Having classified them into three classes, abusive, hateful or neither, we create a baseline model and improve model performance scores using various optimisation techniques. After attaining a competitive performance score, we create a tool that identifies and scores a page with an effective metric in near-real-time and uses the same feedback to re-train our model. We prove the competitive performance of our multilingual model in two languages, English and Hindi. This leads to comparable or superior performance to most monolingual models.

Cross-lingual Training for Multiple-Choice Question Answering

In this work we explore to what extent multilingual models can be trained for one language and applied to a different one for the task of Multiple Choice Question Answering. We employ the RACE dataset to fine-tune both a monolingual and a multilingual models and apply these models to another different collections in different languages. The results show that both monolingual and multilingual models can be zero-shot transferred to a different dataset in the same language maintaining its performance. Besides, the multilingual model still performs good when it is applied to a different target language. Additionally, we find that exams that are more difficult to humans are harder for machines too. Finally, we advance the state-of-the-art for the QA4MRE Entrance Exams dataset in several languages.

Multi-SimLex: A Large-Scale Evaluation of Multilingual and Crosslingual Lexical Semantic Similarity

We introduce Multi-SimLex, a large-scale lexical resource and evaluation benchmark covering data sets for 12 typologically diverse languages, including major languages (e.g., Mandarin Chinese, Spanish, Russian) as well as less-resourced ones (e.g., Welsh, Kiswahili). Each language data set is annotated for the lexical relation of semantic similarity and contains 1,888 semantically aligned concept pairs, providing a representative coverage of word classes (nouns, verbs, adjectives, adverbs), frequency ranks, similarity intervals, lexical fields, and concreteness levels. Additionally, owing to the alignment of concepts across languages, we provide a suite of 66 crosslingual semantic similarity data sets. Because of its extensive size and language coverage, Multi-SimLex provides entirely novel opportunities for experimental evaluation and analysis. On its monolingual and crosslingual benchmarks, we evaluate and analyze a wide array of recent state-of-the-art monolingual and crosslingual representation models, including static and contextualized word embeddings (such as fastText, monolingual and multilingual BERT, XLM), externally informed lexical representations, as well as fully unsupervised and (weakly) supervised crosslingual word embeddings. We also present a step-by-step data set creation protocol for creating consistent, Multi-Simlex–style resources for additional languages. We make these contributions—the public release of Multi-SimLex data sets, their creation protocol, strong baseline results, and in-depth analyses which can be helpful in guiding future developments in multilingual lexical semantics and representation learning—available via a Web site that will encourage community effort in further expansion of Multi-Simlex to many more languages. Such a large-scale semantic resource could inspire significant further advances in NLP across languages.

Comparing pre-trained language models for Spanish hate speech detection

Nowadays, due to the great uncontrolled content posted daily on the Web, there has also been a huge increase in the dissemination of hate speech worldwide. Social media, blogs and community forums are examples where people are freely allowed to communicate. However, freedom of expression is not always respectful since offensive or insulting language is sometimes used. Social media companies often rely on users and content moderators to report on this type of content. Nevertheless, due to the large amount of content generated every day on the Web, automatic systems based on Natural Language Processing techniques are required for identifying abusive language online. To date, most of the systems developed to combat this problem are mainly focused on English content, but this issue is a worldwide concern and therefore other languages such as Spanish are involved. In this paper, we address the task of Spanish hate speech identification on social media and provide a deeper understanding of the capabilities of new techniques based on machine learning. In particular, we compare the performance of Deep Learning methods with recently pre-trained language models based on Transfer Learning as well as with traditional machine learning models. Our main contribution is the achievement of promising results in Spanish by applying multilingual and monolingual pre-trained language models such as BERT, XLM and BETO.

Challenging Native Speakerism in Literacy Research and Education

Scholars have examined the myth of Asian Americans and Pacific Islanders (AAPIs) as model minorities in education and specifically within mathematics education, yet less is known about how this myth reveals an intersection of race and language that shapes the experiences of AAPIs in the literacy field. In this article, I argue that a monolingual model rooted in nativist ideologies of English is part and parcel of AAPIs’ racialization as model minorities and forever foreigners. Drawing from AAPI and literacy studies as well as autoethnographic insights, I further argue that the positioning of AAPIs in literacy research illustrates its Eurocentric legacy. This Insights article seeks to raise awareness of a racialized native speaker ethos of literacy research and education, and to call for more literacy research on AAPIs—an invisible minority within the field. Implications include expanding notions of literacy with varied and global perspectives through more research with and from multilingual nondominant communities.

Translation in Caribbean Literature

This essay weaves together translation and postcolonial literary studies to propose a translational model of reading for Caribbean literature. Translation and creolization provide the conceptual and aesthetic lens for reading Caribbean literary texts: If translation is an apt model, since it captures languages in transit toward other languages and other contexts, creolization embodies the points of contact among what Naoki Sakai calls the “uncountable languages within the literary texts,” unlocking novel ideas of language and literature. The essay offers “translational reading” of texts by Derek Walcott, Velma Pollard, and Dionne Brand as an alternative to the traditionally monolingual model of reading.