Low-resource Languages Research Articles

Toward accurate Amazigh part-of-speech tagging

<span lang="EN-US">Part-of-speech (POS) tagging is the process of assigning to each word in a text its corresponding grammatical information POS. It is an important pre-processing step in other natural language processing (NLP) tasks, so the objective of finding the most accurate one. The previous approaches were based on traditional machine learning algorithms, later with the development of deep learning, more POS taggers were adopted. If the accuracy of POS tagging reaches 97%, even with the traditional machine learning, for high resourced language like English, French, it’s far the case in low resource language like Amazigh. The most used approaches are traditional machine learning, and the results are far from those for rich language. In this paper, we present a new POS tagger based on bidirectional long short-term memory for Amazigh language and the experiments that have been done on real dataset shows that it outperforms the existing machine learning methods.</span>

Impact of Language-Specific Training on Image Caption Synthesis: A Case Study on Low-Resource Assamese Language

Automatic Image Captioning (AIC) refers to the process of synthesizing semantically and syntactically correct descriptions for images. Existing research on AIC has predominantly focused on the English language. Comparatively, lower numbers of works have focused on developing captioning systems for low-resource Indian languages like Assamese. This paper investigates AIC for the Assamese language using two distinct approaches. The first approach involves utilizing state-of-the-art AIC model pretrained on an English image-caption dataset to generate English captions for input images. Next, these English captions are translated to the Assamese language using a publicly available automatic translator. The second approach involves exclusively training the AIC model using an Assamese image-caption dataset to predict captions directly in Assamese. The experiments are performed on two types of state-of-art models, one which uses LSTM as a decoder and the other one uses a transformer. Through extensive experimentation, the performance of these approaches is evaluated both quantitatively and qualitatively. The quantitative results are obtained using automatic metrics such as BLEU-n and CIDEr. For qualitative analysis, human evaluation is performed. The comparative performances between the two approaches reveal that models trained exclusively on Assamese image-caption datasets achieve superior results both in terms of quantitative measures and qualitative assessment when compared to models pretrained on English and subsequently translated into Assamese.

Voices of the Himalayas: Benchmarking Speech Recognition Systems for the Tibetan Language

The Lhasa dialect, the most widely spoken Tibetan dialect in Tibet, is also renowned for its rich historical archive of written scripts. Exploring speech recognition methodologies specific to the Lhasa dialect is paramount in safeguarding Tibet’s distinct linguistic heritage. Previous studies in Tibetan speech recognition have been largely confined to academic research using nonpublic datasets, focusing on elements such as the selection of phone-level acoustic modeling units and the integration of tonal information. However, these studies have not significantly benefited the community due to the scarcity of available data. To mitigate the challenge posed by limited data resources, we present the NICT-Tib1 (phase 1) dataset, a new open-source dataset collected from native speakers dedicated to investigating speech recognition for the Lhasa dialect. Speech recognition with deep neural networks (DNNs) evolved three generations from systems hybrid with hidden Markov model (HMM) (e.g., DNN-HMM) to End-to-End systems (e.g., Transformer), and finally to self-supervised learning (SSL) systems (e.g., Wav2Vec2.0), each generation improving accuracy and simplifying the training process, with the latest generation achieving state-of-the-art performance, especially for low-resource languages. Besides the early DNN-HMM-based system using Kaldi, we further update benchmark systems with the Conformer and Wav2Vec2.0 trained by ESPnet and Huggingface on this dataset, respectively. Experimental results show that these state-of-the-art models outperformed the models in previous work.

Multi-level multilingual semantic alignment for zero-shot cross-lingual transfer learning

Recently, cross-lingual transfer learning has attracted extensive attention from both academia and industry. Previous studies usually focus only on the single-level alignment (e.g., word-level, sentence-level), based on pre-trained language models. However, it leads to suboptimal performance in downstream tasks of the low-resource language due to the missing correlation of hierarchical semantic information (e.g., sentence-to-word, word-to-word). Therefore, in this paper, we propose a novel multi-level alignment framework, which hierarchically learns the semantic correlation between multiple levels by leveraging well-designed alignment training tasks. In addition, we devise an attention-based fusion mechanism (AFM) to infuse semantic information from high levels. Extensive experiments on mainstream cross-lingual tasks (e.g., text classification, paraphrase identification, and named entity recognition) demonstrate the effectiveness of our proposed method, and also show that our model achieves state-of-the-art performance across various benchmarks compared to other strong baselines.

Language fusion via adapters for low-resource speech recognition

Data scarcity makes low-resource speech recognition systems suffer from severe overfitting. Although fine-tuning addresses this issue to some extent, it leads to parameter-inefficient training. In this paper, a novel language knowledge fusion method, named LanFusion, is proposed. It is built on the recent popular adapter-tuning technique, thus maintaining better parameter efficiency compared with conventional fine-tuning methods. LanFusion is a two-stage method. Specifically, multiple adapters are first trained on several source languages to extract language-specific and language-invariant knowledge. Then, the trained adapters are re-trained on the target low-resource language to fuse the learned knowledge. Compared with Vanilla-adapter, LanFusion obtains a relative average word error rate (WER) reduction of 9.8% and 8.6% on the Common Voice and FLEURS corpora, respectively. Extensive experiments demonstrate the proposed method is not only simple and effective but also parameter-efficient. Besides, using source languages that are geographically similar to the target language yields better results on both datasets.

Extraction and attribution of public figures statements for journalism in Indonesia using deep learning

News articles are usually written by journalists based on statements taken from interviews with public figures. Attribution from such statements provides important information and it can be extracted from news articles to build a knowledge base by developing a sequential tagging scheme such as entity recognition. This research applies two deep learning architectures: recurrent neural networks-based and transformer-based, to establish public figures statement attribution and extraction models in the Indonesian Language. The experiments are conducted using five deep-learning model architectures with two different corpus sizes to investigate the impact of corpus size on each model's performance. The experiments show that the best model for the RNN-based architecture is PFSA-ID-BLWCA which achieves 81.34 % F1 score, and the best model for the transformer-based is PFSA-ID-TWCA which obtains 81.01 % F1 score. This research also discovers that the size of the corpus influences the model performances. Furthermore, the study lays a foundation to overcome the attribution extraction in another language, especially low-resource languages, with some necessary adjustments.

Cyberbullying Text Identification based on Deep Learning and Transformer-based Language Models

In the contemporary digital age, social media platforms like Facebook, Twitter, and YouTube serve as vital channels for individuals to express ideas and connect with others. Despite fostering increased connectivity, these platforms have inadvertently given rise to negative behaviors, particularly cyberbullying. While extensive research has been conducted on high-resource languages such as English, there is a notable scarcity of resources for low-resource languages like Bengali, Arabic, Tamil, etc., particularly in terms of language modeling. This study addresses this gap by developing a cyberbullying text identification system called BullyFilterNeT tailored for social media texts, considering Bengali as a test case. The intelligent BullyFilterNeT system devised overcomes Out-of-Vocabulary (OOV) challenges associated with non-contextual embeddings and addresses the limitations of context-aware feature representations. To facilitate a comprehensive understanding, three non-contextual embedding models GloVe, FastText, and Word2Vec are developed for feature extraction in Bengali. These embedding models are utilized in the classification models, employing three statistical models (SVM, SGD, Libsvm), and four deep learning models (CNN, VDCNN, LSTM, GRU). Additionally, the study employs six transformer-based language models: mBERT, bELECTRA, IndicBERT, XML-RoBERTa, DistilBERT, and BanglaBERT, respectively to overcome the limitations of earlier models. Remarkably, BanglaBERT-based BullyFilterNeT achieves the highest accuracy of 88.04% in our test set, underscoring its effectiveness in cyberbullying text identification in the Bengali language.

Automatic generation of conclusions from neuroradiology MRI reports through natural language processing.

The conclusion section of a radiology report is crucial for summarizing the primary radiological findings in natural language and essential for communicating results to clinicians. However, creating these summaries is time-consuming, repetitive, and prone to variability and errors among different radiologists. To address these issues, we evaluated a fine-tuned Text-To-Text Transfer Transformer (T5) model for abstractive summarization to automatically generate conclusions for neuroradiology MRI reports in a low-resource language. We retrospectively applied our method to a dataset of 232,425 neuroradiology MRI reports in Spanish. We compared various pre-trained T5 models, including multilingual T5 and those newly adapted for Spanish. For precise evaluation, we employed BLEU, METEOR, ROUGE-L, CIDEr, and cosine similarity metrics alongside expert radiologist assessments. The findings are promising, with the models specifically fine-tuned for neuroradiology MRI achieving scores of 0.46, 0.28, 0.52, 2.45, and 0.87 in the BLEU-1, METEOR, ROUGE-L, CIDEr, and cosine similarity metrics, respectively. In the radiological experts' evaluation, they found that in 75% of the cases evaluated, the conclusions generated by the system were as good as or even better than the manually generated conclusions. The methods demonstrate the potential and effectiveness of customizing state-of-the-art pre-trained models for neuroradiology, yielding automatic MRI report conclusions that nearly match expert quality. Furthermore, these results underscore the importance of designing and pre-training a dedicated language model for radiology report summarization.

Collaborative Encoding Method for Scene Text Recognition in Low Linguistic Resources: The Uyghur Language Case Study

Current research on scene text recognition primarily focuses on languages with abundant linguistic resources, such as English and Chinese. In contrast, there is relatively limited research dedicated to low-resource languages. Advanced methods for scene text recognition often employ Transformer-based architectures. However, the performance of Transformer architectures is suboptimal when dealing with low-resource datasets. This paper proposes a Collaborative Encoding Method for Scene Text Recognition in the low-resource Uyghur language. The encoding framework comprises three main modules: the Filter module, the Dual-Branch Feature Extraction module, and the Dynamic Fusion module. The Filter module, consisting of a series of upsampling and downsampling operations, performs coarse-grained filtering on input images to reduce the impact of scene noise on the model, thereby obtaining more accurate feature information. The Dual-Branch Feature Extraction module adopts a parallel structure combining Transformer encoding and Convolutional Neural Network (CNN) encoding to capture local and global information. The Dynamic Fusion module employs an attention mechanism to dynamically merge the feature information obtained from the Transformer and CNN branches. To address the scarcity of real data for natural scene Uyghur text recognition, this paper conducted two rounds of data augmentation on a dataset of 7267 real images, resulting in 254,345 and 3,052,140 scene images, respectively. This process partially mitigated the issue of insufficient Uyghur language data, making low-resource scene text recognition research feasible. Experimental results demonstrate that the proposed collaborative encoding approach achieves outstanding performance. Compared to baseline methods, our collaborative encoding approach improves accuracy by 14.1%.

Exploring ASR fine-tuning on limited domain-specific data for low-resource languages

Exploring ASR fine-tuning on limited domain-specific data for low-resource languages

Deep Learning-based POS Tagger and Chunker for Odia Language Using Pre-trained Transformers

Developing effective natural language processing (NLP) tools for low-resourced languages poses significant challenges. This article centers its attention on the task of Part-of-speech (POS) tagging and chunking, which pertains to the identification and categorization of linguistic units within sentences. POS tagging and Chunking have already produced positive results in English and other European languages. However, in Indian languages, particularly in Odia language, it is not yet well explored because of the lack of supporting tools, resources, and its complex linguistic morphology. This study presents the building of a manually annotated dataset for Odia phrase chunking task and the development of a deep learning-based model specifically tailored to accommodate the distinctive properties of the language. The process of annotating the Odia chunking corpus involved the utilization of inside-outside-begin labels, which were tagged by using designed Odia chunking tagset. We utilize the constructed Odia chunking dataset to build Odia chunker based on deep learning techniques, employing state-of-the-art architectures. Various techniques, such as Recurrent Neural Networks, Convolutional Neural Networks, and transformer-based models, are investigated to determine the most effective approach for Odia POS tagging and chunking. In addition, we conduct experiments utilizing diverse input representations, including Odia word embeddings, character-level representations, and sub-word units, to effectively capture the complex linguistic characteristics of the Odia language. Numerous experiments are conducted that evaluate the performance of our Odia POS tagger and chunker, employing standard evaluation metrics and making comparisons with existing approaches. The results demonstrate that our transformer-based tagger and chunker achieves superior accuracy and robustness in identifying and categorizing linguistic POS tags and chunks within Odia sentences. It outperforms existing work and exhibits consistent performance across diverse linguistic contexts and sentence structures. The developed Odia POS tagger and chunker have enormous potential for a variety of NLP applications, including information extraction, syntactic parsing, and machine translation, all of which are tailored to the low-resource Odia language. This work contributes to developing NLP tools and technologies for low-resource languages, thereby facilitating enhanced language processing capabilities in various linguistic contexts.

Autoregressive Feature Extraction with Topic Modeling for Aspect-based Sentiment Analysis of Arabic as a Low-resource Language

This paper proposes an approach for aspect-based sentiment analysis of Arabic social data, especially the considerable text corpus generated through communications on X (formerly known as Twitter) for expressing opinions in Arabic-language tweets during the COVID-19 pandemic. The proposed approach examines the performance of several pre-trained predictive and autoregressive language models; namely, Bidirectional Encoder Representations from Transformers (BERT) and XLNet, along with topic modeling algorithms; namely, Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF), for aspect-based sentiment analysis of online Arabic text. In addition, Bidirectional Long Short Term Memory (Bi-LSTM) deep learning model is used to classify the extracted aspects from online reviews. Obtained experimental results indicate that the combined XLNet-NMF model outperforms other implemented state-of-the-art methods through improving the feature extraction of unstructured social media text with achieving values of 0.946 and 0.938, for average sentiment classification accuracy and F-measure, respectively.

Research of types and current state of machine translation

The background of machine translation dates back to the 1950s when scientists began exploring the use of computers for translation. Motivated by various challenges and needs, such as the high cost of manual translation and cross-cultural communication, machine translation has become a pivotal field. This overview delves into the research background, content, methods, key figures, conclusions, and future prospects of machine translation. It summarizes automatic evaluation metrics, corpus construction, and transfer learning, all of which contribute to enhancing translation performance. Currently, there are three mainstream categories of methods, which include rule-based translation, statistical translation, and neural network-based translation. The rule-based translation method relies on language rules and dictionaries for translation. Statistical machine translation involves the use of extensive bilingual corpora for identification and translation. The conclusion emphasizes the potential of neural machine translation, yet acknowledges challenges in diverse languages, low-resource languages, and specialized terminology.

A Literature Survey on SpeakSmart: AI – Enhanced language learning guide

This extensive review of the literature examines the pervasive impact of artificial intelligence (AI) on language learning and teaching, covering innovative uses like real-time language practice, personalized language learning, and automated testing methods in the teaching of English as a foreign language (EFL). The survey covers a wide range of topics, such as the use of Natural Language Processing (NLP) to extract Business Process Models in Business Process Management (BPM), the deployment of English-language chat-bots operating on the WeChat platform that are based on transfer learning, and the incorporation of NLP tools and virtual reality in the e-learning platform Exills. Additionally, it delves into cutting-edge technologies such as chat-bots for language learning, low-resource spoken language learning applications with AI support, and the broad application of AI in EFL teaching, tackling issues and offering encouraging results. The poll also addresses the use of AI in language learning tools, focusing on transparency and ethical issues while utilizing chat-bots, machine translation, speech recognition, and AI-generated content. The study highlights the importance of exposure to different cultures, community interaction, and multilingual voice recognition in creating a comprehensive language learning environment. It also highlights the limitations and enormous possibilities of integrating AI into language instruction.

A Survey on Challenges and Advances in Natural Language Processing with a Focus on Legal Informatics and Low-Resource Languages

The field of Natural Language Processing (NLP) has experienced significant growth in recent years, largely due to advancements in Deep Learning technology and especially Large Language Models. These improvements have allowed for the development of new models and architectures that have been successfully applied in various real-world applications. Despite this progress, the field of Legal Informatics has been slow to adopt these techniques. In this study, we conducted an extensive literature review of NLP research focused on legislative documents. We present the current state-of-the-art NLP tasks related to Law Consolidation, highlighting the challenges that arise in low-resource languages. Our goal is to outline the difficulties faced by this field and the methods that have been developed to overcome them. Finally, we provide examples of NLP implementations in the legal domain and discuss potential future directions.

Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection

Hate speech in social media is a growing phenomenon, and detecting such toxic content has recently gained significant traction in the research community. Existing studies have explored fine-tuning language models (LMs) to perform hate speech detection, and these solutions have yielded significant performance. However, most of these studies are limited to detecting hate speech only in English, neglecting the bulk of hateful content that is generated in other languages, particularly in low-resource languages. Developing a classifier that captures hate speech and nuances in a low-resource language with limited data is extremely challenging. To fill the research gap, we propose HateMAML, a model-agnostic meta-learning (MAML)-based framework that effectively performs hate speech detection in low-resource languages. HateMAML utilizes a self-supervision strategy to overcome the limitation of data scarcity and produces better LM initialization for fast adaptation to an unseen target language (i.e., cross-lingual transfer) or other hate speech datasets (i.e., domain generalization). Extensive experiments are conducted on five datasets across eight different low-resource languages. The results show that HateMAML outperforms the state-of-the-art baselines by more than 3% in the cross-domain multilingual transfer setting. We also conduct ablation studies to analyze the characteristics of HateMAML.

Machine learning based framework for fine-grained word segmentation and enhanced text normalization for low resourced language

In text applications, pre-processing is deemed as a significant parameter to enhance the outcomes of natural language processing (NLP) chores. Text normalization and tokenization are two pivotal procedures of text pre-processing that cannot be overstated. Text normalization refers to transforming raw text into scriptural standardized text, while word tokenization splits the text into tokens or words. Well defined normalization and tokenization approaches exist for most spoken languages in world. However, the world’s 10th most widely spoken language has been overlooked by the research community. This research presents improved text normalization and tokenization techniques for the Urdu language. For Urdu text normalization, multiple regular expressions and rules are proposed, including removing diuretics, normalizing single characters, separating digits, etc. While for word tokenization, core features are defined and extracted against each character of text. Machine learning model is considered with specified handcrafted rules to predict the space and to tokenize the text. This experiment is performed, while creating the largest human-annotated dataset composed in Urdu script covering five different domains. The results have been evaluated using precision, recall, F-measure, and accuracy. Further, the results are compared with state-of-the-art. The normalization approach produced 20% and tokenization approach achieved 6% improvement.

MGPT: Few-Shot Learners Go Multilingual

Abstract This paper introduces mGPT, a multilingual variant of GPT-3, pretrained on 61 languages from 25 linguistically diverse language families using Wikipedia and the C4 Corpus. We detail the design and pretraining procedure. The models undergo an intrinsic and extrinsic evaluation: language modeling in all languages, downstream evaluation on cross-lingual NLU datasets and benchmarks in 33 languages, and world knowledge probing in 23 languages. The in-context learning abilities are on par with the contemporaneous language models while covering a larger number of languages, including underrepresented and low-resource languages of the Commonwealth of Independent States and the indigenous peoples in Russia. The source code and the language models are publicly available under the MIT license.

Improving neural machine translation for low resource languages through non-parallel corpora: a case study of Egyptian dialect to modern standard Arabic translation

Machine translation for low-resource languages poses significant challenges, primarily due to the limited availability of data. In recent years, unsupervised learning has emerged as a promising approach to overcome this issue by aiming to learn translations between languages without depending on parallel data. A wide range of methods have been proposed in the literature to address this complex problem. This paper presents an in-depth investigation of semi-supervised neural machine translation specifically focusing on translating Arabic dialects, particularly Egyptian, to Modern Standard Arabic. The study employs two distinct datasets: one parallel dataset containing aligned sentences in both dialects, and a monolingual dataset where the source dialect is not directly connected to the target language in the training data. Three different translation systems are explored in this study. The first is an attention-based sequence-to-sequence model that benefits from the shared vocabulary between the Egyptian dialect and Modern Arabic to learn word embeddings. The second is an unsupervised transformer model that depends solely on monolingual data, without any parallel data. The third system starts with the parallel dataset for an initial supervised learning phase and then incorporates the monolingual data during the training process.

Negative Stances Detection from Multilingual Data Streams in Low-Resource Languages on Social Media Using BERT and CNN-Based Transfer Learning Model

Online social media allows users to connect with a large number of people across the globe and facilitate the exchange of information efficiently. These platforms cater to many of our day-to-day needs. However, at the same time, social media have been increasingly used to transmit negative stances such as derogatory language, hate speech, and cyberbullying. The task of identifying the negative stances from social media posts or comments or tweets is termed negative stance detection . One of the major challenges associated with negative stance detection is that most of the content published on social media is often in a multilingual format. This work aims to identify negative stances from multilingual data streams in low-resource languages on social media using a hybrid transfer learning and deep convolutional neural network approach. The proposed work starts by preprocessing the multilingual datasets by removing irrelevant information such as special characters and hyperlinks. The processed dataset is then passed through a pretrained BERT (bidirectional encoder representations from Transformers) model to generate embeddings by fine-tuning the model as per the dataset under consideration. The generated word embeddings are then passed to a deep convolutional neural network for extracting the latent features from the texts and removing the unessential information. This helps our model to achieve robustness and effectiveness for efficient learning on the given dataset and make appropriate predictions on zero-shot data. The article utilizes several optimization strategies for examining the impact of fine-tuning different BERT layers on the model’s performance. Intensive experiments on a variety of languages — namely, English, French, Italian, Danish, Arabic, Spanish, Indonesian, German, and Portuguese — are performed. The experimental results demonstrate the effectiveness and efficiency of the proposed framework.