Cross-lingual Natural Language Processing Research Articles

A Benchmark Evaluation of Multilingual Large Language Models for Arabic Cross-Lingual Named-Entity Recognition

Multilingual large language models (MLLMs) have demonstrated remarkable performance across a wide range of cross-lingual Natural Language Processing (NLP) tasks. The emergence of MLLMs made it possible to achieve knowledge transfer from high-resource to low-resource languages. Several MLLMs have been released for cross-lingual transfer tasks. However, no systematic evaluation comparing all models for Arabic cross-lingual Named-Entity Recognition (NER) is available. This paper presents a benchmark evaluation to empirically investigate the performance of the state-of-the-art multilingual large language models for Arabic cross-lingual NER. Furthermore, we investigated the performance of different MLLMs adaptation methods to better model the Arabic language. An error analysis of the different adaptation methods is presented. Our experimental results indicate that GigaBERT outperforms other models for Arabic cross-lingual NER, while language-adaptive pre-training (LAPT) proves to be the most effective adaptation method across all datasets. Our findings highlight the importance of incorporating language-specific knowledge to enhance the performance in distant language pairs like English and Arabic.

Cross-lingual Natural Language Processing on Limited Annotated Case/Radiology Reports in English and Japanese: Insights from the Real-MedNLP Workshop.

Textual datasets (corpora) are crucial for the application of natural language processing (NLP) models. However, corpus creation in the medical field is challenging, primarily because of privacy issues with raw clinical data such as health records. Thus, the existing clinical corpora are generally small and scarce. Medical NLP (MedNLP) methodologies perform well with limited data availability. We present the outcomes of the Real-MedNLP workshop, which was conducted using limited and parallel medical corpora. Real-MedNLP exhibits three distinct characteristics: (1) Limited Annotated Documents: The training data comprises only a small set (approximately 100) of case reports (CRs) and radiology reports (RRs) that have been annotated. (2) Bilingually Parallel: The constructed corpora are parallel in Japanese and English. (3) Practical Tasks: The workshop addresses fundamental tasks, such as named entity recognition and applied practical tasks. We propose three tasks: named entity recognition (NER) of approximately 100 available documents (Task 1), NER based only on annotation guidelines for humans (Task 2), and clinical applications (Task 3) consisting of adverse drug effects (ADE) detection for CRs and identical case identification (CI) for RRs. Nine teams participated in this study. The best systems achieved 0.65 and 0.89 F1-scores for CRs and RRs in Task 1, whereas the top scores in Task 2 decreased by 50-70%. In Task 3, ADE reports were detected by up to 0.64 F1-score, and CI scored up to 0.96 binary accuracy. Most systems adopt medical-domain-specific pre-trained language models using data augmentation methods. Despite the challenge of limited corpus size in Tasks 1 and 2, recent approaches are promising because the partial match scores reached approximately 0.8-0.9 F1-scores. Task 3 applications revealed that the different availabilities of external language resources affected the performance per language.

Language relatedness evaluation for multilingual neural machine translation

Multilingual neural machine translation (MNMT) has attracted more and more attention in recent days because it can use a single neural machine translation (NMT) model to translate between multiple languages. As several languages are involved in MNMT, recent studies have shown that using part of these languages rather than all of them to train the model leads to comparable results. However, previous work on this topic mainly focuses on language clustering and features defined by linguists. The semantic relationship and language distance are not fully considered. How to select the most related language pairs to current low-resource pair to optimize the performance of MNMT is still an open question. In this paper, we propose to take language relatedness computation as a ranking problem, where features such as language distance, linguistic typological information and semantic relatedness features are incorporated into a random decision forest to improve the language relatedness evaluation (LRE) for MNMT. Since the model only focuses on monolingual LRE in general cross-lingual natural language processing tasks, we also propose two features related to machine translation (data size and bilingual relatedness) to predict the final language pairs. Experimental results on IWSLT and WMT datasets show that our proposed LRE method can achieve significant improvements compared with other models. We also conducted several groups of experiments on IWSLT and WMT datasets to further evaluate the effectiveness of the proposed method on MNMT. The results show that the MNMT model trained on language pairs predicted by the LRE method outperforms other language selection methods.

Improving the Robustness of Loanword Identification in Social Media Texts

As a potential bilingual resource, loanwords play a very important role in many natural language processing tasks. If loanwords in a low-resource language can be identified effectively, the generated donor-receipt word pairs will benefit many cross-lingual natural language processing tasks. However, most studies on loanword identification mainly focus on formal texts such as news and government documents. Loanword identification in social media texts is still an under-studied field. Since it faces many challenges and can be widely used in several downstream tasks, more efforts should be put on loanword identification in social media texts. In this study, we present a multi-task learning architecture with deep bi-directional recurrent neural networks for loanword identification in social media texts, where different task supervision can happen at different layers. The multi-task neural network architecture learns higher-order feature representations from word and character sequences along with basic spell error checking, part-of-speech tagging, and named entity recognition information. Experimental results on Uyghur loanword identification in social media texts in five donor languages (Chinese, Arabic, Russian, Turkish, and Farsi) show that our method achieves the best performance compared with several strong baseline systems. We also combine the loanword detection results into the training data of neural machine translation for low-resource language pairs. Experiments show that models trained on the extended datasets achieve significant improvements compared with the baseline models in all language pairs.

Word-Pair Relevance Modeling with Multi-View Neural Attention Mechanism for Sentence Alignment

Sentence alignment provides multi-lingual or cross-lingual natural language processing (NLP) applications with high-quality parallel sentence pairs. Normally, an aligned sentence pair contains multiple aligned words, which intuitively play different roles during sentence alignment. Inspired by this intuition, we propose to deal with the problem of sentence alignment by exploring the semantic interactionship among fine-grained word pairs within the framework of neural network. In particular, we first employ various relevance measures to capture various kinds of semantic interactions among word pairs by using a word-pair relevance network, and then model their importance by using a multi-view attention network. Experimental results on both monotonic and non-monotonic bitexts show that our proposed approach significantly improves the performance of sentence alignment.

Exploring Implicit Semantic Constraints for Bilingual Word Embeddings

Bilingual word embeddings (BWEs) have proven to be useful in many cross-lingual natural language processing tasks. Previous studies often require bilingual texts or dictionaries that are scarce res...

Bilingual lexicon induction from non-parallel corpora

In cross-lingual natural language processing, the lack of parallel data is a serious problem. However, this is common in scenarios with scarce language resources. In this case, better utilizing translational equivalence encoded in non-parallel corpora becomes more important. Owing to the non-parallelism of the corpora, acquiring translational equivalence faces the challenging problem of small data or unsupervised learning, and the result usually takes the form of a bilingual lexicon. Not only is this an important research problem in the field of artificial intelligence, but it also has significant application value in scenarios with scarce language resources. This paper introduces a series of studies that address problems in prior research, exploring how to obtain better bilingual lexica with non-parallel corpora from various perspectives.

Exploring Implicit Semantic Constraints for Bilingual Word Embeddings

Bilingual word embeddings (BWEs) have proven to be useful in many cross-lingual natural language processing tasks. Previous studies often require bilingual texts or dictionaries that are scarce resources. As a result, in these studies, the exploited explicit semantic information, such as monolingual word co-occurrences and cross-lingual semantic equivalences, is often insufficient for BWE learning, leading to the limitation of learned word representations. To overcome this problem, in this paper, we study how to exploit implicit semantic constraints for better BWEs. Concretely, we first discover implicit monolingual word-level semantic equivalences by pivoting their translations in the other language. Then, we perform BWE learning under various semantic constraints. Experimental results on machine translation and cross-lingual document classification demonstrate the effectiveness of our model.

A neural generative autoencoder for bilingual word embeddings

Bilingual word embeddings (BWEs) have been shown to be useful in various cross-lingual natural language processing tasks. To accurately learn BWEs, previous studies often resort to discriminative approaches which explore semantic proximities between translation equivalents of different languages. Instead, in this paper, we propose a neural generative bilingual autoencoder (NGBAE) which introduces a latent variable to explicitly induce the underlying semantics of bilingual text. In this way, NGBAE is able to obtain better BWEs from more robust bilingual semantics by modeling the semantic distributions of bilingual text. In order to facilitate scalable inference and learning, we utilize deep neural networks to perform the recognition and generation procedures, and then employ stochastic gradient variational Bayes algorithm to optimize them jointly. We validate the proposed model via both extrinsic (cross-lingual document classification and translation probability modeling) and intrinsic (word embedding analysis) evaluations. Experimental results demonstrate the effectiveness of NGBAE on learning BWEs.

Introduction to the Special Issue on Cross-Language Algorithms and Applications

With the increasingly global nature of our everyday interactions, the need for multilin- gual technologies to support efficient and effective information access and communication cannot be overemphasized. Computational modeling of language has been the focus of Natural Language Processing, a subdiscipline of Artificial Intelligence. One of the current challenges for this discipline is to design methodologies and algorithms that are cross- language in order to create multilingual technologies rapidly. The goal of this JAIR special issue on Cross-Language Algorithms and Applications (CLAA) is to present leading re- search in this area, with emphasis on developing unifying themes that could lead to the development of the science of multi- and cross-lingualism. In this introduction, we provide the reader with the motivation for this special issue and summarize the contributions of the papers that have been included. The selected papers cover a broad range of cross-lingual technologies including machine translation, domain and language adaptation for sentiment analysis, cross-language lexical resources, dependency parsing, information retrieval and knowledge representation. We anticipate that this special issue will serve as an invaluable resource for researchers interested in topics of cross-lingual natural language processing.

A statistical approach to crosslingual natural language tasks

The existence of huge volumes of documents written in multiple languages on Internet leads to investigate novel algorithmic approaches to deal with information of this kind. However, most crosslingual natural language processing (NLP) tasks consider a decoupled approach in which monolingual NLP techniques are applied along with an independent translation process. This two-step approach is too sensitive to translation errors, and in general to the accumulative effect of errors. To solve this problem, we propose to use a direct probabilistic crosslingual NLP system which integrates both steps, translation and the specific NLP task, into a single one. In order to perform this integrated approach to crosslingual tasks, we propose to use the statistical IBM 1 word alignment model (M1). The M1 model may show a non-monotonic behaviour when aligning words from a sentence in a source language to words from another sentence in a different, target language. This is the case of languages with different word order. In English, for instance, adjectives appear before nouns, whereas in Spanish it is exactly the opposite. The successful experimental results reported in three different tasks – text classification, information retrieval and plagiarism analysis – highlight the benefits of the statistical integrated approach proposed in this work.