Multilingual Models Research Articles

CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer

Cross-lingual cross-modal retrieval has garnered increasing attention recently, which aims to achieve the alignment between vision and target language (V-T) without using any annotated V-T data pairs. Current methods employ machine translation (MT) to construct pseudo-parallel data pairs, which are then used to learn a multi-lingual and multi-modal embedding space that aligns visual and target-language representations. However, the large heterogeneous gap between vision and text, along with the noise present in target language translations, poses significant challenges in effectively aligning their representations. To address these challenges, we propose a general framework, Cross-Lingual to Cross-Modal (CL2CM), which improves the alignment between vision and target language using cross-lingual transfer. This approach allows us to fully leverage the merits of multi-lingual pre-trained models (e.g., mBERT) and the benefits of the same modality structure, i.e., smaller gap, to provide reliable and comprehensive semantic correspondence (knowledge) for the cross-modal network. We evaluate our proposed approach on two multilingual image-text datasets, Multi30K and MSCOCO, and one video-text dataset, VATEX. The results clearly demonstrate the effectiveness of our proposed method and its high potential for large-scale retrieval.

CONSIDER: Commonalities and Specialties Driven Multilingual Code Retrieval Framework

Multilingual code retrieval aims to find code snippets relevant to a user's query from a multilingual codebase, which plays a crucial role in software development and expands their application scenarios compared to classical monolingual code retrieval. Despite the performance improvements achieved by previous studies, two crucial problems are overlooked in the multilingual scenario. First, certain programming languages face data scarcity in specific domains, resulting in limited representation capabilities within those domains. Second, different programming languages can be used interchangeably within the same domain, making it challenging for multilingual models to accurately identify the intended programming language of a user's query. To address these issues, we propose the CommONalities and SpecIalties Driven Multilingual CodE Retrieval Framework (CONSIDER), which includes two modules. The first module enhances the representation of various programming languages by modeling pairwise and global commonalities among them. The second module introduces a novel contrastive learning negative sampling algorithm that leverages language confusion to automatically extract specific language features. Through our experiments, we confirm the significant benefits of our model in real-world multilingual code retrieval scenarios in various aspects. Furthermore, an evaluation demonstrates the effectiveness of our proposed CONSIDER framework in monolingual scenarios as well. Our source code is available at https://github.com/smsquirrel/consider.

Multi-class hate speech detection in the Norwegian language using FAST-RNN and multilingual fine-tuned transformers

The growth of social networks has provided a platform for individuals with prejudiced views, allowing them to spread hate speech and target others based on their gender, ethnicity, religion, or sexual orientation. While positive interactions within diverse communities can considerably enhance confidence, it is critical to recognize that negative comments can hurt people’s reputations and well-being. This emergence emphasizes the need for more diligent monitoring and robust policies on these platforms to protect individuals from such discriminatory and harmful behavior. Hate speech is often characterized as an intentional act of aggression directed at a specific group, typically meant to harm or marginalize them based on certain aspects of their identity. Most of the research related to hate speech has been conducted in resource-aware languages like English, Spanish, and French. However, low-resource European languages, such as Irish, Norwegian, Portuguese, Polish, Slovak, and many South Asian, present challenges due to limited linguistic resources, making information extraction labor-intensive. In this study, we present deep neural networks with FastText word embeddings using regularization methods for multi-class hate speech detection in the Norwegian language, along with the implementation of multilingual transformer-based models with hyperparameter tuning and generative configuration. FastText outperformed other deep learning models when stacked with Bidirectional LSTM and GRU, resulting in the FAST-RNN model. In the concluding phase, we compare our results with the state-of-the-art and perform interpretability modeling using Local Interpretable Model-Agnostic Explanations to achieve a more comprehensive understanding of the model’s decision-making mechanisms.

Construction and analysis of uncertainty indices based on multilingual text representations

The work by Baker et al. (2016), who propose a dictionary based method and estimate the level of economic policy uncertainty (EPU) based on the occurrence of specific terms in ten leading newspapers in the USA, is among the first ones to detect the potential of text data in economic research. Following this line of research, this paper proposes automated approaches to construction of EPU indices for different countries based on newspapers’ texts. Multilingual fastText word embeddings, (S)BERT embeddings, and a novel multilingual topic modeling approach are used to construct EPU indices for Germany, Russia, and Ukraine. It is shown that constructed EPU indices based on multilingual word embeddings are Granger causal to the economic activity in all of the considered countries.

BioLORD-2023: semantic textual representations fusing large language models and clinical knowledge graph insights.

In this study, we investigate the potential of large language models (LLMs) to complement biomedical knowledge graphs in the training of semantic models for the biomedical and clinical domains. Drawing on the wealth of the Unified Medical Language System knowledge graph and harnessing cutting-edge LLMs, we propose a new state-of-the-art approach for obtaining high-fidelity representations of biomedical concepts and sentences, consisting of 3 steps: an improved contrastive learning phase, a novel self-distillation phase, and a weight averaging phase. Through rigorous evaluations of diverse downstream tasks, we demonstrate consistent and substantial improvements over the previous state of the art for semantic textual similarity (STS), biomedical concept representation (BCR), and clinically named entity linking, across 15+ datasets. Besides our new state-of-the-art biomedical model for English, we also distill and release a multilingual model compatible with 50+ languages and finetuned on 7 European languages. Many clinical pipelines can benefit from our latest models. Our new multilingual model enables a range of languages to benefit from our advancements in biomedical semantic representation learning, opening a new avenue for bioinformatics researchers around the world. As a result, we hope to see BioLORD-2023 becoming a precious tool for future biomedical applications. In this article, we introduced BioLORD-2023, a state-of-the-art model for STS and BCR designed for the clinical domain.

A Bilingual Basque–Spanish Dataset of Parliamentary Sessions for the Development and Evaluation of Speech Technology

The development of speech technology requires large amounts of data to estimate the underlying models. Even when relying on large multilingual pre-trained models, some amount of task-specific data on the target language is needed to fine-tune those models and obtain competitive performance. In this paper, we present a bilingual Basque–Spanish dataset extracted from parliamentary sessions. The dataset is designed to develop and evaluate automatic speech recognition (ASR) systems but can be easily repurposed for other speech-processing tasks (such as speaker or language recognition). The paper first compares the two target languages, emphasizing their similarities at the acoustic-phonetic level, which sets the basis for sharing data and compensating for the relatively small amount of spoken resources available for Basque. Then, Basque Parliament plenary sessions are characterized in terms of organization, topics, speaker turns and the use of the two languages. The paper continues with the description of the data collection procedure (involving both speech and text), the audio formats and conversions along with the creation and postprocessing of text transcriptions and session minutes. Then, it describes the semi-supervised iterative procedure used to cut, rank and select the training segments and the manual supervision process employed to produce the test set. Finally, ASR experiments are presented using state-of-the-art technology to validate the dataset and to set a reference for future works. The datasets, along with models and recipes to reproduce the experiments reported in the paper, are released through Hugging Face.

Neural machine translation of clinical text: an empirical investigation into multilingual pre-trained language models and transfer-learning.

Clinical text and documents contain very rich information and knowledge in healthcare, and their processing using state-of-the-art language technology becomes very important for building intelligent systems for supporting healthcare and social good. This processing includes creating language understanding models and translating resources into other natural languages to share domain-specific cross-lingual knowledge. In this work, we conduct investigations on clinical text machine translation by examining multilingual neural network models using deep learning such as Transformer based structures. Furthermore, to address the language resource imbalance issue, we also carry out experiments using a transfer learning methodology based on massive multilingual pre-trained language models (MMPLMs). The experimental results on three sub-tasks including (1) clinical case (CC), (2) clinical terminology (CT), and (3) ontological concept (OC) show that our models achieved top-level performances in the ClinSpEn-2022 shared task on English-Spanish clinical domain data. Furthermore, our expert-based human evaluations demonstrate that the small-sized pre-trained language model (PLM) outperformed the other two extra-large language models by a large margin in the clinical domain fine-tuning, which finding was never reported in the field. Finally, the transfer learning method works well in our experimental setting using the WMT21fb model to accommodate a new language space Spanish that was not seen at the pre-training stage within WMT21fb itself, which deserves more exploitation for clinical knowledge transformation, e.g. to investigate into more languages. These research findings can shed some light on domain-specific machine translation development, especially in clinical and healthcare fields. Further research projects can be carried out based on our work to improve healthcare text analytics and knowledge transformation. Our data is openly available for research purposes at: https://github.com/HECTA-UoM/ClinicalNMT.

Multilingualism in Kazakhstan�s education system: Implementation and challenges

Relevance. Kazakhstan�s linguistic landscape reflects its diverse cultural heritage and contemporary geopolitical position. As the country navigates economic development and global integration, the implementation of multilingual education becomes crucial for fostering national unity while preparing citizens for international engagement. Purpose. This purpose of the study is to analyze the effectiveness and challenges of multilingual education models in Kazakhstan�s educational institutions, from primary schools to universities. Methodology. The research employs a mixed-methods approach, combining sociological analysis, statistical data evaluation, and comparative studies of educational practices. Results. The study reveals that Kazakhstan has adopted various models for implementing trilingual education, including Content and Language Integrated Learning (CLIL). While 117 schools and 42 universities have implemented trilingual programs, significant challenges persist, particularly in teacher training and resource allocation. The language of instruction in higher education shows a predominance of Kazakh (63%) and Russian (34%), with English (3%) gaining traction. Conclusions. The implementation of multilingual education in Kazakhstan reflects a delicate balance between preserving national identity and fostering global integration. The study highlights the positive impact of multilingualism on individual development and societal progress. Keywords: educational policy; linguistic diversity; teacher training; cultural identity; trilingual education

Beyond Language Barriers: Allowing Multiple Languages in Postsecondary Chemistry Classes Through Multilingual Machine Learning

AbstractStudents who learn the language of instruction as an additional language represent a heterogeneous group with varying linguistic and cultural backgrounds, contributing to classroom diversity. Because of the manifold challenges these students encounter while learning the language of instruction, additional barriers arise for them when engaging in chemistry classes. Adapting teaching practices to the language skills of these students, for instance, in formative assessments, is essential to promote equity and inclusivity in chemistry learning. For this reason, novel educational practices are needed to meet each student’s unique set of language capabilities, irrespective of course size. In this study, we propose and validate several approaches to allow undergraduate chemistry students who are not yet fluent in the language of instruction to complete a formative assessment in their preferred language. A technically easy-to-implement option for instructors is to use translation tools to translate students’ reasoning in any language into the instructor’s language. Besides, instructors could also establish multilingual machine learning models capable of automatically analyzing students’ reasoning regardless of the applied language. Herein, we evaluated both opportunities by comparing the reliability of three translation tools and determining the degree to which multilingual machine learning models can simultaneously assess written arguments in different languages. The findings illustrate opportunities to apply machine learning for analyzing students’ reasoning in multiple languages, demonstrating the potential of such techniques in ensuring equal access for learners of the language of instruction.

Определение коэффициента билингвизма будущих учителей-предметников методом ассоциативного эксперимента

The paper describes the essence of the phenomenon of bilingualism, types of bilingualism, approaches to its quantitative assessment, as well as the results of an associative experiment aimed at determining the level of bilingualism of future teachers studying at Kazan Federal University on a bilingual basis using Russian and Tatar languages. The experiment involved 40 students – future mathematics teachers in the 2nd and 3rd years of study. The scientific novelty of the research lies in determining that the level of the academic component of bilingual communicative competence among future bilingual teachers is higher than the level of their general communicative competence. As a result, it was found that at this stage of the implementation of the multilingual education model at Kazan Federal University, most future subject teachers have formed the coordinate and the compound type of Tatar-Russian bilingualism both in the field of general communication and academic language in the sphere of mathematics on the topic “Function”.

Assessing the risks and opportunities posed by AI-enhanced influence operations on social media

AbstractLarge language models (LLMs) like GPT-4 have the potential to dramatically change the landscape of influence operations. They can generate persuasive, tailored content at scale, making campaigns using falsified content, such as disinformation and fake accounts, easier to produce. Advances in self-hosted open-source models have meant that adversaries can evade content moderation and security checks built into large commercial models such as those commercialised by Anthropic, Google, and OpenAI. New multi-lingual models make it easier than ever for foreign adversaries to pose as local actors. This article examines the heightened threats posed by synthetic media, as well as the potential that these tools hold for creating effective countermeasures. It begins with assessing the challenges posed by a toxic combination of automated bots, human-controlled troll accounts, and more targeted social engineering operations. However, the second part of the article assesses the potential for these same tools to improve detection. Promising countermeasures include running internal generative models to bolster training data for internal classifiers, detecting statistical anomalies, identifying output from common prompts, and building specialised classifiers optimised for specific monitoring needs.

DReAMy: a library for the automatic analysis and annotation of dream reports with multilingual large language models

DReAMy: a library for the automatic analysis and annotation of dream reports with multilingual large language models

Initial Exploration of the Transformation of Technology Museums Empowered by Artificial Intelligence

Traditional science and technology museums are facing challenges in the rapid advancements in science and technology. These challenges include limited exhibition formats, outdated methods of conveying information, and inefficient resource management. However, with the rapid development of artificial intelligence technology, there is an opportunity to overcome these challenges. Artificial intelligence technology, specifically in the field of natural language processing, has made significant progress in improving multilingual understanding and generative models. This advancement can greatly enhance the learning experience for diverse audiences. In addition, the development of computer vision technology focuses on scene understanding, video analysis, and visual inference. These advancements can provide a more immersive and engaging exhibition experience at science and technology museums. This paper aims to explore how traditional science and technology museums can integrate and redesign artificial intelligence technology to better meet the needs of their audiences. By doing so, it can improve the effectiveness of science and technology communication and enable museums to adapt more flexibly to the rapidly changing science and technology landscape. The findings of this study can provide strategic considerations and guidance for the future development of science and technology museums.

A comparative study of cross-lingual sentiment analysis

This paper presents a detailed comparative study of the zero-shot cross-lingual sentiment analysis. Namely, we use modern multilingual Transformer-based models and linear transformations combined with CNN and LSTM neural networks. We evaluate their performance in Czech, French, and English. We aim to compare and assess the models’ ability to transfer knowledge across languages and discuss the trade-off between their performance and training/inference speed. We build strong monolingual baselines comparable with the current SotA approaches, achieving state-of-the-art results in Czech (96.0% accuracy) and French (97.6% accuracy). Next, we compare our results with the latest large language models (LLMs), i.e., Llama 2 and ChatGPT.We show that the large multilingual Transformer-based XLM-R model consistently outperforms all other cross-lingual approaches in zero-shot cross-lingual sentiment classification, surpassing them by at least 3%. Next, we show that the smaller Transformer-based models are comparable in performance to older but much faster methods with linear transformations. The best-performing model with linear transformation achieved an accuracy of 92.1% on the French dataset, compared to 90.3% received by the smaller XLM-R model. Notably, this performance is achieved with just approximately 0.01 of the training time required for the XLM-R model. It underscores the potential of linear transformations as a pragmatic alternative to resource-intensive and slower Transformer-based models in real-world applications. The LLMs achieved impressive results that are on par or better, at least by 1%–3%, but with additional hardware requirements and limitations. Overall, this study contributes to understanding cross-lingual sentiment analysis and provides valuable insights into the strengths and limitations of cross-lingual approaches for sentiment analysis.

On Language Policy in the Education of Young People in Multinational States

This article presents theoretical aspects of language policy in the education of young people in multinational states, drawing on the experience of the Republic of Moldova. An emphasis is placed on the interaction between language and state education policy, clarifying the legal framework in the multinational state in question. The opprtunities different ethnic groups, which historically live on the territory of the Republic of Moldova, have with regard to studying their mother tongue in the country‘s schools are outlined. In order to look for opportunities to implement a language policy in accordance with the new educational realities in European countries, a study is organized in the Republic of Moldova, and part of the results are presented in the article. The obtained empirical data on attitudes to language learning expressed by students of various ethnic origins give reason to conceptualize the idea of implementing multilingual learning models in the educational system of the Republic of Moldova. These results provide important guidelines about continuing studies in this direction and about developing language policies in the field of education.

CodeFed: Federated Speech Recognition for Low-Resource Code-Switching Detection

One common constraint in the practical application of speech recognition is Code Switching. The issue of code-switched languages is especially aggravated in the context of Indian languages – since most massively multilingual models are trained on corpora that are not representative of the diverse set of Indian languages. An associated constraint with such systems is the privacy-intrusive nature of the applications that aim to collate such representative data. To collectively mitigate both problems, this work presents CodeFed: A federated learning-based code-switching detection model that can be deployed to collaboratively be trained by leveraging private data from multiple users, without compromising their privacy. Using a representative low-resource Indic dataset, we demonstrate the superior performance of a collaboratively trained global model that is trained using federated learning on three low-resource Indic languages – Gujarati, Tamil and Telugu and draw a comparison of the model with respect to the most current work in the field. Finally, to evaluate the practical realizability of the proposed system, CodeFed also discusses the system overview of the label generation architecture which may accompany CodeFed’s possible real-time deployment.

Are Emotions Conveyed Across Machine Translations? Establishing an Analytical Process for the Effectiveness of Multilingual Sentiment Analysis with Italian Text


 
 
 Abstract Natural language processing (NLP) is being widely used globally for a variety of value-creation tasks ranging from chat-bots and machine translations to sentiment and topic analysis and multilingual large language models (LLMs). However, most of the advances are initially implemented within the English language framework, and it takes time and resources to develop comparable resources in other languages. The advances in machine translations have enabled the rapid and effective conversion of content in global languages into English and vice-versa. This creates potential opportunities to apply English language NLP methods and tools to other languages via machine translations. However, although this idea is powerful, it needs to be validated and processes and best practices need to be developed and kept updated. The present research is an effort to contribute to the development of best practices and an evaluation framework. We present a systematic and repeatable state-of-the-art process to evaluate the viability of applying English language sentiment analysis tools to Italian text by using multiple English language machine translation mechanisms such that it can be easily extended to other languages.
 
 

Few-shot learning for automated content analysis: Efficient coding of arguments and claims in the debate on arms deliveries to Ukraine

Pre-trained language models (PLM) based on transformer neural networks developed in the field of natural language processing (NLP) offer great opportunities to improve automatic content analysis in communication science, especially for the coding of complex semantic categories in large datasets via supervised machine learning. However, three characteristics so far impeded the widespread adoption of the methods in the applying disciplines: the dominance of English language models in NLP research, the necessary computing resources, and the effort required to produce training data to fine-tune PLMs. In this study, we address these challenges by using a multilingual transformer model in combination with the adapter extension to transformers, and few-shot learning methods. We test our approach on a realistic use case from communication science to automatically detect claims and arguments together with their stance in the German news debate on arms deliveries to Ukraine. In three experiments, we evaluate (1) data preprocessing strategies and model variants for this task, (2) the performance of different few-shot learning methods, and (3) how well the best setup performs on varying training set sizes in terms of validity, reliability, replicability and reproducibility of the results. We find that our proposed combination of transformer adapters with pattern exploiting training provides a parameter-efficient and easily shareable alternative to fully fine-tuning PLMs. It performs on par in terms of validity, while overall, provides better properties for application in communication studies. The results also show that pre-fine-tuning for a task on a near-domain dataset leads to substantial improvement, in particular in the few-shot setting. Further, the results indicate that it is useful to bias the dataset away from the viewpoints of specific prominent individuals.

AGI-P: A Gender Identification Framework for Authorship Analysis Using Customized Fine-Tuning of Multilingual Language Model

AGI-P: A Gender Identification Framework for Authorship Analysis Using Customized Fine-Tuning of Multilingual Language Model

Terminology in the Age of AI: The Transformation of Terminology Theory and Practice

Abstract This article explores the necessary transformation of terminology theory and practice induced by advances in artificial intelligence (AI), in particular generative AI and large language models (LLMs). It underlines the centrality of terminology for accurate and efficient multilingual communication, and it studies the interaction between two different forms of language, the embodied language of humans and the algorithmic and mathematical language of AI. The analysis traces the evolution of terminology, from its normative, concept-centric origins to contemporary challenges such as the concept-context gap, multilingual conceptual modeling and enriching terminologies with contextual relations. The article then highlights how AI and LLM are revolutionizing terminology work, by proposing the construction of relations between terms around prototypical contexts. This innovative approach aims to infuse a more deterministic understanding into human-AI interactions, guiding the latter to generate responses more aligned with human intent and context. Advocating a re-evaluation of current methodologies to interact better with AI's algorithmic mathematical interpretation of natural language, the article envisions a future in which terminology science and AI co-create a framework for improving multilingual knowledge exchange and understanding.