Multilingual Sentence Research Articles

A data-driven approach to aligning academic offerings with industry needs for business and economy in Slovenia

ABSTRACT The success of graduates and their employability hinge on a crucial alignment between the skills and competencies sought by employers and those possessed by job seekers. This study delves into the extent to which the competencies cultivated in the curricula of selected Slovenian universities align with the ever-evolving requirements of the Slovenian labour market. Employing a combination of LLaMA-2-7B-chat, a generative language model, and the multilingual Sentence Transformer, a contextual language model, we identify skill gaps and pinpoint the most in-demand competencies that align with those generated by the faculty. This data-driven approach offers a roadmap for confirming the relevance and enhancing the university curricula, thus fostering a stronger connection between education and employment as well as improving the labour market efficiency.

Semantically Enriched Cross-Lingual Sentence Embeddings for Crisis-related Social Media Texts

Tasks such as semantic search and clustering on crisis-related social media texts enhance our comprehension of crisis discourse, aiding decision-making and targeted interventions. Pre-trained language models have advanced performance in crisis informatics, but their contextual embeddings lack semantic meaningfulness. Although the CrisisTransformers family includes a sentence encoder to address the semanticity issue, it remains monolingual, processing only English texts. Furthermore, employing separate models for different languages leads to embeddings in distinct vector spaces, introducing challenges when comparing semantic similarities between multi-lingual texts. Therefore, we propose multi-lingual sentence encoders (CT-XLMR-SE and CT-mBERT-SE) that embed crisis-related social media texts for over 50 languages, such that texts with similar meanings are in close proximity within the same vector space, irrespective of language diversity. Results in sentence encoding and sentence matching tasks are promising, suggesting these models could serve as robust baselines when embedding multi-lingual crisis-related social media texts. The models are publicly available at: https://huggingface.co/crisistransformers.

EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning

EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning

Examination of language, cognitive, and mathematical skills in childhood endocrine diseases.

Children diagnosed with endocrine disorders may exhibit atypical development and may encounter challenges in language, academic, and cognitive skills, as well as social-emotional issues. The objective of this study was to identify potential therapeutic requirements in the areas of language, cognition, and mathematical skills among children with endocrine disorders who experience school failure. This will enable an early evaluation of speech and language disorders and the planning of interventions to be possible. In this study, children with endocrine disorders were compared with their normally developing peers in terms of language, cognition, mathematical skills, and psychosocial characteristics. In this study, 15 children diagnosed with endocrine disorders (8 females, 7 males; mean age: 10, SD: 2) and 15 children with normal development (8 females, 7 males; mean age: 10, SD: 2) participated. The participants were subjected to the Test of Language Development-Primary: Fourth Edition Turkish Revision (TOLDP-4:T), the Turkish Nonword Repetition Test (TNRT), the Turkish Multilingual Sentence Repetition Test (LITMUS-TR), the Wechsler Intelligence Scale for Children (WISC-R), the Problem-Solving Test (PST), the Revised Child Anxiety and Depression Scale-Child Version (RCADS-CV), the Coopersmith Self-Esteem Inventory (CSEI), and the Child Behavior Checklist (CBCL). The findings of the study indicate that children with endocrine disease have lower performance in language, cognition, and mathematical skills compared to their healthy peers. Otherwise, they do not differ in terms of social-emotional status assessed by psychological scales. These findings suggest that while children with endocrine disorders face challenges in academic and cognitive domains, their social-emotional development remains relatively unaffected. Early identification and intervention in language, cognition, and mathematical skills may help address the academic struggles of these children, potentially improving their school performance and overall well-being.

Defining Semantically Close Words of Kazakh Language with Distributed System Apache Spark

This work focuses on determining semantically close words and using semantic similarity in general in order to improve performance in information retrieval tasks. The semantic similarity of words is an important task with many applications from information retrieval to spell checking or even document clustering and classification. Although, in languages with rich linguistic resources, the methods and tools for this task are well established, some languages do not have such tools. The first step in our experiment is to represent the words in a collection in a vector form and then define the semantic similarity of the terms using a vector similarity method. In order to tame the complexity of the task, which relies on the number of word (and, consequently, of the vector) pairs that have to be combined in order to define the semantically closest word pairs, A distributed method that runs on Apache Spark is designed to reduce the calculation time by running comparison tasks in parallel. Three alternative implementations are proposed and tested using a list of target words and seeking the most semantically similar words from a lexicon for each one of them. In a second step, we employ pre-trained multilingual sentence transformers to capture the content semantics at a sentence level and a vector-based semantic index to accelerate the searches. The code is written in MapReduce, and the experiments and results show that the proposed methods can provide an interesting solution for finding similar words or texts in the Kazakh language.

Cross-Lingual Classification of Political Texts Using Multilingual Sentence Embeddings

AbstractEstablished approaches to analyze multilingual text corpora require either a duplication of analysts’ efforts or high-quality machine translation (MT). In this paper, I argue that multilingual sentence embedding (MSE) is an attractive alternative approach to language-independent text representation. To support this argument, I evaluate MSE for cross-lingual supervised text classification. Specifically, I assess how reliably MSE-based classifiers detect manifesto sentences’ topics and positions compared to classifiers trained using bag-of-words representations of machine-translated texts, and how this depends on the amount of training data. These analyses show that when training data are relatively scarce (e.g., 20K or less-labeled sentences), MSE-based classifiers can be more reliable and are at least no less reliable than their MT-based counterparts. Furthermore, I examine how reliable MSE-based classifiers label sentences written in languages not in the training data, focusing on the task of discriminating sentences that discuss the issue of immigration from those that do not. This analysis shows that compared to the within-language classification benchmark, such “cross-lingual transfer” tends to result in fewer reliability losses when relying on the MSE instead of the MT approach. This study thus presents an important addition to the cross-lingual text analysis toolkit.

Investigating Language Relationships in Multilingual Sentence Encoders Through the Lens of Linguistic Typology

Abstract Multilingual sentence encoders have seen much success in cross-lingual model transfer for downstream NLP tasks. The success of this transfer is, however, dependent on the model’s ability to encode the patterns of cross-lingual similarity and variation. Yet, we know relatively little about the properties of individual languages or the general patterns of linguistic variation that the models encode. In this article, we investigate these questions by leveraging knowledge from the field of linguistic typology, which studies and documents structural and semantic variation across languages. We propose methods for separating language-specific subspaces within state-of-the-art multilingual sentence encoders (LASER, M-BERT, XLM, and XLM-R) with respect to a range of typological properties pertaining to lexical, morphological, and syntactic structure. Moreover, we investigate how typological information about languages is distributed across all layers of the models. Our results show interesting differences in encoding linguistic variation associated with different pretraining strategies. In addition, we propose a simple method to study how shared typological properties of languages are encoded in two state-of-the-art multilingual models—M-BERT and XLM-R. The results provide insight into their information-sharing mechanisms and suggest that these linguistic properties are encoded jointly across typologically similar languages in these models.

The news media and its audience: Agenda setting on organic food in the United States and Germany

What are the agenda-setting effects between the news media and its audience regarding organic food? This longitudinal text-mining study investigates the relationship between topics mentioned in news articles and reader comments published the online news outlets nytimes.com (USA) and spiegel.de (Germany) from 2007 to 2020. Topics are modeled using a neural network approach based on clustered multilingual sentence embeddings. Results show that the salience of topics in news articles significantly influences their salience in reader comments but not vice versa. Metrics for agenda distance and agenda diversity confirm the media’s agenda-setting role and additionally point out periods of time when events caused the media and public attention to diverge. The news media drives public opinion on organic food in the US and Germany by determining the discussion topics and is thus an important player in the promotion of organic food consumption to be considered by marketers and policy makers.

Gated recurrent unit with multilingual universal sentence encoder for Arabic aspect-based sentiment analysis

The increasing interactive content in the Internet motivated researchers and data scientists to conduct Aspect-Based Sentiment Analysis (ABSA) research to understand the various sentiments and the different aspects of a product in a single user’s comment. Determining the various aspects along with their polarities (positive, negative, or neutral) from a single comment is a challenging problem. To this end, we have designed and developed a deep learning model based on Gated Recurrent Units (GRU) and features extracted using the Multilingual Universal Sentence Encoder (MUSE). The proposed Pooled-GRU model trained on a Hotels’ Arabic reviews to address two ABSA tasks: (1) aspect extraction, and (2) aspect polarity classification. The proposed model achieved high results with 93.0% F1 score in the former task and 90.86% F1 score in the latter task. Our experimental results show that our proposed model outperforms the baseline model and the related research methods evaluated on the same dataset. More precisely, our proposed model showed 62.1% improvement in the F1 score over the baseline model for the aspect extraction task and 15% improvement in the accuracy over the baseline model for the aspect polarity classification task.

Preliminary Findings on a Newly Developed Malaysian Multilingual Sentence Repetition Task for Multilingual Chinese Children in Malaysia

INTRODUCTION: Even though multilingualism is the basic fabric of linguistic diversity of mainstream communities in Malaysia, there is generally a lack of assessment tool to assess the language abilities of multilingual children at risk of language impairment. This study is aimed at developing a Malaysian Multilingual Sentence Repetition (MMSR) task which can be used to assess the morphosyntactic abilities of multilingual Chinese children speaking three languages i.e., English-Mandarin-Malay. MATERIALS AND METHOD: 10 typically developing Chinese children within the ages of 4 years until 6 years and 11 months old were recruited. Each child was presented with sentences auditorily using a laptop with a microphone headset. The child was then asked to repeat verbatim the sentences they just heard. RESULTS: Findings showed that the subjects performed equally well in English and Mandarin but lower in Malay (p < 0.01). Correlation analyses revealed that the older children performed better for English (p < 0.05) and Mandarin (p < 0.01) sentence repetitions (SRs). Performance on SR tasks decreased with the increase in sentence length and complexity. CONCLUSION: The newly developed multilingual SR task proved reliable and valid as a tool and has the potential to be developed into a standardized test to assess multilingual Malaysian children at risk of language impairment.

Embeddings-Based Clustering for Target Specific Stances: The Case of a Polarized Turkey

On June 24, 2018, Turkey conducted a highly consequential election in which the Turkish people elected their president and parliament in the first election under a new presidential system. During the election period, the Turkish people extensively shared their political opinions on Twitter. One aspect of polarization among the electorate was support for or opposition to the reelection of Recep Tayyip Erdoğan. In this paper, we present an unsupervised method for target-specific stance detection in a polarized setting, specifically Turkish politics, achieving 90% precision in identifying user stances, while maintaining more than 80% recall. The method involves representing users in an embedding space using Google's Convolutional Neural Network (CNN) based multilingual universal sentence encoder. The representations are then projected onto a lower dimensional space in a manner that reflects similarities and are consequently clustered. We show the effectiveness of our method in properly clustering users of divergent groups across multiple targets that include political figures, different groups, and parties. We perform our analysis on a large dataset of 108M Turkish election-related tweets along with the timeline tweets of 168k Turkish users, who authored 213M tweets. Given the resultant user stances, we are able to observe correlations between topics and compute topic polarization.

AraCap: A hybrid deep learning architecture for Arabic Image Captioning

Automatic captioning of images no only enrich multimedia content with descriptive features, but also helps in detecting patterns, trends, and events of interest. Particularly, Arabic Image Caption Generation is a very challenging topic in the machine learning field. This paper presents, AraCap, a hybrid object-based, attention-enriched image captioning architecture, with a focus on Arabic language. Three models are demonstrated, all of them are implemented and trained on COCO and Flickr30k datasets, and then tested by building an Arabic version of a subset of COCO dataset. The first model is an object-based captioner that can handle one or multiple detected objects. The second is a combined pipeline that uses both object detector and attention-based captioning; while the third one is based on a pure soft attention mechanism. The models are evaluated using multi-lingual semantic sentence similarity techniques to assess the generated captions accuracy against the actual ground truth captions. Results show that similarity scores for Arabic generated captions from all three proposed models outperformed the basic captioning technique.

Monolingual and multilingual topic analysis using LDA and BERT embeddings

Analyzing research topics offers potential insights into the direction of scientific development. In particular, analyzing multilingual research topics can help researchers grasp the evolution of topics globally, revealing topic similarity among scientific publications written in different languages. Most studies to date on topic analysis have been based on English-language publications and have relied heavily on citation-based topic evolution analysis. However, since it can be challenging for English publications to cite non-English sources and since many languages do not offer English translations of abstracts, citation-based methodologies are not suitable for analyzing multilingual research topic relations. Since multilingual sentence embeddings can effectively preserve word semantics in multilingual translation tasks, a topic model based on multilingual sentence embeddings could potentially generate topic–word distributions for publications in multilingual analysis. In this paper, which is situated in the field of library and information science, we use multilingual pretrained Bidirectional Encoder Representations from Transformers (BERT) embeddings and the Latent Dirichlet Allocation (LDA) topic model to analyze topic evolution in monolingual and multilingual topic similarity settings. For each topic, we multiply its LDA probability value by the averaged tensor similarity of BERT embeddings to explore the evolution of the topic in scientific publications. As our proposed method does not rely on a machine translator or the author's subjective translation, it avoids confusion and misusages caused by either machine error or the author's subjectively chosen English keywords. Our results show that the proposed approach is well-suited to analyzing the scientific evolutions in monolingual and scientific multilingual topic similarity relations.

A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings

The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.

Emu: Enhancing Multilingual Sentence Embeddings with Semantic Specialization

We present Emu, a system that semantically enhances multilingual sentence embeddings. Our framework fine-tunes pre-trained multilingual sentence embeddings using two main components: a semantic classifier and a language discriminator. The semantic classifier improves the semantic similarity of related sentences, whereas the language discriminator enhances the multilinguality of the embeddings via multilingual adversarial training. Our experimental results based on several language pairs show that our specialized embeddings outperform the state-of-the-art multilingual sentence embedding model on the task of cross-lingual intent classification using only monolingual labeled data.

Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond

We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared byte-pair encoding vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI data set), cross-lingual document classification (MLDoc data set), and parallel corpus mining (BUCC data set) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low- resource languages. Our implementation, the pre-trained encoder, and the multilingual test set are available at https://github.com/facebookresearch/LASER .

Analysis of Joint Multilingual Sentence Representations and Semantic K-Nearest Neighbor Graphs

Multilingual sentence and document representations are becoming increasingly important. We build on recent advances in multilingual sentence encoders, with a focus on efficiency and large-scale applicability. Specifically, we construct and investigate the k-nn graph over the joint space of 566 million news sentences in seven different languages. We show excellent multilingual retrieval quality on the UN corpus of 11.3M sentences, which extends to the zero-shot case where we have never seen a language. We provide a detailed analysis of both the multilingual sentence encoder for twenty-one European languages and the learned graph. Our sentence encoder is language agnostic and supports code switching.

Tagging Urdu Sentences from English POS Taggers

Being a global language, English has attracted a majority of researchers and academia to work on several Natural Language Processing (NLP) applications. The rest of the languages are not focused as much as English. Part-of-speech (POS) Tagging is a necessary component for several NLP applications. An accurate POS Tagger for a particular language is not easy to construct due to the diversity of that language. The global language English, POS Taggers are more focused and widely used by the researchers and academia for NLP processing. In this paper, an idea of reusing English POS Taggers for tagging non-English sentences is proposed. On exemplary basis, Urdu sentences are processed to tagged from 11 famous English POS Taggers. State-of-the-art English POS Taggers were explored from the literature, however, 11 famous POS Taggers were being input to Urdu sentences for tagging. A famous Google translator is used to translate the sentences across the languages. Data from twitter.com is extracted for evaluation perspective. Confusion matrix with kappa statistic is used to measure the accuracy of actual Vs predicted tagging. The two best English POS Taggers which tagged Urdu sentences were Stanford POS Tagger and MBSP POS Tagger with an accuracy of 96.4% and 95.7%, respectively. The system can be generalized for multi-lingual sentence tagging.

Adaptive HMM based Speech Recognition to Recognize Multi-lingual Sentence

Hidden Markov Models (HMMs) provides an effective framework for the modeling of time-varying sequence of spectral vector. An approach of mapping signal to discrete signal is to define it as a set of acoustic featured symbol over a minimal but constant time interval. The aim of proposing this paper is to recognize the speech sample using hidden markov model (HMM) with the use of cepstrum feature of our given speech sample within an adaptive interval of time for which pitch period is determined and divides the sample in accordance with this period. Secondly the phonetics or exact “pronunciation” of the word needs to be defined. These are established by associated rule probability where probability is done on word‟s pronunciation. General Terms Multilingual Speech Recognition

ISentenizer-μ: multilingual sentence boundary detection model.

Sentence boundary detection (SBD) system is normally quite sensitive to genres of data that the system is trained on. The genres of data are often referred to the shifts of text topics and new languages domains. Although new detection models can be retrained for different languages or new text genres, previous model has to be thrown away and the creation process has to be restarted from scratch. In this paper, we present a multilingual sentence boundary detection system (iSentenizer-μ) for Danish, German, English, Spanish, Dutch, French, Italian, Portuguese, Greek, Finnish, and Swedish languages. The proposed system is able to detect the sentence boundaries of a mixture of different text genres and languages with high accuracy. We employ i +Learning algorithm, an incremental tree learning architecture, for constructing the system. iSentenizer-μ, under the incremental learning framework, is adaptable to text of different topics and Roman-alphabet languages, by merging new data into existing model to learn the new knowledge incrementally by revision instead of retraining. The system has been extensively evaluated on different languages and text genres and has been compared against two state-of-the-art SBD systems, Punkt and MaxEnt. The experimental results show that the proposed system outperforms the other systems on all datasets.