Bilingual Word Embeddings Research Articles

Bilingual word embedding fusion for robust unsupervised bilingual lexicon induction

Great progress has been made in unsupervised bilingual lexicon induction (UBLI) by aligning the source and target word embeddings independently trained on monolingual corpora. The common assumption of most UBLI models is that the embedding spaces of two languages are approximately isomorphic (i.e., similar in geometric structure). Therefore, the performance is bound by the degree of isomorphism, especially on etymologically and typologically distant languages. Near-zero UBLI results have been reported for them. To address this problem, we propose a method to increase the isomorphism based on bilingual word embedding fusion. In particular, the features from the source embeddings are integrated into the target embeddings, and vice versa. Therefore, the resulting structures of source and target embeddings are similar to each other. The method does not require any form of supervision and can be applied to any language pair. On a benchmark dataset of bilingual lexicon induction, our approach can achieve competitive or superior performance compared to the state-of-the-art methods, with particularly strong results being found on distant languages.

Learning bilingual word embedding for automatic text summarization in low resource language

Studies in low-resource languages have become more challenging with the increasing volume of texts in today's digital era. Also, the lack of labeled data and text processing libraries in a language further widens the research gap between high and low-resource languages, such as English and Indonesian. This has led to the use of a transfer learning approach, which applies pre-trained models to solve similar problems, even in different languages by using bilingual or cross-lingual word embedding. Therefore, this study aims to investigate two bilingual word embedding methods, namely VecMap and BiVec, for Indonesian – English language and evaluates them for bilingual lexicon induction and text summarization tasks. The generated bilingual embedding was compared with MUSE (Multilingual Unsupervised and Supervised Embeddings) as the existing multilingual word created with the generative adversarial network method. Furthermore, the VecMap was improved by creating shared vocabulary spaces and mapping the unshared ones between languages. The result showed the embedding produced by the joint methods of BiVec, performed better in intrinsic evaluation, especially with CSLS (Cross-Domain Similarity Local Scaling) retrieval. Meanwhile, the improved VecMap outperformed the regular type by 16.6% without surpassing the BiVec evaluation score. These methods enabled model transfer between languages when applied to cross-lingual-based text summarization. Moreover, the ROUGE score outperformed classical text summarization by adding only 10% of the training dataset of the target language.

Unsupervised Parallel Sentences of Machine Translation for Asian Language Pairs

Parallel sentence pairs play a very important role in many natural language processing tasks, especially cross-lingual tasks such as machine translation. So far, many Asian language pairs lack bilingual parallel sentences. As collecting bilingual parallel data is very time-consuming and difficult, it is very important for many low-resource Asian language pairs. While existing methods have shown encouraging results, they rely on bilingual data seriously or have some drawbacks in an unsupervised situation. To address these issues, we propose a new unsupervised similarity calculation and dynamic selection metric to obtain parallel sentence pairs in an unsupervised situation. First, our method maps bilingual word embedding by postdoc adversarial training, which rotates the source space to match the target without parallel data. Then, we introduce a new cross-domain similarity adaption to obtain parallel sentence pairs. Experimental results on real-world datasets show that our model can obtain better accuracy and recall on mining parallel sentence pairs. We also show that the extracted bilingual sentence corpora can significantly improve the performance of neural machine translation.

Unsupervised English Intelligent Machine Translation in Wireless Network Environment

Researchers suggest unsupervised English machine translation to address the absence of parallel corpus in English translation. Unsupervised pretraining techniques, denoising autoencoders, back translation, and shared latent representation mechanisms are used to simulate the translation task using just monolingual corpora. This paper uses pseudo-parallel data to construct unsupervised neural machine translation (NMT) and dissimilar language pair analysis. This paper firstly analyzes the low performance of unsupervised translation on dissimilar language pairs from three aspects: bilingual word embedding quality, shared words, and word order. And artificial shared word replacement and preordering strategies are proposed to increase the shared words between dissimilar language pairs and reduce the difference in their syntactic structure, thereby improving the translation performance on dissimilar language pairs. The denoising autoencoder and shared latent representation mechanism in unsupervised English machine translation are only required in the early stage of training, and learning the shared latent representation limits the further improvement of performance in different directions. While training the denoising autoencoder by repeatedly altering the training data slows down the convergence of the model, this is especially true for divergent languages. This paper presents an unsupervised NMT model based on pseudo-parallel data to address this issue. It trains two standard supervised neural machine translation models using the pseudo-parallel corpus generated by the unsupervised neural machine translation system, which enhances translation performance and speeds convergence. Finally, the English intelligent translation model is deployed in the wireless network server, and users can access it through the wireless network.

Cross-Linguistic Similarity Evaluation Techniques Based on Deep Learning

For the cross-linguistic similarity problem, a twin network model with ordered neuron long- and short-term memory neural network as a subnet is proposed. The model fuses bilingual word embeddings and encodes the representation of input sequences by ordered neuron long- and short-term memory neural networks. Based on this, the distributed semantic vector representation of the sentences is jointly constructed by using the global modelling capability of the fully connected network for higher-order semantic extraction. The final output part is the similarity of the bilingual sentences and is optimized by optimizing the parameters of each layer in the framework. Multiple experiments on the dataset show that the model achieves 81.05% accuracy, which effectively improves the accuracy of text similarity and converges faster and improves the semantic text analysis to some extent.

Learning Bilingual Word Embedding Mappings with Similar Words in Related Languages Using GAN

ABSTRACT Cross-lingual word embeddings display words from different languages in the same vector space. They provide reasoning about semantics, compare the meaning of words across languages and word meaning in multilingual contexts, necessary to bilingual lexicon induction, machine translation, and cross-lingual information retrieval. This paper proposes an efficient approach to learn bilingual transform mapping between monolingual word embeddings in language pairs. We choose ten different languages from three different language families and downloaded their last update Wikipedia dumps1 1. https://dumps.wikimedia.org. Then, with some pre-processing steps and using word2vec, we produce word embeddings for them. We select seven language pairs from chosen languages. Since the selected languages are relative, they have thousands of identical words with similar meanings. With these identical dictation words and word embedding models of each language, we create training, validation and, test sets for the language pairs. We then use a generative adversarial network (GAN) to learn the transform mapping between word embeddings of source and target languages. The average accuracy of our proposed method in all language pairs is 71.34%. The highest accuracy is achieved for the Turkish-Azerbaijani language pair with the accuracy 78.32%., which is noticeably higher than prior methods.

Unsupervised word translation for English-Hindi with different retrieval techniques

Word translation or incorporation of bilingual dictionaries is an important capability that impacts many multilingual language processing tasks. For translation from one language to another language, we either relied on parallel data or bilingual dictionaries. In this paper, we address this problem and generate best cross-lingual word embedding for English-Hindi language pair. Here, we neither use an aligned document or sentence aligned corpus, nor any bilingual dictionary. We are following the assumption of intra lingual similarity distribution that for the most frequent word the distribution graph is similar between Hindi and English corpus and embeddings are isometric. These cross-lingual words embedding can be used for unsupervised neural machine translation and cross-lingual transfer learning. Different retrieval techniques nearest neighbour, inverted nearest neighbours retrieval, inverted Softmax, and cross-lingual word scaling are performed and compared for the bi-lingual embedding of English-Hindi, which is trained for unsupervised and semi-supervised ways by passing seed dictionary. Bi-lingual word embedding is tested on generated English-Hindi dictionary.

Jointly learning bilingual word embeddings and alignments

Jointly learning bilingual word embeddings and alignments

Leveraging bilingual-view parallel translation for code-switched emotion detection with adversarial dual-channel encoder

Code-switched emotion detection, a task analyzing the emotion in code-switched texts, has gain increasing research attention within recent years. Prior works utilize various neural models with sophisticated features to pursuit high performance of the task, while they still overlook some crucial characteristics of the code-switched texts. In this work, we present an innovative approach for improving code-switched emotion detection. We first consider a bilingual-view parallel translation for code-switched text enhancement, i.e., translating the code-switched texts into two languages. Then we propose an adversarial dual-channel encoder architecture, where two private encoders take as inputs the parallel texts in two languages, respectively. The private encoders and the shared encoder work collaboratively, and effectively retrieve the features from monolingual and bilingual perspectives under adversarial training. We conduct extensive experiments on five code-switched benchmark datasets. Results show that our model outperforms the strongly-performing baselines that leverage external code-switched or bilingual word embedding with over 1.5% F1 score on the Chinese–English, Spanish–English and Hindi–English code-mixed data, becoming the new state-of-the-art system. Further analyses including ablation, qualitative and error studies, demonstrate the effectiveness of our proposed encoder for code-switched texts, as well as the bilingual-view parallel translation strategy.

Document-Level Machine Translation – Ensuring Translational Consistency of Non-Local Phenomena

In this thesis, we study the automatic translation of documents by taking into account cross-sentence phenomena. This document-level information is typically ignored by most of the standard state-of-the-art Machine Translation (MT) systems, which focus on translating texts processing each of their sentences in isolation. Translating each sentence without looking at its surrounding context can lead to certain types of translation errors, such as inconsistent translations for the same word or for elements in a coreference chain. We introduce methods to attend to document-level phenomena in order to avoid those errors, and thus, reach translations that properly convey the original meaning. Our research starts by identifying the translation errors related to such document-level phenomena that commonly appear in the output of state-of-the-art Statistical Machine Translation (SMT) systems. For two of those errors, namely inconsistent word translations as well as gender and number disagreements among words, we design simple and yet effective post-processing techniques to tackle and correct them. Since these techniques are applied a posteriori, they can access the whole source and target documents, and hence, they are able to perform a global analysis and improve the coherence and consistency of the translation. Nevertheless, since following such a two-pass decoding strategy is not optimal in terms of efficiency, we also focus on introducing the context-awareness during the decoding process itself. To this end, we enhance a document-oriented SMT system with distributional semantic information in the form of bilingual and monolingual word embeddings. In particular, these embeddings are used as Semantic Space Language Models (SSLMs) and as a novel feature function. The goal of the former is to promote word translations that are semantically close to their preceding context, whereas the latter promotes the lexical choice that is closest to its surrounding context, for those words that have varying translations throughout the document. In both cases, the context extends beyond sentence boundaries. Recently, the MT community has transitioned to the neural paradigm. The finalstep of our research proposes an extension of the decoding process for a Neural Machine Translation (NMT) framework, independent of the model architecture, by shallow fusing the information from a neural translation model and the context semantics enclosed in the previously studied SSLMs. The aim of this modification is to introduce the benefits of context information also into the decoding process of NMT systems, as well as to obtain an additional validation for the techniques we explored. The automatic evaluation of our approaches does not reflect significant variations. This is expected since most automatic metrics are neither context- nor semantic-aware and because the phenomena we tackle are rare, leading to few modifications with respect to the baseline translations. On the other hand, manual evaluations demonstrate the positive impact of our approaches since human evaluators tend to prefer the translations produced by our document-aware systems. Therefore, the changes introduced by our enhanced systems are important since they are related to how humans perceive translation quality for long texts.

Learning to Translate Kannada and English Queries for Mixed Script Information Retrieval

Due to increase in the availability of numerous languages in the Web, cross language information retrieval is one of the happening issues in the field of natural language processing and information retrieval. Nowadays, people are habituated to combine two or more language words during oral or written discourse. Speakers have also employed intermixing of different languages and scripts in digital media while querying, blogging and on social media platforms. The way of representing two different language words of an utterance in their native scripts is known as mixed scripting. In the present work, we attempted to translate mixed script queries of Kannada and English languages into monolingual queries. We proposed three approaches for translation by constructing bilingual dictionary, word embeddings and Google translate. The proposed method outperforms the conventional dictionary based approach, when word embeddings were combined with the translations learnt from Google Translate and Dictionary.

A teacher-student model with the bilingual word embedding for zero-shot machine translation

Neural machine translation has achieved state-of-art performance with sufficient data, but it still suffers from the data scarcity problem for low-resource language pairs. Teacher-student model that transfers knowledge from a pivot→target model to a source→target model with the source↔pivot parallel data has shown its effectiveness. In this paper, we improve the model with the bilingual word embedding and transformer architecture. Experiments are carried out on three commonly used translation datasets and the result shows the improvement over a baseline teacher-student model.

Learning Japanese-English Bilingual Word Embeddings by Using Language Specificity

Cross-lingual word embeddings have been gaining attention because they can capture the semantic meaning of words across languages, which can be applied to cross-lingual tasks. Most methods learn a single mapping (e.g., a linear mapping) to transform a word embedding space from one language to another. To improve bilingual word embeddings, we propose an advanced method that adds a language-specific mapping. We focus on learning Japanese-English bilingual word embedding mapping by considering the specificity of the Japanese language. We evaluated our method by comparing it with single mapping-based-models on bilingual lexicon induction between Japanese and English. We determined that our method was more effective, with significant improvements on words of Japanese origin.

Unsupervised Translation Quality Estimation for Digital Entertainment Content Subtitles

We demonstrate the potential for using aligned bilingual word embeddings in developing an unsupervised method to evaluate machine translations without a need for parallel translation corpus or reference corpus. We explain different aspects of digital entertainment content subtitles. We share our experimental results for four languages pairs English to French, German, Portuguese, Spanish, and present findings on the shortcomings of Neural Machine Translation for subtitles. We propose several improvements over the system designed by Gupta et al. [P. Gupta, S. Shekhawat and K. Kumar, Unsupervised quality estimation without reference corpus for subtitle machine translation using word embeddings, IEEE 13th Int. Conf. Semantic Computing, 2019, pp. 32–38.] by incorporating custom embedding model curated to subtitles, compound word splits and punctuation inclusion. We show a massive run time improvement of the order of [Formula: see text] by considering three types of edits, removing Proximity Intensity Index (PII) and changing post-edit score calculation from their system.

Tibetan-Chinese cross-lingual word embeddings based on MUSE

The idea of word embedding is based on the semantic distribution hypothesis of linguist Harris (1954), who believes that words with the same semantics are distributed in similar contexts. The learning of word embedding is a crucial technology in natural language processing. In recent years, cross-language word vectors have received more and more attention. Cross-language Word vectors can transfer knowledge between different languages. Most importantly, this transfer can occur between rich-resource and low-resource languages. This paper uses the Tibetan-Chinese Wikipedia corpus to train monolingual word vectors. Based on the Tibetan-Chinese bilingual translations, we use the supervised method in the MUSE library to train the Tibetan-Chinese bilingual cross-language word embeddings. In the experiment, we evaluate the result of word representation on the standard lexical semantic evaluation task. The results show that the method has a certain improvement in the semantic representation of the word embedding.

Unsupervised Neural Machine Translation With Cross-Lingual Language Representation Agreement

Unsupervised cross-lingual language representation initialization methods such as unsupervised bilingual word embedding (UBWE) pre-training and cross-lingual masked language model (CMLM) pre-training, together with mechanisms such as denoising and back-translation, have advanced unsupervised neural machine translation (UNMT), which has achieved impressive results on several language pairs, particularly French-English and German-English. Typically, UBWE focuses on initializing the word embedding layer in the encoder and decoder of UNMT, whereas the CMLM focuses on initializing the entire encoder and decoder of UNMT. However, UBWE/CMLM training and UNMT training are independent, which makes it difficult to assess how the quality of UBWE/CMLM affects the performance of UNMT during UNMT training. In this paper, we first empirically explore relationships between UNMT and UBWE/CMLM. The empirical results demonstrate that the performance of UBWE and CMLM has a significant influence on the performance of UNMT. Motivated by this, we propose a novel UNMT structure with cross-lingual language representation agreement to capture the interaction between UBWE/CMLM and UNMT during UNMT training. Experimental results on several language pairs demonstrate that the proposed UNMT models improve significantly over the corresponding state-of-the-art UNMT baselines.

Improving Neural Machine Translation by Filtering Synthetic Parallel Data

Synthetic data has been shown to be effective in training state-of-the-art neural machine translation (NMT) systems. Because the synthetic data is often generated by back-translating monolingual data from the target language into the source language, it potentially contains a lot of noise—weakly paired sentences or translation errors. In this paper, we propose a novel approach to filter this noise from synthetic data. For each sentence pair of the synthetic data, we compute a semantic similarity score using bilingual word embeddings. By selecting sentence pairs according to these scores, we obtain better synthetic parallel data. Experimental results on the IWSLT 2017 Korean→English translation task show that despite using much less data, our method outperforms the baseline NMT system with back-translation by up to 0.72 and 0.62 Bleu points for tst2016 and tst2017, respectively.

Using Communities of Words Derived from Multilingual Word Vectors for Cross-Language Information Retrieval in Indian Languages

We investigate the use of word embeddings for query translation to improve precision in cross-language information retrieval (CLIR). Word vectors represent words in a distributional space such that syntactically or semantically similar words are close to each other in this space. Multilingual word embeddings are constructed in such a way that similar words across languages have similar vector representations. We explore the effective use of bilingual and multilingual word embeddings learned from comparable corpora of Indic languages to the task of CLIR. We propose a clustering method based on the multilingual word vectors to group similar words across languages. For this we construct a graph with words from multiple languages as nodes and with edges connecting words with similar vectors. We use the Louvain method for community detection to find communities in this graph. We show that choosing target language words as query translations from the clusters or communities containing the query terms helps in improving CLIR. We also find that better-quality query translations are obtained when words from more languages are used to do the clustering even when the additional languages are neither the source nor the target languages. This is probably because having more similar words across multiple languages helps define well-defined dense subclusters that help us obtain precise query translations. In this article, we demonstrate the use of multilingual word embeddings and word clusters for CLIR involving Indic languages. We also make available a tool for obtaining related words and the visualizations of the multilingual word vectors for English, Hindi, Bengali, Marathi, Gujarati, and Tamil.

Improving Word Embedding Coverage in Less-Resourced Languages Through Multi-Linguality and Cross-Linguality

In the era of deep learning-based systems, efficient input representation is one of the primary requisites in solving various problems related to Natural Language Processing (NLP), data mining, text mining, and the like. Absence of adequate representation for an input introduces the problem of data sparsity, and it poses a great challenge to solve the underlying problem. The problem is more intensified with resource-poor languages due to the absence of a sufficiently large corpus required to train a word embedding model. In this work, we propose an effective method to improve the word embedding coverage in less-resourced languages by leveraging bilingual word embeddings learned from different corpora. We train and evaluate deep Long Short Term Memory (LSTM)-based architecture and show the effectiveness of the proposed approach for two aspect-level sentiment analysis tasks (i.e., aspect term extraction and sentiment classification). The neural network architecture is further assisted by hand-crafted features for prediction. We apply the proposed model in two experimental setups: multi-lingual and cross-lingual. Experimental results show the effectiveness of the proposed approach against the state-of-the-art methods.

Exploring Implicit Semantic Constraints for Bilingual Word Embeddings

Bilingual word embeddings (BWEs) have proven to be useful in many cross-lingual natural language processing tasks. Previous studies often require bilingual texts or dictionaries that are scarce res...