English-Hindi Language Pair Research Articles

NLP-reliant Neural Machine Translation techniques used in smart city applications

For smart city applications, Neural Machine Translation (NMT) methods based on Natural Language Processing (NLP) are crucial as they facilitate information sharing and communication among diverse populations. NLP techniques are used in many domains related to smart cities, such as development and research, business, industries, media, healthcare, and residences and communities. The majority of people in India communicate using their regional languages. The majority of applications used by users in smart cities will mostly accept English as input. These people will be able to interact with these smart city devices in their native tongues more effectively with the help of effective machine translation. Just 10% of Indians use English as their primary language of communication; there are 22 official regional languages in India. So, there is requirement of better machine translation using Natural language processing (NLP). Natural language processing for Indian regional languages has a very long way to go until it surpassing the abilities of existing rich NLP applications and techniques for English language. Machine Translation is technique of Natural Language Processing (NLP) which provides better inter-lingual communication. For low resourced Indian languages effective machine translation systems became important for establishing proper communication. Machine Transliteration is a technique to convert source language into target language using machine. The developed system takes English language as input and then applies machine translation techniques to translate the source language into multiple languages using trained RNN model and multi-lingual search model which search the input word across all the datasets and generate the output into other Indian languages such as Hindi, Tamil. Our approach achieves top performance for English-Hindi language pair and comparable results for other cases.

Do cues in a video help in handling rare words in a machine translation system under a low-resource setting?

Video modality is an emerging research area among the numerous modalities utilized for multimodal machine translation. Multimodal machine translation uses multiple modalities to improve the machine-translated target language from the source language. However, the currently available multimodal dataset is focused on a few well-studied languages. In this paper, we propose a video-guided multimodal machine translation (VMMT) model under a low-resource setting by building a synthetic multimodal dataset of the English-Hindi language pair, the first one of its kind for this language pair. The VMMT system employs spatio-temporal video context as an additional input modality along with the source text. The spatio-temporal video context is extracted using a pre-trained 3D convolutional neural network. We report how well the VMMT systems outperform the text-only neural machine translation (NMT) system using automatic evaluation metrics and human evaluation on two test datasets: one in-domain and another out-domain. Our results indicate that the use of video context as an additional input modality enhances the performance of the MT system in resolving the various MT challenges, such as handling rare words, ambiguity, etc., in both English→Hindi and Hindi→English translations. Our experimental results show a significant improvement of up to +4.2 BLEU and +0.07 chrF scores in English→Hindi and +5.4 BLEU and +0.07 chrF scores in Hindi→English with our VMMT system over unimodal NMT system. Our findings highlight the potential of visual cues as an additional modality for improving machine translation systems especially in low-resource settings and emphasize the importance of synthetic multimodal datasets in addressing the scarcity of diverse data for less-studied language pairs.

Unsupervised word translation for English-Hindi with different retrieval techniques

Word translation or incorporation of bilingual dictionaries is an important capability that impacts many multilingual language processing tasks. For translation from one language to another language, we either relied on parallel data or bilingual dictionaries. In this paper, we address this problem and generate best cross-lingual word embedding for English-Hindi language pair. Here, we neither use an aligned document or sentence aligned corpus, nor any bilingual dictionary. We are following the assumption of intra lingual similarity distribution that for the most frequent word the distribution graph is similar between Hindi and English corpus and embeddings are isometric. These cross-lingual words embedding can be used for unsupervised neural machine translation and cross-lingual transfer learning. Different retrieval techniques nearest neighbour, inverted nearest neighbours retrieval, inverted Softmax, and cross-lingual word scaling are performed and compared for the bi-lingual embedding of English-Hindi, which is trained for unsupervised and semi-supervised ways by passing seed dictionary. Bi-lingual word embedding is tested on generated English-Hindi dictionary.

Context-based Machine Translation of English-Hindi using CE-Encoder

The difficulty in obtaining accurate word alignment and determining a target word that is the best candidate for a source context in machine translation leads to different translations. In this study, we propose a method with a more accurate context model. Our Neural Machine Translation (NMT) approach focuses on the encoder to apprehend the meaning of source sentences for improved translation. The recurrent encoder works by taking into consideration the history and future information of the source context. In this study, we implement the proposed approach into three steps. Firstly, we learn the representation of future context in advance. Secondly, a context-based recurrent encoder called as CE-Encoder with two-level Gated Recurrent Unit (GRU) is used. In this, the bottom-level GRU gathers history data of a sentence and top-level GRU assembles future data information. Finally, the future learned context and the history information from the opposite direction is integrated. The distinguishing factor of the proposed framework from the existing models, specifically Bidirectional Recurrent Neural Network (BiRNN) is that, the current models have not spent substantial time and capacity in learning future context or disambiguating source and target words based on the context which is defined by source sentence. We conduct experiments on the datasets from ILCC and CFILT for the English-Hindi language pair. From the comparative evaluation, we observed that the proposed model outperforms the Bidirectional RNN encoder in terms of translation quality. The proposed model has shown the improvement of 7 Bleu points using the ILCC dataset and 9 points using the CFILT dataset over BiRNN.

Source-side Reordering to Improve Machine Translation between Languages with Distinct Word Orders

English and Hindi have significantly different word orders. English follows the subject-verb-object (SVO) order, while Hindi primarily follows the subject-object-verb (SOV) order. This difference poses challenges to modeling this pair of languages for translation. In phrase-based translation systems, word reordering is governed by the language model, the phrase table, and reordering models. Reordering in such systems is generally achieved during decoding by transposing words within a defined window. These systems can handle local reorderings, and while some phrase-level reorderings are carried out during the formation of phrases, they are weak in learning long-distance reorderings. To overcome this weakness, researchers have used reordering as a step in pre-processing to render the reordered source sentence closer to the target language in terms of word order. Such approaches focus on using parts-of-speech (POS) tag sequences and reordering the syntax tree by using grammatical rules, or through head finalization. This study shows that mere head finalization is not sufficient for the reordering of sentences in the English-Hindi language pair. It describes various grammatical constructs and presents a comparative evaluation of reorderings with the original and the head-finalized representations. The impact of the reordering on the quality of translation is measured through the BLEU score in phrase-based statistical systems and neural machine translation systems. A significant gain in BLEU score was noted for reorderings in different grammatical constructs.

Supervised Classification Based Machine Translation Quality Estimation

This submission describes the study of linguistically motivated features to estimate the translated sentence quality at sentence level on English-Hindi language pair. Several classification algorithms are employed to build the Quality Estimation (QE) models using the extracted features. We used source language text and the MT output to extract these features. Experiments show that our proposed approach is robust and producing competitive results for the DT based QE model on neural machine translation system.

Pre-processing English-Hindi Corpus for Statistical Machine Translation

Corpus may be considered as fuel for the data driven approaches of machine translation. Parallel corpus building is a labour intensive task, thus making it a costly and scarce resource. Full potential of available data needs to be exploited and this can be ensured by removing different types of inconsistencies as being faced throughout the NLP domain. The paper presented here describes the experiments carried out on corpus text pre-processing for building the baseline Statistical Machine Translation (SMT) system. Text pre-processing performed here is classified in two stages – i. the first one relates to handling of orthographic representation of content and ii. the second stage relates to handling of non-lexical words. The first stage covers punctuation symbols, casing, word spellings and their normalization while second stage covers handling of numbers and named entities (NEs) applied on the best settings observed in first stage. The motivation behind performing these experiments was to derive a relationship and gauge the extent of pre-processing the corpus, thereby building a considerably optimized baseline SMT system. This baseline system would provide platform for performing further experiments with different syntactic and semantic factors in future. The findings presented here is for English-Hindi language pair, however, the concept of pre-processing is language neutral and can be transcended to any other language pair. The best performance is reported with retaining the punctuation symbols, lower-cased English corpus and spell normalized Hindi corpus for English to Hindi translation. Further to these, in the second stage of experiments, handling numbers and Named Entities have been described wherein these are mapped to unique class labels. The impact of these experiments have been explained with their appropriateness for the concerned language pair.

Efficient Use of Resources for Statistical Machine Translation

<div class="page" title="Page 1"><div class="layoutArea"><div class="column"><p><span>Machine translation has great potential to expand the audience for ever increasing digital collections. Success of data driven machine translation systems is governed by the volume of parallel data on which these systems are being modelled. The languages which do not have such resources in huge quantity, the optimum utilisation of them can only be assured through their quality. Morphologically rich language like Hindi poses further challenge, due to </span><span>having more number of orthographic inflections for a given word and presence of non-standard word spellings in </span><span>the corpus. This increases the chances of getting more number of words which are unseen in the training corpus. In this paper, the objective is to reduce redundancy of available corpus and utilise the other resources as well, to make best use of resources. Reduction in number of words unseen to the translation model is achieved through text noise removal, spell normalisation and utilising English WordNet (EWN). The test case presented here is for English-Hindi language pair. The results achieved are promising and set example for other morphological rich languages to optimise the resources to improve the performance of the translation system. </span></p></div></div></div>

Efficient Use of Resources for Statistical Machine Translation

<div class="page" title="Page 1"><div class="layoutArea"><div class="column"><p><span>Machine translation has great potential to expand the audience for ever increasing digital collections. Success of data driven machine translation systems is governed by the volume of parallel data on which these systems are being modelled. The languages which do not have such resources in huge quantity, the optimum utilisation of them can only be assured through their quality. Morphologically rich language like Hindi poses further challenge, due to </span><span>having more number of orthographic inflections for a given word and presence of non-standard word spellings in </span><span>the corpus. This increases the chances of getting more number of words which are unseen in the training corpus. In this paper, the objective is to reduce redundancy of available corpus and utilise the other resources as well, to make best use of resources. Reduction in number of words unseen to the translation model is achieved through text noise removal, spell normalisation and utilising English WordNet (EWN). The test case presented here is for English-Hindi language pair. The results achieved are promising and set example for other morphological rich languages to optimise the resources to improve the performance of the translation system. </span></p></div></div></div>

Continuous space models for CLIR

We present and evaluate a novel technique for learning cross-lingual continuous space models to aid cross-language information retrieval (CLIR). Our model, which is referred to as external-data composition neural network (XCNN), is based on a composition function that is implemented on top of a deep neural network that provides a distributed learning framework. Different from most existing models, which rely only on available parallel data for training, our learning framework provides a natural way to exploit monolingual data and its associated relevance metadata for learning continuous space representations of language. Cross-language extensions of the obtained models can then be trained by using a small set of parallel data. This property is very helpful for resource-poor languages, therefore, we carry out experiments on the English-Hindi language pair. On the conducted comparative evaluation, the proposed model is shown to outperform state-of-the-art continuous space models with statistically significant margin on two different tasks: parallel sentence retrieval and ad-hoc retrieval.

Exploration of Corpus Augmentation Approach for English-Hindi Bidirectional Statistical Machine Translation System

Even though lot of Statistical Machine Translation(SMT) research work is happening for English-Hindi language pair, there is no effort done to standardize the dataset. Each of the research work uses different dataset, different parameters and different number of sentences during various phases of translation resulting in varied translation output. So comparing these models, understand the result of these models, to get insight into corpus behavior for these models, regenerating the result of these research work becomes tedious. This necessitates the need for standardization of dataset and to identify the common parameter for the development of model. The main contribution of this paper is to discuss an approach to standardize the dataset and to identify the best parameter which in combination gives best performance. It also investigates a novel corpus augmentation approach to improve the translation quality of English-Hindi bidirectional statistical machine translation system. This model works well for the scarce resource without incorporating the external parallel data corpus of the underlying language. This experiment is carried out using Open Source phrase-based toolkit Moses. Indian Languages Corpora Initiative (ILCI) Hindi-English tourism corpus is used. With limited dataset, considerable improvement is achieved using the corpus augmentation approach for the English-Hindi bidirectional SMT system.

Improving the Quality of English-Hindi Name Entity Translation

Machine Translation is a novel step in overcoming the language barrier. The results of the research in this area has started to show its results with some good machine translators being available. In the context of English-Hindi language pair, due to bad name entity translations, the quality of translation deteriorates. In this paper we have addressed this issue where we show that only a single approach is not effective in dealing with this issue. We need to devise mechanism and address this problem in a multi pronged approach. General Terms Natural Language Processing, Machine Translation

Quality Estimation of Machine Translation Outputs Through Stemming

Machine Translation is the challenging problem for Indian languages. Every day we can see some machine translators being developed, but getting a high quality automatic translation is still a very distant dream . The correct translated sentence for Hindi language is rarely found. In this paper, we are emphasizing on English-Hindi language pair, so in order to preserve the correct MT output we present a ranking system, which employs some machine learning techniques and morphological features. In ranking no human intervention is required. We have also validated our results by comparing it with human ranking.