Machine Transliteration System Research Articles

Transliteration Characteristics in Romanized Assamese Language Social Media Text and Machine Transliteration

This article aims to understand different transliteration behaviors of Romanized Assamese text on social media. Assamese, a language that belongs to the Indo-Aryan language family, is also among the 22 scheduled languages in India. With the increasing popularity of social media in India and also the common use of the English Qwerty keyboard, Indian users on social media express themselves in their native languages, but using the Roman/Latin script. Unlike some other popular South Asian languages (say Pinyin for Chinese), Indian languages do not have a common standard romanization convention for writing on social media platforms. Assamese and English are two very different orthographical languages. Thus, considering both orthographic and phonemic characteristics of the language, this study tries to explain how Assamese vowels, vowel diacritics, and consonants are represented in Roman transliterated form. From a dataset of romanized Assamese social media texts collected from three popular social media sites: (Facebook, YouTube, and X (formerly known as Twitter)), 1 we have manually labeled them with their native Assamese script. A comparison analysis is also carried out between the transliterated Assamese social media texts with six different Assamese romanization schemes that reflect how Assamese users on social media do not adhere to any fixed romanization scheme. We have built three separate character-level transliteration models from our dataset. One using a traditional phrase-based statistical machine transliteration model, (1) PBSMT model and two separate neural transliteration models, (2) BiLSTM neural seq2seq model with attention, and (3) Neural transformer model. A thorough error analysis has been performed on the transliteration result obtained from the three state-of-the-art models mentioned above. This may help to build a more robust machine transliteration system for the Assamese social media domain in the future. Finally, an attention analysis experiment is also carried out with the help of attention weight scores taken from the character-level BiLSTM neural seq2seq transliteration model built from our dataset.

State-of-the-art automatic machine transliteration systems for Indic scripts: a comparative report

State-of-the-art automatic machine transliteration systems for Indic scripts: a comparative report

State of the Art Automatic Machine Transliteration Systems for Indic Scripts: A Comparative Report

State of the Art Automatic Machine Transliteration Systems for Indic Scripts: A Comparative Report

Roman-Urdu-Parl: Roman-Urdu and Urdu Parallel Corpus for Urdu Language Understanding

Availability of corpora is a basic requirement for conducting research in a particular language. Unfortunately, for a morphologically rich language like Urdu, despite being used by over a 100 million people around the globe, the dearth of corpora is a major reason for the lack of attention and advancement in research. To this end, we present the first-ever large-scale publicly available Roman-Urdu parallel corpus, Roman-Urdu-Parl, with 6.37 million sentence-pairs. It is a huge corpus collected from diverse sources, annotated using crowd-sourcing techniques, and also assured for quality. It has a total of 92.76 million Roman-Urdu words, 92.85 million Urdu words, Roman-Urdu vocabulary of 42.9 K words, and Urdu vocabulary of 43.8 K words. Roman-Urdu-Parl has been built to ensure that it not only captures the morphological and linguistic features of the language but also the heterogeneity and variations arising due to demographic conditions. We validate the authenticity and quality of our corpus by using it to address two natural language processing research problems, that is, on learning word embeddings and building a machine transliteration system. Our contribution of the corpus leads to exceptional results in both settings, for example, our machine transliteration system sets a new state-of-the-art with a Bilingual Evaluation Understudy (BLEU) score of 84.67. We believe that Roman-Urdu-Parl can serve as fuel for igniting and advancing works in many research areas related to the Urdu language.

Bodo to English Machine Translation through Transliteration

Machine Translation (MT) is a technique that automatically translates text from one natural language to another using machine like computer. Machine Transliteration (MTn) is also a technique that converts the script of text from source language to target language without changing the pronunciation of the source text. Both the MT and MTn are the challenging research task in the field of Natural Language Processing (NLP) and Computational Linguistics (CL) globally. English is a high resource natural language, whereas Bodo is a low resource natural language. Though Bodo is a recognized language of India; still not much research work has been done on MT and MTn systems due to the low resources. The primary objective of this paper is to develop Bodo to English Machine Translation system with the help of Bodo to English Machine Transliteration system. The Bodo to English MT system has been developed using the Phrase-based Statistical Machine Translation technique for General and News domains of Bodo-English parallel text corpus. The Bodo to English MTn system has been developed using the Hybrid technique for General and News domains of Bodo-English parallel transliterated words/terms. The translation accuracy of the MT system has been evaluated using BLEU technique.

A Review on a web based Punjabi to English Machine Transliteration System

The paper presents the transliteration of noun phrases from Punjabi to English using statistical machine translation approach.Transliteration maps the letters of source scripts to letters of another language.Forward transliteration converts an original word or phrase in the source language into a word in the target language.Backward transliteration is the reverse process that converts the transliterated word or phrase back into its original word or phrase.Transliteration is an important part of research in NLP.Natural Language Processing (NLP) is the ability of a computer program to understand human speech as it is spoken.NLP is an important component of AI.Artificial Intelligence is a branch of science which deals with helping machines find solutions to complex programs in a human like fashion.The transliteration system is going to developed using SMT.Statistical Machine Translation (SMT) is a data oriented statistical framework for translating text from one natural language to another based on the knowledge. Keyword:Transliteration,Mapping,Translation,Dictionary

Analyzing Probability Vectors for Named Entity Statistical Machine Transliteration

ABSTRACT Machine transliteration systems are classified as either Rule-based methods or statistical methods. A rule-based method focuses on transliterating names using lots of human-made rules set. These systems are simple to implement but require huge amount of language expertise. In statistical methods, the importance is given in converting transliteration problem into a classification problem and employs a statistical model to solve this classification problem. Though these methods don’t require expert knowledge of Language model, they need large amounts of bilingual data and good algorithm for training. Currently, basic Markov Chain Model (MM), Extended Markov Chain (EMC), Hidden Markov Model (HMM), Conditional Random Fields (CRF), Decision Tree (DT), Maximum Entropy Markov Model (MEMM) and Support Vector Machine (SVM) are the popular statistical approaches used by many researchers across the globe. This paper focuses on mathematical analysis of different statistical approaches used in machine transliteration of named entity which would be beneficial for many upcoming researchers to know the mathematics used behind the curtains.

Compositional Machine Transliteration

Machine transliteration is an important problem in an increasingly multilingual world, as it plays a critical role in many downstream applications, such as machine translation or crosslingual information retrieval systems. In this article, we propose compositional machine transliteration systems, where multiple transliteration components may be composed either to improve existing transliteration quality, or to enable transliteration functionality between languages even when no direct parallel names corpora exist between them. Specifically, we propose two distinct forms of composition: serial and parallel. Serial compositional system chains individual transliteration components, say, X → Y and Y → Z systems, to provide transliteration functionality, X → Z. In parallel composition evidence from multiple transliteration paths between X → Z are aggregated for improving the quality of a direct system. We demonstrate the functionality and performance benefits of the compositional methodology using a state-of-the-art machine transliteration framework in English and a set of Indian languages, namely, Hindi, Marathi, and Kannada. Finally, we underscore the utility and practicality of our compositional approach by showing that a CLIR system integrated with compositional transliteration systems performs consistently on par with, and sometimes better than, that integrated with a direct transliteration system.

An ensemble of transliteration models for information retrieval

Transliteration is used to phonetically translate proper names and technical terms especially from languages in Roman alphabets to languages in non-Roman alphabets such as from English to Korean, Japanese, and Chinese. Because transliterations are usually representative index terms for documents, proper handling of the transliterations is important for an effective information retrieval system. However, there are limitations on handling transliterations depending on dictionary lookup, because transliterations are usually not registered in the dictionary. For this reason, many researchers have been trying to overcome the problem using machine transliteration. In this paper, we propose a method for improving machine transliteration using an ensemble of three different transliteration models. Because one transliteration model alone has limitation on reflecting all possible transliteration behaviors, several transliteration models should be complementary used in order to achieve a high-performance machine transliteration system. This paper describes a method about transliteration production using the several machine transliteration models and transliteration ranking with web data and relevance scores given by each transliteration model. We report evaluation results for our ensemble transliteration model and experimental results for its impact on IR effectiveness. Machine transliteration tests on English-to-Korean transliteration and English-to-Japanese transliteration show that our proposed method achieves 78–80% word accuracy. Information retrieval tests on KTSET and NTCIR-1 test collection show that our transliteration model can improve the performance of an information retrieval system about 10–34%.

Machine Transliteration from Roman to Devanagari and Devanagari to Roman

Design and implementation of a machine transliteration system from Roman to Devanagari and vice-versa is presented. The Roman characters are grouped into 12 categories depending upon their phonetic correspondences with Devanagari. A state transition table is utilized to transform the Roman character string to a linearized Devanagari representation. Target Devanagari script is composed through a composition processor on a Devanagari terminal. The Devanagari to Roman conversion is primarily table driven, however, a number of heuristics are used to get a more acceptable form. The system produces outputs which are largely acceptable, however, pre-editing by way of insertion of phonetic symbols are required for error-free transliteration.