Construction Of Language Models Research Articles

American Sign Language Fingerspelling Recognition in the Wild with Iterative Language Model Construction

American Sign Language Fingerspelling Recognition in the Wild with Iterative Language Model Construction

The Construction of Machine Translation Model and Its Application in English Grammar Error Detection

In order to solve the problems of low accuracy, recall rate, and F1 value of traditional English grammar error detection methods, a new machine translation model is constructed and applied to English grammar error detection. In the encoder-decoder framework, the machine translation model is constructed through the steps of word vector generation, encoder language model construction, decoder language model construction, word alignment, output module, and so on. On this basis, the machine translation model is trained to detect English grammatical errors through dependency analysis and alternative word generation. Experimental results show that the accuracy, recall rate, and F1 value of the proposed method are higher than those of the experimental comparison method for detecting English grammatical errors such as articles, prepositions, nouns, verbs, and subject-verb agreement, indicating that the proposed method is of high practical value.

The Relevance of the Source Language in Transfer Learning for ASR

This study presents new experiments on Zyrian Komi speech recognition. We use Deep-Speech to train ASR models from a language documentation corpus that contains both contemporary and archival recordings. Earlier studies have shown that transfer learning from English and using a domain matching Komi language model both improve the CER and WER. In this study we experiment with transfer learning from a more relevant source language, Russian, and including Russian text in the language model construction. The motivation for this is that Russian and Komi are contemporary contact languages, and Russian is regularly present in the corpus. We found that despite the close contact of Russian and Komi, the size of the English speech corpus yielded greater performance when used as the source language. Additionally, we can report that already an update in DeepSpeech version improved the CER by 3.9% against the earlier studies, which is an important step in the development of Komi ASR.

Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction

Neural architectures are prominent in the construction of language models (LMs). However, word-level prediction is typically agnostic of subword-level information (characters and character sequences) and operates over a closed vocabulary, consisting of a limited word set. Indeed, while subword-aware models boost performance across a variety of NLP tasks, previous work did not evaluate the ability of these models to assist next-word prediction in language modeling tasks. Such subword-level informed models should be particularly effective for morphologically-rich languages (MRLs) that exhibit high type-to-token ratios. In this work, we present a large-scale LM study on 50 typologically diverse languages covering a wide variety of morphological systems, and offer new LM benchmarks to the community, while considering subword-level information. The main technical contribution of our work is a novel method for injecting subword-level information into semantic word vectors, integrated into the neural language modeling training, to facilitate word-level prediction. We conduct experiments in the LM setting where the number of infrequent words is large, and demonstrate strong perplexity gains across our 50 languages, especially for morphologically-rich languages. Our code and data sets are publicly available.

Modeling positive and negative feedback for improving document retrieval

Pseudo-relevance feedback (PRF) has evident potential for enriching the representation of short queries. Traditional PRF methods treat top-ranked documents as feedback, since they are assumed to be relevant to the query. However, some of these feedback documents may actually distract from the query topic for a range of reasons and accordingly downgrade PRF system performance. Such documents constitute negative examples (negative feedback) but could also be valuable in retrieval. In this paper, a novel framework of query language model construction is proposed in order to improve retrieval performance by integrating both positive and negative feedback. First, an improvement-based method is proposed to automatically identify the types of feedback documents (i.e. positive or negative) according to whether the document enhances the retrieval’s effectiveness. Subsequently, based on the learned positive and negative examples, the positive feedback models and the negative feedback models are estimated using an Expectation-Maximization algorithm with the assumptions: the positive term distribution is affected by the context term distribution and the negative term distribution is affected by both the positive term distribution and the context term distribution (such that the positive feedback model upgrades the rankings of relevant documents and the negative feedback model prunes the irrelevant documents from a query). Finally, a content-based representativeness criterion is proposed in order to obtain the representative negative feedback documents. Experiments conducted on the TREC collections demonstrate that our proposed approach results in better retrieval accuracy and robustness than baseline methods.

Converting Continuous-Space Language Models into N -gram Language Models with Efficient Bilingual Pruning for Statistical Machine Translation

The Language Model (LM) is an essential component of Statistical Machine Translation (SMT). In this article, we focus on developing efficient methods for LM construction. Our main contribution is that we propose a Natural N -grams based Converting (NNGC) method for transforming a Continuous-Space Language Model (CSLM) to a Back-off N -gram Language Model (BNLM). Furthermore, a Bilingual LM Pruning (BLMP) approach is developed for enhancing LMs in SMT decoding and speeding up CSLM converting. The proposed pruning and converting methods can convert a large LM efficiently by working jointly. That is, a LM can be effectively pruned before it is converted from CSLM without sacrificing performance, and further improved if an additional corpus contains out-of-domain information. For different SMT tasks, our experimental results indicate that the proposed NNGC and BLMP methods outperform the existing counterpart approaches significantly in BLEU and computational cost.

Pipelined language model construction for Polish speech recognition

Abstract The aim of works described in this article is to elaborate and experimentally evaluate a consistent method of Language Model (LM) construction for the sake of Polish speech recognition. In the proposed method we tried to take into account the features and specific problems experienced in practical applications of speech recognition in the Polish language, reach inflection, a loose word order and the tendency for short word deletion. The LM is created in five stages. Each successive stage takes the model prepared at the previous stage and modifies or extends it so as to improve its properties. At the first stage, typical methods of LM smoothing are used to create the initial model. Four most frequently used methods of LM construction are here. At the second stage the model is extended in order to take into account words indirectly co-occurring in the corpus. At the next stage, LM modifications are aimed at reduction of short word deletion errors, which occur frequently in Polish speech recognition. The fourth stage extends the model by insertion of words that were not observed in the corpus. Finally the model is modified so as to assure highly accurate recognition of very important utterances. The performance of the methods applied is tested in four language domains.

문장음성인식을 위한 VCCV 기반의 언어모델과 Smoothing 기법 평가

본 논문에서는 언어모델의 언어처리 단위로 VCCV(vowel consonant consonant vowel) 단위를 제안하구 기존의 언어처리 단위인 어적 형태소 단위와 비교한다. 어절과 형태소는 어휘수가 많고 높은 복잡도를 가진다. 그러나 VCCV 단위는 작은 사전과 제한된 어휘를 가지므로 복잡도가 적다. 언어모델 구성에 smoothing은 반드시 필요하다. smoothing 기법은 정확한 확률 예측이 불확실한 데이터가 있을 때 더 나은 확률 예측을 위해 사용된다. 본 논문에서는 형태소, 어절, VCCV 단위에 대해 언어모델을 구성하여 복잡도를 계산하였다. 그 결과 VCCV 단위의 복잡도가 형태소나 어절보다 적게 나오는 것을 볼 수 있었다. 복잡도가 적게 나온 VCCV를 기반으로 N-gram을 구성하고 Katz. Witten-Bell, absolute, modified Kneser-Ney smoothing 등의 방법을 이용한 언어 모델에 대해 평가하였다. 그 결과 VCCV 단위의 언어모델에 적합한 smoothing 기법은 modified Kneser-Ney 방법으로 평가되었다. In this paper, we propose VCCV units as a processing unit of language model and compare them with clauses and morphemes of existing processing units. Clauses and morphemes have many vocabulary and high perplexity. But VCCV units have low perplexity because of the small lexicon and the limited vocabulary. The construction of language models needs an issue of the smoothing. The smoothing technique used to better estimate probabilities when there is an insufficient data to estimate probabilities accurately. This paper made a language model of morphemes, clauses and VCCV units and calculated their perplexity. The perplexity of VCCV units is lower than morphemes and clauses units. We constructed the N-grams of VCCV units with low perplexity and tested the language model using Katz, absolute, modified Kneser-Ney smoothing and so on. In the experiment results, the modified Kneser-Ney smoothing is tested proper smoothing technique for VCCV units.

Acquisition of language models from text

Statistical language models that assign a probability of being generated to each possible string of words are of interest here. Grammatical language models may, moreover, assign probabilities to parses of the given string of words. Language models are used as components of recognition systems, speech and handwriting being current examples of interest. Another use of language models is in text understanding systems and machine translation systems, where grammatical information is very desirable. Because of the richness of language, language models are usually constructed for particular discourse domains, such as medical reports, office correspondence, legal documents, etc. The construction is on the basis of large amounts of sample text, and should be as automatic as possible. For grammatical language models, it is desirable to estimate not only their statistical parameters from the supplied text, but even the grammatical rules themselves. The talk will discuss various methods of language model construction, their relative performance characteristics, and some of the many open problems.