N-gram Language Models Research Articles

An empirical study of statistical language models: n-gram language models vs. neural network language models

Statistical language models are an important module in many areas of successful applications such as speech recognition and machine translation. And n-gram models are basically the state-of-the-art. However, due to sparsity of data, the modelled language cannot be completely represented in the n-gram language model. In fact, if new words appear in the recognition or translation steps, we need to provide a smoothing method to distribute the model probabilities over the unknown values. Recently, neural networks were used to model language based on the idea of projecting words onto a continuous space and performing the probability estimation in this space. In this experimental work, we compare the behaviour of the most popular smoothing methods with statistical n-gram language models and neural network language models in different situations and with different parameters. The language models are trained on two corpora of French and English texts. Good empirical results are obtained by the recurrent neural network language models.

An empirical study of statistical language models: n-gram language models vs. neural network language models

An empirical study of statistical language models: n-gram language models vs. neural network language models

Domain-specific language models training methodology for the in-car infotainment

The proposed paper focuses on the methodology for training small domain-specific language models. The methodology has been applied for creating a language model for the demonstration version of the in-vehicle infotainment system speech interface. The proposed methodology is based on constructing an initial n-gram language model only from the vocabulary. Such “random” model can be easily adapted to the user’s speaking style. The methodology is well suitable for situations when constructing the accurate deterministic grammar is not a trivial task or the higher flexibility is required. In the case of in-vehicle infotainment, the key idea is to decrease the cognitive load of the driver during controlling such a system and also to make designing process less time consuming. Evaluation of the proposed approach was done on the INCARSCOM corpus (Slovak In-car Command Database). Obtained results favour designed method instead of constructing the appropriate deterministic grammar.

A Novel Approach Research on Chinese Language Model Fusion Based on RNN

In the light of the poor ability to describe the long-distance information of a sentence and the serious data sparse phenomenon of the mainstream n-gram language model, the RNN modeling method which can capture the inherent rules of natural language better and overcome the inadequacy of n-gram model was firstly used for Chinese language. To further improve the model performance, a model combination method was introduced so that RNN and the n-gram model can be merged together respectively. This algorithm can take full advantage of each model combining the language score information of the confused network marked by different models. The experiment results show that the RNN language model has a great superiority with better modeling performance. Meanwhile, with the combination of the models, the system recognition rate increases effectively on the task of Chinese telephone speech recognition.

Smoothed n-gram based models for tweet language identification: A case study of the Brazilian and European Portuguese national varieties

Identifying the language of a text is an important step for several natural language processing applications. State-of-the-art language identification (LID) systems perform very well when discriminating between unrelated languages on standard datasets. However, the LID task has a bottleneck when discriminating between similar languages or language varieties. Furthermore, LID has also proven to be very challenging when dealing with short texts such as the ones from Twitter. In this paper, we propose the use of smoothed n-gram language models to classify tweets in both Brazilian and European Portuguese variants. Word and character n-gram language models were combined and evaluated through five different classifiers. We have compared the smoothed n-gram language models together with the Term Frequency and Inverse Document Frequency weighting scheme. This paper also proposes an ensemble model, in which the class labels output were combined using majority voting and algebraic combiners. The best configuration reached accuracy of 92.71% using an ensemble model, which combines Lidstone (0.1) character 6-gram, Good–Turing word unigram, and Witten–Bell word bigram models, together with the Log-Likelihood Ratio estimation method.

Improving handwritten Chinese text recognition using neural network language models and convolutional neural network shape models

Handwritten Chinese text recognition based on over-segmentation and path search integrating multiple contexts has been demonstrated successful, wherein the language model (LM) and character shape models play important roles. Although back-off N-gram LMs (BLMs) have been used dominantly for decades, they suffer from the data sparseness problem, especially for high-order LMs. Recently, neural network LMs (NNLMs) have been applied to handwriting recognition with superiority to BLMs. With the aim of improving Chinese handwriting recognition, this paper evaluates the effects of two types of character-level NNLMs, namely, feedforward neural network LMs (FNNLMs) and recurrent neural network LMs (RNNLMs). Both FNNLMs and RNNLMs are also combined with BLMs to construct hybrid LMs. For fair comparison with BLMs and a state-of-the-art system, we evaluate in a system with the same character over-segmentation and classification techniques as before, and compare various LMs using a small text corpus used before. Experimental results on the Chinese handwriting database CASIA-HWDB validate that NNLMs improve the recognition performance, and hybrid RNNLMs outperform the other LMs. To report a new benchmark, we also evaluate selected LMs on a large corpus, and replace the baseline character classifier, over-segmentation, and geometric context models with convolutional neural network (CNN) based models. The performance on both the CASIA-HWDB and the ICDAR-2013 competition dataset are improved significantly. On the CASIA-HWDB test set, the character-level accurate rate (AR) and correct rate (CR) achieve 95.88% and 95.95%, respectively.

57 A predictable smart home integrated with cloud computing and long-term care

Objectives This paper introduces a predictable smart home which was integrated with cloud computing. A Hakka pinyin input method for Android mobile/cell phones has been provided. Our system achieved the goals of energy saving and carbon reduction, and can be employed in medical care applications, such as long-term care. Methods This system provides a diversified smart home interface. By mining power consumption data, we can predict people’s future behavior for controlling home appliances. In this paper, we use an n-gram language model for training an (n-1)th-order Markov model. Based on statistical models, we can predict the behavior of residents and patients, and achieve energy saving and carbon reduction. In our system, mobile/cell phone and smart watch apps were developed to provide residents or patients with a convenient interface to control IR appliances, ZigBee appliances and appliances with a WiFi smart plug. The power consumption data must first be obtained by calling the WiFi smart plug’s open API. The open API’s power consumption data is presented in JSON format, as show below: {”result”:true,”data”:[{”outletid”:”PT020020F85EAA766A01”,”attridval”:”2_0,5_0,7_4517,8_0,12_0”}]} The attribute of the result is present calling status. The outlet id is the WiFi smart plug’s id. Data is the power consumption string which is shown as 2_Current A, 1/100 = 0.01 A for this case. This should be divided by 100. The home appliance condition should then be determined based on the current information, and the following rule used to generate one string for the present condition at different times. In the rule, the string consists of two parts: the prefix and the time index. In this system, the Hakka pinyin input method for Android mobile/cell phones that enable users to input Hakka words quickly and conveniently has been provided. It is toneless while the user inputs the pinyin, and several response messages for typing errors are included. The smart home apps on mobile/cell phones employ a socket client connecting to the cloud server, allowing residents to use their smart devices and web applications anywhere with an internet connection to control appliances. Results The experimental data were generated from six appliances generating 4320 sequential states over 1 month. Several experiments were conducted using three n-gram models on six home appliances. A prediction precision of 95.7% on average was achieved for outside testing. We furthermore compare performance and analyze various features to promote the results. Conclusions A predictable smart home interface integrated with cloud computing and long-term care is implemented using mobile/cell phone apps and various applications. The predictable models for controlling appliances are based on Markov technology. The best prediction rate for outside testing is 95.7% on average. The Hakka pinyin input method for Android mobile/cell phones that enables users to input Hakka words has been provided. Our system achieved the goal of energy saving and carbon reduction, and can be employed for medical care applications in patient’s rooms, for instance for patients in long-term care.

Similar Word Model for Unfrequent Word Enhancement in Speech Recognition

The popular n-gram language model (LM) is weak for unfrequent words. Conventional approaches such as class-based LMs pre-define some sharing structures (e.g., word classes) to solve the problem. However, defining such structures requires prior knowledge, and the context sharing based on these structures is generally inaccurate. This paper presents a novel similar word model to enhance unfrequent words. In principle, we enrich the context of an unfrequent word by borrowing context information from some “similar words.” Compared to conventional class-based methods, this new approach offers a fine-grained context sharing by referring to words that best match the target word, and it is more flexible as no sharing structures need to be defined by hand. Experiments on a large-scale Chinese speech recognition task demonstrated that the similar word approach can improve performance on unfrequent words significantly, while keeping the performance on general tasks almost unchanged.

Optimization of topic estimation for the domain adapted neural network language model

We present a neural network language model adapted for topics fluctuating in broadcast programs. Topic adapted n-gram language models constructed by using latent Dirichlet allocation for topic estimation are widely used. The conventional method estimates topics by separating the corpora into chunks that have few sentences. While the n-gram model uses several preceding words, the recurrent neural network and long short-term memory can learn to store huge amounts of past information in the hidden layers. Consequently, chunks for language models trained by using neural networks may have a longer optimal length than the chunks for language models trained by using the conventional methods. In this paper, the length of chunks and topic estimation process are optimized for the neural network language models. For the topic estimation, k-mean clustering, latent Dirichlet allocation, and word2vec were compared. On the basis of the results of comparison, we designed a neural network language model.

HMM word graph based keyword spotting in handwritten document images

Line-level keyword spotting (KWS) is presented on the basis of frame-level word posterior probabilities. These posteriors are obtained using word graphs derived from the recognition process of a full-fledged handwritten text recognizer based on hidden Markov models and N-gram language models. This approach has several advantages. First, since it uses a holistic, segmentation-free technology, it does not require any kind of word or character segmentation. Second, the use of language models allows the context of each spotted word to be taken into account, thereby considerably increasing KWS accuracy. And third, the proposed KWS scores are based on true posterior probabilities, taking into account all (or most) possible word segmentations of the input image. These scores are properly bounded and normalized. This mathematically clean formulation lends itself to smooth, threshold-based keyword queries which, in turn, permit comfortable trade-offs between search precision and recall. Experiments are carried out on several historic collections of handwritten text images, as well as a well-known data set of modern English handwritten text. According to the empirical results, the proposed approach achieves KWS results comparable to those obtained with the recently-introduced “BLSTM neural networks KWS” approach and clearly outperform the popular, state-of-the-art “Filler HMM” KWS method. Overall, the results clearly support all the above-claimed advantages of the proposed approach.

워드 임베딩과 품사 태깅을 이용한 클래스 언어모델 연구

Recurrent neural network based language models (RNN LM) have shown improved results in language model researches. The RNN LMs are limited to post processing sessions, such as the N-best rescoring step of the wFST based speech recognition. However, it has considerable vocabulary problems that require large computing powers for the LM training. In this paper, we try to find the 1st pass N-gram model using word embedding, which is the simplified deep neural network. The class based language model (LM) can be a way to approach to this issue. We have built class based vocabulary through word embedding, by combining the class LM with word N-gram LM to evaluate the performance of LMs. In addition, we propose that part-of-speech (POS) tagging based LM shows an improvement of perplexity in all types of the LM tests.

Stepwise API usage assistance using n -gram language models

An IDE code completion mechanism for stepwise API exploration.A method for building a language model for supporting (1).a detailed evaluation of (2), showing how n-grams perform against existing client code.Results of (3) revealing a 90% chance of getting the expected result in the 10-top-most code completion proposals. Reusing software involves learning third-party APIs, a process that is often time-consuming and error-prone. Recommendation systems for API usage assistance based on statistical models built from source code corpora are capable of assisting API users through code completion mechanisms in IDEs. A valid sequence of API calls involving different types may be regarded as a well-formed sentence of tokens from the API vocabulary. In this article we describe an approach for recommending subsequent tokens to complete API sentences using n-gram language models built from source code corpora. The provided system was integrated in the code completion facilities of the Eclipse IDE, providing contextualized completion proposals for Java taking into account the nearest lines of code. The approach was evaluated against existing client code of four widely used APIs, revealing that in more than 90% of the cases the expected subsequent token is within the 10-top-most proposals of our models. The high score provides evidence that the recommendations could help on API learning and exploration, namely through the assistance on writing valid API sentences.

Word graphs size impact on the performance of handwriting document applications

Two document processing applications are considered: computer-assisted transcription of text images (CATTI) and Keyword Spotting (KWS), for transcribing and indexing handwritten documents, respectively. Instead of working directly on the handwriting images, both of them employ meta-data structures called word graphs (WG), which are obtained using segmentation-free handwritten text recognition technology based on N-gram language models and hidden Markov models. A WG contains most of the relevant information of the original text (line) image required by CATTI and KWS but, if it is too large, the computational cost of generating and using it can become unafordable. Conversely, if it is too small, relevant information may be lost, leading to a reduction of CATTI or KWS performance. We study the trade-off between WG size and performance in terms of effectiveness and efficiency of CATTI and KWS. Results show that small, computationally cheap WGs can be used without loosing the excellent CATTI and KWS performance achieved with huge WGs.

Perspell: A New Persian Semantic-Based Spelling Correction System

Due to the special features of Persian, developing natural language processing tools for it involves an array of challenges. Lack of efficient Persian knowledge sources is another obstacle to research this language. The goal of this article was to overcome these problems by implementing spelling correction task. The main outputs of this study included a parallel corpus, an N-gram language model for Persian, and a semantic-based spelling correction system named Perspell, which made use of extracted language model. Compared to its rival software (including Vafa spellchecker), Perspell could detect and correct nonword and real word errors more successfully. The rate of real word error detection in Perspell was 95%. In fact, its outstanding ability to detect real word errors as well as its significant improvement in terms of F-measure were the two advantages of the proposed system.

A Novel Approach by Injecting CCG Supertags into an Arabic–English Factored Translation Machine

This study addresses the integration and incorporation of rich additional information into the phrase-based approach, aptly called factored translation, which is an extension of phrase-based statistic machine translation (PBSMT). This approach was proven successful when translating English into a morphologically rich language. PBSMT represents the baseline of this work. We extend the phrase-based translation approach by integrating additional linguistic knowledge, namely part-of-speech (POS) tags, to create a factored model. The main contribution of this study is the creation of a new approach for Arabic–English translation via the injection of the factored model into Combinatory Categorial Grammar (CCG) supertags to form an integrated model (POS + CCG). The system was trained on a freely available multi-UN corpus on Arabic–English language pairs. Moses decoder, which is an open-source factored SMT system, was used to integrate these data into the target language model and the target side of the translation model. Results showed improvements to the BLEU automatic score via various high n-gram language models (LMs). The integration of the featured factors (POS + CCG) of the translation has been successfully tested. Overall, the 3-, 5-, 7-, and 9-g LM evaluation with BLEU scores proved that our integrated model performed better than PBSMT. Compared with three other models (PBSMT, POS, and CCG models), the integrated model improved the translation quality by 1.54, 1.29, and 0.21 %, respectively, over the 3-g LM.

Contextualized Latent Semantic Indexing: A New Approach to Automated Chinese Essay Scoring

AbstractThe writing part in Chinese language tests is badly in need of a mature automated essay scoring system. In this paper, we propose a new approach applied to automated Chinese essay scoring (ACES), called contextualized latent semantic indexing (CLSI), of which Genuine CLSI and Modified CLSI are two versions. The n-gram language model and the weighted finite-state transducer (WFST), two critical components, are used to extract context information in our ACES system. Not only does CLSI improve conventional latent semantic indexing (LSI), but bridges the gap between latent semantics and their context information, which is absent in LSI. Moreover, CLSI can score essays from the perspectives of language fluency and contents, and address the local overrating and underrating problems caused by LSI. Experimental results show that CLSI outperforms LSI, Regularized LSI, and latent Dirichlet allocation in many aspects, and thus, proves to be an effective approach.

Improved Neural Machine Translation with SMT Features

Neural machine translation (NMT) conducts end-to-end translation with a source language encoder and a target language decoder, making promising translation performance. However, as a newly emerged approach, the method has some limitations. An NMT system usually has to apply a vocabulary of certain size to avoid the time-consuming training and decoding, thus it causes a serious out-of-vocabulary problem. Furthermore, the decoder lacks a mechanism to guarantee all the source words to be translated and usually favors short translations, resulting in fluent but inadequate translations. In order to solve the above problems, we incorporate statistical machine translation (SMT) features, such as a translation model and an n-gram language model, with the NMT model under the log-linear framework. Our experiments show that the proposed method significantly improves the translation quality of the state-ofthe-art NMT system on Chinese-to-English translation tasks. Our method produces a gain of up to 2.33 BLEU score on NIST open test sets.

Modelling and Optimizing on Syntactic N-Grams for Statistical Machine Translation

The role of language models in SMT is to promote fluent translation output, but traditional n-gram language models are unable to capture fluency phenomena between distant words, such as some morphological agreement phenomena, subcategorisation, and syntactic collocations with string-level gaps. Syntactic language models have the potential to fill this modelling gap. We propose a language model for dependency structures that is relational rather than configurational and thus particularly suited for languages with a (relatively) free word order. It is trainable with Neural Networks, and not only improves over standard n-gram language models, but also outperforms related syntactic language models. We empirically demonstrate its effectiveness in terms of perplexity and as a feature function in string-to-tree SMT from English to German and Russian. We also show that using a syntactic evaluation metric to tune the log-linear parameters of an SMT system further increases translation quality when coupled with a syntactic language model.

Toward Normalizing Romanized Gurumukhi Text from Social Media

Roman characters are used to write Indian language text on social media like facebook and twitter. Processing this text for NLP applications is not a trivial task. This text needs to be transliterated as well as conversion to canonical form. This paper discusses the various issues involved in normalizing such text in the domain of Punjabi Language. An algorithm is proposed to normalize Punjabi language text which is written using roman script. The proposed algorithm tries to find out all possible combinations and then filter using n-gram language model.

Bilingual Language Model for English Arabic Technical Translation

The massive fast of new scientific publications increase the need to a reliable effective automatic machine translation (AMT) system, which translates from English, as the common language of publications, to other different languages. Statistical machine translation (SMT) model crafted to deal with certain domain of text often fails when subjected to another domain. The paper addresses the characterization of language domains and their behavior in SMT, experiments the management of SMT model to translate scientific text collected from artificial intelligence publications. The effectiveness of Bilingual language model is tested against the typical N-gram language model, in addition to utilizing the fill-up and back-off techniques to handle different phrase tables from different domains. As not every human capable to translate artificial intelligence book, should have strong knowledge in the field, We suggest that in order AMT can handle different domains it must be trained by in-domain parallel data, adjusting weights for the words on different domains to learn the model how to differentiate between different meaning of same word in different domains.