N-gram Model Research Articles

A private DNA motif finding algorithm

With the increasing availability of genomic sequence data, numerous methods have been proposed for finding DNA motifs. The discovery of DNA motifs serves a critical step in many biological applications. However, the privacy implication of DNA analysis is normally neglected in the existing methods. In this work, we propose a private DNA motif finding algorithm in which a DNA owner’s privacy is protected by a rigorous privacy model, known as ∊-differential privacy. It provides provable privacy guarantees that are independent of adversaries’ background knowledge. Our algorithm makes use of the n-gram model and is optimized for processing large-scale DNA sequences. We evaluate the performance of our algorithm over real-life genomic data and demonstrate the promise of integrating privacy into DNA motif finding.

Algorithm for Fast Finding High-Frequency Strings from Large-Scale Corpus

In high-frequency string extraction, there exists enormous time and memory waste in taking statistics of tremendous low-frequency strings, which causes low efficiency. Based on the incremental n-gram model, this paper puts forward Hierarchical Pruning Algorithm (HPA) to filter out low-frequency garbage strings and to extract candidate repeats for reducing I/O reading-writing times and enhancing efficiency of memory usage. On the basis of candidate repeats, external sort method is applied to merge all of them in order to obtain the final repeat set. For improving the efficiency of candidate repeats merging, this paper proposes to employ improved Radix Sort method to process strings in O(dn). With 32 gigabyte plain text corpus, experiments show that the relationship between I/O reading-writing times of HPA and the corpus size is nearly linear, and the algorithm can efficiently extract repeats from corpus whose size is much larger than that of memory. Index Terms—repeats, hash table, low-frequency string, hierarchical pruning algorithm

2A1-S06 過去の操作系列と状況変化の履歴に基づく隠れマルコフモデルを用いた人の操作予測

The future human's operation correlates with the past sequential operations. Based on the assumption, we propose the human operation model for predicting the sequential operation. First, the reactive operation models are built by Hidden Markov Model, and the meta- operation models are configured with the tree structure by combining the close reactive operations. Using the meta-operation models, the N-gram model is configured to predict the transition of the operation. The model can narrow down the options for predicting next operation and it also can predict the two or more minions. Experimental result shows the effectiveness of the proposed human operation model.

Exploiting contextual information for prosodic event detection using auto-context

Prosody and prosodic boundaries carry significant information regarding linguistics and paralinguistics and are important aspects of speech. In the field of prosodic event detection, many local acoustic features have been investigated; however, contextual information has not yet been thoroughly exploited. The most difficult aspect of this lies in learning the long-distance contextual dependencies effectively and efficiently. To address this problem, we introduce the use of an algorithm called auto-context. In this algorithm, a classifier is first trained based on a set of local acoustic features, after which the generated probabilities are used along with the local features as contextual information to train new classifiers. By iteratively using updated probabilities as the contextual information, the algorithm can accurately model contextual dependencies and improve classification ability. The advantages of this method include its flexible structure and the ability of capturing contextual relationships. When using the auto-context algorithm based on support vector machine, we can improve the detection accuracy by about 3% and F-score by more than 7% on both two-way and four-way pitch accent detections in combination with the acoustic context. For boundary detection, the accuracy improvement is about 1% and the F-score improvement reaches 12%. The new algorithm outperforms conditional random fields, especially on boundary detection in terms of F-score. It also outperforms an n-gram language model on the task of pitch accent detection.

Incremental Tree Substitution Grammar for Parsing and Sentence Prediction

In this paper, we present the first incremental parser for Tree Substitution Grammar (TSG). A TSG allows arbitrarily large syntactic fragments to be combined into complete trees; we show how constraints (including lexicalization) can be imposed on the shape of the TSG fragments to enable incremental processing. We propose an efficient Earley-based algorithm for incremental TSG parsing and report an F-score competitive with other incremental parsers. In addition to whole-sentence F-score, we also evaluate the partial trees that the parser constructs for sentence prefixes; partial trees play an important role in incremental interpretation, language modeling, and psycholinguistics. Unlike existing parsers, our incremental TSG parser can generate partial trees that include predictions about the upcoming words in a sentence. We show that it outperforms an n-gram model in predicting more than one upcoming word.

MicroWindows-based multi-device support intelligent Chinese input system

The existing embedded Chinese input systems are restricted by problems such as single type of input device,low inefficiency and poor user experience.To solve these problems,this paper put forward a MicroWindows-based intelligent Chinese input system.First messages from different types of device were packed and delivered in the device input layer.Then the delivered messages were centrally treated by uniformly encoding and distributing in the message processing center.Finally the improved N-gram model and user model were combined to implement Chinese input method.The experiments in Microprocessor without Interlocked Piped Stages(MIPS) and other platform show that this system runs well with fluent and fast Chinese input.The efficiency of input is raised by 35% compared with traditional Chinese input.

N-gram distribution and unification gain problem and its optimal solution

N-gram distribution and unification gain is a type of problem in which objects obtain gain after a series of regular actions, such as ‘distributions’ and ‘unifications’. The uncertainty return forms of the gain lead to the complexity of the whole gain process in the problem we propose. The gain path is usually concurrent and consecutive to the timeline in practice; thus, we are unable to solve the problem and obtain the optimal path or overall gain at a certain time using the optimal path algorithm alone. Therefore, the N-gram distribution and unification gain model, which utilises a new dynamic programming algorithm in solving problems, is proposed. This procedure facilitates the solving of similar comprehensive gain problems and obtaining important information, such as the optimal gain path and the overall gain.

A context evaluation approach for structural comparison of proteins using cross entropy over n-gram modelling

The structural comparison of proteins is a vital step in structural biology that is used to predict and analyse a new unknown protein function. Although a number of different techniques have been explored, the study to develop new alternative methods is still an active research area. The present paper introduces a text modelling-based technique for the structural comparison of proteins. The method models the secondary and tertiary structure of proteins in two linear sequences and then applies them to the comparison of two structures. The technique used for pairwise comparison of the sequences has been adopted from computational linguistics and its well-known techniques for analysing and quantifying textual sequences. To this end, an n-gram modelling technique is used to capture regularities between sequences, and then, the cross-entropy concept is employed to measure their similarities. Several experiments are conducted to evaluate the performance of the method and compare it with other commonly used programs. The assessments for information retrieval evaluation demonstrate that the technique has a high running speed, which is similar to other linear encoding methods, such as 3D-BLAST, SARST, and TS-AMIR, whereas its accuracy is comparable to CE and TM-align, which are high accuracy comparison tools. Accordingly, the results demonstrate that the algorithm has high efficiency compared with other state-of-the-art methods.

Semantic spaces for improving language modeling

Language models are crucial for many tasks in NLP (Natural Language Processing) and n-grams are the best way to build them. Huge effort is being invested in improving n-gram language models. By introducing external information (morphology, syntax, partitioning into documents, etc.) into the models a significant improvement can be achieved. The models can however be improved with no external information and smoothing is an excellent example of such an improvement.In this article we show another way of improving the models that also requires no external information. We examine patterns that can be found in large corpora by building semantic spaces (HAL, COALS, BEAGLE and others described in this article). These semantic spaces have never been tested in language modeling before. Our method uses semantic spaces and clustering to build classes for a class-based language model. The class-based model is then coupled with a standard n-gram model to create a very effective language model.Our experiments show that our models reduce the perplexity and improve the accuracy of n-gram language models with no external information added. Training of our models is fully unsupervised. Our models are very effective for inflectional languages, which are particularly hard to model. We show results for five different semantic spaces with different settings and different number of classes. The perplexity tests are accompanied with machine translation tests that prove the ability of proposed models to improve performance of a real-world application.

Leveraging topical and positional cues for language modeling in speech recognition

This paper investigates language modeling with topical and positional information for large vocabulary continuous speech recognition. We first compare among a few topic models both theoretically and empirically, including document topic models and word topic models. On the other hand, since for some spoken documents such as broadcast news stories, the composition and the word usage of documents of the same style are usually similar, the documents hence can be separated into partitions consisting of identical rhetoric or topic styles by the literary structures, like introductory remarks, elucidations of methodology or affairs, conclusions of the articles, references or footnotes of reporters, etc. We hence present two position-dependent language models for speech recognition by integrating word positional information into the exiting n-gram and topic models. The experiments conducted on broadcast news transcription seem to indicate that such position-dependent models obtain comparable results to the existing n-gram and topic models.

Combining compound and single terms under language model framework

Most existing Information Retrieval model including probabilistic and vector space models are based on the term independence hypothesis. To go beyond this assumption and thereby capture the semantics of document and query more accurately, several works have incorporated phrases or other syntactic information in IR, such attempts have shown slight benefit, at best. Particularly in language modeling approaches this extension is achieved through the use of the bigram or n-gram models. However, in these models all bigrams/n-grams are considered and weighted uniformly. In this paper we introduce a new approach to select and weight relevant n-grams associated with a document. Experimental results on three TREC test collections showed an improvement over three strongest state-of-the-art model baselines, which are the original unigram language model, the Markov Random Field model, and the positional language model.

Leveraging relevance cues for language modeling in speech recognition

Language modeling (LM), providing a principled mechanism to associate quantitative scores to sequences of words or tokens, has long been an interesting yet challenging problem in the field of speech and language processing. The n-gram model is still the predominant method, while a number of disparate LM methods, exploring either lexical co-occurrence or topic cues, have been developed to complement the n-gram model with some success. In this paper, we explore a novel language modeling framework built on top of the notion of relevance for speech recognition, where the relationship between a search history and the word being predicted is discovered through different granularities of semantic context for relevance modeling. Empirical experiments on a large vocabulary continuous speech recognition (LVCSR) task seem to demonstrate that the various language models deduced from our framework are very comparable to existing language models both in terms of perplexity and recognition error rate reductions.

A Method of Hot Topic Detection in Blogs Using N-gram Model

Over the last few years, blogs (web logs) have gained massive popularity and have become one of the most important web social media, through which people can get and release information. Hot topic detection in blogs is most commonly used in analyzing network public opinion. A method of hot topic detection using n -gram model and hotness of topic evaluation is proposed in this paper. Our approach consists of three steps. First of all, keywords during a given time period are obtained by means of calculating word's weight, and hot keywords are collected by combining keywords. Secondly, based on hot keywords, hot keyword groups are extracted using n -gram model. In the third step, hot keyword groups are extracted and hot topics are detected. The hotness of hot topic is evaluated by the value of keywords’ weight, which is got in the second step. Evaluations on Chinese corpus show that when the size of n for n -gram is five, the proposed method is most effective.

Online Review Spam Detection using Language Model and Feature Selection

In today’s age of web 2.0, large numbers of product reviews posted on the Internet. Such reviews are important to customers or users and to companies. Customers use the reviews for deciding quality of product to buy. Companies or vendors use opinions to take a decision to improve their sales according to intelligent things done by other competitors. However, all reviews are given by customers or users are not true reviews. These reviews are given to promote or to demote the product. Some reviews are given on brand of product, and others are related to advertising of another product. There is need to find how many reviews are spam or non spam. In this paper, the system is proposed for detecting untruthful spam reviews using n-gram language model and reviews on brand spam detection using Feature Selection. Given system separately identifies spam and joined the result showing spam and non spam reviews.

FASTSUBS: An Efficient and Exact Procedure for Finding the Most Likely Lexical Substitutes Based on an N-Gram Language Model

Lexical substitutes have found use in areas such as paraphrasing, text simplification, machine translation, word sense disambiguation, and part of speech induction. However the computational complexity of accurately identifying the most likely substitutes for a word has made large scale experiments difficult. In this paper I introduce a new search algorithm, FASTSUBS, that is guaranteed to find the K most likely lexical substitutes for a given word in a sentence based on an n-gram language model. The computation is sub-linear in both K and the vocabulary size V. An implementation of the algorithm and a dataset with the top 100 substitutes of each token in the WSJ section of the Penn Treebank are available at http://goo.gl/jzKH0.

A Scalable Distributed Syntactic, Semantic, and Lexical Language Model

This paper presents an attempt at building a large scale distributed composite language model that is formed by seamlessly integrating an n-gram model, a structured language model, and probabilistic latent semantic analysis under a directed Markov random field paradigm to simultaneously account for local word lexical information, mid-range sentence syntactic structure, and long-span document semantic content. The composite language model has been trained by performing a convergent N-best list approximate EM algorithm and a follow-up EM algorithm to improve word prediction power on corpora with up to a billion tokens and stored on a supercomputer. The large scale distributed composite language model gives drastic perplexity reduction over n-grams and achieves significantly better translation quality measured by the Bleu score and “readability” of translations when applied to the task of re-ranking the N-best list from a state-of-the-art parsing-based machine translation system.

Common Sense Knowledge for Handwritten Chinese Text Recognition

Compared to human intelligence, computers are far short of common sense knowledge which people normally acquire during the formative years of their lives. This paper investigates the effects of employing common sense knowledge as a new linguistic context in handwritten Chinese text recognition. Three methods are introduced to supplement the standard n-gram language model: embedding model, direct model, and an ensemble of these two. The embedding model uses semantic similarities from common sense knowledge to make the n-gram probabilities estimation more reliable, especially for the unseen n-grams in the training text corpus. The direct model, in turn, considers the linguistic context of the whole document to make up for the short context limit of the n-gram model. The three models are evaluated on a large unconstrained handwriting database, CASIA-HWDB, and the results show that the adoption of common sense knowledge yields improvements in recognition performance, despite the reduced concept list hereby employed.

Approximate inference: A sampling based modeling technique to capture complex dependencies in a language model

In this paper, we present strategies to incorporate long context information directly during the first pass decoding and also for the second pass lattice re-scoring in speech recognition systems. Long-span language models that capture complex syntactic and/or semantic information are seldom used in the first pass of large vocabulary continuous speech recognition systems due to the prohibitive increase in the size of the sentence-hypotheses search space. Typically, n-gram language models are used in the first pass to produce N-best lists, which are then re-scored using long-span models. Such a pipeline produces biased first pass output, resulting in sub-optimal performance during re-scoring. In this paper we show that computationally tractable variational approximations of the long-span and complex language models are a better choice than the standard n-gram model for the first pass decoding and also for lattice re-scoring.

Classification of Tandem Repeats in the Human Genome

Tandem repeats in DNA sequences are extremely relevant in biological phenomena and diagnostic tools. Computational programs that discover these tandem repeats generate a huge volume of data, which is often difficult to decipher without further organization. In this paper, the authors describe a new method for post-processing tandem repeats through clustering and classification. Their work presents multiple ways of expressing tandem repeats using the n-gram model with different clustering distance measures. Analysis of the clusters for the tandem repeats in the human genome shows that the method yields a well-defined grouping in which similarity among repeats is apparent. The authors’ new, alignment-free method facilitates the analysis of the myriad of tandem repeats that occur in the human genome and they believe that this work will lead to new discoveries on the roles, origins, and significance of tandem repeats.

Gamelyn’s Place among the Early Exemplars for Chaucer’s Canterbury Tales

Application of standard techniques from natural language processing on n-gram models of spelling enables quantification of the similarity between Middle English texts despite their lexical differences. Three studies employing similarity metrics confirm that a scribe’s spelling always is biased in the direction of his exemplars. This bias opens up a window on the number of scribes behind the exemplars for a text executed in a single hand, when other variables such as authorship and poetic form are held constant. A fourth study addresses nine manuscripts of Geoffrey Chaucer’s poem the Canterbury Tales with early textual contents. A one-way ANOVA/Tukey’s Range Test shows that none of these manuscripts is based on exemplars written in more than three hands, when allowance is made for variation due to poetic form. The results point to unified exemplars for the full text as the normal format for the poem’s transmission. The discussion suggests that the final tale ordering found in the first manuscripts is a product of collaboration between the poem’s first two scribes, probably working after Chaucer’s death and spuriously adding the Tale of Gamelyn.