N-gram Frequency Research Articles

The subjective frequency of word n-grams

When asked to think about the subjective frequency of an n-gram (a group of n words), what properties of the n-gram influence the respondent? It has been recently shown that n-grams that occurred more frequently in a large corpus of English were read faster than n-grams that occurred less frequently (Arnon & Snider, 2010), an effect that is analogous to the frequency effects in word reading and lexical decision. The subjective frequency of words has also been extensively studied and linked to performance on linguistic tasks. We investigated the capacity of people to gauge the absolute and relative frequencies of n-grams. Subjective frequency ratings collected for 352 n-grams showed a strong correlation with corpus frequency, in particular for n-grams with the highest subjective frequency. These n-grams were then paired up and used in a relative frequency decision task (e.g. Is green hills more frequent than weekend trips?). Accuracy on this task was reliably above chance, and the trial-level accuracy was best predicted by a model that included the corpus frequencies of the whole n-grams. A computational model of word recognition (Baayen, Milin, Djurdjevic, Hendrix, & Marelli, 2011) was then used to attempt to simulate subjective frequency ratings, with limited success. Our results suggest that human n-gram frequency intuitions arise from the probabilistic information contained in n-grams.

ThetextcatPackage forn-Gram Based Text Categorization inR

Identifying the language used will typically be the first step in most natural language processing tasks. Among the wide variety of language identification methods discussed in the literature, the ones employing the Cavnar and Trenkle (1994) approach to text categorization based on character n-gram frequencies have been particularly successful. This paper presents the R extension package textcat for n-gram based text categorization which implements both the Cavnar and Trenkle approach as well as a reduced n-gram approach designed to remove redundancies of the original approach. A multi-lingual corpus obtained from the Wikipedia pages available on a selection of topics is used to illustrate the functionality of the package and the performance of the provided language identification methods. (authors' abstract)

SUITE OF TOOLS FOR STATISTICAL N-GRAM LANGUAGE MODELING FOR PATTERN MINING IN WHOLE GENOME SEQUENCES

Genome sequences contain a number of patterns that have biomedical significance. Repetitive sequences of various kinds are a primary component of most of the genomic sequence patterns. We extended the suffix-array based Biological Language Modeling Toolkit to compute n-gram frequencies as well as n-gram language-model based perplexity in windows over the whole genome sequence to find biologically relevant patterns. We present the suite of tools and their application for analysis on whole human genome sequence.

Early Fusion of Low Level Features for Emotion Mining

We study the discrimination of emotions annotated in free texts at the sentence level: a sentence can either be associated with no emotion (neutral) or multiple labels of emotion. The proposed system relies on three characteristics. We implement an early fusion of grams of increasing orders transposing an approach successfully employed in the related task of opinion mining. We apply a filtering process that consists in extracting frequent n-grams and making use of the Shannon’s entropy measure to respectively maintain dictionaries at balanced sizes and keep emotion specific features. Finally the overall system is implemented as a 2-step decision process: a first classifier discriminates between neutral and emotion bearing sentences, then one classifier per emotion is applied on emotion bearing sentences. The final decision is given by the classifier holding the maximum confidence. Results obtained on the testing set are promising.

The effects of N-gram probabilistic measures on the recognition and production of four-word sequences

The present study investigates the processing and production of four-word sequences such as I don’t really know, at the age of, and I think it’s the. Specifically, we investigate the influence of families of probabilistic measures such as unigram, bigram, trigram, and quadgram frequency of occurrence, logarithmic (log) probability of occurrence, and mutual information. Log probability of occurrence emerged as the predominant predictor family in the onset latency analysis, suggesting that recognition is mainly underpinned by competition between a target N-gram and its family members. In contrast, the amount of experience one has with an N-gram (frequency of occurrence) surfaced as the most prominent predictor in production. Further, probabilistic measures tied to trigrams surfaced as the best predictors in the onset latency analysis, while the measures tied to unigrams were most predictive of production durations.Finally, the interactions between probabilistic measures tied to unigrams, bigrams, trigrams, and quadgrams suggest that N-grams of different lengths are processed in parallel in both recognition and production.

Culturomic Analysis of Trends in Otolaryngology

Objective: 1) Quantitatively describe historical trends in diagnoses and treatments in otolaryngology as measured by frequency of word use. 2) Quantitatively describe historical trends in terminology used in otolaryngology. Method: A variety of search terms of n-grams (words or short phrases) specific to the practice of otolaryngology was used to search the English Language Corpus of Google Books. Trends in n-gram frequency across years were correlated to known historical milestones in otolaryngology. Results: A variety of search terms was used, the results of which quantitatively document historical trends in otolaryngology. For example the frequency of “stapes” in the corpus increased sharply around 1957, correlated with increases in the frequency of the terms “otosclerosis” and “stapedectomy,” whereas “incus” showed no similar increase in frequency. These frequencies correlate well with the popularization of stapedectomy for otosclerosis. Many additional analyses are presented with similarly strong correlations. In addition, trends of word use in otolaryngology were quantified by the same technique. Conclusion: Culturomic searches accurately demonstrate historical trends in otolaryngology, permitting both quantitative analysis of known trends and revelation of previously unknown trends in diagnosis, treatment, and terminology use in otolaryngology.

An Effective Method of Arbitrary Length N-gram Statistics for Chinese Text

N-gram frequency is always an important indicator in corpus-based natural language processing. As the corpus being larger, time and space consumption is always a serious issue. In this paper, we proposed a two-stage inverted index based on bigrams. The proposed index is constructed by two levels of variable length vectors and the two characters of a bigram are hashed to different levels. In order to ensure an efficient indexing, some null units are reserved in the vectors. By this way, unigrams and bigrams can be indexed directly. With the help of the index, the frequency of arbitrary length ngram can be calculated efficiently by intersection operation. The two-stage index has excellent properties: (1) it can be constructed efficiently; (2) the space consumption is significantly reduced with no compression; (3) when n=1 and n=2, time complexity of statistical processing decreases significantly. When n>2, it is more efficient than suffix array in most cases. This method is suitable for Chinese and some other Asian languages.

N-gram analysis of 970 microbial organisms reveals presence of biological language models

BackgroundIt has been suggested previously that genome and proteome sequences show characteristics typical of natural-language texts such as "signature-style" word usage indicative of authors or topics, and that the algorithms originally developed for natural language processing may therefore be applied to genome sequences to draw biologically relevant conclusions. Following this approach of 'biological language modeling', statistical n-gram analysis has been applied for comparative analysis of whole proteome sequences of 44 organisms. It has been shown that a few particular amino acid n-grams are found in abundance in one organism but occurring very rarely in other organisms, thereby serving as genome signatures. At that time proteomes of only 44 organisms were available, thereby limiting the generalization of this hypothesis. Today nearly 1,000 genome sequences and corresponding translated sequences are available, making it feasible to test the existence of biological language models over the evolutionary tree.ResultsWe studied whole proteome sequences of 970 microbial organisms using n-gram frequencies and cross-perplexity employing the Biological Language Modeling Toolkit and Patternix Revelio toolkit. Genus-specific signatures were observed even in a simple unigram distribution. By taking statistical n-gram model of one organism as reference and computing cross-perplexity of all other microbial proteomes with it, cross-perplexity was found to be predictive of branch distance of the phylogenetic tree. For example, a 4-gram model from proteome of Shigellae flexneri 2a, which belongs to the Gammaproteobacteria class showed a self-perplexity of 15.34 while the cross-perplexity of other organisms was in the range of 15.59 to 29.5 and was proportional to their branching distance in the evolutionary tree from S. flexneri. The organisms of this genus, which happen to be pathotypes of E.coli, also have the closest perplexity values with E. coli.ConclusionWhole proteome sequences of microbial organisms have been shown to contain particular n-gram sequences in abundance in one organism but occurring very rarely in other organisms, thereby serving as proteome signatures. Further it has also been shown that perplexity, a statistical measure of similarity of n-gram composition, can be used to predict evolutionary distance within a genus in the phylogenetic tree.

Improve feature selection method of web page language identification using fuzzy ARTMAP

The information available in languages other than English on the World Wide Web and global information systems is increasing significantly. Different languages can be produced by using one particular script such as Arabic, Persian, Urdu and Pashto that use Arabic script letters. The issue is how to produce reliable features of a web page that is to undergo language identification. Incorrectly identifying the language results in garbled translations as well as faulty and incomplete analyses. The aim of this study is to enhance the effectiveness of feature selection method of web page language identification. We have investigated total N-grams, N-grams frequency, N-grams frequency document frequency, and N-grams frequency inverse document frequency of web page language identification. From the experimental results, it is proven that N-grams frequency gives the most promising result compared to other feature selection methods.

N-grams based feature selection and text representation for Chinese Text Classification

In this paper, text representation and feature selection strategies for Chinese text classification based on n-grams are discussed. Two steps feature selection strategy is proposed which combines the preprocess within classes with the feature selection among classes. Four different feature selection methods and three text representation weights are compared by exhaustive experiments. Both C-SVC classifier and Naive bayes classifier are adopted to assess the results. All experiments are performed on Chinese corpus TanCorpV1.0 which includes more than 14,000 texts divided in 12 classes. Our experiments concern: (1) the performance comparison among different feature selection strategies: absolute text frequency, relative text frequency, absolute n-gram frequency and relative n-gram frequency; (2) the comparison of the sparseness and feature correlation in the “text by feature” matrices produced by four feature selection methods; (3) the performance comparison among three term weights: 0/1 logical value, n-gr...

Patterns of online behaviour in the United Kingdom and Japan: Insights based on asynchronous online conversations

This comparative study examines aspects of online behaviour exhibited by participants in online discussion groups in the United Kingdom and Japan. The primary data consists of message board threads gathered from U.K. and Japanese Internet forum sites with the analysis focusing on hyperlinks contained in the forum messages as well as dates and times of posting extracted from the message heads. A 'reading' of hyperlinks is undertaken through consulting N-gram frequencies obtained from each data set, juxtaposed and compared with the help of a coefficient of difference and the chi-square test. Contrasts in Internet surfing patterns, information-gathering preferences, and references to video, audio, pictorial and sexual content are examined; post times are used to compare daily and weekly patterns of posting activity between the two countries. This study also provides an overview of the uses that N-grams have in natural language processing and argues for their analytical potential in sociolinguistic and CMC-related research.

A machine learning approach for Arabic text classification using N-gram frequency statistics

In this paper a machine learning approach for classifying Arabic text documents is presented. To handle the high dimensionality of text documents, embeddings are used to map each document (instance) into R (the set of real numbers) representing the tri-gram frequency statistics profiles for a document. Classification is achieved by computing a dissimilarity measure, called the Manhattan distance, between the profile of the instance to be classified and the profiles of all the instances in the training set. The class (category) to which an instance (document) belongs is the one with the least computed Manhattan measure. The Dice similarity measure is used to compare the performance of method. Results show that tri-gram text classification using the Dice measure outperforms classification using the Manhattan measure.

N-Gram characterization of genomic islands in bacterial genomes

The paper presents a novel, n-gram-based method for analysis of bacterial genome segments known as genomic islands (GIs). Identification of GIs in bacterial genomes is an important task since many of them represent inserts that may contribute to bacterial evolution and pathogenesis. In order to characterize and distinguish GIs from rest of the genome, binary classification of islands based on n-gram frequency distribution have been performed. It consists of testing the agreement of islands n-gram frequency distributions with the complete genome and backbone sequence. In addition, a statistic based on the maximal order Markov model is used to identify significantly overrepresented and underrepresented n-grams in islands. The results may be used as a basis for Zipf-like analysis suggesting that some of the n-grams are overrepresented in a subset of islands and underrepresented in the backbone, or vice versa, thus complementing the binary classification. The method is applied to strain-specific regions in the Escherichia coli O157:H7 EDL933 genome (O-islands), resulting in two groups of O-islands with different n-gram characteristics. It refines a characterization based on other compositional features such as G + C content and codon usage, and may help in identification of GIs, and also in research and development of adequate drugs targeting virulence genes in them.

Parametric analysis of EEG and fMRI data reveals the neural basis of visual word processing

Parametric analysis of EEG and fMRI data reveals the neural basis of visual word processing

Speaker Verification Using Support Vector Machines and High-Level Features

High-level characteristics such as word usage, pronunciation, phonotactics, prosody, etc., have seen a resurgence for automatic speaker recognition over the last several years. With the availability of many conversation sides per speaker in current corpora, high-level systems now have the amount of data needed to sufficiently characterize a speaker. Although a significant amount of work has been done in finding novel high-level features, less work has been done on modeling these features. We describe a method of speaker modeling based upon support vector machines. Current high-level feature extraction produces sequences or lattices of tokens for a given conversation side. These sequences can be converted to counts and then frequencies of <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</i> -gram for a given conversation side. We use support vector machine modeling of these n-gram frequencies for speaker verification. We derive a new kernel based upon linearizing a log likelihood ratio scoring system. Generalizations of this method are shown to produce excellent results on a variety of high-level features. We demonstrate that our methods produce results significantly better than standard log-likelihood ratio modeling. We also demonstrate that our system can perform well in conjunction with standard cesptral speaker recognition systems.

The Quantitative Comparison of the Mongolian Dialects and Languages Using the N-gram Frequencies

The Quantitative Comparison of the Mongolian Dialects and Languages Using the N-gram Frequencies

N-gram analysis for computer virus detection

Generic computer virus detection is the need of the hour as most commercial antivirus software fail to detect unknown and new viruses. Motivated by the success of datamining/machine learning techniques in intrusion detection systems, recent research in detecting malicious executables is directed towards devising efficient non-signature-based techniques that can profile the program characteristics from a set of training examples. Byte sequences and byte n-grams are considered to be basis of feature extraction. But as the number of n-grams is going to be very large, several methods of feature selections were proposed in literature. A recent report on use of information gain based feature selection has yielded the best-known result in classifying malicious executables from benign ones. We observe that information gain models the presence of n-gram in one class and its absence in the other. Through a simple example we show that this may lead to erroneous results. In this paper, we describe a new feature selection measure, class-wise document frequency of byte n-grams. We empirically demonstrate that the proposed method is a better method for feature selection. For detection, we combine several classifiers using Dempster Shafer Theory for better classification accuracy instead of using any single classifier. Our experimental results show that such a scheme detects virus program far more efficiently than the earlier known methods.

The time course of visual word recognition as revealed by linear regression analysis of ERP data

EEG correlates of a range of psycholinguistic word properties were used to investigate the time course of access to psycholinguistic information during visual word recognition. Neurophysiological responses recorded in a visual lexical decision task were submitted to linear regression analysis. First, 10 psycholinguistic features of each of 300 stimulus words were submitted to a principal component analysis, which yielded four orthogonal variables likely to reflect separable processes in visual word recognition: Word length, Letter n-gram frequency, Lexical frequency and Semantic coherence of a word's morphological family. Since the lexical decision task required subjects to distinguish between words and pseudowords, the binary variable Lexicality was also investigated using a factorial design. Word–pseudoword differences in the event-related potential first appeared at 160 ms after word onset. However, regression analysis of EEG data documented a much earlier effect of both Word length and Letter n-gram frequency around 90 ms. Lexical frequency showed its earliest effect slightly later, at 110 ms, and Semantic coherence significantly correlated with neurophysiological measures around 160 ms, simultaneously with the lexicality effect. Source estimates indicated parieto-temporo-occipital generators for the factors Length, Letter n-gram frequency and Word frequency, but widespread activation with foci in left anterior temporal lobe and inferior frontal cortex related to Semantic coherence. At later stages (>200 ms), all variables exhibited simultaneous EEG correlates. These results indicate that information about surface form and meaning of a lexical item is first accessed at different times in different brain systems and then processed simultaneously, thus supporting cascaded interactive processing models.

Comparing Simple Recurrent Networks and n -Grams in a Large Corpus

The increased availability of text corpora and the growth of connectionism has stimulated a renewed interest in probabilistic models of language processing in computational linguistics and psycholinguistics. The Simple Recurrent Network (SRN) is an important connectionist model because it has the potential to learn temporal dependencies of unspecified length. In addition, many computational questions about the SRN's ability to learn dependencies between individual items extend to other models. This paper will report on experiments with an SRN trained on a large corpus and examine the ability of the network to learn bigrams, trigrams, etc., as a function of the size of the corpus. The performance is evaluated by an information theoretic measure of prediction (or guess) ranking and output vector entropy. With enough training and hidden units the SRN shows the ability to learn 5 and 6-gram dependencies, although learning an n-gram is contingent on its frequency and the relative frequency of other n-grams. In some cases, the network will learn relatively low frequency deep dependencies before relatively high frequency short ones if the deep dependencies do not require representational shifts in hidden unit space.

Comparing Corpora

Corpus linguistics lacks strategies for describing and comparing corpora. Currently most descriptions of corpora are textual, and questions such as ‘what sort of a corpus is this?’, or ‘how does this corpus compare to that?’ can only be answered impressionistically. This paper considers various ways in which different corpora can be compared more objectively. First we address the issue, ‘which words are particularly characteristic of a corpus?’, reviewing and critiquing the statistical methods which have been applied to the question and proposing the use of the Mann-Whitney ranks test. Results of two corpus comparisons using the ranks test are presented. Then, we consider measures for corpus similarity. After discussing limitations of the idea of corpus similarity, we present a method for evaluating corpus similarity measures. We consider several measures and establish that a\chi\tsup{2}-based one performs best. All methods considered in this paper are based on word and ngram frequencies; the strategy is defended.