Word Frequency Distributions Research Articles

A novel density-based clustering method using word embedding features for dialogue intention recognition

In dialogue systems, understanding user utterances is crucial for providing appropriate responses. Various classification models have been proposed to deal with natural language understanding tasks related to user intention analysis, such as dialogue acts or emotion recognition. However, models that use original lexical features without any modifications encounter the problem of data sparseness, and constructing sufficient training data to overcome this problem is labor-intensive, time-consuming, and expensive. To address this issue, word embedding models that can learn lexical synonyms using vast raw corpora have recently been proposed. However, the analysis of embedding features is not yet sufficient to validate the efficiency of such models. Specifically, using the cosine similarity score as a feature in the embedding space neglects the skewed nature of the word frequency distribution, which can affect the improvement of model performance. This paper describes a novel density-based clustering method that efficiently integrates word embedding vectors into dialogue intention recognition. Experimental results show that our proposed model helps overcome the data sparseness problem seen in previous classification models and can assist in improving the classification performance.

Hoaxing statistical features of the Voynich Manuscript

ABSTRACTIn a previous article, the first author demonstrated that simple materials and techniques could produce meaningless text of comparable complexity to the text in the Voynich Manuscript, at a speed which made a hoax a feasible explanation. The table and grille method described in that article also replicated the main qualitative features of the text in the Voynich Manuscript. In this article, the authors demonstrate that the same table and grille method can also replicate the main quantitative statistical features of the text in the Voynich Manuscript, namely a distribution of word frequencies that mimics Zipf’s distribution, a symmetrical distribution of word length frequencies, and a non-homogeneous distribution of words and of syllables across a corpus of text produced using this method. The main unusual qualitative and quantitative features of the Voynich Manuscript are therefore explicable as products of a low-technology hoax, with no need to invoke an undiscovered new type of code and/or the presence of meaningful text in the manuscript.

On variation of word frequencies in Russian literary texts

We study the variation of word frequencies in Russian literary texts. Our findings indicate that the standard deviation of a word’s frequency across texts depends on its average frequency according to a power law with exponent 12<α<1, which shows that the rarer words have a relatively larger degree of frequency volatility (that is, higher “burstiness”).A latent factor model has been estimated to investigate the structure of the word frequency distribution. The findings suggest that the dependence of a word’s frequency volatility on its average frequency can be explained by the asymmetry in the distribution of latent factors.

A New Hybrid Stemming Method for Persian Language

One of the important issues in natural language processing and information retrieval is the automatic extraction of the word’s stem. Both statistical and rule-based approaches for stemming have their own advantages and limitations. The statistical stemmers are not accurate and fail to take advantage of some language phenomenon which can be easily expressed by simple rules. On the other hand, handcrafting the stemming rules in the rule-based stemmers is a time-consuming, tedious, and impractical task. In this regard, we propose a new hybrid stemming method based on a combination of affix stripping and statistical techniques for Persian language. The proposed method combines cues from the orthography, word frequency, and syntactic distributions to induce the stemming rules. In general, the proposed method is divided into two main parts. In the first part, all words of the annotated text corpus are used to automatically induce the stemming rules; while in the second part, the rule-based stemmer uses the induced stemming rules to discover the word's stem. We test the performance of the proposed scheme on two different data sets. The encouraging results indicate the superior performance of the proposed method compared with its counterparts.

Randomness versus specifics for word-frequency distributions

The text-length-dependence of real word-frequency distributions can be connected to the general properties of a random book. It is pointed out that this finding has strong implications, when deciding between two conceptually different views on word-frequency distributions, i.e. the specific ‘Zipf’s-view’ and the non-specific ‘Randomness-view’, as is discussed. It is also noticed that the text-length transformation of a random book does have an exact scaling property precisely for the power-law index γ=1, as opposed to the Zipf’s exponent γ=2 and the implication of this exact scaling property is discussed. However a real text has γ>1 and as a consequence γ increases when shortening a real text. The connections to the predictions from the RGF (Random Group Formation) and to the infinite length-limit of a meta-book are also discussed. The difference between ‘curve-fitting’ and ‘predicting’ word-frequency distributions is stressed. It is pointed out that the question of randomness versus specifics for the distribution of outcomes in case of sufficiently complex systems has a much wider relevance than just the word-frequency example analyzed in the present work.

Zipf's Law for Word Frequencies: Word Forms versus Lemmas in Long Texts.

Zipf’s law is a fundamental paradigm in the statistics of written and spoken natural language as well as in other communication systems. We raise the question of the elementary units for which Zipf’s law should hold in the most natural way, studying its validity for plain word forms and for the corresponding lemma forms. We analyze several long literary texts comprising four languages, with different levels of morphological complexity. In all cases Zipf’s law is fulfilled, in the sense that a power-law distribution of word or lemma frequencies is valid for several orders of magnitude. We investigate the extent to which the word-lemma transformation preserves two parameters of Zipf’s law: the exponent and the low-frequency cut-off. We are not able to demonstrate a strict invariance of the tail, as for a few texts both exponents deviate significantly, but we conclude that the exponents are very similar, despite the remarkable transformation that going from words to lemmas represents, considerably affecting all ranges of frequencies. In contrast, the low-frequency cut-offs are less stable, tending to increase substantially after the transformation.

An Efficient Approach to Construct Object Model of Static Textual Structure with Dynamic Behavior Based on Q-learning

Developing information technology led to raise the intricacy of information systems intensive, hence techniques with effectiveness and efficiency are required. These techniques are used to support users in using the information for rapid and correct decision-making. Conventional text mining and managing systems mainly use the presence or absence of keywords to discover and analyze useful information from textual documents. However, simple word counting and frequency distributions of term appearances do not capture the meaning behind the words, which results in limiting the ability to mine the texts. This paper has been primarily concerned with constructing text representation model and exploiting that in mining and managing operations such as gathering, searching, filtering, retrieving, extracting, clustering, classifying, and summarizing. This representation model is based on semantic notions to represent text in documents, to infer unknown dependencies and relationships among concepts in a text, to measure the relatedness between text documents, and to apply mining processes using the representation and the relatedness measure. This model reflects the existing relations among concepts and facilitates accurate relatedness measurements that result in better mining performance. The experimental evaluations were carried out on real datasets from various domains, showing the importance of the proposed model.

Understanding Zipf's law of word frequencies through sample-space collapse in sentence formation.

The formation of sentences is a highly structured and history-dependent process. The probability of using a specific word in a sentence strongly depends on the 'history' of word usage earlier in that sentence. We study a simple history-dependent model of text generation assuming that the sample-space of word usage reduces along sentence formation, on average. We first show that the model explains the approximate Zipf law found in word frequencies as a direct consequence of sample-space reduction. We then empirically quantify the amount of sample-space reduction in the sentences of 10 famous English books, by analysis of corresponding word-transition tables that capture which words can follow any given word in a text. We find a highly nested structure in these transition tables and show that this 'nestedness' is tightly related to the power law exponents of the observed word frequency distributions. With the proposed model, it is possible to understand that the nestedness of a text can be the origin of the actual scaling exponent and that deviations from the exact Zipf law can be understood by variations of the degree of nestedness on a book-by-book basis. On a theoretical level, we are able to show that in the case of weak nesting, Zipf's law breaks down in a fast transition. Unlike previous attempts to understand Zipf's law in language the sample-space reducing model is not based on assumptions of multiplicative, preferential or self-organized critical mechanisms behind language formation, but simply uses the empirically quantifiable parameter 'nestedness' to understand the statistics of word frequencies.

Maximum entropy, word-frequency, Chinese characters, and multiple meanings.

The word-frequency distribution of a text written by an author is well accounted for by a maximum entropy distribution, the RGF (random group formation)-prediction. The RGF-distribution is completely determined by the a priori values of the total number of words in the text (M), the number of distinct words (N) and the number of repetitions of the most common word (kmax). It is here shown that this maximum entropy prediction also describes a text written in Chinese characters. In particular it is shown that although the same Chinese text written in words and Chinese characters have quite differently shaped distributions, they are nevertheless both well predicted by their respective three a priori characteristic values. It is pointed out that this is analogous to the change in the shape of the distribution when translating a given text to another language. Another consequence of the RGF-prediction is that taking a part of a long text will change the input parameters (M, N, kmax) and consequently also the shape of the frequency distribution. This is explicitly confirmed for texts written in Chinese characters. Since the RGF-prediction has no system-specific information beyond the three a priori values (M, N, kmax), any specific language characteristic has to be sought in systematic deviations from the RGF-prediction and the measured frequencies. One such systematic deviation is identified and, through a statistical information theoretical argument and an extended RGF-model, it is proposed that this deviation is caused by multiple meanings of Chinese characters. The effect is stronger for Chinese characters than for Chinese words. The relation between Zipf’s law, the Simon-model for texts and the present results are discussed.

A corpus-based lexical analysis of subject-specific university textbooks for English majors

This study is a corpus-based lexical analysis of subject-specific university textbooks which purports to explore lexical text coverage and frequency distribution of words from the Academic Word List and the British National Corpus frequency-based word families. For this study a 508,802-word corpus was created, the findings of which reflect that the Academic Word List word families constitute only a small coverage (6.5%) of the words in the entire corpus, whereas the first two thousand high-frequency word families give the coverage of 88.92%. In terms of the text coverage, the results reveal that if 98% coverage of a text is needed for unassisted comprehension, then a vocabulary size of 9000 word families is required. The results also substantiate the claims that the Academic Word List is not as general an academic vocabulary as it was initially intended to be and, more importantly, supports the assumption that students need a more restricted core academic vocabulary. It is therefore argued that 127 academic word families which are relatively frequent in the overall university textbook corpus can be used as a part of the university word list for second-year English majors who have to read and comprehend university textbooks.

Exploiting semantic annotations and Q-learning for constructing an efficient hierarchy/graph texts organization.

Tremendous growth in the number of textual documents has produced daily requirements for effective development to explore, analyze, and discover knowledge from these textual documents. Conventional text mining and managing systems mainly use the presence or absence of key words to discover and analyze useful information from textual documents. However, simple word counts and frequency distributions of term appearances do not capture the meaning behind the words, which results in limiting the ability to mine the texts. This paper proposes an efficient methodology for constructing hierarchy/graph-based texts organization and representation scheme based on semantic annotation and Q-learning. This methodology is based on semantic notions to represent the text in documents, to infer unknown dependencies and relationships among concepts in a text, to measure the relatedness between text documents, and to apply mining processes using the representation and the relatedness measure. The representation scheme reflects the existing relationships among concepts and facilitates accurate relatedness measurements that result in a better mining performance. An extensive experimental evaluation is conducted on real datasets from various domains, indicating the importance of the proposed approach.

Universals versus historical contingencies in lexical evolution.

The frequency with which we use different words changes all the time, and every so often, a new lexical item is invented or another one ceases to be used. Beyond a small sample of lexical items whose properties are well studied, little is known about the dynamics of lexical evolution. How do the lexical inventories of languages, viewed as entire systems, evolve? Is the rate of evolution of the lexicon contingent upon historical factors or is it driven by regularities, perhaps to do with universals of cognition and social interaction? We address these questions using the Google Books N-Gram Corpus as a source of data and relative entropy as a measure of changes in the frequency distributions of words. It turns out that there are both universals and historical contingencies at work. Across several languages, we observe similar rates of change, but only at timescales of at least around five decades. At shorter timescales, the rate of change is highly variable and differs between languages. Major societal transformations as well as catastrophic events such as wars lead to increased change in frequency distributions, whereas stability in society has a dampening effect on lexical evolution.

Scaling laws and fluctuations in the statistics of word frequencies

In this paper, we combine statistical analysis of written texts and simple stochastic models to explain the appearance of scaling laws in the statistics of word frequencies. The average vocabulary of an ensemble of fixed-length texts is known to scale sublinearly with the total number of words (Heaps’ law). Analyzing the fluctuations around this average in three large databases (Google-ngram, English Wikipedia, and a collection of scientific articles), we find that the standard deviation scales linearly with the average (Taylorʼs law), in contrast to the prediction of decaying fluctuations obtained using simple sampling arguments. We explain both scaling laws (Heaps’ and Taylor) by modeling the usage of words using a Poisson process with a fat-tailed distribution of word frequencies (Zipfʼs law) and topic-dependent frequencies of individual words (as in topic models). Considering topical variations lead to quenched averages, turn the vocabulary size a non-self-averaging quantity, and explain the empirical observations. For the numerous practical applications relying on estimations of vocabulary size, our results show that uncertainties remain large even for long texts. We show how to account for these uncertainties in measurements of lexical richness of texts with different lengths.

When a text is translated does the complexity of its vocabulary change? Translations and target readerships.

In linguistic studies, the academic level of the vocabulary in a text can be described in terms of statistical physics by using a “temperature” concept related to the text's word-frequency distribution. We propose a “comparative thermo-linguistic” technique to analyze the vocabulary of a text to determine its academic level and its target readership in any given language. We apply this technique to a large number of books by several authors and examine how the vocabulary of a text changes when it is translated from one language to another. Unlike the uniform results produced using the Zipf law, using our “word energy” distribution technique we find variations in the power-law behavior. We also examine some common features that span across languages and identify some intriguing questions concerning how to determine when a text is suitable for its intended readership.

Measurement of word frequencies in genomic DNA sequences based on partial alignment and fuzzy set.

Accompanied with the rapid increase of the amount of data registered in the databases of biological sequences, the need for a fast method of sequence comparison applicable to sequences of large size is also increasing. In general, alignment is used for sequence comparison. However, the alignment may not be appropriate for comparison of sequences of large size such as whole genome sequences due to its large time complexity. In this article, we propose a semi alignment-free method of sequence comparison based on word frequency distributions, in which we partially use the alignment to measure word frequencies along with the idea of fuzzy set theory. Experiments with ten bacterial genome sequences demonstrated that the fuzzy measurements has the effect that facilitates discrimination between close relatives and distant relatives.

Zipf’s word frequency law in natural language: A critical review and future directions

The frequency distribution of words has been a key object of study in statistical linguistics for the past 70 years. This distribution approximately follows a simple mathematical form known as Zipf's law. This article first shows that human language has a highly complex, reliable structure in the frequency distribution over and above this classic law, although prior data visualization methods have obscured this fact. A number of empirical phenomena related to word frequencies are then reviewed. These facts are chosen to be informative about the mechanisms giving rise to Zipf's law and are then used to evaluate many of the theoretical explanations of Zipf's law in language. No prior account straightforwardly explains all the basic facts or is supported with independent evaluation of its underlying assumptions. To make progress at understanding why language obeys Zipf's law, studies must seek evidence beyond the law itself, testing assumptions and evaluating novel predictions with new, independent data.

Zipf's law and the grammar of languages: A quantitative study of Old and Modern English parallel texts

This paper reports a quantitative analysis of the relationship between word frequency distributions and morphological features in languages. We analyze a commonly-observed process of historical language change: The loss of inflected forms in favour of ‘analytic’ periphrastic constructions. These tendencies are observed in parallel translations of the Book of Genesis in Old English and Modern English. We show that there are significant differences in the frequency distributions of the two texts, and that parts of these differences are independent of total number of words, style of translation, orthography or contents. We argue that they derive instead from the trade-off between synthetic inflectional marking in Old English and analytic constructions in Modern English. By exploiting the earliest ideas of Zipf, we show that the syntheticity of the language in these texts can be captured mathematically, a property we tentatively call their grammatical fingerprint. Our findings suggest implications for both the specific historical process of inflection loss and more generally for the characterization of languages based on statistical properties.

A scaling law beyond Zipf's law and its relation to Heaps' law

The dependence on text length of the statistical properties of word occurrences has long been considered a severe limitation on the usefulness of quantitative linguistics. We propose a simple scaling form for the distribution of absolute word frequencies that brings to light the robustness of this distribution as text grows. In this way, the shape of the distribution is always the same, and it is only a scale parameter that increases (linearly) with text length. By analyzing very long novels we show that this behavior holds both for raw, unlemmatized texts and for lemmatized texts. In the latter case, the distribution of frequencies is well approximated by a double power law, maintaining the Zipf's exponent value γ ≃ 2 for large frequencies but yielding a smaller exponent in the low-frequency regime. The growth of the distribution with text length allows us to estimate the size of the vocabulary at each step and to propose a generic alternative to Heaps' law, which turns out to be intimately connected to the distribution of frequencies, thanks to its scaling behavior.

The Negative Binomial Model of Word Usage

How people make texts is a rarely asked but central question in linguistics. From the language engineering perspective texts are outcomes of stochastic processes. A cognitive approach in linguistics holds that the speaker’s intention is the key to text formation. We propose a biologically inspired statistical formulation of word usage that brings together these views. We have observed that in several multilingual text collections1 word frequency distributions in a majority of non-rare words fit the negative binomial distribution (NBD). However, word counts in artificially randomized (permuted) texts agree with the geometric distribution. We conclude that the parameters of NBD deal with linguistic features of words. Specifically, named entities tend to have similar parameter values. We suggest that the NDB in natural texts accounts for intentionality of the speaker.

Gamma-Poisson Distribution Model for Text Categorization

We introduce a new model for describing word frequency distributions in documents for automatic text classification tasks. In the model, the gamma-Poisson probability distribution is used to achieve better text modeling. The framework of the modeling and its application to text categorization are demonstrated with practical techniques for parameter estimation and vector normalization. To investigate the efficiency of our model, text categorization experiments were performed on 20 Newsgroups, Reuters-21578, Industry Sector, and TechTC-100 datasets. The results show that the model allows performance comparable to that of the support vector machine and clearly exceeding that of the multinomial model and the Dirichlet-multinomial model. The time complexity of the proposed classifier and its advantage in practical applications are also discussed.