N-gram Distributions Research Articles

A treatise on the Rilke cryptogram

The ‘Rilke Cryptogram’ is a possible cipher of German, WWII origin. This study summarises the text’s history, and examines explanations for its nature. Statistical techniques are applied to an updated transcription. The text consists of 18,760 characters in 3,305 unique groups of four, arranged into 670 rows by seven columns on 33 pages of an otherwise ordinary book. The distribution of these groups is not particularly Zipfian, suggesting the text is likely not a code or monoalphabetic substitution cipher. Its alphabet is longer than English and German (46 characters). Character and N-gram entropy analyses suggest the text is less ordered than English and German, but more ordered than random text. Machine-learning modelling suggests the text is a substitution cipher, but frequency and Kasiski-Kerckhoff analyses suggest otherwise. Sukhotin’s algorithm results for the text are somewhat consistent with results for German, though N-gram distributions do not strongly resemble English or German. Statistical analysis of physical typewriter key distances suggest the text’s groupings are highly unlikely to appear at random, and are consistent with intentional ‘lazy’ typing. The text is inconsistent with machine ciphers of its era (e.g., Enigma), and other features are not entirely consistent with non-cipher explanations. Further investigation is required.

On divergence-based author obfuscation: An attack on the state of the art in statistical authorship verification

Abstract Authorship verification is the task of determining whether two texts were written by the same author based on a writing style analysis. Author obfuscation is the adversarial task of preventing a successful verification by altering a text’s style so that it does not resemble that of its original author anymore. This paper introduces new algorithms for both tasks and reports on a comprehensive evaluation to ascertain the merits of the state of the art in authorship verification to withstand obfuscation. After introducing a new generalization of the well-known unmasking algorithm for short texts, thus completing our collection of state-of-the-art algorithms for verification, we introduce an approach that (1) models writing style difference as the Jensen-Shannon distance between the character n-gram distributions of texts, and (2) manipulates an author’s writing style in a sophisticated manner using heuristic search. For obfuscation, we explore the huge space of textual variants in order to find a paraphrased version of the to-be-obfuscated text that has a sufficiently high Jensen-Shannon distance at minimal costs in terms of text quality loss. We analyze, quantify, and illustrate the rationale of this approach, define paraphrasing operators, derive text length-invariant thresholds for termination, and develop an effective obfuscation framework. Our authorship obfuscation approach defeats the presented state-of-the-art verification approaches, while keeping text changes at a minimum. As a final contribution, we discuss and experimentally evaluate a reverse obfuscation attack against our obfuscation approach as well as possible remedies.

Detecting Impoliteness and Incivility in Online Discussions

Abstract Impoliteness and incivility in online discussions have recently been discussed as relevant issues in communication science. However, automatically detecting these concepts with computational methods is challenging. In our study, we build and compare supervised classification models to predict impoliteness and incivility in online discussions on German media outlets on Facebook. Using a sample of 10,000 hand-coded user comments and a theory-grounded coding scheme, we develop classifiers on different feature sets including unigram and n-gram distributions as well as various dictionary-based features. Our findings show that impoliteness and incivility can be measured to a certain extent on the word level of a comment, but the models suffer from high misclassification rates, even if lexical resources are included. This is mainly because the classifiers cannot reveal subtle forms of incivility and because comment authors often use predictive words of incivility or impoliteness in non-offensive ways or in different contexts. Still, when applying the classifiers to a comparable set of comments, we find that the machine-coded categories and the hand-coded categories reveal similar patterns regarding the distribution of and the user reactions to uncivil/impolite comments. The findings of our study therefore provide new insights into the supervised machine learning approach to the detection of different forms of offensive language.

Stylometry and Automatic Attribution of Medieval Liturgical Monodies

While automatic attribution of literary text as well as stylometry evaluation are nowadays well-established research areas, much less has been done in the field of musicology. Here we present the results of the implementation of an automatic stylometric attribution technique to a corpus of liturgical monodies of medieval origin (the so-called Gregorian Chant, Old Roman Chant and Ambrosian Chant). The ‘unidimensional’ nature of the musical repertoires investigated (rhythm-free melody without accompaniment) allows the adoption of a known method based on a pseudo-distance between frequency-vectors of n-gram of consecutive symbols. Finally, we show that some specific features of musicological interest inside the three liturgical families can be naturally extracted using a statistical analysis of n-gram distributions. The results presented show that a quantitative approach is well suited to support and accompany the investigation of refined problems in musicology.

Exploiting Content Spatial Distribution to Improve Detection of Intrusions

We present PCkAD, a novel semisupervised anomaly-based IDS (Intrusion Detection System) technique, detecting application-level content-based attacks. Its peculiarity is to learn legitimate payloads by splitting packets into chunks and determining the within-packet distribution of n-grams. This strategy is resistant to evasion techniques as blending. We prove that finding the right legitimate content is NP-hard in the presence of chunks. Moreover, it improves the false-positive rate for a given detection rate with respect to the case where the spatial information is not considered. Comparison with well-known IDSs using n-grams highlights that PCkAD achieves state-of-the-art performances.

Quantiprot - a Python package for quantitative analysis of protein sequences

BackgroundThe field of protein sequence analysis is dominated by tools rooted in substitution matrices and alignments. A complementary approach is provided by methods of quantitative characterization. A major advantage of the approach is that quantitative properties defines a multidimensional solution space, where sequences can be related to each other and differences can be meaningfully interpreted.ResultsQuantiprot is a software package in Python, which provides a simple and consistent interface to multiple methods for quantitative characterization of protein sequences. The package can be used to calculate dozens of characteristics directly from sequences or using physico-chemical properties of amino acids. Besides basic measures, Quantiprot performs quantitative analysis of recurrence and determinism in the sequence, calculates distribution of n-grams and computes the Zipf’s law coefficient.ConclusionsWe propose three main fields of application of the Quantiprot package. First, quantitative characteristics can be used in alignment-free similarity searches, and in clustering of large and/or divergent sequence sets. Second, a feature space defined by quantitative properties can be used in comparative studies of protein families and organisms. Third, the feature space can be used for evaluating generative models, where large number of sequences generated by the model can be compared to actually observed sequences.

N-Gram Approaches to the Historical Dynamics of Basic Vocabulary*

ABSTRACTIn this paper, we apply an information theoretic measure, self-entropy of phoneme n-gram distributions, for quantifying the amount of phonological variation in words for the same concepts across languages, thereby investigating the stability of concepts in a standardized concept list – based on the 100-item Swadesh list – specifically designed for automated language classification. Our findings are consistent with those of the ASJP project (Automated Similarity Judgment Program; Holman et al. 2008a). The correlation of our ranking with that of ASJP is statistically highly significant. Our ranking also largely agrees with two other reduced concept lists proposed in the literature. Our results suggest that n-gram analysis works at least as well as other measures for investigating the relation of phonological similarity to geographical spread, automatic language classification, and typological similarity, while being computationally considerably cheaper than the most widespread method (normalized Levenshtein distance), very important when processing large quantities of language data.

Character usage in Chinese short message service (SMS): a real-world study in Mainland China

Short message service SMS is an important component of modern mobile services. Given unique characteristics of Chinese language, it is imperative to conduct study to understand characteristic of language usage patterns in Chinese SMS so that important facts like why and how people in China use SMS can be discovered. In this paper, we report an analysis of Chinese SMS logs from three different provinces in China. A computational approach was applied to extract n-grams from logs of SMS. The language usage patterns reported in this paper consist of two aspects: 1 most popular n-grams that represent what types of information were transmitted via SMS; 2 distribution of n-grams in comparison with Zipf laws. We discovered that, compared with other forms of free text in Chinese, SMS contains more conversational elements, which are expressed mostly in bigrams. Trigrams, 4-and 5-grams are less frequent but are closely connected to commercial activities, which may indicate the commercial needs of SMS users.

Estimation of Subcellular Proteomes in Bacterial Species

Computational methods for predicting the subcellular localization of bacterial proteins play a crucial role in the ongoing efforts to annotate the function of these proteins and to suggest potential drug targets. These methods, used in combination with other experimental and computational methods, can play an important role in biomedical research by annotating the proteomes of a wide variety of bacterial species. We use the ngLOC method, a Bayesian classifier that pre- dicts the subcellular localization of a protein based on the distribution of n-grams in a curated dataset of experimentally- determined proteins. Subcellular localization was predicted with an overall accuracy of 89.7% and 89.3% for Gram- negative and Gram-positive bacteria protein sequences, respectively. Through the use of a confidence score threshold, we improve the precision to 96.6% while covering 84.4% of Gram-negative bacterial data, and 96.0% while covering 87.9% of Gram-positive data. We use this method to estimate the subcellular proteomes of ten Gram-negative species and five Gram-positive species, covering an average of 74.7% and 80.6% of the proteome for Gram-negative and Gram-positive sequences, respectively. The current method is useful for large-scale analysis and annotation of the subcellular proteomes of bacterial species. We demonstrate that our method has excellent predictive performance while achieving superior pro- teome coverage compared to other popular methods such as PSORTb and PLoc.

Characteristics of character usage in Chinese Web searching

The use of non-English Web search engines has been prevalent. Given the popularity of Chinese Web searching and the unique characteristics of Chinese language, it is imperative to conduct studies with focuses on the analysis of Chinese Web search queries. In this paper, we report our research on the character usage of Chinese search logs from a Web search engine in Hong Kong. By examining the distribution of search query terms, we found that users tended to use more diversified terms and that the usage of characters in search queries was quite different from the character usage of general online information in Chinese. After studying the Zipf distribution of n-grams with different values of n, we found that the curve of unigram is the most curved one of all while the bigram curve follows the Zipf distribution best, and that the curves of n-grams with larger n ( n = 3–6) had similar structures with β-values in the range of 0.66–0.86. The distribution of combined n-grams was also studied. All the analyses are performed on the data both before and after the removal of function terms and incomplete terms and similar findings are revealed. We believe the findings from this study have provided some insights into further research in non-English Web searching and will assist in the design of more effective Chinese Web search engines.

An insight into the entropy and redundancy of the English dictionary

The inherent statistical characteristics, including the economy, entropy, and redundancy, of a very large set containing 93681 words from the Shorter Oxford English Dictionary is investigated. Analytical n-gram statistics are also presented for applications in natural language understanding, text processing, test compression, error detection and correction, and speech synthesis and recognition. Experimental results show how the distribution of n-grams in the dictionary varies from the ideal as n increases from 2 to 5, that is, from bigrams to pentagrams; it is shown that the corresponding redundancy increases from 0.1067 to 0.3409. The results are of interest because, (1) the dictionary provides a finite list for deterministic analyses, (2) each entry (word) appears once, compared to free-running text where words are repeated, and (3) all entries, even rarely occurring ones, have equal weight.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The Sensitivity of the Modified Viterbi Algorithm to the Source Statistics

The modified Viterbi algorithm is a powerful, and increasingly used, tool for using contextual information in text recognition in its various forms. As yet, no known studies have been published concerning its robustness with respect to source statistics. This paper describes experiments performed to determine the sensitivity of the algorithm to variations in source statistics. The results of the experiments show that a character-recognition machine incorporating the modified Viterbi algorithm, using N-gram statistics estimated from source A does not deteriorate in performance when operating on a passage from source B even if A and B differ significantly in N-gram distributions or entropy.