Zipf-Mandelbrot Law Research Articles

N-convexity and weighted majorization with applications to f-divergences and Zipf–Mandelbrot law

n-convexity and weighted majorization with applications to f-divergences and Zipf–Mandelbrot law

Fundamental limits to economic development in developing and underdeveloped countries imposed by global hierarchy

Despite great leaps in science, technology, and human civilization since World War II, many countries in the world have been stuck in “middle income” traps or remained hopelessly poor. To understand this reality,we introduce a new metric, the revealed comparative wealth (RCW), and develop a theory of global hierarchical order (GHO), which is characterized by a Zipf–Mandelbrot law (ZML) for the yearly ranked RCW for all the countries in the world. Major factors contributing to the existence of GHO have been identified. Various types of behavior, including equilibria characterized by largely constant RCW, strongly coupled advanced economies modulated by business cycles, strongly competing developing and underdeveloped economies opposite-phase locked to advanced economies, and steadily growing or decaying economies synergistically interact to create ZML. The mathematical structure of ZML imposes fundamental limits to world economic development and sheds light on why “middle income” traps have persisted.

Generalizations of Levinson type inequalities via new Green functions with applications to information theory

In this paper, four new Green functions are used to generalize Levinson-type inequalities for the class of 3-convex functions. The f-divergence, Renyi entropy, Renyi divergence, Shannon entropy, and the Zipf–Mandelbrot law are also used to apply the main results to information theory.

Diversity of diversity indices: Which diversity measure is better?

The article evaluates the dependence of the most common indices of species diversity on sample size and determines their ability to differentiate between different types of ecosystems, with a special emphasis on discriminating between natural and anthropogenic ecosystems. An approach to adjusting the indices to reduce their dependence on sample size was also proposed. The study was conducted in seven types of ecosystems: four were natural and three were anthropogenically transformed. Samples of soil animals were selected in 2011–2013 and 2021 using the same methods. A total of 20,518 soil animal specimens belonging to 202 species were collected in all study locations. The null alternative was generated by randomly selecting samples containing 2, 3, ..., 110 soil animals from the combined soil animal sample. For each gradation of sample size, 200 sample variants were formed. The density of soil macrofauna in natural ecosystems ranged from 3.6 ± 1.5 to 15.2 ± 6.9 specimens per sample, and in artificial ecosystems – from 13.2 ± 7.6 to 21.0 ± 11.9 specimens per sample. The number of species ranged from 22–80 species, and in artificial ecosystems it was 38–99 species. Indicators of species diversity correlated with each other. A high level of correlation was observed between indicators within groups of indices: indices of species richness and indices of heterogeneity and evenness. Fisher’s log-series alpha and the fundamental parameter of biodiversity were highly correlated with each other, as well as with the Margalef, species richness, and Chao’s species abundance indices. The log-normal distribution best describes the dominance patterns in terms of abundance in the natural ecosystems, and the Zipf-Mandelbrot distribution best describes the dominance patterns in terms of abundance in the artificial ecosystems. Diversity indices were ordered in the space of two dimensions, one explaining the variation between ecosystems and the other depending on sample size. The ordering of the traditional indices showed that there is a vacancy for the best index in the sense that such an index should best explain differences between ecosystems and differences between natural and artificial ecosystems. It should also be independent of sample size. The Simpson heterogeneity index and the Simpson evenness index were the best of the traditional indices, but they did not explain differences between ecosystems very well, especially when it came to distinguishing between natural and artificial ecosystems. The Margalef index, which is supposed to be independent of sample size, on the other hand, showed a very high level of dependence. Such a dependence was also found for the Menhinick index, though to a lesser extent. Obviously, size dependence negatively affects the differential ability of the indices. The corrected indices of species richness and the Shannon index are practically independent of sample size and have a greater ability to differentiate ecosystems by the level of diversity, with natural ecosystems characterized by consistently higher values of the corrected indices than artificial ecosystems. The dependence on the sample size makes indices from different ecosystems practically incomparable, which makes their use meaningless. Even minor differences in sample size can lead to significant deviations in the values of diversity indices. The application of the Michaelis-Menten model allowed us to suggest a method of correction of species richness indices and the Shannon index. After the correction, the indices are practically independent of the sample size, and their differential ability to characterize individual ecosystems and the level of anthropogenic transformation increases significantly.

Cardiac dynamics evaluation with the application of methodologies based on proportional entropy and on Zipf-Mandelbrot law

Cardiac dynamics evaluation with the application of methodologies based on proportional entropy and on Zipf-Mandelbrot law

Random sampling of the Zipf–Mandelbrot distribution as a representation of vocabulary growth

We develop a discrete model of type-token dynamics based on random type selection from the Zipf–Mandelbrot probability distribution, with a view to examining the relationships between the constants of Zipf’s and Heaps’ laws. Analysis of items randomly selected items from the Standardised Project Gutenberg Corpus (SPGC) reveal a significant low-frequency “droop” in the β-slope of the types vs. frequency distribution, inconsistent with the model when vocabulary is unlimited: when a finite vocabulary limit is imposed, optimal parameter selection allows the droop to be reproduced. We adjust the parameters of both the limited and unlimited vocabulary models to obtain optimal agreement with the vocabulary growth curves: the limited vocabulary model usually yields the best optimised agreement, but a sizeable minority of items are better represented by an unlimited vocabulary. While the optimised Zipf α indices correlate strongly with the corresponding values obtained directly from document statistics, the former are generally larger than the latter (though this is partially explained by the distorting effect of large values of the Mandelbrot parameter m). The β indices optimised from the limited vocabulary model are also compared with their directly measured equivalents, showing significant positive correlation. The relationship between optimised α and β agrees plausibly with the well-known continuum model, though the degree of agreement depends on how β is defined. The experiments yield repeatable results from each of three 100-item samples, demonstrating the statistical significance of the experiments.

Quantifying the unexpected: A scientific approach to Black Swans

Many natural and socio-economic systems are characterized by power-law distributions that make the occurrence of extreme events not negligible. Such events are sometimes referred to as Black Swans, but a quantitative definition of a Black Swan is still lacking. Here, by leveraging on the properties of Zipf-Mandelbrot law, we investigate the relations between such extreme events and the dynamics of the upper cutoff of the inherent distribution. This approach permits a quantification of extreme events and allows to classify them as White, Grey, or Black Swans. Our criterion is in accordance with some previous findings, but also allows us to spot new examples of Black Swans, such as Lionel Messi and the Turkish Airline Flight 981 disaster. The systematic and quantitative methodology we developed allows a scientific and immediate categorization of rare events, providing also new insight into the generative mechanism behind Black Swans.

On the Converse Jensen-Type Inequality for Generalized f-Divergences and Zipf–Mandelbrot Law

Motivated by some recent investigations about the sharpness of the Jensen inequality, this paper deals with the sharpness of the converse of the Jensen inequality. These results are then used for deriving new inequalities for different types of generalized f-divergences. As divergences measure the differences between probability distributions, these new inequalities are then applied on the Zipf–Mandelbrot law as a special kind of a probability distribution.

NDN Content Store and Caching Policies: Performance Evaluation

Among various factors contributing to performance of named data networking (NDN), the organization of caching is a key factor and has benefited from intense studies by the networking research community. The performed studies aimed at (1) finding the best strategy to adopt for content caching; (2) specifying the best location, and number of content stores (CS) in the network; and (3) defining the best cache replacement policy. Accessing and comparing the performance of the proposed solutions is as essential as the development of the proposals themselves. The present work aims at evaluating and comparing the behavior of four caching policies (i.e., random, least recently used (LRU), least frequently used (LFU), and first in first out (FIFO)) applied to NDN. Several network scenarios are used for simulation (2 topologies, varying the percentage of nodes of the content stores (5–100), 1 and 10 producers, 32 and 41 consumers). Five metrics are considered for the performance evaluation: cache hit ratio (CHR), network traffic, retrieval delay, interest re-transmissions, and the number of upstream hops. The content request follows the Zipf–Mandelbrot distribution (with skewness factor α=1.1 and α=0.75). LFU presents better performance in all considered metrics, except on the NDN testbed, with 41 consumers, 1 producer and a content request rate of 100 packets/s. For the level of content store from 50% to 100%, LRU presents a notably higher performance. Although the network behavior is similar for both skewness factors, when α=0.75, the CHR is significantly reduced, as expected.

Statistical tests for text homogeneity: using forward and backward processes of numbers of different words

The processes of growth in the number of diverse words in a text, when reading in the forward and backward directions, are studied in this article. Based upon the statistics achieved from the difference between these two processes, we construct a statistical test. This statistical test is used for text homogeneity checks. The elementary model states that words in a text are selected from some dictionary independent of each other according to the Zipf–Mandelbrot law. P-values of the statistical test are calculated based on the elementary probabilistic model using the asymptotic normality of corresponding statistics. At last but not least, this statistical test is applied for the analysis of homogeneity of sequences of sonnets.

Random Sampling of the Zipf-Mandelbrot Distribution as a Representation of Vocabulary Growth by Martin Tunnicliffe and Gordon Hunter

Random Sampling of the Zipf-Mandelbrot Distribution as a Representation of Vocabulary Growth by Martin Tunnicliffe and Gordon Hunter

Zipf’s law for cosmic structures: How large are the greatest structures in the universe?

The statistical characterization of the distribution of visible matter in the universe is a central problem in modern cosmology. In this respect, a crucial question still lacking a definitive answer concerns how large the greatest structures in the universe are. This point is closely related to whether or not such a distribution can be approximated as being homogeneous on large enough scales. Here we assess this problem by considering the size distribution of superclusters of galaxies and by leveraging the properties of Zipf–Mandelbrot law, providing a novel approach which complements the standard analysis based on the correlation functions. We find that galaxy superclusters are well described by a pure Zipf’s law with no deviations and this implies that all the catalogs currently available are not sufficiently large to spot a truncation in the power-law behavior. This finding provides evidence that structures larger than the greatest superclusters already observed are expected to be found when deeper redshift surveys will be completed. As a consequence, the scale beyond which galaxy distribution crossovers toward homogeneity, if any, should increase accordingly.

Initial and Final Syllables in Tatar- from Phonotactics to Morphology

The paper proposes a methodology for analyzing the syllabic structure of Tatar words using fiction text data. Syllable construction rules are unique for each language as they are determined by the laws that govern its specific internal structure. However, the issue of the syllable finds a rather superficial description in Tatar grammars. Thus, possible correlations of the syllable structure with morphological features of the language will be examined in this paper. We analyze the distribution of syllable types in Tatar texts and represent their ranked frequencies and theoretical values fitted by means of the Zipf Mandelbrot distribution. The main part of the study is devoted to inquiry into the structure of initial and final syllables. We proceed from the hypothesis that distributions of syllable structures in word-initial and word-final positions should be marked by statistically important differences due to discriminative structural features of stems and affixal chains. The study is based on a selection of obstruent and sonorant consonants. To evaluate statistical significance of these differences, the well-known chi square test is applied.

Why Do Parameter Values in the Zipf-Mandelbrot Distribution Sometimes Explode?

ABSTRACT The Zipf-Mandelbrot distribution serves as a mathematical model for ranked frequencies in many areas of scientific research, including linguistics. Many linguistic units, like e.g., words or word n-grams, follow this distribution. However, in some cases, such as for graphemes in linguistics or species abundance and diversity data in biology, the parameters of the Zipf-Mandelbrot distribution are virtually uninterpretable, as their values strongly depend on the precision of numerical methods used to estimate them (values from several tens to several hundreds are not uncommon). It is shown in the paper that these values can be explained by the convergence to the geometric distribution, which forces both parameters of the Zipf-Mandelbrot distribution to increase to infinity while their ratio converges to a constant. Some examples which illustrate this limit behaviour are presented.

A statistical test for correspondence of texts to the Zipf—Mandelbrot law

We analyse correspondence of a text to a simple probabilistic model. The model assumes that the words are selected independently from an infinite dictionary. The probability distribution correspond to the Zipf---Mandelbrot law. We count sequentially the numbers of different words in the text and get the process of the numbers of different words. Then we estimate Zipf---Mandelbrot law parameters using the same sequence and construct an estimate of the expectation of the number of different words in the text. Then we subtract the corresponding values of the estimate from the sequence and normalize along the coordinate axes, obtaining a random process on a segment from 0 to 1. We prove that this process (the empirical text bridge) converges weakly in the uniform metric on $C (0,1)$ to a centered Gaussian process with continuous a.s. paths. We develop and implement an algorithm for approximate calculation of eigenvalues of the covariance function of the limit Gaussian process, and then an algorithm for calculating the probability distribution of the integral of the square of this process. We use the algorithm to analyze uniformity of texts in English, French, Russian and Chinese.

Dynamic Modeling of the Effects of Site Placement on Environmental Suitability: A Theoretical Example from Northwest Morocco

ABSTRACT This paper offers a new numerical approach to model the effects of archaeological site placement and population density on environmental suitability using two ecological models, the Ideal Free Distribution (IFD) and Ideal Despotic Distribution (IDD), treating the Oued Loukkos in northern Morocco as an example. This method incorporates local resource depletion with increasing population density consistent with the predictions of the IFD. It also incorporates the potential for the exclusion of part of the population from certain productive areas consistent with the IDD. In this study, we propose a dynamic approach to the impact of site placement on habitat suitability and therefore broader application for understanding changes in settlement distribution with population density through time. Furthermore, resulting settlements can be parameterized by a Zipf-Mandelbrot distribution, entailing a direct relationship between the ecological, demographic, and political factors and the resulting rank-size distribution of sites. This approach allows for a broader range of applications among foraging and agricultural economies.

Words ranking and Hirsch index for identifying the core of the hapaxes in political texts

This paper deals with a quantitative analysis of the content of official political speeches. We study a set of about one thousand talks pronounced by the US Presidents, ranging from Washington to Trump. In particular, we search for the relevance of the rare words, i.e. those said only once in each speech -- the so-called hapaxes. We implement a rank-size procedure of Zipf-Mandelbrot type for discussing the hapaxes' frequencies regularity over the overall set of speeches. Starting from the obtained rank-size law, we define and detect the core of the hapaxes set by means of a procedure based on an Hirsch index variant. We discuss the resulting list of words in the light of the overall US Presidents' speeches. We further show that this core of hapaxes itself can be well fitted through a Zipf-Mandelbrot law and that contains elements producing deviations at the low ranks between scatter plots and fitted curve -- the so-called king and vice-roy effect. Some socio-political insights are derived from the obtained findings about the US Presidents messages.

Interactions between soccer teams reveal both design and emergence: Cooperation, competition and Zipf-Mandelbrot regularity

Abstract Considering soccer matches as complex systems facilitates the identification of properties that emerge from the interactions between players. Such properties include the regularities and statistical patterns that characterize couplings and sets between players established during matches. Empirical studies on the statistical distributions of number of items (e.g., words in texts) have shown that these distributions follow scaling properties according to empirical laws known as Zipf-Mandelbrot. Here we investigate whether the (re)occurrence of pitch location of sets of players in a soccer match also obeys these empirical laws. Data collected from 10 soccer matches shows that, for most sets of players, this seems to be the case. Exceptions were found in particular types of sets, such as goalkeeper and goal, and left defender and right attacker from opposite teams. Rather than challenging the hypothesis that a Zipf-Mandelbrot law defines this system, these exceptions may be explained by the players configuration design, which is a trait of human interaction within complex systems. This design expresses match strategy, before the team enters in such dynamical processes (the game).

Estimation of different entropies via Lidstone polynomial using Jensen-type functionals

In this work, some new functional of Jensen-type inequalities are constructed using Shannon entropy, f-divergence, and Rényi divergence, and some estimates are obtained for these new functionals. Also using the Zipf–Mandelbrot law and hybrid Zipf–Mandelbrot law, we investigate some bounds for these new functionals. Furthermore, we generalize these new functionals for m-convex function using Lidstone polynomial.

Soil fungal assemblage complexity is dependent on soil fertility and dominated by deterministic processes.

In the processes controlling ecosystem fertility, fungi are increasingly acknowledged as key drivers. However, our understanding of the rules behind fungal community assembly regarding the effect of soil fertility level remains limited. Using soil samples from typical tea plantations spanning c. 2167km north-east to south-west across China, we investigated the assemblage complexity and assembly processes of 140 fungal communities along a soil fertility gradient. The community dissimilarities of total fungi and fungal functional guilds increased with increasing soil fertility index dissimilarity. The symbiotrophs were more sensitive to variations in soil fertility compared with pathotrophs and saprotrophs. Fungal networks were larger and showed higher connectivity as well as greater potential for inter-module connection in more fertile soils. Environmental factors had a slightly greater influence on fungal community composition than spatial factors. Species abundance fitted the Zipf-Mandelbrot distribution (niche-based mechanisms), which provided evidence for deterministic-based processes. Overall, the soil fungal communities in tea plantations responded in a deterministic manner to soil fertility, with high fertility correlated with complex fungal community assemblages. This study provides new insights that might contribute to predictions of fungal community complexity.