Shannon Information Measures Research Articles

Sampling bias corrections for accurate neural measures of redundant, unique, and synergistic information.

Shannon Information theory has long been a tool of choice to measure empirically how populations of neurons in the brain encode information about cognitive variables. Recently, Partial Information Decomposition (PID) has emerged as principled way to break down this information into components identifying not only the unique information carried by each neuron, but also whether relationships between neurons generate synergistic or redundant information. While it has been long recognized that Shannon information measures on neural activity suffer from a (mostly upward) limited sampling estimation bias, this issue has largely been ignored in the burgeoning field of PID analysis of neural activity. We used simulations to investigate the limited sampling bias of PID computed from discrete probabilities (suited to describe neural spiking activity). We found that PID suffers from a large bias that is uneven across components, with synergy by far the most biased. Using approximate analytical expansions, we found that the bias of synergy increases quadratically with the number of discrete responses of each neuron, whereas the bias of unique and redundant information increase only linearly or sub-linearly. Based on the understanding of the PID bias properties, we developed simple yet effective procedures that correct for the bias effectively, and that improve greatly the PID estimation with respect to current state-of-the-art procedures. We apply these PID bias correction procedures to datasets of 53117 pairs neurons in auditory cortex, posterior parietal cortex and hippocampus of mice performing cognitive tasks, deriving precise estimates and bounds of how synergy and redundancy vary across these brain regions.

Генетическое разнообразие российских и европейских пред-ставителей Dryopteris expansa (C. Presl) Fraser-Jenkins et Jermy (Dryopteridaceae)

The results of molecular genetic analysis (ISSR-PCR) of 112 Dryopteris expansa (C. Presl) Fraser-Jenkins et Jermy specimens collected in Russia and Europe are presented. DNA was extracted using fresh material and material from the herbarium of the South Siberian Botanical Garden (ALTB): the collection of fresh material was carried out during expeditions (40 samples from Perm Krai, Moscow, Smolensk and Tver regions), the selection of material from the herbarium was carried out by removing a part of the dried frond (72 samples from various regions of Russia and Europe). An analysis of DNA polymorphism and the genetic structure of the studied groups with the calculation of the proportion (P95) of polymorphic loci, the absolute (na) number of alleles, the effective (ne) number of alleles, the expected (HE) heterozygosity and the information-entropy measure (I) of Shannon was carried out using the POPGENE 1.31 program. To describe the genetic structure of the studied groups, the following parameters were used: the expected proportion of heterozygous genotypes (HT) in the entire group as a measure of its overall diversity; the expected proportion of heterozygous genotypes in a subgroup (HS) as a measure of its internal diversity; the proportion of intergroup genetic diversity in the total diversity or the subdivision index (GST). In addition to this, the AMOVA package was used with the calculation of the subdivision index of groups using 1000 rounds of permutations. Based on a comparison of the indicators of the Shannon information measure (I), Nei fixation index (FST), as well as intra- and intergroup variability based on the results of the analysis of molecular variants (AMOVA) obtained for D. expansa with similar indicators given for D. fragrans (L.) Schott, a conclusion is made about the inexpediency of dividing D. expansa s.l. into two independent species – D. expansa s. str. and D. assimilis S. Walker. The indicators of all indices and calculated values given in the publication can be used as a basis for comparison in other works devoted to the genetic variability of species of the genus Dryopteris in Russia.

Neural network based image classifier resilient to destructive perturbation influences – architecture and training method

Modern methods of image recognition are sensitive to various types of disturbances, which actualize the development of resilient intelligent algorithms for safety-critical applications. The current article develops a model and method of training a classifier that exhibits characteristics of resilience to adversarial attacks, fault injection, and concept drift. The proposed model has a hierarchical structure of prototypes and hyperspherical boundaries of classes formed in the space of high-level features. Class boundaries are optimized during training and provide perturbation absorption and graceful degradation. The proposed learning method involves the use of a combined loss function, which allows the use of both labeled and unlabeled data, implements the compression of the feature representation to a discrete form and ensures the compactness of the distribution of classes and the maximization of the buffer zone between classes. The main component of the loss function is the value of the normalized modification of Shannon's information measure, averaged over the alphabet of the classes, expressed as a function of accuracy characteristics. Simultaneously, accuracy characteristics are calculated on the basis of smoothed versions of the distribution of statistical hypothesis testing results. It is experimentally confirmed that the proposed approach provides a certain level of disturbance absorption, graceful degradation and recovery. During testing of the proposed algorithm on the Cifar10 data set, it was established that the integral metric of resilience to tensor damage by inversion of one randomly selected bit is about 0.95 if the share of damaged tensors does not exceed 30%. Also, during testing of the proposed algorithm, it was established that an adversarial attack with a disturbance that does not exceed the L∞-norm threshold equal to 3 provides resilience that exceeds the value of 0.95 according to the integral metric. Additionally, the integral metric of resilience during adaptation to the appearance of two new classes is 0.959. The integral metric of resilience to the real drift of concepts between the two classes is 0.973. The ability to adapt to the appearance of new classes or the concept drift has been confirmed 8 times faster than learning from scratch.

GEOENT: A Toolbox for Calculating Directional Geological Entropy

Geological entropy is based on Shannon information entropy and measures order in the structure of a spatial random variable. Metrics have been defined to quantify geological entropy in multidimensional (2D and 3D) heterogeneous systems, for instance, porous and fractured geological media. This study introduces GEOENT, a toolbox that can efficiently be used to calculate geological entropy metrics for any kind of input-gridded field. Additionally, the definition of geological entropy metrics is updated to consider anisotropy in the structure of the heterogeneous system. Directional entrograms provide more accurate descriptions of spatial order over different Cartesian directions. This study presents the development of the geological entropy metrics, a description of the toolbox, and examples of its applications in different datasets, including 2D and 3D gridded fields, representing a variety of heterogeneous environments at different scales, from pore-scale microtomography (μCT) images to aquifer analogues.

Information Inequalities via Submodularity and a Problem in Extremal Graph Theory.

The present paper offers, in its first part, a unified approach for the derivation of families of inequalities for set functions which satisfy sub/supermodularity properties. It applies this approach for the derivation of information inequalities with Shannon information measures. Connections of the considered approach to a generalized version of Shearer’s lemma, and other related results in the literature are considered. Some of the derived information inequalities are new, and also known results (such as a generalized version of Han’s inequality) are reproduced in a simple and unified way. In its second part, this paper applies the generalized Han’s inequality to analyze a problem in extremal graph theory. This problem is motivated and analyzed from the perspective of information theory, and the analysis leads to generalized and refined bounds. The two parts of this paper are meant to be independently accessible to the reader.

Shannon (Information) Measures of Symmetry for 1D and 2D Shapes and Patterns

In this paper, informational (Shannon) measures of symmetry are introduced and analyzed for patterns built of 1D and 2D shapes. The informational measure of symmetry Hsym(G) characterizes the averaged uncertainty in the presence of symmetry elements from group G in a given pattern, whereas the Shannon-like measure of symmetry Ωsym(G) quantifies the averaged uncertainty of the appearance of shapes possessing a total of n elements of symmetry belonging to group G in a given pattern. Hsym(G1)=Ωsym(G1)=0 for the patterns built of irregular, non-symmetric shapes, where G1 is the identity element of the symmetry group. Both informational measures of symmetry are intensive parameters of the pattern and do not depend on the number of shapes, their size, and the entire area of the pattern. They are also insensitive to the long-range order (translational symmetry) inherent for the pattern. Additionally, informational measures of symmetry of fractal patterns are addressed, the mixed patterns including curves and shapes are considered, the time evolution of Shannon measures of symmetry are examined, the close-packed and dispersed 2D patterns are analyzed, and an application of the suggested measures of symmetry for the analysis of the chemical reaction is demonstrated.

On the α-q-Mutual Information and the α-q-Capacities.

The measures of information transfer which correspond to non-additive entropies have intensively been studied in previous decades. The majority of the work includes the ones belonging to the Sharma–Mittal entropy class, such as the Rényi, the Tsallis, the Landsberg–Vedral and the Gaussian entropies. All of the considerations follow the same approach, mimicking some of the various and mutually equivalent definitions of Shannon information measures, and the information transfer is quantified by an appropriately defined measure of mutual information, while the maximal information transfer is considered as a generalized channel capacity. However, all of the previous approaches fail to satisfy at least one of the ineluctable properties which a measure of (maximal) information transfer should satisfy, leading to counterintuitive conclusions and predicting nonphysical behavior even in the case of very simple communication channels. This paper fills the gap by proposing two parameter measures named the -q-mutual information and the -q-capacity. In addition to standard Shannon approaches, special cases of these measures include the -mutual information and the -capacity, which are well established in the information theory literature as measures of additive Rényi information transfer, while the cases of the Tsallis, the Landsberg–Vedral and the Gaussian entropies can also be accessed by special choices of the parameters and q. It is shown that, unlike the previous definition, the -q-mutual information and the -q-capacity satisfy the set of properties, which are stated as axioms, by which they reduce to zero in the case of totally destructive channels and to the (maximal) input Sharma–Mittal entropy in the case of perfect transmission, which is consistent with the maximum likelihood detection error. In addition, they are non-negative and less than or equal to the input and the output Sharma–Mittal entropies, in general. Thus, unlike the previous approaches, the proposed (maximal) information transfer measures do not manifest nonphysical behaviors such as sub-capacitance or super-capacitance, which could qualify them as appropriate measures of the Sharma–Mittal information transfer.

Statistical Characteristics of Stationary Flow of Substance in a Network Channel Containing Arbitrary Number of Arms.

We study flow of substance in a channel of network which consists of nodes of network and edges which connect these nodes and form ways for motion of substance. The channel can have arbitrary number of arms and each arm can contain arbitrary number of nodes. The flow of substance is modeled by a system of ordinary differential equations. We discuss first a model for a channel which arms contain infinite number of nodes each. For stationary regime of motion of substance in such a channel we obtain probability distributions connected to distribution of substance in any of channel’s arms and in entire channel. Obtained distributions are not discussed by other authors and can be connected to Waring distribution. Next, we discuss a model for flow of substance in a channel which arms contain finite number of nodes each. We obtain probability distributions connected to distribution of substance in the nodes of the channel for stationary regime of flow of substance. These distributions are also new and we calculate corresponding information measure and Shannon information measure for studied kind of flow of substance.

Is Accounting an Information Science?

Abstract The central result is an equality connecting accounting numbers with information. \ln \left( 1+\frac{income}{assets}\right) =r_{f}+I\left( X;Y\right) , r_{f} is the risk free rate, ln is the natural logarithm, Y is the outcome of interest, X is the information signal about Y , and I(X;Y) is a Shannon information measure. The equality is derived using economic income accounting; it is shown to hold, under appropriate conditions, for declining balance and straight line depreciation methods. Some social welfare implications are explored.

Approximating Information Measures for Fields.

We supply corrected proofs of the invariance of completion and the chain rule for the Shannon information measures of arbitrary fields, as stated by Dębowski in 2009. Our corrected proofs rest on a number of auxiliary approximation results for Shannon information measures, which may be of an independent interest. As also discussed briefly in this article, the generalized calculus of Shannon information measures for fields, including the invariance of completion and the chain rule, is useful in particular for studying the ergodic decomposition of stationary processes and its links with statistical modeling of natural language.

Entropy vs. human ontogeny: the Shannon information measure; a reliable general indicator of human health?

ABSTRACT A measure of health based on exact sciences is not considered by social sciences to evaluate human development. We propose the H measure of information of Shannon ([1948]. “A Mathematical Theory of Communication.” The Bell Systems Technical Journal 27 (3): 379–423, 623–656), renamed as H cto , as a general indicator of human health which can be subsequently aggregated at any statistical level. Several advantages of H cto are explained: (i) connections with thermodynamics and information theory; (ii) combination of physical and statistical elements; (iii) measurements of entropy and/or information depending on the reference point, and (iv) a collection of examples based on the scientific literature that supports the application of H cto . Simplified graphical models of the ontogeny of health condition are also shown. The empirical feasibility of using H cto is also analysed, by concluding that it could be a tangible reality in the near future.

Analyzing the State of Oil Field Development Based on the Fisher and Shannon Information Measures

For the time series that describe the amounts of crude and load oil, the Fisher and Shannon entropies as well as other information measures are calculated. Their dynamics are used for performing an early diagnosis of the temporal limits of the oil field evolution, including its separate zones, and also for evaluating the state of oil production in response to planned bed stimulation. The suggested method is tested on the Forties Oil Field data. In accordance with the experimental results, oil field exploitation without considering the dynamics of these information measures reduces oil production, increasing the output of water-cut oil and causing an inefficient mode of bed stimulation.

Is Accounting an Information Science?

The central result is an equality connecting accounting numbers with information. ln (1 ((income)/(assets)))=r_{f} I(X;Y), r_{f} is the risk free rate, ln is the natural logarithm, Y is the outcome of interest, X is the information signal about Y, and I(X;Y) is a Shannon information measure. The equality is derived using economic income accounting; it is shown to hold, under appropriate conditions, for declining balance and straight line depreciation methods. Some social welfare implications are explored.

Analysis of Remaining Uncertainties and Exponents Under Various Conditional Rényi Entropies

We analyze the asymptotics of the normalized remaining uncertainty of a source when a compressed or hashed version of it and correlated side information is observed. For this system, commonly known as Slepian–Wolf source coding, we establish the optimal (minimum) rate of compression of the source to ensure that the remaining uncertainties vanish. We also study the exponential rate of decay of the remaining uncertainty to zero when the rate is above the optimal rate of compression. In this paper, we consider various classes of random universal hash functions. Instead of measuring remaining uncertainties using traditional Shannon information measures, we do so using two forms of the conditional Renyi entropy. Among other techniques, we employ new one-shot bounds and the moments of type class enumerator method (see Merhav) for these evaluations. We show that these asymptotic results are generalizations of the strong converse exponent and the error exponent of the Slepian–Wolf problem under maximum a posteriori decoding.

Multivariate Dependence beyond Shannon Information

Accurately determining dependency structure is critical to understanding a complex system’s organization. We recently showed that the transfer entropy fails in a key aspect of this—measuring information flow—due to its conflation of dyadic and polyadic relationships. We extend this observation to demonstrate that Shannon information measures (entropy and mutual information, in their conditional and multivariate forms) can fail to accurately ascertain multivariate dependencies due to their conflation of qualitatively different relations among variables. This has broad implications, particularly when employing information to express the organization and mechanisms embedded in complex systems, including the burgeoning efforts to combine complex network theory with information theory. Here, we do not suggest that any aspect of information theory is wrong. Rather, the vast majority of its informational measures are simply inadequate for determining the meaningful relationships among variables within joint probability distributions. We close by demonstrating that such distributions exist across an arbitrary set of variables.

Comments on “Statistical complexity and Fisher–Shannon information measure of [formula omitted]” [Phys. Lett. A 372 (13) (2008) 2271–2273

We point out and correct the erroneous numerical results calculated by H.E. Montgomery Jr. and K.D. Sen concerning the statistical complexity and the Fisher–Shannon information measure for the ground state H2+ molecule ion as a function of internuclear distance in momentum space and as a product in position and momentum spaces. In addition we have shown that Coulson-type wave function approximation used by authors leads to notable (up to almost 20%) discrepancies between such approximated results and exact ones.

Reply to “Comments on ‘Statistical complexity and Fisher–Shannon information measure of [formula omitted]’ [Phys. Lett. A 372 (13) (2008) 2271–2273

Clarifications about the source of error in [Phys. Lett. A 371 (2008) 2271] of momentum-space information measures are given along with the revised estimates which satisfy all the points commented upon by Astashkevich. We note that the position-space quantities calculated previously remain unchanged. Our main conclusion, that the inter-nuclear distance, R, dependence of statistical complexity and Fisher–Shannon measure derived from the simple Coulson wave function contain significant information on molecular bonding in H2+, remains qualitatively valid in the light of calculations using more accurate wave functions as reported by Astashkevich.

Equivocations, Exponents, and Second-Order Coding Rates Under Various Rényi Information Measures

We evaluate the asymptotics of equivocations, their exponents as well as their second-order coding rates under various R\'{e}nyi information measures. Specifically, we consider the effect of applying a hash function on a source and we quantify the level of non-uniformity and dependence of the compressed source from another correlated source when the number of copies of the sources is large. Unlike previous works that use Shannon information measures to quantify randomness, information or uniformity, we define our security measures in terms of a more general class of information measures--the R\'{e}nyi information measures and their Gallager-type counterparts. A special case of these R\'{e}nyi information measure is the class of Shannon information measures. We prove tight asymptotic results for the security measures and their exponential rates of decay. We also prove bounds on the second-order asymptotics and show that these bounds match when the magnitudes of the second-order coding rates are large. We do so by establishing new classes non-asymptotic bounds on the equivocation and evaluating these bounds using various probabilistic limit theorems asymptotically.

Allo-allo-triploid Sphagnum × falcatulum: single individuals contain most of the Holantarctic diversity for ancestrally indicative markers.

Allopolyploids exhibit both different levels and different patterns of genetic variation than are typical of diploids. However, scant attention has been given to the partitioning of allelic information and diversity in allopolyploids, particularly that among homeologous monoploid components of the hologenome. Sphagnum × falcatulum is a double allopolyploid peat moss that spans a considerable portion of the Holantarctic. With monoploid genomes from three ancestral species, this organism exhibits a complex evolutionary history involving serial inter-subgeneric allopolyploidizations. Studying populations from three disjunct regions [South Island (New Zealand); Tierra de Fuego archipelago (Chile, Argentina); Tasmania (Australia)], allelic information for five highly stable microsatellite markers that differed among the three (ancestral) monoploid genomes was examined. Using Shannon information and diversity measures, the holoploid information, as well as the information within and among the three component monoploid genomes, was partitioned into separate components for individuals within and among populations and regions, and those information components were then converted into corresponding diversity measures. The majority (76%) of alleles detected across these five markers are most likely to have been captured by hybridization, but the information within each of the three monoploid genomes varied, suggesting a history of recurrent allopolyploidization between ancestral species containing different levels of genetic diversity. Information within individuals, equivalent to the information among monoploid genomes (for this dataset), was relatively stable, and represented 83% of the grand total information across the Holantarctic, with both inter-regional and inter-population diversification each accounting for about 5% of the total information. Sphagnum × falcatulum probably inherited the great majority of its genetic diversity at these markers by reticulation, rather than by subsequent evolutionary radiation. However, some post-hybridization genetic diversification has become fixed in at least one regional population. Methodology allowing statistical analysis of any ploidy level is presented.

On the entropy of couplings

In this paper, some general properties of Shannon information measures are investigated over sets of probability distributions with restricted marginals. Certain optimization problems associated with these functionals are shown to be NP-hard, and their special cases are found to be essentially information-theoretic restatements of well-known computational problems, such as the Subset sum and the 3-Partition. The notion of minimum entropy coupling is introduced and its relevance is demonstrated in information-theoretic, computational, and statistical contexts. Finally, a family of pseudometrics (on the space of discrete probability distributions) defined by these couplings is studied, in particular their relation to the total variation distance, and a new characterization of the conditional entropy is given.