Shannon Information Measures Research Articles

Genetic diversity of Sinojackia dolichocarpa (Styracaceae), a species endangered and endemic to China, detected by inter-simple sequence repeat (ISSR)

Sinojackia dolichocarpa, a species endangered and endemic to China, is distributed only in the regional area of Shimen and Sangzhi in Hunan Province. Inter-simple sequence repeat (ISSR) markers were used to investigate the genetic diversity within and among the four natural populations of S. dolichocarpa. Leaf samples were collected from 84 individuals. Thirteen ISSR primers selected from 80 primers gave rise to 137 discernible DNA bands of which 100 (72.99%) were polymorphic. On average each primer gave rise to 10.5 bands including 7.7 bands with polymorphic profile. At the species level, high genetic diversity was detected (PPB: 72.99%; H E: 0.2255; Ho: 0.3453). However, relatively low genetic diversity existed within populations. Population in Maozhuhe (MZH) exhibits the greatest level of variability (PPB: 40.38%, H E: 0.1566, Ho: 0.2330), whereas the population in Jingguanmen (JGM) finds its own variability at the lowest level (PPB: 30.66%; H E: 0.1078; Ho: 0.1601). A high level of genetic differentiation among populations was revealed by Nei's gene diversity statistics (45.30%), Shannon's information measure (45.24%) and analysis of molecular variance (AMOVA) (52.88%). The main factors responsible for the high level of differentiation among populations are probably related to the selfing reproductive system and the isolation of populations. The strong genetic differentiation among populations indicates that the management for the conservation of genetic variability in S. dolichocarpa should aim to preserve every population.

THEORETICAL QUANTUM-INFORMATION PROPERTIES OF NUCLEI AND TRAPPED BOSE GASES

We will study fermionic systems like atomic nuclei and bosonic systems like the correlated atoms in a trap from an information-theoretical point of view. The Shannon and Onicescu information measures are calculated for the above systems by comparing the correlated and uncorrelated cases as functions of the strength of the short range correlations. One-body and two-body density and momentum distributions are employed. Thus, the effect of short-range correlations on the information content is evaluated. The magnitude of distinguishability between the correlated and uncorrelated densities is also discussed employing suitable measures for the distance of states i.e. the well known Kullback–Leibler relative entropy and the recently proposed Jensen–Shannon divergence entropy. We will see that the same information-theoretical properties hold for quantum many-body systems obeying Bose–Einstein and Fermi–Dirac (statistics).

The need for alternative information measures

In 1948 Shannon advanced information theory (IT) as a new branch of mathematics and a powerful tool for understanding the intricacies of the communication process. Nine years later Jaynes’ conceived the maximum entropy principle and was able to shed with it much light onto statistical mechanics and thermodynamics. Shannon's logarithmic information measure, the MaxEnt protagonist, was successfully connected with the thermodynamic entropy. Since 1988 a new project, called nonextensive thermostatistics (NET), came into being after the pioneering work by Tsallis and collaborators. It has achieved today a remarkable degree of success, amid a not small degree of controversy. It is also a MaxEnt construction but uses a different information measure in the leading role. We advance here some epistemological considerations on the proper place of any information measure within a theoretical physics’ construct in order put NET into a more solid conceptual framework.

Analysis of genetic diversity and structure in Ethiopian populations of Phytolacca dodecandra using RAPD.

The genetic diversity and structure in 17 wild populations (249 individuals) of Phytolacca dodecandra (endod) sampled along altitudinal gradients of 1600-3000 meters above sea level (m.a.s.l.) in Ethiopia was studied using random amplified polymorphic DNA (RAPD). A total of 70 polymorphic loci (P) scored from 12 RAPD primers were used to calculate different diversity indices within and between populations, habitats, geographical regions, climatic zones and altitude groups. The number of polymorphic loci and overall Shannon information measure (H) in the populations varied from 30 to 55 and from 0.228 to 0.418, respectively. In general, differences in population variability were found significantly correlated to effective population size. Both P and H were significantly higher in an undisturbed than in a disturbed habitat, and in the lowland and central-highland than in the highland altitude group. However, for both parameters the differences were not statistically significant between regions and climatic zones. Genetic distance between populations varied from 0.301 to 0.628. Cluster analysis performed using the genetic distance matrix revealed a clear separation of the highland populations (2501-3000 m.a.s.l.) from those of the lowland/central-highlands (1600-2500 m.a.s.l.) irrespective of their geographical regions and climatic zones. Analysis of molecular variance (AMOVA) indicated that differences in habitat, geographical regions and climatic zones explained 4.6%, 2.5% and 4.6%, respectively. But none of these differences were significant. Altitude explained 17.2% of the total variance and was highly significant. The data, therefore, clearly indicated the association of genetic structure in endod with altitude. The proportion of RAPD variation found among populations (21.2-35.0%) was somewhat intermediate between values reported for selfing and outcrossing species. The fixation index (FST) values (0.350 to 0.384) indicated very high genetic differentiation among populations.

The information analysis in joint problem of continuous-discrete filtering and generalized extrapolation

This paper considers the information aspect of the problem of the joint filtering and extrapolation, when the output of observation channels is the set of realizations of the processes with continuous and discrete time, which depend on both the current and the past values of unobservable process. The expressions, defined time evolution of Shannon information measures, are obtained and the solution structure are researched. The problem of optimal transmission of Gaussian Markov signal on a continuous-discrete time memory channel and the example of information efficiency of observations with memory are considered.

Genetic diversity and differentiation in Ethiopian populations of Phytolacca dodecandra as revealed by AFLP and RAPD analyses

Amplified fragment length polymorphism (AFLP) and random amplified polymorphic DNA (RAPD) analyses were performed on six populations (a total of 89 individuals) of Phytolacca dodecandra (endod) collected in Ethiopia. The populations were selected based on our previous investigation to represent two altitude groups: lowland/central-highland (1600–2500 m) and highland (2501–3000 m). A total of 197 AFLP and 68 RAPD markers were scored from 5 primer pairs and 12 random primers, respectively. The overall patterns obtained for AFLPs and RAPDs from diversity, cluster and principal component analyses were very comparable. However, the moderate correlation (r = 0.56) between AFLP and RAPD similarity matrices as well as the discrepancies in diversity estimates between the two techniques in some populations and in the lowland/central-highland plants could be due to differences in sensitivity of reaction conditions, bias in scoring of bands, number of markers used for analyses, and/or parts of the genome surveyed. For both AFLP and RAPD, the lowland/central-highland populations showed higher polymorphism and Shannon information measure (H) than the highlands. Cluster and principal component analyses performed for both marker types have also clearly demonstrated the differentiation of all the lowland/central highland plants from those of the highlands, in agreement with our previous conclusion. Markers scored from any of the five AFLP primer pairs were sufficient to clearly distinguish the two altitude groups; with RAPD, selection of about 8 informative markers produced by seven random primers was needed for the same purpose.

Genetic differentiation in Sargassum polyceratium (Fucales: Phaeophyceae) around the island of Curaçao (Netherlands Antilles)

The seaweed Sargassum polyceratium Montagne inhabits a broad spectrum of subtidal and intertidal habitats. Genetic diversity and spatial genetic structure were examined within and among 12 stands using random amplified polymorphic DNA (RAPD) phenotypes. Data were analyzed using analysis of molecular variance (AMOVA) and Shannon's information measure. In both analyses, 60-75% of the variation occurred within stands and 25-40% between stands. These values are consistent with out-crossing, high-dispersal species. Significant differentiation was found among bays ca. 25 kin apart (Shannon's G'(st) averaged 0.37 and pairwise AMOVA Phi (st) values averaged 0.272) and among stands 150-200 m apart within bays (AMOVA Phi (st) values averaged 0.149). Effects of shore (windward vs. leeward), depth, and bay on population structure were tested. These analyses revealed that the factor depth is confounded with shore, and that bays show significant differentiation from each other but are not completely isolated from one another. Mantel tests for differentiation-by-distance were significant along both sides of the island but stronger along the windward side. A neighbor-joining analysis of genetic distances among stands showed that the effects of currents around both tips of the island were especially important for shallow populations. For S. polyceratium, depth and bay promote population differentiation along shores, yet dispersal around the tips of the island simultaneously connects these populations to varying degrees. This study highlights the importance of investigating the relative contribution of habitat factorsin relation to island-scale population structure.

A class of non-Shannon-type information inequalities and their applications

Information inequalities form the most important set of tools for proving converse coding theorems in information theory problems. They are sometimes referred to as the Laws of Information Theory, because they govern the impossibilities in information theory. For a long time, all information inequalities we knew were nothing but simple consequences of the nonnegativity of Shannon's information measures. Owing to the recent discovery of a few so-called non-Shannon-type information inequalities, it is now known that there are laws in information theory beyond those laid down by Shannon. In this paper, we show that the unconditional inequality discovered by the authors (1998) in fact implies a class of 2/sup 14/ non-Shannon-type inequalities, and we show possible applications of these inequalities in information theory problems.

Genetic variation in the endemic and endangered Rosmarinus tomentosus Huber-Morath & Maire (Labiatae) using RAPD markers. Random amplified polymorphic DNA.

Rosmarinus tomentosus Huber-Morath & Maire, an endemic species of southern Spain, is critically endangered as a consequence of habitat destruction by anthropogenic activities. Random amplified polymorphic DNA (RAPD) markers were used for initial evaluation of genetic variation in this species; among zones, among populations (within zones and independently of zones), and among individuals (within populations and zones). The eight primers used to study five populations provided 126 reliable bands, of which 109 (86%) were polymorphic. Analyses of the RAPD markers conducted using different approaches (cluster analysis, Shannon's diversity measure and analysis of molecular variance-AMOVA) showed similar results concerning genetic structure of populations of R. tomentosus. The three zones studied, and their corresponding populations, were clearly defined through UPGMA analysis. AMOVA analyses indicated that 18.0% of the total genetic variation was attributable to differences among zones. When total variance was partitioned without considering the zonal distribution of the populations, 34.1% was attributable to differences among populations, and 65.9% to differences among individuals within populations. Similar results were obtained using Shannon's information measure. Significant genetic distances (Ph(ist) values) were obtained between any two populations. This strong genetic differentiation among zones, populations and subpopulations may be explained by geographical distances and limited gene flow. This study will be of considerable help during subsequent formulations of conservation strategies for the endangered species R. tomentosus.

A GENERALIZATION OF SHANNON'S INFORMATION THEORY

A generalized information theory is proposed as a natural extension of Shannon's information theory. It proposes that information comes from forecasts. The more precise and the more unexpected a forecast is, the more information it conveys. If subjective forecast always conforms with objective facts then the generalized information measure will be equivalent to Shannon's information measure. The generalized communication model is consistent with Popper's model of knowledge evolution. The mathematical foundations of the new information theory, the generalized communication model, information measures for semantic information and sensory information, and the coding meanings of generalized entropy and generalized mutual information are introduced. Assessments and optimizations of pattern recognition, predictions, and detection with the generalized information criterion are discussed. For economization of communication, a revised version of rate-distortion theory: rate-of-keeping-precision theory, which is a theory for datum compression and also a theory for matching an objective channels with the subjective understanding of information receivers, is proposed. Applications include stock market forecasting and video image presentation.

On characterization of entropy function via information inequalities

Given n discrete random variables /spl Omega/={X/sub 1/, /spl middot//spl middot//spl middot/, X/sub n/}, associated with any subset /spl alpha/ of (1, 2, /spl middot//spl middot//spl middot/, n), there is a joint entropy H(X/sub /spl alpha//) where X/sub /spl alpha//={X/sub i/:i/spl epsiv//spl alpha/}. This can be viewed as a function defined on 2/sup {1, 2, /spl middot//spl middot//spl middot/, n}/ taking values in (0, +/spl infin/). We call this function the entropy function of /spl Omega/. The nonnegativity of the joint entropies implies that this function is nonnegative; the nonnegativity of the conditional joint entropies implies that this function is nondecreasing; and the nonnegativity of the conditional mutual information implies that this function has the following property: for any two subsets /spl alpha/ and /spl beta/ of {1, 2, /spl middot//spl middot//spl middot/, n} H/sub /spl Omega//(/spl alpha/)+H/sub /spl Omega//(/spl beta/)/spl ges/H/sub /spl Omega//(/spl alpha//spl cup//spl beta/)+H/sub /spl Omega//(/spl alpha//spl cap//spl beta/). These properties are the so-called basic information inequalities of Shannon's information measures. Do these properties fully characterize the entropy function? To make this question more precise, we view an entropy function as a 2/sup n/-1-dimensional vector where the coordinates are indexed by the nonempty subsets of the ground set {1, 2, /spl middot//spl middot//spl middot/, n}. Let /spl Gamma//sub n/ be the cone in R/sup 2n-1/ consisting of all vectors which have these three properties when they are viewed as functions defined on 2/sup {1, 2, /spl middot//spl middot//spl middot/, n}/. Let /spl Gamma//sub n/* be the set of all 2/sup n/-1-dimensional vectors which correspond to the entropy functions of some sets of n discrete random variables. The question can be restated as: is it true that for any n, /spl Gamma/~/sub n/*=/spl Gamma//sub n/? Here /spl Gamma/~/sub n/* stands for the closure of the set /spl Gamma//sub n/*. The answer is "yes" when n=2 and 3 as proved in our previous work. Based on intuition, one may tend to believe that the answer should be "yes" for any n. The main discovery of this paper is a new information-theoretic inequality involving four discrete random variables which gives a negative answer to this fundamental problem in information theory: /spl Gamma/~*/sub n/ is strictly smaller than /spl Gamma//sub n/ whenever n>3. While this new inequality gives a nontrivial outer bound to the cone /spl Gamma/~/sub 4/*, an inner bound for /spl Gamma/~*/sub 4/ is also given. The inequality is also extended to any number of random variables.

An algorithm for designing a precipitation network in the south-western region of Saudi Arabia

The planning and design of a water resources project demands accurate hydrological information. Accurate and reliable regional hydrological information not only reduces the changes of project failure but also minimizes the economic risk. Therefore, it is imperative that a hydrological network be planned and designed scientifically. The criterion for hydrological network design should be the optimum number of stations and their respective locations by which hydrological information can be maximized. In this paper, a hydrological network design method, based on Shannon's Information Measure, was applied to the existing hydrological network in hydrological Area III, located in the south-western region of the Kingdom of Saudi Arabia, to examine their suitability towards providing maximum hydrological information. The study shows that the existing seventy rainfall stations can be reduced to approximately 45 including stations at new sites.

Information Content of Individual Genetic Sequences

Related genetic sequences having a common function can be described by Shannon's information measure and depicted graphically by a sequence logo. Though useful for many purposes, sequence logos only show the average sequence conservation, and inferring the conservation for individual sequences is difficult. This limitation is overcome by the individual information (Ri) technique described here. The method begins by generating a weight matrix from the frequencies of each nucleotide or amino acid at each position of the aligned sequences. This matrix is then applied to the sequences themselves to determine the sequence conservation of each individual sequence. The matrix is unique because the average of these assignments is the total sequence conservation, ad there is only one way to construct such a matrix. For binding sites on polynucleotides, the weight matrix has a natural cut off that distinguishes functional sequences from other sequences.Rivalues are on an absolute scale measured in bits of information so the conservation of different biological functions can be compared with one another. The matrix can be used to rank-order the sequences, to search for new sequences, to compare sequences to other quantitative data such as binding energy or distance between binding sites, to distinguish mutations from polymorphisms, to design sequences of a given strength, and to detect errors in databases. TheRimethod has been used to identify previously undescribed but experimentally verified DNA binding sites. The individual information distribution was determined forE. coliribosome binding sites, bacterial Fis binding sites, and human donor and acceptor splice junctions, among others. The distributions demonstrate clearly that the consensus sequence is highly unusual, and hence is a poor method to describe naturally occurring binding sites.

The Frieden-Soffer principle and wave functions that maximize Shannon's measure

Starting with a wave function ψ of a form that automatically extremizes Shannon's information measure according to Jaynes' maximum entropy principle (MEP), we show how, with the help of Frieden and Soffer's principle of extremal information (EPI), one can devise a potential V( x) that, if taken as a constraint in extremizing the Frieden-Soffer action, transforms ψ into an exact solution of Schrödinger's equation for V( x). The concomitant procedure illustrates the connection between two apparently disparate physical principles (MEP and EPI).

A non-Shannon-type conditional inequality of information quantities

Given n discrete random variables /spl Omega/={X/sub 1/,...,X/sub n/}, associated with any subset /spl alpha/ of {1,2,...,n}, there is a joint entropy H(X/sub /spl alpha//) where X/sub /spl alpha//={X/sub i/: i/spl isin//spl alpha/}. This can be viewed as a function defined on 2/sup {1,2,...,n}/ taking values in [0, +/spl infin/). We call this function the entropy function of /spl Omega/. The nonnegativity of the joint entropies implies that this function is nonnegative; the nonnegativity of the conditional joint entropies implies that this function is nondecreasing; and the nonnegativity of the conditional mutual information implies that this function is two-alternative. These properties are the so-called basic information inequalities of Shannon's information measures. An entropy function can be viewed as a 2/sup n/-1-dimensional vector where the coordinates are indexed by the subsets of the ground set {1,2,...,n}. As introduced by Yeng (see ibid., vol.43, no.6, p.1923-34, 1997) /spl Gamma//sub n/ stands for the cone in IR(2/sup n/-1) consisting of all vectors which have all these properties. Let /spl Gamma//sub n/* be the set of all 2/sup n/-1-dimensional vectors which correspond to the entropy functions of some sets of n discrete random variables. A fundamental information-theoretic problem is whether or not /spl Gamma/~/sub n/*=/spl Gamma//sub n/. Here /spl Gamma/~/sub n/* stands for the closure of the set /spl Gamma//sub n/*. We show that /spl Gamma/~/sub n/* is a convex cone, /spl Gamma//sub 2/*=/spl Gamma//sub 2/, /spl Gamma//sub 3/*/spl ne//spl Gamma//sub 3/, but /spl Gamma/~/sub 3/*=/spl Gamma//sub 3/. For four random variables, we have discovered a conditional inequality which is not implied by the basic information inequalities of the same set of random variables. This lends an evidence to the plausible conjecture that /spl Gamma/~/sub n/*/spl ne//spl Gamma//sub n/ for n>3.

A framework for linear information inequalities

We present a framework for information inequalities, namely, inequalities involving only Shannon's information measures, for discrete random variables. A region in IR(2/sup n/-1), denoted by /spl Gamma/*, is identified to be the origin of all information inequalities involving n random variables in the sense that all such inequalities are partial characterizations of /spl Gamma/*. A product from this framework is a simple calculus for verifying all unconstrained and constrained linear information identities and inequalities which can be proved by conventional techniques. These include all information identities and inequalities of such types in the literature. As a consequence of this work, most identities and inequalities involving a definite number of random variables can now be verified by a software called ITIP which is available on the World Wide Web. Our work suggests the possibility of the existence of information inequalities which cannot be proved by conventional techniques. We also point out the relation between /spl Gamma/* and some important problems in probability theory and information theory.

Three Information–Theoretical Methods to Estimate a Random Variable

Three information–theoretical methods to estimate a continuous univariate distribution are proposed for estimation when the distribution type is uncertain, when data are scarce, or when extremes are important. The first is a new version of Jaynes» MaxEnt Method. The second, minimizing Shannon's information measure, yields a minimally informative estimate. The third analysis produces a pair of distributions that together minimize the relative entropy (=cross-entropy) satisfying the principle of least information. Thus, one distribution (p) provides minimal information among all chosen candidates about these events, while the other (q) minimizes the information among all distributions that satisfies the sample rule, which is an essential constraint. Modelpis also identical to the ««maximum product of spacings»» estimate. Distributionqis determined frompby simple algebra. The principle of least information yields a unique solution (p,q) when other methods fail. The algorithm is the same for all types of distributions; the estimation process introduces a minimum of information, approaching objectivity, and the solution is invariant under monotonic variable transformations. All three methods are computationally simple but involve optimization. There are also several approximate information–theoretical methods that retain some of the advantages cited and are computationally simpler.

Maximum Entropy Aggregation of Expert Predictions

This paper presents a maximum entropy framework for the aggregation of expert opinions where the expert opinions concern the prediction of the outcome of an uncertain event. The event to be predicted and individual predictions rendered are assumed to be discrete random variables. A measure of expert competence is defined using a distance metric between the actual outcome of the event and each expert's predicted outcome. Following Levy and Delic (Levy, W. B., H. Delic. 1994. Maximum entropy aggregation of individual opinions. IEEE Trans. Sys. Man & Cybernetics 24 606–613.), we use Shannon's information measure (Shannon [Shannon, C. E. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27 379–423.], Jaynes [Jaynes, E. T. 1957. Information theory and statistical mechanics. Phys. Rev. 106 Part I: 620–630, 108 Part II: 171–190.]) to derive aggregation rules for combining two or more expert predictions into a single aggregated prediction that appropriately calibrates different degrees of expert competence and reflects any dependence that may exist among the expert predictions. The resulting maximum entropy aggregated prediction is least prejudiced in the sense that it utilizes all information available but remains maximally non committal with regard to information not available. Numerical examples to illuminate the implications of maximum entropy aggregation are also presented.

Observer-participant models of neural processing

A model is proposed in which the neuron serves as an information channel. Channel distortion occurs through the channel since the mapping from input Boolean codes to output codes are many-to-one in that neuron outputs consist of just two distinguished states. Within the described model, the neuron performs a decision-making function. Decisions are made regarding the validity of a question passively posed by the neuron. This question becomes defined through learning hence learning is viewed as the process of determining an appropriate question based on supplied input ensembles. An application of the Shannon information measures of entropy and mutual information taken together in the context of the proposed model lead to the Hopfield neuron model with conditionalized Hebbian learning rules. Neural decisions are shown to be based on a sigmoidal transfer characteristic or, in the limit as computational temperature tends to zero, a maximum likelihood decision rule. The described work is contrasted with the information-theoretic approach of Linsker.

Cortical feedback increases visual information transmitted by monkey parvocellular lateral geniculate nucleus neurons.

We studied the effect of cooling the striate cortex on parvocellular lateral geniculate nucleus (PLGN) neurons in awake monkeys. Cooling the striate cortex produced both facilitation and inhibition of the responses of all neurons, depending on the stimulus presented. Cooling the striate cortex also altered the temporal distribution of spikes in the responses of PLGN neurons. Shannon's information measure revealed that cooling the striate cortex reduced the average stimulus-related information transmitted by all PLGN neurons. The reduction in transmitted information was associated with both facilitation and inhibition of the response. Cooling the striate cortex reduced the amount of information transmitted about all of the stimulus parameters tested: pattern, luminance, spatial contrast, and sequential contrast. The effect of cooling was nearly the same for codes based on the number of spikes in the response as for codes based on their temporal distribution. The reduction in transmitted information occurred because the differences among the responses to different stimuli (signal separation) were reduced, not because the variability of the responses to individual stimuli (noise) was increased. We conclude that one function of corticogeniculate feedback is to improve the ability of PLGN neurons to discriminate among stimuli by enhancing the differences among their responses.