Marginal Entropy Research Articles

Normalized Mutual Entropy in Biology: Quantifying Division of Labor

Division of labor is one of the primary adaptations of sociality and the focus of much theoretical work on self-organization. This work has been hampered by the lack of a quantitative measure of division of labor that can be applied across systems. We divide Shannon's mutual entropy by marginal entropy to quantify division of labor, rendering it robust over changes in number of individuals or tasks. Reinterpreting individuals and tasks makes this methodology applicable to a wide range of other contexts, such as breeding systems and predator-prey interactions.

Information and Coding Discrimination of Pseudo-Additive Entropies (PAE)

PAE cannot be made a basis for either a generalized statistical mechanics or a generalized information theory. Either statistical independence must be waived, or the expression of the averaged conditional probability as the difference between the marginal and joint entropies must be relinquished. The same inequality, relating the PAE to the Rényi entropy, when applied to the mean code length produces an expression that is without bound as the order of the code length approaches infinity. Since the mean code length associated with the Rényi entropy is finite and can be made to come as close to the Hartley entropy as desired in the same limit, the PAE have a more limited range of validity than the Rényi entropy which they approximate.

Characterizing entanglement with global and marginal entropic measures

We qualify the entanglement of arbitrary mixed states of bipartite quantum systems by comparing global and marginal mixednesses quantified by different entropic measures. For systems of two qubits we discriminate the class of maximally entangled states with fixed marginal mixednesses, and determine an analytical upper bound relating the entanglement of formation to the marginal linear entropies. This result partially generalizes to mixed states the quantification of entanglement with marginal mixednesses holding for pure states. We identify a class of entangled states that, for fixed marginals, are globally more mixed than product states when measured by the linear entropy. Such states cannot be discriminated by the majorization criterion.

Uncertainty of Weekly Nitrate-Nitrogen Forecasts Using Artificial Neural Networks

Nonpoint source pollution affects the quality of numerous watersheds in the Midwestern United States. The Illinois State Water Survey conducted this study to ~1! assess the potential of artificial neural networks ~ANNs! in forecasting weekly nitrate-nitrogen ~nitrate-N! concentration; and ~2! evaluate the uncertainty associated with those forecasts. Three ANN models were applied to predict weekly nitrate-N concentrations in the Sangamon River near Decatur, Illinois, based on past weekly precipitation, air temperature, discharge, and past nitrate-N concentrations. Those ANN models were more accurate than the linear regression models having the same inputs and output. Uncertainty of the ANN models was further expressed through the entropy principle, as defined in the information theory. Using several inputs in an ANN-based forecasting model reduced the uncertainty expressed through the marginal entropy of weekly nitrate-N concentrations. The uncertainty of predictions was expressed as conditional entropy of future nitrate concentrations for given past precipitation, temperature, discharge, and nitrate-N concentration. In general, the uncertainty of predictions decreased with model complexity. Including additional input variables produced more accurate predictions. However, using the previous weekly data ~week t21! did not reduce the uncertainty in the predictions of future nitrate concentrations ~week t11! based on current weekly data ~week t!.

Blind source separation using Renyi's α-marginal entropies

We have recently suggested the minimization of a nonparametric estimator of Renyi's mutual information as a criterion for blind source separation. Using a two-stage topology, consisting of spatial whitening and a series of Givens rotations, the cost function reduces to the sum of marginal entropies, just like in the Shannon's entropy case. Since we use a Parzen window density estimator and eliminate the joint entropy by employing an orthonormal demixing matrix, the problems of probability density function inaccuracy due to truncation of series expansion and the estimation of joint pdfs in high-dimensional spaces (given the typical paucity of data) are avoided, respectively. In our previous formulation, the algorithm was restricted to Renyi's second-order entropy and Gaussian kernels for the Parzen window estimator. The present work extends the previous results by formulating a new estimation methodology for Renyi's entropy, which allows the designer to choose any order of entropy and any suitable kernel function. Simulations illustrate that the proposed method compares favorably to Hyvarinen's FastICA, Bell and Sejnowski's Infomax and Common's minimum of mutual information.

Combining belief networks and neural networks for scene segmentation

We are concerned with the problem of image segmentation, in which each pixel is assigned to one of a predefined finite number of labels. In Bayesian image analysis, this requires fusing together local predictions for the class labels with a prior model of label images. Following the work of Bouman and Shapiro (1994), we consider the use of tree-structured belief networks (TSBNs) as prior models. The parameters in the TSBN are trained using a maximum-likelihood objective function with the EM algorithm and the resulting model is evaluated by calculating how efficiently it codes label images. A number of authors have used Gaussian mixture models to connect the label field to the image data. We compare this approach to the scaled-likelihood method of Smyth (1994) and Morgan and Bourlard (1995), where local predictions of pixel classification from neural networks are fused with the TSBN prior. Our results show a higher performance is obtained with the neural networks. We evaluate the classification results obtained and emphasize not only the maximum a posteriori segmentation, but also the uncertainty, as evidenced e.g., by the pixelwise posterior marginal entropies. We also investigate the use of conditional maximum-likelihood training for the TSBN and find that this gives rise to improved classification performance over the ML-trained TSBN.

Dynamics of quantum entanglement

A model of discrete dynamics of entanglement of bipartite quantum state is considered. It involves a global unitary dynamics of the system and periodic actions of local bistochastic or decaying channel. For initially pure states the decay of entanglement is accompanied with an increase of von Neumann entropy of the system. We observe and discuss revivals of entanglement due to unitary interaction of both subsystems. For some mixed states having different marginal entropies of both subsystems (one of them larger than the global entropy and the other one one smaller) we find an asymmetry in speed of entanglement decay. The entanglement of these states decreases faster, if the depolarizing channel acts on the "classical" subsystem, characterized by smaller marginal entropy.

Blind source separation using Renyi's mutual information

A blind source separation algorithm is proposed that is based on minimizing Renyi's mutual information by means of nonparametric probability density function (PDF) estimation. The two-stage process consists of spatial whitening and a series of Givens rotations and produces a cost function consisting only of marginal entropies. This formulation avoids the problems of PDF inaccuracy due to truncation of series expansion and the estimation of joint PDFs in high-dimensional spaces given the typical paucity of data. Simulations illustrate the superior efficiency, in terms of data length, of the proposed method compared to fast independent component analysis (FastICA), Comon's (1994) minimum mutual information, and Bell and Sejnowski's (1995) Infomax.

Marginal maximum entropy partitioning yields asymptotically consistent probability density functions

The marginal maximum entropy criterion has been used to guide recursive partitioning of a continuous sample space. Although the criterion has been successfully applied in pattern discovery applications, its theoretical justification has not been clearly addressed. In the paper, it is shown that the basic marginal maximum entropy partitioning algorithm yields asymptotically consistent density estimates. This result supports the use of the marginal maximum entropy criterion in pattern discovery and implies that an optimal classifier can be constructed.

Maximum Entropy Sampling and Optimal Bayesian Experimental Design

Summary When Shannon entropy is used as a criterion in the optimal design of experiments, advantage can be taken of the classical identity representing the joint entropy of parameters and observations as the sum of the marginal entropy of the observations and the preposterior conditional entropy of the parameters. Following previous work in which this idea was used in spatial sampling, the method is applied to standard parameterized Bayesian optimal experimental design. Under suitable conditions, which include non-linear as well as linear regression models, it is shown in a few steps that maximizing the marginal entropy of the sample is equivalent to minimizing the preposterior entropy, the usual Bayesian criterion, thus avoiding the use of conditional distributions. It is shown using this marginal formulation that under normality assumptions every standard model which has a two-point prior distribution on the parameters gives an optimal design supported on a single point. Other results include a new asymptotic formula which applies as the error variance is large and bounds on support size.

An overlap invariant entropy measure of 3D medical image alignment

This paper is concerned with the development of entropy-based registration criteria for automated 3D multi-modality medical image alignment. In this application where misalignment can be large with respect to the imaged field of view, invariance to overlap statistics is an important consideration. Current entropy measures are reviewed and a normalised measure is proposed which is simply the ratio of the sum of the marginal entropies and the joint entropy. The effect of changing overlap on current entropy measures and this normalised measure are compared using a simple image model and experiments on clinical image data. Results indicate that the normalised entropy measure provides significantly improved behaviour over a range of imaged fields of view.

An Entropy-Based, Air-Side Heat Exchanger Performance Evaluation Method: Application to a Condenser

This paper describes the development of a straightforward, entropy-based method for evaluating air-side heat exchanger performance. Using energy conservation, the appropriate rate equations, and the second law of thermodynamics, all energy interactions were cast into their available- work equivalents with heat transfer rate limitations. The proposed method improved on previous techniques in two ways. First, it placed value explicitly on heat duty, recognizing system design constraints and external entropy generation. Second, it accounted for marginal entropy generation due to coupling between the system and the heat exchanger. The effects of the heat exchanger design on the system can cause significant exergy destruction and must be considered by any good performance measure. The proposed methods were applied to the evaluation of a condenser in a vapor-compression system. The condenser example is discussed here in detail, to explore design tradeoffs.

Phase-intensity uncertainty relation from quasiprobability distributions

Extending the concept of Wehrl's entropy to s-parametrized quasiprobability distributions, as they are observed in optical homodyne detection with realistic detectors, we show that the marginal entropies with respect to phase and amplitude obey a rigourous inequality relation. Relating those entropies to phase and intensity uncertainties, respectively, we arrive at an uncertainty relation for operationally defined phase and intensity that takes explicit account of the noise introduced by non-ideal detectors.

Generalized entropy as a measure of quantum uncertainty

A generalized entropy is used in order to advance a different form of expressing the Uncertainty Principle of Quantum mechanics. We consider the generalized entropic formulation for different pairs of incompatible observables. In particular, we study the number-phase entropic uncertainty measure for the case of coherent states within the Pegg-Barnett theory. We also tackle the situation of operators with continuous spectra, where a correlation functional is calculated in terms of generalized joint and marginal entropies, for harmonic oscillator wavefunctions.

Entropy maximised queueing networks with blocking and multiple job classes

The method of entropy maximisation (MEM) is applied in a state space partitioning mode for the approximation of the joint stationary queue length distribution of an M/M/1/N queue with finite capacity, N( > 1), multiple and distinct classes of jobs, R( > 1), under a complete buffer sharing scheme and mixed service disciplines drawn from the first-come-first-served (FCFS), last-come-first-served with (LCFS-PR) or without (LCFS-NPR) preemption and processor sharing (PS) rules. The marginal and aggregate maximum entropy (ME) queue length distributions and the associated blocking probabilities per class are also determined. These ME results in conjunction with the first moments of the effective flows are used, as building blocks, in order to establish a new product-form approximation for arbitrary exponential open queueing networks with multiple classes of jobs under repetitive-service (RS) blocking with random destination (RD). It is verified that the ME approximation reduces to the exact truncated solution of open multi-class reversible queueing networks. Numerical experiments demonstrate a good accuracy level of ME statistics in relation to simulation. Moreover, recent extentions of MEM for arbitrary GE-type queueing networks with RS-RD blocking and multiple classes of jobs are presented.

Informational Observation Theory: An Extension to Two Discrete Sets of Events

Gives an analysis of the consequences of the limitations inherent in observation processes PA and PB on the joint, marginal and conditional entropies of two discrete sets of events A and B, observed by means of PA and PB. Up to now, unlike the continuous case studied in a previous article, the Shannon theory seemed to apply without any problems. Demonstrates that this is not true in some particular situations. Some properties of dissymmetry, absent from the Shannonian model, appear.

Signal-pump entanglement in quantum k-photon down-conversion.

We study the quantum dynamics of the k-photon down-conversion process. We concentrate our attention on the entanglement between the pump and the signal modes. We show that the degree of the entanglement between the pump and the signal depends on the initial statistics of the light field in both modes. Moreover, the higher the nonlinear process the stronger the entanglement is. The entanglement between the modes is related to the marginal entropy in the pump (signal) mode; i.e., the larger the entanglement, the larger the entropy is. In the quantum k-photon down-conversion the signal mode at t>0 is not generally in a pure state which restricts applicability of the parametric approximation.

Informational Observation Theory: An Extension to Two Continuous Sets of Events

An analysis is proposed of the consequences of the limitations inherent in observation processes PA and PB on the joint, marginal and conditional entropies of two continuous sets A and B of events, observed by means of PA and PB. These events can be statistically dependent. Properties of asymmetry appear which do not exist in the Shannonian model. They induce a new definition of mutual information.

Some informational properties of the lognormal distribution (Corresp.)

The marginal, joint, and conditional entropy and the trans-information are derived for random variables with lognormal probability distributions, revealing some interesting deviations from its sister distribution, the normal. A maximal entropy property, capacity theorem for the lognormal channel, and implications for some nonlinear transformations are also presented. Potential applications of the measures to psychophysics are mentioned.

Entropy in the assessment of uncertainty in hydrologic systems and models

In the representation of natural catchment systems with mathematical models, a relatively wide range of choice exists among models of various degrees of completeness and sophistication. When the cost, as well as the appropriateness, of a model is considered for a particular purpose, it is desirable to have an objective criterion to make this selection. The concept of entropy as used in information theory provides one such criterion. Entropy is a measure of the degree of uncertainty of a particular outcome in a process; therefore, in dealing with the prediction of a hydrologie variable such as streamflow, one can compute the entropy of this variable from historical data and thus characterize the unexpectedness or variability inherent in the process. This represents a property of the system and is called marginal entropy. It is likewise possible to evaluate the uncertainty of the predictions made by a given mathematical model of the catchment by comparing these predictions with the measured flow. This is done through the conditional entropy. By combination of these two entropy functions a criterion called ‘transinformation,’ providing an objective evaluation of the goodness of the model, is obtained. The important point to note is that this assessment depends not only on the model but on the characteristics of the output series resulting from the natural process. The above criteria, which have been discussed in other contexts, were applied to a basin for which a number of years of historical flow record and concurrent model simulations were available. The results show the value of the entropy criterion in judging model performance as well as the difficulties and limitations of the method.