Maximum Entropy Inference Research Articles

Inference of metabolic fluxes in nutrient-limited continuous cultures: A Maximum Entropy approach with the minimum information.

The study of cellular metabolism is limited by the amount of experimental data available. Formulations able to extract relevant predictions from accessible measurements are needed. Maximum Entropy (ME) inference has been successfully applied to genome-scale models of cellular metabolism, and recent data-driven studies have suggested that in chemostat cultures of Escherichia coli (E.coli), the growth rate and uptake rates of limiting nutrients are the most informative observables. We propose the thesis that this can be explained by the chemostat dynamics, which typically drives nutrient-limited cultures toward observable metabolic states maximally restricted in the dimensions of those fluxes. A practical consequence is that relevant flux observables can now be replaced by culture parameters usually controlled. We test our model by using simulations, and then we apply it to E.coli experimental data where we evaluate the quality of the inference, comparing it to alternative formulations that rest on convex optimization.

Maximum entropy inference of seabed properties using waveguide invariant features from surface ships.

Acoustic data were recorded on two vertical line arrays (VLAs) deployed in the New England Mud Patch during the Seabed Characterization Experiment 2017 in about 75 m of water. The sound recorded during the passage of merchant ships permits identification of singular points for the waveguide invariant β for mode pairs (1,n):β1,n,for n=2,3,4,5, in the 15-80 Hz band. Using prior geophysical information and an acoustic data sample from the merchant ship KALAMATA, a geoacoustic model M of the seabed was developed. Then, using data samples from other merchant ships, a feature-ensemble maximum entropy method is employed to infer the statistical properties of geoacoustic parameter values for the sound speeds in a surface mud layer and a deep sand layer. Technical challenges include a sparsity of observed singular points, the unique identification of mode pairs for an observed singular point, and the deviation of the waveguide from horizontal stratification. A geoacoustic model M is developed that reproduced the observed β≈-1 for f < 20 Hz and mode cutoff features at about 15 Hz. The statistical low-frequency inference of the singular point structure from multiple ships provides evidence of an angle of intromission at the water sediment interface with an average sound speed ratio of about 0.986 and an average sound speed for the deeper sand layer of about 1775 m/s.

Relationship between fitness and heterogeneity in exponentially growing microbial populations

Despite major environmental and genetic differences, microbial metabolic networks are known to generate consistent physiological outcomes across vastly different organisms. This remarkable robustness suggests that, at least in bacteria, metabolic activity may be guided by universal principles. The constrained optimization of evolutionarily motivated objective functions, such as the growth rate, has emerged as the key theoretical assumption for the study of bacterial metabolism. While conceptually and practically useful in many situations, the idea that certain functions are optimized is hard to validate in data. Moreover, it is not always clear how optimality can be reconciled with the high degree of single-cell variability observed in experiments within microbial populations. To shed light on these issues, we develop an inverse modeling framework that connects the fitness of a population of cells (represented by the mean single-cell growth rate) to the underlying metabolic variability through the maximum entropy inference of the distribution of metabolic phenotypes from data. While no clear objective function emerges, we find that, as the medium gets richer, the fitness and inferred variability for Escherichia coli populations follow and slowly approach the theoretically optimal bound defined by minimal reduction of variability at given fitness. These results suggest that bacterial metabolism may be crucially shaped by a population-level trade-off between growth and heterogeneity.

Multiple metastable states in an off-lattice Potts model

The interactions between a group of components are commonly studied in several areas of science (social science, biology, material science, complex dynamical systems, among others) using the methods of thermodynamics and statistical mechanics. In this work we study the properties of the recently proposed off-lattice, two-dimensional Potts model (Davis et al., 2014), originally motivated by the dynamics of agent opinions, and which is described by a Hamiltonian obtained by a maximum entropy inference procedure. We performed microcanonical and canonical Monte Carlo simulations of the first-order phase transition in the model, revealing a caloric curve with metastable regions. Furthermore, we report a “switching” behavior between multiple metastable states. We also note that the thermodynamics of the model has striking similarities with systems having long-range interactions, even though the interactions are short-ranged.

Optimal Transport in Systems and Control

Optimal transport began as the problem of how to efficiently redistribute goods between production and consumers and evolved into a far-reaching geometric variational framework for studying flows of distributions on metric spaces. This theory enables a class of stochastic control problems to regulate dynamical systems so as to limit uncertainty to within specified limits. Representative control examples include the landing of a spacecraft aimed probabilistically toward a target and the suppression of undesirable effects of thermal noise on resonators; in both of these examples, the goal is to regulate the flow of the distribution of the random state. A most unlikely link turned up between transport of probability distributions and a maximum entropy inference problem posed by Erwin Schrödinger, where the latter is seen as an entropy-regularized version of the former. These intertwined topics of optimal transport, stochastic control, and inference are the subject of this review, which aims to highlight connections, insights, and computational tools while touching on quadratic regulator theory and probabilistic flows in discrete spaces and networks.

Energy and entropy: Path from game theory to statistical mechanics

Statistical mechanics is based on interplay between energy minimization and entropy maximization. Here we formalize this interplay via axioms of cooperative game theory (Nash bargaining) and apply it out of equilibrium. These axioms capture basic notions related to joint maximization of entropy and minus energy, formally represented by utilities of two different players. We predict thermalization of a non-equilibrium statistical system employing the axiom of affine covariance|related to the freedom of changing initial points and dimensions for entropy and energy|together with the contraction invariance of the entropy-energy diagram. Whenever the initial non-equilibrium state is active, this mechanism allows thermalization to negative temperatures. Demanding a symmetry between players fixes the final state to a specific positive-temperature (equilibrium) state. The approach solves an important open problem in the maximum entropy inference principle, {\it viz.} generalizes it to the case when the constraint is not known precisely.

An Entropy-Based Machine Learning Algorithm for Combining Macroeconomic Forecasts

This paper applies a Machine Learning approach with the aim of providing a single aggregated prediction from a set of individual predictions. Departing from the well-known maximum-entropy inference methodology, a new factor capturing the distance between the true and the estimated aggregated predictions presents a new problem. Algorithms such as ridge, lasso or elastic net help in finding a new methodology to tackle this issue. We carry out a simulation study to evaluate the performance of such a procedure and apply it in order to forecast and measure predictive ability using a dataset of predictions on Spanish gross domestic product.

Maximum entropy principle and Landau free energy inequality

ABSTRACT In this paper, the following problem is considered: given two Hermitian matrices H and K and two real numbers x and y, determine a positive semidefinite matrix ρ such that the von Neumann entropy is maximum, subject to the condition that and . This question arises in information theory and in statistical mechanics in connection with the maximum-entropy inference principle. To answer it, we use this principle and numerical range methods.

Signatures of quantum phase transitions from the boundary of the numerical range

The ground state energy of a finite-dimensional one-parameter Hamiltonian and the continuity of a maximum-entropy inference map are discussed in the context of quantum critical phenomena. The domain of the inference map is a convex compact set in the plane, called the numerical range. We study the differential geometry of its boundary in relation to the ground state energy. We prove that discontinuities of the inference map correspond to $C^1$-smooth crossings of the ground state energy with a higher energy level. Discontinuities may appear only at $C^1$-smooth points of the boundary of the numerical range considered as a manifold. Discontinuities exist at all $C^2$-smooth non-analytic boundary points and are essentially stronger than at analytic points or at points which are merely $C^1$-smooth (non-exposed points).

Computational design of hepatitis C virus immunogens from host-pathogen dynamics over empirical viral fitness landscapes

Hepatitis C virus (HCV) afflicts 170 million people and kills 700 000 annually. Vaccination offers the most realistic and cost effective hope of controlling this epidemic, but despite 25 years of research, no vaccine is available. A major obstacle is HCV’s extreme genetic variability and rapid mutational escape from immune pressure. Coupling maximum entropy inference with population dynamics simulations, we have employed a computational approach to translate HCV sequence databases into empirical landscapes of viral fitness and simulate the intrahost evolution of the viral quasispecies over these landscapes. We explicitly model the coupled host-pathogen dynamics by combining agent-based models of viral mutation with stochastically-integrated coupled ordinary differential equations for the host immune response. We validate our model in predicting the mutational evolution of the HCV RNA-dependent RNA polymerase (protein NS5B) within seven individuals for whom longitudinal sequencing data is available. We then use our approach to perform exhaustive in silico evaluation of putative immunogen candidates to rationally design tailored vaccines to simultaneously cripple viral fitness and block mutational escape within two selected individuals. By systematically identifying a small number of promising vaccine candidates, our empirical fitness landscapes and host-pathogen dynamics simulator can guide and accelerate experimental vaccine design efforts.

Mutations in bacterial genes induce unanticipated changes in the relationship between bacterial pathogens in experimental otitis media.

Otitis media (OM) is a common polymicrobial infection of the middle ear in children under the age of 15 years. A widely used experimental strategy to analyse roles of specific phenotypes of bacterial pathogens of OM is to study changes in co-infection kinetics of bacterial populations in animal models when a wild-type bacterial strain is replaced by a specific isogenic mutant strain in the co-inoculating mixtures. As relationships between the OM bacterial pathogens within the host are regulated by many interlinked processes, connecting the changes in the co-infection kinetics to a bacterial phenotype can be challenging. We investigated middle ear co-infections in adult chinchillas (Chinchilla lanigera) by two major OM pathogens: non-typeable Haemophilus influenzae (NTHi) and Moraxella catarrhalis (Mcat), as well as isogenic mutant strains in each bacterial species. We analysed the infection kinetic data using Lotka–Volterra population dynamics, maximum entropy inference and Akaike information criteria-(AIC)-based model selection. We found that changes in relationships between the bacterial pathogens that were not anticipated in the design of the co-infection experiments involving mutant strains are common and were strong regulators of the co-infecting bacterial populations. The framework developed here allows for a systematic analysis of host–host variations of bacterial populations and small sizes of animal cohorts in co-infection experiments to quantify the role of specific mutant strains in changing the infection kinetics. Our combined approach can be used to analyse the functional footprint of mutant strains in regulating co-infection kinetics in models of experimental OM and other polymicrobial diseases.

Deterministic physical systems under uncertain initial conditions: the case of maximum entropy applied to projectile motion

The kinematics and dynamics of deterministic physical systems have been a foundation of our understanding of the world since Galileo and Newton. For real systems, however, uncertainty is largely present via external forces such as friction or lack of precise knowledge about the initial conditions of the system. In this work we focus on the latter case and describe the use of inference methodologies in solving the statistical properties of classical systems subject to uncertain initial conditions. In particular we describe the application of the formalism of maximum entropy (MaxEnt) inference to the problem of projectile motion, given information about the average horizontal range over many realizations. By using MaxEnt we can invert the problem and use the provided information on the average range to reduce the original uncertainty in the initial conditions. Also, additional insight into the initial condition's probabilities, and the projectile path distribution itself, can be achieved based on the value of the average horizontal range. The wide applicability of this procedure, as well as its ease of use, reveals a useful tool with which to revisit a large number of physics problems, from classrooms to frontier research.

Maximum Entropy Theory of Ecology: A Reply to Harte.

In a paper published in this journal, I addressed the following problem: under which conditions will two scientists, observing the same system and sharing the same initial information, reach the same probabilistic description upon the application of the Maximum Entropy inference principle (MaxEnt) independent of the probability distribution chosen to set up the MaxEnt procedure. This is a minimal objectivity requirement which is generally asked for scientific investigation. In the same paper, I applied the findings to a critical examination of the application of MaxEnt made in Harte’s Maximum Entropy Theory of Ecology (METE). Prof. Harte published a comment to my paper and this is my reply. For the sake of the reader who may be unaware of the content of the papers, I have tried to make this reply self-contained and to skip technical details. However, I invite the interested reader to consult the previously published papers.

Maximum Entropy and Theory Construction: A Reply to Favretti.

In the maximum entropy theory of ecology (METE), the form of a function describing the distribution of abundances over species and metabolic rates over individuals in an ecosystem is inferred using the maximum entropy inference procedure. Favretti shows that an alternative maximum entropy model exists that assumes the same prior knowledge and makes predictions that differ from METE’s. He shows that both cannot be correct and asserts that his is the correct one because it can be derived from a classic microstate-counting calculation. I clarify here exactly what the core entities and definitions are for METE, and discuss the relevance of two critical issues raised by Favretti: the existence of a counting procedure for microstates and the choices of definition of the core elements of a theory. I emphasize that a theorist controls how the core entities of his or her theory are defined, and that nature is the final arbiter of the validity of a theory.

Entropy and Insufficient Reason: A Note on the Judy Benjamin Problem

One well-known objection to the principle of maximum entropy is the so-called Judy Benjamin problem, first introduced by van Fraassen ([1981]). The problem turns on the apparently puzzling fact that, on the basis of information relating an event’s conditional probability, the maximum entropy distribution will almost always assign to the event conditionalized on a probability strictly less than that assigned to it by the uniform distribution. In this article, I present an analysis of the Judy Benjamin problem that can help to make sense of this seemingly odd feature of maximum entropy inference. My analysis is based on the claim that, in applying the principle of maximum entropy, Judy Benjamin is not acting out of a concern to maximize uncertainty in the face of new evidence, but is rather exercising a certain brand of epistemic charity towards her informant. This epistemic charity takes the form of an assumption on the part of Judy Benjamin that her informant’s evidential report leaves out no relevant information. Such a reconceptualization of the motives underlying Judy Benjamin’s appeal to the principle of maximum entropy can help to further our understanding of the true epistemological grounds of this principle and correct a common misapprehension regarding its relationship to the principle of insufficient reason. 1. Introduction2. The Principle of Maximum Entropy3. An Apologia for Judy Benjamin4. Conclusion: Entropy and Insufficient Reason

Rhythms of the Collective Brain: Metastable Synchronization and Cross-Scale Interactions in Connected Multitudes

Crowd behaviour challenges our fundamental understanding of social phenomena. Involving complex interactions between multiple temporal and spatial scales of activity, its governing mechanisms defy conventional analysis. Using 1.5 million Twitter messages from the 15M movement in Spain as an example of multitudinous self-organization, we describe the coordination dynamics of the system measuring phase-locking statistics at different frequencies using wavelet transforms, identifying 8 frequency bands of entrained oscillations between 15 geographical nodes. Then we apply maximum entropy inference methods to describe Ising models capturing transient synchrony in our data at each frequency band. The models show that (1) all frequency bands of the system operate near critical points of their parameter space and (2) while fast frequencies present only a few metastable states displaying all-or-none synchronization, slow frequencies present a diversity of metastable states of partial synchronization. Furthermore, describing the state at each frequency band using the energy of the corresponding Ising model, we compute transfer entropy to characterize cross-scale interactions between frequency bands, showing (1) a cascade of upward information flows in which each frequency band influences its contiguous slower bands and (2) downward information flows where slow frequencies modulate distant fast frequencies.

A Direct Mapping of Max k-SAT and High Order Parity Checks to a Chimera Graph.

We demonstrate a direct mapping of max k-SAT problems (and weighted max k-SAT) to a Chimera graph, which is the non-planar hardware graph of the devices built by D-Wave Systems Inc. We further show that this mapping can be used to map a similar class of maximum satisfiability problems where the clauses are replaced by parity checks over potentially large numbers of bits. The latter is of specific interest for applications in decoding for communication. We discuss an example in which the decoding of a turbo code, which has been demonstrated to perform near the Shannon limit, can be mapped to a Chimera graph. The weighted max k-SAT problem is the most general class of satisfiability problems, so our result effectively demonstrates how any satisfiability problem may be directly mapped to a Chimera graph. Our methods faithfully reproduce the low energy spectrum of the target problems, so therefore may also be used for maximum entropy inference.

On the applicability of the ‘number of possible states’ argument in multi-expert reasoning

The aim of this paper is to explore the applicability of the ‘number of possible states’ argument in inferential problems in multi-expert reasoning. The argument is Bayesian and it is similar in spirit to the one used to derive the maximum entropy inference process. Under certain conditions a particular way of applying it surprisingly suggests that the weighted arithmetic mean should be used in meta-analysis with unexplained heterogeneity.

Maximum-Entropy Inference and Inverse Continuity of the Numerical Range

We study the continuity of the maximum-entropy inference map for two observables in finite dimensions. We prove that the continuity is equivalent to the strong continuity of the set-valued inverse numerical range map. This gives a continuity condition in terms of analytic eigenvalue functions which implies that discontinuities are very rare. It shows also that the continuity of the MaxEnt inference method is independent of the prior state.

Maximum-Entropy Inference with a Programmable Annealer.

Optimisation problems typically involve finding the ground state (i.e. the minimum energy configuration) of a cost function with respect to many variables. If the variables are corrupted by noise then this maximises the likelihood that the solution is correct. The maximum entropy solution on the other hand takes the form of a Boltzmann distribution over the ground and excited states of the cost function to correct for noise. Here we use a programmable annealer for the information decoding problem which we simulate as a random Ising model in a field. We show experimentally that finite temperature maximum entropy decoding can give slightly better bit-error-rates than the maximum likelihood approach, confirming that useful information can be extracted from the excited states of the annealer. Furthermore we introduce a bit-by-bit analytical method which is agnostic to the specific application and use it to show that the annealer samples from a highly Boltzmann-like distribution. Machines of this kind are therefore candidates for use in a variety of machine learning applications which exploit maximum entropy inference, including language processing and image recognition.