Constrained Maximum Entropy Research Articles

MaxEnt Loss: Constrained Maximum Entropy for Calibration under Out-of-Distribution Shift

We present a new loss function that addresses the out-of-distribution (OOD) network calibration problem. While many objective functions have been proposed to effectively calibrate models in-distribution, our findings show that they do not always fare well OOD. Based on the Principle of Maximum Entropy, we incorporate helpful statistical constraints observed during training, delivering better model calibration without sacrificing accuracy. We provide theoretical analysis and show empirically that our method works well in practice, achieving state-of-the-art calibration on both synthetic and real-world benchmarks. Our code is available at https://github.com/dexterdley/MaxEnt-Loss.

Node fault diagnosis algorithm for wireless sensor networks based on BN and WSN

Wireless sensor networks, as an emerging information exchange technology, have been widely applied in many fields. However, nodes tend to become damaged in harsh and complex environmental conditions. In order to effectively diagnose node faults, a Bayesian model-based node fault diagnosis model was proposed. Firstly, a comprehensive analysis was conducted into the operative principles of wireless sensor systems, whereby fault-related features were then extrapolated. A Bayesian diagnostic model was constructed using the maximum likelihood method with sufficient sample features, and a joint tree model was introduced for node diagnosis. Due to the insufficient accuracy of Bayesian models in processing small sample data, a constrained maximum entropy method was proposed as the prediction module of the model. The use of small sample data to obtain the initial model parameters leads to improved performance and accuracy of the model. During parameter learning tests, the limited maximum entropy model outperformed the other two learning models on a smaller dataset of 35 with a distance value of 2.65. In node fault diagnosis, the diagnostic time of the three models was compared, and the average diagnostic time of the proposed diagnostic model was 41.2 seconds. In the node diagnosis accuracy test, the proposed model has the highest node fault diagnosis accuracy, with an average diagnosis accuracy of 0.946, which is superior to the other two models. In summary, the node fault diagnosis model based on Bayesian model proposed in this study has important research significance and practical application value in wireless sensor networks. By improving the reliability and maintenance efficiency of the network, this model provides strong support for the development and application of wireless sensor networks.

Saccadic trajectories deviate toward or away from optimally informative visual features

The saccades' path is influenced by visual distractors, making their trajectory curve away or toward them. Previous research suggested that the more salient the distractor, the more pronounced is the curvature. We investigate the saliency of spatial visual features, predicted by a constrained maximum entropy model to be optimal or non-optimal information carriers in fast vision, by using them as distractors in a saccadic task. Their effect was compared to that of luminance-based control distractors. Optimal features evoke a larger saccadic curvature compared to non-optimal features, and the magnitude and direction of deviation change as a function of the delay between distractor and saccade onset. Effects were similar to those found with high-luminance versus low-luminance distractors. Therefore, model-predicted information optimal features interfere with target-oriented saccades, following a dynamic attraction-repulsion pattern. This suggests that the visuo-oculomotor system rapidly and automatically processes optimally informative features while programming visually guided eye movements.

CEM: Constrained Entropy Maximization for Task-Agnostic Safe Exploration

In the absence of assigned tasks, a learning agent typically seeks to explore its environment efficiently. However, the pursuit of exploration will bring more safety risks. An under-explored aspect of reinforcement learning is how to achieve safe efficient exploration when the task is unknown. In this paper, we propose a practical Constrained Entropy Maximization (CEM) algorithm to solve task-agnostic safe exploration problems, which naturally require a finite horizon and undiscounted constraints on safety costs. The CEM algorithm aims to learn a policy that maximizes state entropy under the premise of safety. To avoid approximating the state density in complex domains, CEM leverages a k-nearest neighbor entropy estimator to evaluate the efficiency of exploration. In terms of safety, CEM minimizes the safety costs, and adaptively trades off safety and exploration based on the current constraint satisfaction. The empirical analysis shows that CEM enables the acquisition of a safe exploration policy in complex environments, resulting in improved performance in both safety and sample efficiency for target tasks.

On Bayesian mechanics: a physics of and by beliefs.

The aim of this paper is to introduce a field of study that has emerged over the last decade, called Bayesian mechanics. Bayesian mechanics is a probabilistic mechanics, comprising tools that enable us to model systems endowed with a particular partition (i.e. into particles), where the internal states (or the trajectories of internal states) of a particular system encode the parameters of beliefs about external states (or their trajectories). These tools allow us to write down mechanical theories for systems that look as if they are estimating posterior probability distributions over the causes of their sensory states. This provides a formal language for modelling the constraints, forces, potentials and other quantities determining the dynamics of such systems, especially as they entail dynamics on a space of beliefs (i.e. on a statistical manifold). Here, we will review the state of the art in the literature on the free energy principle, distinguishing between three ways in which Bayesian mechanics has been applied to particular systems (i.e. path-tracking, mode-tracking and mode-matching). We go on to examine a duality between the free energy principle and the constrained maximum entropy principle, both of which lie at the heart of Bayesian mechanics, and discuss its implications.

Goal Orientation in Music Composition and Other Social Behaviors Leading to the Common Quantitative Law

The power-law distribution is ubiquitous and common in many natural, engineering, and social systems. In this article, we found that goal orientation is the driving reason on the power-law distribution in typical social behaviors. A unified model is constructed to formulate the goal orientation mechanism in many social systems. Three mathematical characteristics of tonal music melody are found based on music composition theory. The constrained entropy maximization model (CEMM) is constructed, and the power-law distribution of melodic intervals is the result of solving the problem formulated in this model. Three other typical power-law models in economics, linguistics/informatics, and design of complex engineering systems are reviewed, and their goal orientation behaviors are analyzed. It is found that the models quantifying different social behaviors with goal orientation are consistent with the unified model. The power law is actually the result of the well-known social distribution mechanisms with limited resources, such as the maximal wealth with limited economic capability, the minimal cost under constrained resources, and the maximal entropy with restrictions. If without limit or constraint, these social systems would not observe the power law. This finding could inspire the study on other common laws in computational social systems.

Local reliability sensitivity method by the reconstructed fraction moment constrained maximum entropy

The local sensitivity of the failure probability (LS-FP) with respect to the distribution parameter of the random input is very important in the field of structural reliability analysis and the reliability based design optimization (RBDO). It can be used not only to measure the effect of distribution parameters of random inputs on the failure probability, but also to provide the exact gradient for the RBDO. In this paper, the LS-FP with respect to the mean parameters and standard deviation parameters are originally derived as the weighted sum of two types of failure probability, one is the original failure probability and the other one is the integration of reconfigurable probability density function (PDF) in failure domain. By reconstructing the optimal PDF, the maximum entropy method based on the fraction moment constraints is then proposed to estimate the sensitivity of the failure probability, which transforms the LS-FP as a byproduct of the original failure probability estimation and avoids the additional computational cost.

Asymptotic method for entropic multiple relaxation time model in lattice Boltzmann method.

To improve the numerical stability of the lattice Boltzmann method, Karlin etal. [Phys. Rev. E 90, 031302(R) (2014)10.1103/PhysRevE.90.031302] proposed the entropic multiple relaxation time (EMRT) collision model. The idea behind EMRT is to construct an optimal postcollision state by maximizing its local entropy value. The critical step of the EMRT model is to solve the entropy maximization problem under certain constraints, which is often computationally expensive and even not feasible. In this paper, we propose to employ perturbation theory and obtain an asymptotic solution to the maximum entropy state. With mathematical analysis of particular cases under relaxed constraints, we obtain the unperturbed form of the original problem and derive the asymptotic solution. We show that the asymptotic solution well approximates the optimal states; thus, our approach provides an efficient way to solve the constrained maximum entropy problem in the EMRT model. Also, we use the same idea of the EMRT model for the initial condition of the distribution function and propose to leave the entropy function to determine the missing information at the initial nodes. Finally, we numerically verify that the simulation results of the EMRT model obtained via the perturbation theory agree well with the exact solution to the Taylor-Green vortex problem. Furthermore, we also demonstrate that the EMRT model exhibits excellent stability performance for under-resolved simulations in the doubly periodic shear layer flow problem.

Common quantitative characteristics of music melodies — pursuing the constrained entropy maximization casually in composition

Common quantitative characteristics of music melodies — pursuing the constrained entropy maximization casually in composition

On the solution of the population balance equation: From global to local constrained maximum entropy method

We propose continuous approximations to the population balance equation based on maximization of the Shannon entropy subject to the expected properties of the particle size distribution (PSD). This solution is used to close the source term of the PBE with careful sampling of the PSD at prescribed points as roots of the Nth-degree Legendre polynomial. Being a maximum entropy functional, the solution is unique and converges to the exact solution as the number of sampling points increases with accurate calculation of PSD integral properties. The accuracy and efficiency of the method are demonstrated by trying different analytical case studies (particle aggregation, aggregation and growth, and particle breakage) where we show it is not restricted to prespecified particle kinetics and functional forms. As practical case study, we modelled the coupled hydrodynamics and mass transfer in different liquid extraction columns and compared the calculated results with published steady state experimental data.

Optimal privacy control for transport network data sharing

In the era of smart cities, Internet of Things, and Mobility-as-a-Service, private operators need to share data with public agencies to support data exchanges for “living lab” ecosystems more than ever before. However, it is still problematic for private operators to share data with the public due to risks to competitive advantages. A privacy control algorithm is proposed to overcome this key obstacle for private operators sharing complex network-oriented data objects. The algorithm is based on information-theoretic k-anonymity and, using tour data as an example, where an operator’s data is used in conjunction with performance measure accuracy controls to synthesize a set of alternative tours with diffused probabilities for sampling during a query. The algorithm is proven to converge sublinearly toward a constrained maximum entropy under certain asymptotic conditions with measurable gap. Computational experiments verify the applicability to multi-vehicle fleet tour data; they confirm that reverse engineered parameters from the diffused data result in controllable sampling error; and tests conducted on a set of realistic routing records from travel data in Long Island, NY, demonstrate the use of the methodology from both the adversary and user perspectives.

A radiative-convective model based on constrained maximum entropy production

Abstract. The representation of atmospheric convection induced by radiative forcing is a long-standing question mainly because turbulence plays a key role in the transport of energy as sensible heat, geopotential, and latent heat. Recent works have tried using the maximum entropy production (MEP) conjecture as a closure hypothesis in 1-D simple climate models to compute implicitly temperatures and the vertical energy flux. However, these models fail to reproduce realistic profiles. To solve the problem, we describe the energy fluxes as a product of a positive mass mixing coefficient with the corresponding energy gradient. This appears as a constraint which imposes the direction and/or limits the amplitude of the energy fluxes. It leads to a different MEP steady state which naturally depends on the considered energy terms in the model. Accounting for this additional constraint improves the results. Temperature and energy flux are closer to observations, and we reproduce stratification when we consider the geopotential. Variations in the atmospheric composition, such as a doubling of the carbon dioxide concentration, are also investigated.

Boosted negative sampling by quadratically constrained entropy maximization

Learning probability densities for natural language representations is a difficult problem because language is inherently sparse and high-dimensional. Negative sampling is a popular and effective way to avoid intractable maximum likelihood problems, but it requires correct specification of the sampling distribution. Previous state of the art methods rely on heuristic distributions that appear to do well in practice. In this work, we define conditions for optimal sampling distributions and demonstrate how to approximate them using Quadratically Constrained Entropy Maximization(QCEM). Our analysis shows that state of the art heuristics are restrictive approximations to our proposed framework. To demonstrate the merits of our formulation, we apply QCEM to matching synthetic exponential family distributions and to finding high dimensional word embedding vectors for English. We are able to achieve faster inference on synthetic experiments and improve the correlation on semantic similarity evaluations on the Rare Words dataset by 4.8%.

Learning Bayesian network parameters via minimax algorithm

Parameter learning is an important aspect of learning in Bayesian networks. Although the maximum likelihood algorithm is often effective, it suffers from overfitting when there is insufficient data. To address this, prior distributions of model parameters are often imposed. When training a Bayesian network, the parameters of the network are optimized to fit the data. However, imposing prior distributions can reduce the fitness between parameters and data. Therefore, a trade-off is needed between fitting and overfitting. In this study, a new algorithm, named MiniMax Fitness (MMF) is developed to address this problem. The method includes three main steps. First, the maximum a posterior estimation that combines data and prior distribution is derived. Then, the hyper-parameters of the prior distribution are optimized to minimize the fitness between posterior estimation and data. Finally, the order of posterior estimation is checked and adjusted to match the order of the statistical counts from the data. In addition, we introduce an improved constrained maximum entropy method, named Prior Free Constrained Maximum Entropy (PF-CME), to facilitate parameter learning when domain knowledge is provided. Experiments show that the proposed methods outperforms most of existing parameter learning methods.

Optimal privacy control for transport network data sharing

Abstract In the era of smart cities, Internet of Things, and Mobility-as-a-Service, private operators need to share data with public agencies to support data exchanges for “living lab” ecosystems more than ever before. However, it is still problematic for private operators to share data with the public due to risks to competitive advantages. A privacy control algorithm is proposed to overcome this key obstacle for private operators sharing complex network-oriented data objects. The algorithm is based on information-theoretic k-anonymity and, using tour data as an example, where an operator’s data is used in conjunction with performance measure accuracy controls to synthesize a set of alternative tours with diffused probabilities for sampling during a query. The algorithm is proven to converge sublinearly toward a constrained maximum entropy under certain asymptotic conditions with measurable gap. Computational experiments verify the applicability to multi-vehicle fleet tour data; they confirm that reverse engineered parameters from the diffused data result in controllable sampling error; and tests conducted on a set of realistic routing records from travel data in Long Island, NY, demonstrate the use of the methodology from both the adversary and user perspectives.

Constrained maximum entropy models to select genotype interactions associated with censored failure times.

We propose a novel screening method targeting genotype interactions associated with disease risks. The proposed method extends the maximum entropy conditional probability model to address disease occurrences over time. Continuous occurrence times are grouped into intervals. The model estimates the conditional distribution over the disease occurrence intervals given individual genotypes by maximizing the corresponding entropy subject to constraints linking genotype interactions to time intervals. The EM algorithm is employed to handle observations with uncertainty, for which the disease occurrence is censored. Stepwise greedy search is proposed to screen a large number of candidate constraints. The minimum description length is employed to select the optimal set of constraints. Extensive simulations show that five or so quantile-dependent intervals are sufficient to categorize disease outcomes into different risk groups. Performance depends on sample size, number of genotypes, and minor allele frequencies. The proposed method outperforms the likelihood ratio test, Lasso, and a previous maximum entropy method with only binary (disease occurrence, non-occurrence) outcomes. Finally, a GWAS study for type 1 diabetes patients is used to illustrate our method. Novel one-genotype and two-genotype interactions associated with neuropathy are identified.

An equation-by-equation method for solving the multidimensional moment constrained maximum entropy problem

An equation-by-equation (EBE) method is proposed to solve a system of nonlinear equations arising from the moment constrained maximum entropy problem of multidimensional variables. The design of the EBE method combines ideas from homotopy continuation and Newton's iterative methods. Theoretically, we establish the local convergence under appropriate conditions and show that the proposed method, geometrically, finds the solution by searching along the surface corresponding to one component of the nonlinear problem. We will demonstrate the robustness of the method on various numerical examples, including: (1) A six-moment one-dimensional entropy problem with an explicit solution that contains components of order $10^0-10^3$ in magnitude; (2) Four-moment multidimensional entropy problems with explicit solutions where the resulting systems to be solved ranging from $70-310$ equations; (3) Four- to eight-moment of a two-dimensional entropy problem, which solutions correspond to the densities of the two leading EOFs of the wind stress-driven large-scale oceanic model. In this case, we find that the EBE method is more accurate compared to the classical Newton's method, the MATLAB generic solver, and the previously developed BFGS-based method, which was also tested on this problem. (4) Four-moment constrained of up to five-dimensional entropy problems which solutions correspond to multidimensional densities of the components of the solutions of the Kuramoto-Sivashinsky equation. For the higher dimensional cases of this example, the EBE method is superior because it automatically selects a subset of the prescribed moment constraints from which the maximum entropy solution can be estimated within the desired tolerance. This selection feature is particularly important since the moment constrained maximum entropy problems do not necessarily have solutions in general.

Safety assessment for seawall based on constrained maximum entropy projection pursuit model

The safety of seawalls is crucial. A scientific and effective safety evaluation method for seawalls is an important measure to protect seawall safety. Based on the monitoring information, seepage pressure, tide level, and rainfall were taken as the safety assessment indexes of seawalls. The projection pursuit model was improved by taking the primary and secondary relations of the assessment indexes known in the professional field as constraints. The water cycle algorithm was used to solve the projection vector, and the weights of the safety assessment indexes of the seawalls were calculated using the improved projection pursuit model. The safety assessment method of seawalls was established by Fuzzy Theory. A method where the maximum membership degree principle was replaced by the comprehensive safety grade value was put forward to determine the seawall safety grade. The analysis of examples showed that the seawall safety assessment results of the constrained maximum entropy projection pursuit model conformed to the objective reality, and were more scientific and reasonable than the maximum entropy projection pursuit model and the analytical hierarchy process model.

Enhanced capital-asset pricing model for the reconstruction of bipartite financial networks.

Reconstructing patterns of interconnections from partial information is one of the most important issues in the statistical physics of complex networks. A paramount example is provided by financial networks. In fact, the spreading and amplification of financial distress in capital markets are strongly affected by the interconnections among financial institutions. Yet, while the aggregate balance sheets of institutions are publicly disclosed, information on single positions is mostly confidential and, as such, unavailable. Standard approaches to reconstruct the network of financial interconnection produce unrealistically dense topologies, leading to a biased estimation of systemic risk. Moreover, reconstruction techniques are generally designed for monopartite networks of bilateral exposures between financial institutions, thus failing in reproducing bipartite networks of security holdings (e.g., investment portfolios). Here we propose a reconstruction method based on constrained entropy maximization, tailored for bipartite financial networks. Such a procedure enhances the traditional capital-asset pricing model (CAPM) and allows us to reproduce the correct topology of the network. We test this enhanced CAPM (ECAPM) method on a dataset, collected by the European Central Bank, of detailed security holdings of European institutional sectors over a period of six years (2009-2015). Our approach outperforms the traditional CAPM and the recently proposed maximum-entropy CAPM both in reproducing the network topology and in estimating systemic risk due to fire sales spillovers. In general, ECAPM can be applied to the whole class of weighted bipartite networks described by the fitness model.

Learning Bayesian network parameters from small data sets: A further constrained qualitatively maximum a posteriori method

To improve the learning accuracy of the parameters in a Bayesian network from a small data set, domain knowledge is normally incorporated into the learning process as parameter constraints. MAP-based (Maximum a Posteriori) methods that utilize both sample data and domain knowledge have been well studied in the literature. Among all the MAP-based methods, the QMAP (Qualitatively Maximum a Posteriori) method exhibits the best learning performance. However, when the data is insufficient, the estimation given by the QMAP often fails to satisfy all the convex parameter constraints, and this has made the overall QMAP estimation unreliable. To ensure that QMAP estimation does not violate any given parameter constraints and to further improve the learning accuracy, a FC-QMAP (Further Constrained Qualitatively Maximum a Posteriori) algorithm is proposed in this paper. The algorithm regulates QMAP estimation by replacing data estimation with a further constrained estimation via convex optimization. Experiments and theoretical analysis show that the proposed algorithm outperforms most of the existing parameter learning methods (namely, Maximum Likelihood, Constrained Maximum Likelihood, Maximum Entropy, Constrained Maximum Entropy, Maximum a Posteriori, and Qualitatively Maximum a Posteriori).