Maximum Entropy Problem Research Articles

Numerical simulation of rarefied supersonic flows using a fourth-order maximum-entropy moment method with interpolative closure

Maximum-entropy moment methods allow for the modeling of gases from the continuum regime to strongly rarefied conditions. The development of approximated solutions to the entropy maximization problem has made these methods computationally affordable. In this work, we apply a fourth-order maximum-entropy moment method to the study of supersonic rarefied flows. For such conditions, we compare the maximum-entropy solutions to results obtained from the kinetic theory of gases at different Knudsen numbers. The analysis is performed for both a simplified model of a gas with a single translational degree of freedom (5-moment system) and for a typical gas with three degrees of freedom (14-moment system). The maximum-entropy method is applied to the study of the Sod shock-tube problem at various rarefaction levels, and to the simulation of two-dimensional low-collisional crossed supersonic jets. We show that, in rarefied supersonic conditions, it is important to employ accurate estimates of the wave speeds. Since analytical expressions are not presently available, we propose an approximation, valid for the 14-moment system. In these conditions, the solution of the maximum-entropy system is shown to realize large degrees of non-equilibrium and to approach the Junk subspace, yet provides a good overall accuracy and agreement with the kinetic theory. Numerical procedures for reaching second-order accurate discretizations are discussed, as well as the implementation of the 14-moment solver on Graphics Processing Units (GPUs).

An Irregular Tiled Array Technique for Microwave Wireless Power Transmission

An irregular tiled array technique with the maximization of beam collection efficiency (BCE) for microwave wireless power transmission (WPT) is addressed in this paper, where the radiating elements are irregularly partitioned into multiple subarrays for reducing the number of radio frequency (RF) chain. The system architectures of WPT based on irregular tiled arrays are analyzed for the exhibition of its low-cost advantages. To achieve a good trade-off between the manufacturing cost and beampattern, a mixed-integer optimization problem (MIOP) accounting for the maximization of BCE satisfying the subarray tiling configuration, scanning direction and peak sidelobe level (SLL) constraints is established. By equivalently reformulating the beampattern, the MIOP in terms of subarray tiling configuration and subarray excitations is decoupled, and a two-stage solution method can be implemented. Firstly, resort to ‘generalized BCE’, the binary quadratic optimization problem (BQOP) for maximizing the BCE at considered scanning angles while guaranteeing the full aperture coverage is reduced to a binary second-order cone optimization problem. After the subarray tiling configuration is determined, the non-convex Rayleigh entropy maximization problem at a given scanning angle with respect to subarray complex excitations while satisfying peak SLL constraint is globally solved by using iterative convex approximation algorithm. Owing to the high efficiency of convex optimization, the BCE is able to monotonically increase to a stable value. Several numerical results with different array size and subarray shapes are provided to assess the effectiveness of the proposed method by comparing to the competitive counterparts in the open literatures.

Risk-adjusted geometric diversified portfolios

In this paper, exploiting a geometric argument, a novel and intuitive approach to portfolio diversification is proposed. The risk-adjusted geometric diversified portfolio is obtained as the point that is equally distant, for a given distance, from the vertices of the simplex, as they represent the single asset portfolios, the worst portfolios in terms of diversification. The definition of risk-adjusted distance as a special case of weighted Euclidean distance permits to introduce the information on the risks of the assets composing the portfolio in a very general way. The closed form solution for the allocation problem is provided and interesting theoretical properties are proved. Further, a direct comparison with Rao’s Quadratic Entropy maximization problem is outlined, thus yielding a different perspective to the use of such entropy as a diversification measure. Finally, the effectiveness of our proposal is emphasized through a comprehensive empirical out-of-sample exercise on real financial data.

Entropy Maximization for Partially Observable Markov Decision Processes

We study the problem of synthesizing a controller that maximizes the entropy of a partially observable Markov decision process (POMDP) subject to a constraint on the expected total reward. Such a controller minimizes the predictability of an agent’s trajectories to an outside observer while guaranteeing the completion of a task expressed by a reward function. Focusing on finite-state controllers (FSCs) with deterministic memory transitions, we show that the maximum entropy of a POMDP is lower bounded by the maximum entropy of the parameteric Markov chain (pMC) induced by such FSCs. This relationship allows us to recast the entropy maximization problem as a so-called parameter synthesis problem for the induced pMC. We then present an algorithm to synthesize an FSC that locally maximizes the entropy of a POMDP over FSCs with the same number of memory states. In a numerical example, we highlight the benefit of using an entropy-maximizing FSC compared with an FSC that simply finds a feasible policy for accomplishing a task.

Integer programming topology optimization for subsonic compressible flows with geometry trimming

The incompressibility hypothesis is successfully applied in a broad range of engineering problems simulated via computational fluid dynamics (CFD) codes. However, depending on the Mach number, the effects of compressibility are no longer negligible. Recently, the first papers about topology optimization considering compressible flows were published. All of them use the continuous design variables method which has some known drawbacks. The present work proposes topology optimization considering the compressible Navier-Stokes equations and discrete variables with a step of geometry trimming. This approach constrains the design variables to a binary set and solves the problem by using sequential integer linear programming, being a feasible alternative to classic density-based methods. Additionally, the solid regions are trimmed out from the physics analysis. A material model is inserted in the governing equations in such a way that, it is possible to calculate the sensitivity and, at the same time, guarantee the adiabatic condition. In this work, it is observed that the minimization of entropy generation as the objective function tends to converge to an undesired trivial solution. Alternatively, this work proposes the maximization of mass flow rate as the objective function to, indirectly, minimize the entropy. Furthermore, it is proposed the use of a new set of inlet boundary conditions. Three optimization cases are solved: the double-pipe, the pipe-bend, and an entropy maximization problem. The optimized solutions for the pipe-bend and double-pipe problems present geometries where the constraints are satisfied with the minimum entropy generation. Using the double-pipe example, it is conducted a study to understand the impacts of TOBS-GT parameters in the solutions. The results evidenced that, in some conditions, the implementation is prone to fall in a local minimum. In the third example, entropy maximization subjected to volume constraint is successfully solved. Moreover, a minimization case is also solved to aid in the interpretation of the results.

New numerical methods for calculating statistical equilibria of two-dimensional turbulent flows, strictly based on the Miller–Robert–Sommeria theory

New numerical methods are proposed for the mixing entropy maximization problem in the context of Miller–Robert–Sommeria’s (MRS) statistical mechanics theory of two-dimensional turbulence, particularly in the case of spherical geometry. Two of the methods are for the canonical problem; the other is for the microcanonical problem. The methods are based on the original MRS theory and thus take into account all Casimir invariants. Compared to the methods proposed in previous studies, our new methods make it easier to detect multiple statistical equilibria and to search for solutions with broken zonal symmetry. The methods are applied to a zonally symmetric initial vorticity distribution which is barotropically unstable. Two statistical equilibria are obtained, one of which has a wave-like structure with zonal wavenumber 1, and the other has a wave-like structure with zonal wavenumber 2. While the former is the maximum point of the mixing entropy, the wavenumber 2 structure of the latter is nearly the same as the structure that appears in the end state of the time integration of the vorticity equation. The new methods allow for efficient computation of statistical equilibria for initial vorticity distributions consisting of many levels of vorticity patches without losing information about all the conserved quantities. This means that the statistical equilibria can be obtained from an arbitrary initial vorticity distribution, which allows for the application of statistical mechanics to interpret a wide variety of flow patterns appearing in geophysical fluids.

Asymptotic method for entropic multiple relaxation time model in lattice Boltzmann method.

To improve the numerical stability of the lattice Boltzmann method, Karlin etal. [Phys. Rev. E 90, 031302(R) (2014)10.1103/PhysRevE.90.031302] proposed the entropic multiple relaxation time (EMRT) collision model. The idea behind EMRT is to construct an optimal postcollision state by maximizing its local entropy value. The critical step of the EMRT model is to solve the entropy maximization problem under certain constraints, which is often computationally expensive and even not feasible. In this paper, we propose to employ perturbation theory and obtain an asymptotic solution to the maximum entropy state. With mathematical analysis of particular cases under relaxed constraints, we obtain the unperturbed form of the original problem and derive the asymptotic solution. We show that the asymptotic solution well approximates the optimal states; thus, our approach provides an efficient way to solve the constrained maximum entropy problem in the EMRT model. Also, we use the same idea of the EMRT model for the initial condition of the distribution function and propose to leave the entropy function to determine the missing information at the initial nodes. Finally, we numerically verify that the simulation results of the EMRT model obtained via the perturbation theory agree well with the exact solution to the Taylor-Green vortex problem. Furthermore, we also demonstrate that the EMRT model exhibits excellent stability performance for under-resolved simulations in the doubly periodic shear layer flow problem.

Option pricing with maximum entropy densities: The inclusion of higher‐order moments

AbstractEntropy pricing applies notions of information theory to derive the theoretical value of options. This paper employs the maximum entropy (ME) formulation of option pricing, given risk‐neutral moment constraints computed directly from the observed prices. First, higher‐order moments are used to generate option prices. Then a generalization of Shannon entropy, known as Renyi entropy, is studied to account for extreme events. This ME problem provides a class of heavy‐tailed distributions. Examples and Monte Carlo simulations are provided to examine the effects of moment constraints on option prices. The call option values are then constructed using daily Standard and Poor's 500 index options. The findings suggest that entropy pricing with higher‐order moment constraints provides higher forecasting accuracy.

Technical Note—Masking Anstreicher’s linx Bound for Improved Entropy Bounds

A fundamental NP-hard combinatorial-optimization in the area of statistical designs is the maximum-entropy sampling problem (MESP), which seeks to maximize Shannon's “differential entropy” over all subsets of a prespecified cardinality from a set of n Gaussian random variables. This problem has applications in many areas, such as the redesign of environmental-monitoring networks. Most algorithms for exact solution of MESP are branch-and-bound based, and one of the best upper bounds is based on Anstrecher's recent concave “linx relaxation” of differential entropy. A key paradigm for improving bounds is by “masking” the covariance of the random variables with a correlation matrix. The main result establishes that in the best case, the linx bound can be improved by an amount that is at least linear in n by masking. These and other recent results on the hot topic of MESP are leading to practical algorithms for exact solution of meaningful design problems in applied areas such as environmental statistics.

Maximum Entropy Technique and Regularization Functional for Determining the Pharmacokinetic Parameters in DCE-MRI

This paper aims to solve the arterial input function (AIF) determination in dynamic contrast-enhanced MRI (DCE-MRI), an important linear ill-posed inverse problem, using the maximum entropy technique (MET) and regularization functionals. In addition, estimating the pharmacokinetic parameters from a DCE-MR image investigations is an urgent need to obtain the precise information about the AIF–the concentration of the contrast agent on the left ventricular blood pool measured over time. For this reason, the main idea is to show how to find a unique solution of linear system of equations generally in the form of y=Ax+b, named an ill-conditioned linear system of equations after discretization of the integral equations, which appear in different tomographic image restoration and reconstruction issues. Here, a new algorithm is described to estimate an appropriate probability distribution function for AIF according to the MET and regularization functionals for the contrast agent concentration when applying Bayesian estimation approach to estimate two different pharmacokinetic parameters. Moreover, by using the proposed approach when analyzing simulated and real datasets of the breast tumors according to pharmacokinetic factors, it indicates that using Bayesian inference—that infer the uncertainties of the computed solutions, and specific knowledge of the noise and errors—combined with the regularization functional of the maximum entropy problem, improved the convergence behavior and led to more consistent morphological and functional statistics and results. Finally, in comparison to the proposed exponential distribution based on MET and Newton’s method, or Weibull distribution via the MET and teaching–learning-based optimization (MET/TLBO) in the previous studies, the family of Gamma and Erlang distributions estimated by the new algorithm are more appropriate and robust AIFs.

Generalized maximum entropy based identification of graphical ARMA models

This paper focuses on the joint estimation of parameters and topologies of multivariate graphical autoregressive moving-average (ARMA) processes. Since the graphical structure of a model can be characterized by the sparsity pattern of its inverse spectral density matrix, a generalized maximum entropy optimization model is derived by imposing a sparse regularization on the inverse spectrum from which the parameters of the AR part and the graph topology can be simultaneously estimated. Then, the whole graphical ARMA model is identified by alternatingly estimating the graphical AR part by solving the regularized maximum entropy problem and the MA part by applying the moment estimation method. The weight to the sparsity regularization term is chosen according to the information theoretic model selection criterion, and simulation examples are provided to illustrate the effectiveness of the proposed identification approach.

On reliability analysis method through rotational sparse grid nodes

This study aims to develop a rotational sparse grid (R-SPGR) method for statistical moment evaluation and structural reliability analysis with enhanced accuracy and efficiency. The optimal rotational angle in the proposed R-SPGR method is determined by an optimization approach, in which the objective function is constructed in terms of discrepancy between the marginal moments of each input uncertain variables calculated from these rotational sparse grid nodes and their exact values. The R-SPGR nodes allow capturing more information of exact probability distribution for the system response, which is considered critically important to the calculation of statistical moments accurately. Following this, the probability density function (PDF) and failure probability of system response can be efficiently determined by using the saddlepoint approximation technique. To demonstrate the effectiveness of the proposed method, four benchmark examples, one structural analysis example and one practical example of industrial robot are provided here, in which the results obtained from the proposed R-SPGR method are compared with those calculated from the conventional sparse grid (SPGR) method, the maximum entropy problem with fractional moments (ME-FM) method and Monte Carlo simulation (MCS) method. From these illustrative examples involving a wide range of complexity, it is demonstrated that the proposed R-SPGR method has fairly high accuracy and efficiency for structural reliability analysis.

Challenge codes for physically unclonable functions with Gaussian delays: A maximum entropy problem

Motivated by a security application on physically unclonable functions, we evaluate the probability distributions and Renyi entropies of signs of scalar products of i.i.d. Gaussian random variables against binary codewords in \begin{document}$ \{\pm1\}^n $\end{document} . The exact distributions are determined for small values of \begin{document}$ n $\end{document} and upper bounds are provided by linking this problem to the study of Boolean threshold functions. Finally, Monte-Carlo simulations are used to approximate entropies up to \begin{document}$ n = 10 $\end{document} .

Maximal entropy solutions under prescribed mass and energy

We consider a non-local elliptic equation with exponential nonlinearity, closely related to the mean field Liouville equation. The motivation for this equation is a variational entropy maximization problem under prescribed mass and energy. We provide an unconditional existence proof in case of electrostatic (repulsive) self-interaction, and conditional existence and uniqueness in dimension two in the case of gravitational (attractive) self-interaction.

Hybrid master equation for jump-diffusion approximation of biomolecular reaction networks

Cellular reactions have a multi-scale nature in the sense that the abundance of molecular species and the magnitude of reaction rates can vary across orders of magnitude. This diversity naturally leads to hybrid models that combine continuous and discrete modeling regimes. In order to capture this multi-scale nature, we proposed jump-diffusion approximations in a previous study. The key idea was to partition reactions into fast and slow groups, and then to combine a Markov jump updating scheme for the slow group with a diffusion (Langevin) updating scheme for the fast group. In this study we show that the joint probability density function of the jump-diffusion approximation over the reaction counting process satisfies a hybrid master equation that combines terms from the chemical master equation and from the Fokker–Planck equation. Inspired by the method of conditional moments, we propose a efficient method to solve this master equation using the moments of reaction counters of the fast reactions given the reaction counters of the slow reactions. For each time point of interest, we then solve a set of maximum entropy problems in order to recover the conditional probability density from its moments. This finally allows us to reconstruct the complete joint probability density over all reaction counters and hence obtain an approximate solution of the hybrid master equation. Finally, we show the accuracy of the method applied to a simple multi-scale conversion process.

Relaxed Schrödinger bridges and robust network routing.

We seek network routing towards a desired final distribution that can mediate possible random link failures. In other words, we seek a routing plan that utilizes alternative routes so as to be relatively robust to link failures. To this end, we provide a mathematical formulation of a relaxed transport problem where the final distribution only needs to be close to the desired one. The problem is cast as a maximum entropy problem for probability distributions on paths with an added terminal cost. The entropic regularizing penalty aims at distributing the choice of paths amongst possible alternatives. We prove that the unique solution may be obtained by solving a generalized Schrödinger system of equations. An iterative algorithm to compute the solution is provided. Each iteration of the algorithm contracts the distance (in the Hilbert metric) to the optimal solution by more than 1/2, leading to extremely fast convergence.

A Moment Approach to Positioning Accuracy Reliability Analysis for Industrial Robots

The uncertain variables of the link dimensions and joint clearances, whose deviation is caused by manufacturing and assembling errors, have a considerable influence on the positioning accuracy of industrial robots. Understanding how these uncertain variables affect the positioning accuracy of industrial robots is very important to select appropriate parameters during design process. In this paper, the positioning accuracy reliability of industrial robots is analyzed considering the influence of uncertain variables. First, the kinematic models of industrial robots are established based on the Denavit–Hartenberg method, in which the link lengths and joint rotation angles are treated as uncertain variables. Second, the Sobol’ method is used to analyze the sensitivity of uncertain variables for the positioning accuracy of industrial robots, by which the sensitive variables are determined to perform the reliability analysis. Finally, in view of the sensitive variables, the first-four order moments and probability density function of the manipulator's positioning point are assessed by the point estimation method (PEM) in three examples. The Monte Carlo simulation method, the maximum entropy problem with fractional order moments (maximum entropy problem with fractional order moments method (ME-FM) method), and the experimental method are also performed as comparative methods. All the results demonstrate that the proposed PEM has a higher accuracy and efficiency to assess the positioning accuracy reliability of industrial robots.

A Direct Link between Rényi-Tsallis Entropy and Hölder's Inequality-Yet Another Proof of Rényi-Tsallis Entropy Maximization.

The well-known Hölder’s inequality has been recently utilized as an essential tool for solving several optimization problems. However, such an essential role of Hölder’s inequality does not seem to have been reported in the context of generalized entropy, including Rényi–Tsallis entropy. Here, we identify a direct link between Rényi–Tsallis entropy and Hölder’s inequality. Specifically, we demonstrate yet another elegant proof of the Rényi–Tsallis entropy maximization problem. Especially for the Tsallis entropy maximization problem, only with the equality condition of Hölder’s inequality is the q-Gaussian distribution uniquely specified and also proved to be optimal.

Efficient Ensemble Refinement by Reweighting.

Ensemble refinement produces structural ensembles of flexible and dynamic biomolecules by integrating experimental data and molecular simulations. Here we present two efficient numerical methods to solve the computationally challenging maximum-entropy problem arising from a Bayesian formulation of ensemble refinement. Recasting the resulting constrained weight optimization problem into an unconstrained form enables the use of gradient-based algorithms. In two complementary formulations that differ in their dimensionality, we optimize either the log-weights directly or the generalized forces appearing in the explicit analytical form of the solution. We first demonstrate the robustness, accuracy, and efficiency of the two methods using synthetic data. We then use NMR J-couplings to reweight an all-atom molecular dynamics simulation ensemble of the disordered peptide Ala-5 simulated with the AMBER99SB*-ildn-q force field. After reweighting, we find a consistent increase in the population of the polyproline-II conformations and a decrease of α-helical-like conformations. Ensemble refinement makes it possible to infer detailed structural models for biomolecules exhibiting significant dynamics, such as intrinsically disordered proteins, by combining input from experiment and simulation in a balanced manner.

Convenient Interface to Inverse Ising (ConIII): A Python 3 Package for Solving Ising-Type Maximum Entropy Models

ConIII (pronounced CON-ee) is an open-source Python project providing a simple interface to solving the pairwise and higher order Ising model and a base for extension to other maximum entropy models. We describe the maximum entropy problem and give an overview of the algorithms that are implemented as part of ConIII ( https://github.com/eltrompetero/coniii ) including Monte Carlo histogram, pseudolikelihood, minimum probability flow, a regularized mean field method, and a cluster expansion method. Our goal is to make a variety of maximum entropy techniques accessible to those unfamiliar with the techniques and accelerate workflow for users. Funding Statement: EDL was supported by an NSF Graduate Fellowship under grant no. DGE-1650441. This research was supported in part by a congressional research grant provided by the Dirksen Congressional Center.