Maximum Entropy Estimation Research Articles

The Estimation of Distributions and the Minimum Relative Entropy Principle

Estimation of Distribution Algorithms (EDA) have been proposed as an extension of genetic algorithms. In this paper we explain the relationship of EDA to algorithms developed in statistics, artificial intelligence, and statistical physics. The major design issues are discussed within a general interdisciplinary framework. It is shown that maximum entropy approximations play a crucial role. All proposed algorithms try to minimize the Kullback-Leibler divergence KLD between the unknown distribution p(x) and a class q(x) of approximations. However, the Kullback-Leibler divergence is not symmetric. Approximations which suppose that the function to be optimized is additively decomposed (ADF) minimize KLD(q||p), the methods which learn the approximate model from data minimize KLD(p||q). This minimization is identical to maximizing the log-likelihood. In the paper three classes of algorithms are discussed. FDA uses the ADF to compute an approximate factorization of the unknown distribution. The factors are marginal distributions, whose values are computed from samples. The second class is represented by the Bethe-Kikuchi approach which has recently been rediscovered in statistical physics. Here the values of the marginals are computed from a difficult constrained minimization problem. The third class learns the factorization from the data. We analyze our learning algorithm LFDA in detail. It is shown that learning is faced with two problems: first, to detect the important dependencies between the variables, and second, to create an acyclic Bayesian network of bounded clique size.

Functional significance of a deep intronic mutation in the ATM gene and evidence for an alternative exon 28a

Screening for ATM mutations is usually performed using genomic DNA as a template for PCR amplification across exonic regions, with the consequence that deep intronic sequences are not analyzed. Here we report a novel pseudoexon-retaining deep intronic mutation (IVS28-159A>G; g.75117A>G based on GenBank U82828.1) in a patient with ataxia-telangiectasia (A-T), as well as the identification of a previously unrecognized alternative exon in the ATM gene (exon 28a) expressed in lymphoblastoid cell lines (LCL) derived from normal individuals. cDNA analysis using the A-T patient's LCL showed the retention of two aberrant intronic segments of 112 and 190 nt between exons 28 and 29. Minigenes were constructed to determine the functional significance of two genomic changes in the region of aberrant splicing: IVS28-193C>T (g.75083C>T) and IVS28-159A>G, revealing that: 1) the first is a polymorphism; 2) IVS28-159A>G weakens the 5' splice site of the alternative exon 28a and activates a cryptic 5' splice site (ss) 83 nt downstream; and 3) wild-type constructs also retain a 29-nt segment (exon 28a) as part of both the 112- and 190-nt segments. Maximum entropy estimates of ss strengths corroborate the cDNA and minigene findings. Such mutations may prove relevant in planning therapy that targets specific splicing aberrations.

Partially Adaptive Estimation via Maximum Entropy Densities

We propose a partially adaptive estimator based on information theoretic maximum entropy estimates of the error distribution. The maximum entropy (maxent) densities have simple yet flexible functional forms to nest most of the mathematical distributions. Unlike the nonparametric fully adaptive estimators, our parametric estimators do not involve choosing a bandwidth or trimming, and only require estimating a small number of nuisance parameters, which is desirable when the sample size is small. Monte Carlo simulations suggest that the proposed estimators fare well with non-normal error distributions. When the errors are normal, the efficiency loss due to redundant nuisance parameters is negligible as the proposed error densities nest the normal. The proposed partially adaptive estimator compares favorably with existing methods, especially when the sample size is small. We apply the estimator to a bio-pharmaceutical example and a stochastic frontier model.

Bayesian estimation for land surface temperature retrieval: the nuisance of emissivities

An approach to the remote sensing of land surface temperature is developed using the methods of Bayesian inference. The starting point is the maximum entropy estimate for the posterior distribution of radiance in multiple bands. In order to convert this quantity to an estimator for surface temperature and emissivity with Bayes' theorem, it is necessary to obtain the joint prior probability for surface temperature and emissivity, given available prior knowledge. The requirement that any pair of distinct observers be able to relate their descriptions of radiance under arbitrary Lorentz transformations uniquely determines the prior probability. Perhaps surprisingly, surface temperature acts as a scale parameter, while emissivity acts as a location parameter, giving the prior probability P(T,emissivity|K)=const./T dT d(emissivity). Given this result, it is a simple matter to construct estimators for surface temperature and emssivity. Monte Carlo simulations of land surface temeprature retrieval in selected MODIS bands are presented as examples of the utility of the approach.

On entropy estimates of contact forces in static granular assemblies

As an extension of previous work by others, this articles deals with maximum-entropy estimates for the statistics of static contact forces in assemblies of nearly rigid grains. As found in the previous works, the constraint of mean stress leads provisionally to a distribution of contact force that is exponential at large force. This behavior, found in various experiments and numerical simulations, does not depend on special models of force propagation postulated in the contemporary physics literature. Following K. Bagi [Behringer, R. Jenkins, J.T. (Eds.), Powders and Grains, Balkema, 1997, p. 251] consideration is also given to entropy maximization under the constraint of constant mean strain, which leads to a similar exponential tail in the distribution of particle displacements. In contrast to the previous works, it is emphasized that the exact form of the probability density depends on the statistical weight (a priori probability) assigned to elementary volumes in the state-space of contact forces or displacements. This leads to the conclusion that the large-force exponential is a general representation of maximum-entropy statistics arising from global constraints, whereas the state-space measure is dictated by local mechanics. A few examples of state-space measure are considered, and the resulting distributions are compared to previous experiment and simulation. A striking analogy is revealed between the force distribution in a static sphere assembly and the Maxwell–Boltzmann velocity distribution for gases. Based on the methods of statistical thermodynamics, a virtual-thermodynamic formalism is presented for complementary strain energies in granular statics. This involves no direct appeal to the concept of (static) granular temperature favored in certain statistical-physics literature. As a possible test of the general validity of the entropy principle in granular statics, the question is raised as to whether it can describe the heterogeneous two-phase structure found in photoelastic experiments and numerical simulations of granular assemblies.

Estimation under Multicollinearity: Application of Restricted Liu and Maximum Entropy Estimators to the Portland Cement Dataset

A high degree of multicollinearity among the explanatory variables severely impairs estimation of regression coefficients by the Ordinary Least Squares. Several methods have been suggested to ameliorate the deleterious effects of multicollinearity. In this paper we aim at comparing the Restricted Liu estimates of regression coefficients with those obtained by applying the Maximum Entropy Leuven (MEL) family of estimators on the widely analyzed dataset on Portland cement. dataset has been obtained from an experimental investigation of the heat evolved during the setting and hardening of Portland cements of varied composition and the dependence of this heat on the percentage of four compounds in the clinkers from which the cement was produced. The relevance of the relationship between the heat evolved and the chemical processes undergone while setting takes place is best stated in the words of Woods et al.: This property is of interest in the construction of massive works as dams, in which the great thickness severely hinder the outflow of the heat. The consequent rise in temperature while the cement is hardening may result in contractions and cracking when the eventual cooling to the surrounding temperature takes place. Two alternative models have been formulated, the one with an intercept term (non-homogenous) that exhibits a very high degree of multicollinearity and the other with no intercept term (extended homogenous) that characterizes perfect multicollinearity. Our findings suggest that several members of the MEL family of estimators outperform the OLS and the Restricted Liu estimators. The MEL estimators perform well even when perfect multicollinearity is there. A few of them may outperform the Minimum Norm LS (OLS+) estimator. Since the MEL estimators do not seek extra information from the analyst, they are easy to apply. Therefore, one may rely on the MEL estimators for obtaining the coefficients of a linear regression model under the conditions of severe (including perfect) multicollinearity among the explanatory variables.

Multicollinearity and Modular Maximum Entropy Leuven Estimator

A high degree of multicollinearity often has a detrimental effects on the estimation of a linear econometric (regression) model due to an intricate internecine sharing among the estimated regression coefficients (beta). Paris (2001) introduced the Maximum Entropy Leuven (MEL) estimator. It exploits the information available in the sample data more efficiently than the OLS does, and unlike GME estimator, it does not require any additional information to be supplied by the researcher. Paris concludes: under any level of multicollinearity, MEL estimator uniformly dominates the OLS estimator according to the mean squared error criterion. It rivals also the GME estimator without requiring any subjective additional information. Paris used the Euclidean norm to obtain prob(beta). In this paper we obtain prob(beta) using the absolute norm of beta and investigate into its effects on the performance of the Maximum Entropy estimator. We obtain a new estimator of regression coefficients by solving the reformulated problem. This new estimator is not fully a la Paris (2001) and hence we name it as the Modular Maximum Entropy Leuven (MMEL) estimator. Monte Carlo study has been conducted to compare the performance of the MEL estimator (Paris) and the new MMEL estimator. We observe that overall, the performance of MMEL (in terms of mean estimated coefficients as well as the RMS values) is much superior to that of the MEL estimator. It is pertinent to note that obtaining prob(beta) is the most crucial task before the scientist if he chooses to use the maximum entropy estimator (MEL, MMEL or any variant thereof). After all, the mathematics of probability suggests us that given a sample description space S, probability is a function which assigns a non-negative real number to every event A, denoted by P(A) and it is called the probability of the event A. The probability function is defined on a Borel field of events conformal to the axioms of positiveness, certainty and union. Under these axioms, there could be several different rules of assignment, ranging from subjective judgement backed up by a rational belief (JM Keynes) to counting the number of success in the repeated trials. In our study MEL does this assignment in the one way and the MMEL does that in the other way. There could be many more (possibly better) rules of assignment of probability. Thus, the subjective (or exogenous) element that was explicit in Golan et al. reappears in the MEL family of estimators, although in another garb.

A Neural Networks–Based Fusion Technique to Estimate Half-Hourly Rainfall Estimates at 0.1° Resolution from Satellite Passive Microwave and Infrared Data

Abstract The purpose of this paper is to evaluate a new operational procedure to produce half-hourly rainfall estimates at 0.1° spatial resolution. Rainfall is estimated using a neural networks (NN)–based approach utilizing passive microwave (PMW) and infrared satellite measurements. Several neural networks are tested, from multilayer perceptron to adaptative resonance theory architectures. The NN analytical selection process is explained. Half- hourly rain gauge data over Andalusia, Spain, are used for validation purposes. Several interpolation procedures are tested to transform point to areal measurements, including the maximum entropy estimation method. Rainfall estimations are also compared with Geostationary Operational Environmental Satellite precipitation index and histogram-matching results. Half-hourly rainfall estimates give ∼0.6 correlations with PMW data (∼0.2 with gauge), and average correlations of up to 0.7 and 0.6 are obtained for 0.5° and 0.1° monthly accumulated estimates, respectively.

Information-Theoretic Distribution Tests with Application to Symmetry and Normality

We derive distribution free tests based on the Maximum Entropy densities to test the null hypotheses of symmetry and normality. The proposed tests are derived from maximizing the difierential entropy subject to moment constraints. By exploiting the equivalence between Maximum Entropy and Maximum Likelihood estimates of the exponential families, we can use the conventional Likelihood Ratio, Wald and Lagrange Multiplier testing principles in the maximum entropy framework. Monte Carlo evidence suggests that they have desirable small sample properties, when compared with the standard parametric tests used in the literature, such as the standardized skewness coe‐cient test for symmetry or the Jarque and Bera (1980) test for normality. We apply the proposed symmetry tests to test the nominal wage rigidity hypothesis of wage determination process.

Farm‐specific Adjustment Costs in Dutch Pig Farming

This paper develops a dynamic model of investment under rational expectations, assuming farm‐specific production technologies and adjustment cost structures. The model distinguishes regimes of negative, zero and positive investments and maintains that it is optimal for a farmer not to invest for a range of shadow prices, depending on thresholds for positive and negative investments. The model is applied to a rotating sample of Dutch pig farms over the period 1980–1996. Farm‐specific parameters of the adjustment cost function and production technology are obtained using Generalised Maximum Entropy estimation. Cluster analysis using the farm‐specific adjustment cost parameters indicates that five groups of farms with distinct adjustment cost structures can be identified. A tobit regression analysis is used to explain the impact of different socio‐economic factors on the size of the threshold between positive and negative investments.

The Bayesian Hierarchical Classifier (BHC) and Its Application to Short Vegetation Using Multifrequency Polarimetric SAR

Given an image of a scene comprised of a number of distinct terrain classes, the optimum Bayesian classifier (OBC) provides the highest possible classification accuracy of the imaged scene, provided we have a priori knowledge of the probability density function (pdf) of the sensor's output for each terrain class. If the imaging sensor consists of multiple channels, application of OBC requires knowledge of the joint pdf of the observations made by all the channels. In practice, the volume of data needed in order to generate an accurate multidimensional pdf far exceeds the size of available datasets. The data-size requirement may be relaxed by assuming the pdfs to be Gaussian in form, but such an assumption leads to suboptimum classification performance. This paper addresses the data size issue by (1) taking advantage of the maximum-entropy density estimation (MEDE) technique introduced in a companion paper and (2) using marginal pdfs in a hierarchical approach. Using multidate synthetic aperture radar observations, it was shown that the Bayesian hierarchical classifier introduced in this paper can classify short vegetation classes with an accuracy of 93%, without retraining, compared with an accuracy of 84% for the maximum-likelihood estimator (with Gaussian assumption) and only 74% with ISODATA.

A Computationally Efficient Multivariate Maximum-Entropy Density Estimation (MEDE) Technique

Density estimation is the process of taking a set of multivariate data and finding an estimate for the probability density function (pdf) that produced it. One approach for obtaining an accurate estimate of the true density f(x) is to use the polynomial-moment method with Boltzmann-Shannon entropy. Although rigorous mathematically, the method is difficult to implement in practice because the solution involves a large set of simultaneous nonlinear integral equations, one for each moment or joint moment constraint. Solutions available in the literature are generally not easily applicable to multivariate data, nor computationally efficient. In this paper, we take the functional form that was developed in this problem and apply pointwise estimates of the pdf as constraints. These pointwise estimates are transformed into basis coefficients for a set of Legendre polynomials. The procedure is mathematically similar to the multidimensional Fourier transform, although with different basis functions. We apply this technique, called the maximum-entropy density estimation (MEDE) technique, to a series of multivariate datasets.

Maximun entropy Leuven estimators and multicollinearity

A novel class of estimators, called maximum entropy Leuven (MEL) estimators, is pre-sented and its performance is illustrated by Monte Carlo experiments. These estimators are inspired by the theory of light. The MEL estimators are consistent and asymptotically normal. They rival the generalized maximum entropy estimator (GME). Based on the mean squared error criterion, the MEL estimators outperform the ordinary least-squares estimator in the presence of multicollinerity.

Nonclassical splicing mutations in the coding and noncoding regions of the ATM Gene: maximum entropy estimates of splice junction strengths.

Ataxia-telangiectasia (A-T) is an autosomal recessive neurological disorder caused by mutations in the ATM gene. Classical splicing mutations (type I) delete entire exons during pre-mRNA splicing. In this report, we describe nine examples of nonclassical splicing mutations in 12 A-T patients and compare cDNA changes to estimates of splice junction strengths based on maximum entropy modeling. These mutations fall into three categories: pseudoexon insertions (type II), single nucleotide changes within the exon (type III), and intronic changes that disrupt the conserved 3' splice sequence and lead to partial exon deletion (type IV). Four patients with a previously reported type II (pseudoexon) mutation all shared a common founder haplotype. Three patients with apparent missense or silent mutations actually had type III aberrant splicing and partial deletion of an exon. Five patients had type IV mutations that could have been misinterpreted as classical splicing mutations. Instead, their mutations disrupt a splice site and use another AG splice site located nearby within the exon; they lead to partial deletions at the beginning of exons. These nonclassical splicing mutations create frameshifts that result in premature termination codons. Without screening cDNA or using accurate models of splice site strength, the consequences of these genomic mutations cannot be reliably predicted. This may lead to further misinterpretation of genotype-phenotype correlations and may subsequently impact upon gene-based therapeutic approaches.

Symmetric Positive Equilibrium Problem: A Framework for Rationalizing Economic Behavior with Limited Information: Comment

In a recent contribution to this journal, Paris suggests a framework which extends positive mathematical programming (PMP)—a widely used calibration methodology for agricultural supply models—to a symmetric positive equilibrium problem (SPEP). He stresses three main contributions: (1) The PMP methodology is modified to incorporate more than one observation on production programs; (2) A solution to the “self-selection problem” with respect to the choice of crops produced by each farm is provided; (3) “Limiting inputs” are no longer considered fixed quantities as in PMP. We address several conceptual concerns with respect to the SPEP methodology and the presented application. We consider these to be substantial enough to question Paris’ claim to present “ ... a general framework of analysis that is capable of reproducing economic behavior in a consistent way ... ” (p. 1049). Our discussion is structured along Paris’ presentation: The next three sections represent the core of the comment and deal with the methodology itself. They refer to the three phases of SPEP: (i) recovery of unknown variable marginal costs and shadow prices of limited resources, (ii) use of these results to specify data constraints and parameter supports for generalized maximum entropy (GME) estimation of a cost function, and (iii) definition of a simulation model. Finally, concluding remarks are made regarding the application of SPEP to an analysis of the EU Common Agricultural Policy (CAP) based on Italian farm data. Throughout the comment we use the same

Consistent Estimation Of State-Price Distribution Functions From Noisy Price Data

For the accurate pricing of contingent securities, it is often not necessary to have a consistent estimate of the state-price density: It is enough to have a good estimate of the state-price distribution function. It is shown in this paper that the inverse problem of finding the state-price distribution function from the prices of put options is well-posed (as opposed to the problem of finding the state-price density, which is ill-posed), and that the constrained least squares method provides a consistent estimate of the state-price distribution function. It is also suggested that small sample properties of the least squares estimator can be improved by a modification that takes into account the deviation from a prior state-price distribution. The modified method is called the relaxed maximum entropy by analogy with the maximum entropy method by Buchen and Kelly. By generalizing a Csiszar theorem about the existence of relative entropy projections, it is proved that the relaxed maximum entropy estimator exists. The performance of the estimation methods is evaluated by Monte-Carlo simulations. Both methods are found to be fast and accurate, with the constrained least squares having advantage in speed, and the relaxed maximum entropy in accuracy of pricing.

Generalized Maximum Entropy Estimation of Dynamic Programming Models with Sample Selection Bias

In this study, an alternative estimation strategy is suggested, which requires the solution of a dynamic programming problem, expressed in term of conditional choice value functions. The problem of defining the Euler equation for the corner solution case is overcome by introducing an additional constraint on the conditional choice value functions, which provides a correction for the sample selection bias. The problem of estimating the parameters of the resulting objective function and of the state equation is solved using numerical methods in a linear-quadratic inverse control problem which consists of an objective function together with the set of introduced constraints. First step estimates are computed by generalized maximum entropy estimation techniques.

A Small-Sample Estimator for the Sample-Selection Model

A semiparametric estimator for evaluating the parameters of data generated under a sample selection process is developed. This estimator is based on the generalized maximum entropy estimator and performs well for small and ill-posed samples. Theoretical and sampling comparisons with parametric and semiparametric estimators are given. This method and standard ones are applied to three small-sample empirical applications of the wage-participation model for female teenage heads of households, immigrants, and Native Americans.

Estimation of constrained optimisation models for agricultural supply analysis based on generalised maximum entropy

The paper introduces a general methodological approach for the estimation of constrained optimisation models in agricultural supply analysis. It is based on optimality conditions of the desired programming model and shows a conceptual advantage compared with Positive Mathematical Programming in the context of well-posed estimation problems. Moreover, it closes the empirical and methodological gap between programming models and duality-based models with explicit allocation of fixed factors. Monte Carlo simulations are performed with a maximum entropy estimator to evaluate the functionality of the approach as well as the impact of empirically relevant prior information with small samples.

Maximum entropy method for solving the collisional Vlasov equation

By means of the maximum entropy method the reduced Vlasov–Fokker–Planck equation has been solved explicitly. The behaviour of the approximate maximum entropy distribution functions are shown graphically, and compared with that obtained by using Lie group method. The results reported in this article provide further evidence of the usefulness of both maximum entropy and Lie group methods for obtaining time dependent solution in compact form.