Maximum Entropy Estimation Research Articles

Feature Forest Models for Probabilistic HPSG Parsing

Probabilistic modeling of lexicalized grammars is difficult because these grammars exploit complicated data structures, such as typed feature structures. This prevents us from applying common methods of probabilistic modeling in which a complete structure is divided into sub-structures under the assumption of statistical independence among sub-structures. For example, part-of-speech tagging of a sentence is decomposed into tagging of each word, and CFG parsing is split into applications of CFG rules. These methods have relied on the structure of the target problem, namely lattices or trees, and cannot be applied to graph structures including typed feature structures. This article proposes the feature forest model as a solution to the problem of probabilistic modeling of complex data structures including typed feature structures. The feature forest model provides a method for probabilistic modeling without the independence assumption when probabilistic events are represented with feature forests. Feature forests are generic data structures that represent ambiguous trees in a packed forest structure. Feature forest models are maximum entropy models defined over feature forests. A dynamic programming algorithm is proposed for maximum entropy estimation without unpacking feature forests. Thus probabilistic modeling of any data structures is possible when they are represented by feature forests. This article also describes methods for representing HPSG syntactic structures and predicate-argument structures with feature forests. Hence, we describe a complete strategy for developing probabilistic models for HPSG parsing. The effectiveness of the proposed methods is empirically evaluated through parsing experiments on the Penn Treebank, and the promise of applicability to parsing of real-world sentences is discussed.

Anomaly detection in hyperspectral imagery based on maximum entropy and nonparametric estimation

This paper presents several maximum entropy and nonparametric estimation detectors (MENEDs) which belong to two categories to detect anomaly targets in hyperspectral imagery. First, probability density of target is estimated using Principle of Maximum Entropy according to the low-probability occurrence of target, which simplifies the generalize likelihood ratio test to merely testing background likelihood. Then considering the high complexity of hyperspectral data, in conjunction with the low-probability occurrence of target, sample-depended multimode model (SDMM) is presented to obtain the probability density of the background. Finally, the estimated probability density of the background is tested to detect targets. The proposed MENEDs greatly depend on hyperspectral data sample, rather than the statistical model, to extract the statistical information, which alleviates statistical model discrepancy and has explicit physical mechanism on detection. Experimental results on visible/near-infrared hyperspectral imagery of type I Operational Modular Imaging Spectrometer (OMIS-I) demonstrate that MENEDs perform better than several known detectors, including RX detector (RXD), normalized RXD (NRXD), modified RXD (MRXD), correlation matrix based NRXD (CNRXD), correlation matrix based MRXD (CMRXD), unified target detector (UTD) and low probability detection (LPD).

Maximum entropy solution to ill-posed inverse problems with approximately known operator

We consider the linear inverse problem of reconstructing an unknown finite measure μ from a noisy observation of a generalized moment of μ defined as the integral of a continuous and bounded operator Φ with respect to μ. Motivated by various applications, we focus on the case where the operator Φ is unknown; instead, only an approximation Φ m to it is available. An approximate maximum entropy solution to the inverse problem is introduced in the form of a minimizer of a convex functional subject to a sequence of convex constraints. Under several assumptions on the convex functional, the convergence of the approximate solution is established.

Comparative statics of the generalized maximum entropy estimator of the general linear model

The generalized maximum entropy method of information recovery requires that an analyst provides prior information in the form of finite bounds on the permissible values of the regression coefficients and error values for its implementation. Using a new development in the method of comparative statics, the sensitivity of the resulting coefficient and error estimates to the prior information is investigated. A negative semidefinite matrix reminiscent of the Slutsky-matrix of neoclassical microeconomic theory is shown to characterize the said sensitivity, and an upper bound for the rank of the matrix is derived.

Note on the article: Maximum entropy analysis of the M [ x] /M/1 queueing system with multiple vacations and server breakdowns

Wang et al. [Wang, K. H., Chan, M. C., & Ke, J. C. (2007). Maximum entropy analysis of the M [ x] /M/1 queueing system with multiple vacations and server breakdowns. Computers & Industrial Engineering, 52, 192–202] elaborate on an interesting approach to estimate the equilibrium distribution for the number of customers in the M [ x] /M/1 queueing model with multiple vacations and server breakdowns. Their approach consists of maximizing an entropy function subject to constraints, where the constraints are formed by some known exact results. By a comparison between the exact expression for the expected delay time and an approximate expected delay time based on the maximum entropy estimate, they argue that their maximum entropy estimate is sufficiently accurate for practical purposes. In this note, we show that their maximum entropy estimate is easily rejected by simulation. We propose a minor modification of their maximum entropy method that significantly improves the quality of the estimate.

An extension to Sun's decomposition methodology: The Path Based approach

This paper proposes the use of the Path Based approach as a general and flexible method to carry out exact decompositions when some additional information for intermediate periods is available. This additional information is used in a Maximum Entropy estimation procedure to arrive at parameter estimates that determine the temporal paths followed by the decomposition factors. An empirical illustration, decomposing changes in greenhouse gas emissions in the EU 15 from 1990 to 2002 by both the Path Based and Sun's periodwise method, is carried out. The outcome shows that the Path Based approach leads to more accurate results than Sun's periodwise method when both are compared with a complete time series decomposition.

On estimation error using maximum entropy density estimates

PurposeIn many problems involving decision‐making under uncertainty, the underlying probability model is unknown but partial information is available. In some approaches to this problem, the available prior information is used to define an appropriate probability model for the system uncertainty through a probability density function. When the prior information is available as a finite sequence of moments of the unknown probability density function (PDF) defining the appropriate probability model for the uncertain system, the maximum entropy (ME) method derives a PDF from an exponential family to define an approximate model. This paper, aims to investigate some optimality properties of the ME estimates.Design/methodology/approachFor n>m, when the exact model can be best approximated by one of an infinite number of unknown PDFs from an n parameter exponential family. The upper bound of the divergence distance between any PDF from this family and the m parameter exponential family PDF defined by the ME method are derived. A measure of adequacy of the model defined by ME method is thus provided.FindingsThese results may be used to establish confidence intervals on the estimate of a function of the random variable when the ME approach is employed. Additionally, it is shown that when working with large samples of independent observations, a probability density function (PDF) can be defined from an exponential family to model the uncertainty of the underlying system with measurable accuracy. Finally, a relationship with maximum likelihood estimation for this case is established.Practical implicationsThe so‐called known moments problem addressed in this paper has a variety of applications in learning, blind equalization and neural networks.Originality/valueAn upper bound for error in approximating an unknown density function, f(x) by its ME estimate based on m moment constraints, obtained as a PDF p(x, α) from an m parameter exponential family is derived. The error bound will help us decide if the number of moment constraints is adequate for modeling the uncertainty in the system under study. In turn, this allows one to establish confidence intervals on an estimate of some function of the random variable, X, given the known moments. It is also shown how, when working with a large sample of independent observations, instead of precisely known moment constraints, a density from an exponential family to model the uncertainty of the underlying system with measurable accuracy can be defined. In this case, a relationship to ML estimation is established.

Information-Theoretic Deconvolution Approximation of Treatment Effect Distribution

This study proposes an information-theoretic deconvolution method to approximate the entire distribution of individual treatment effect. This method uses higher-order information implied by the standard average treatment effect estimator to construct a maximum entropy approximation to the treatment effect distribution. This method is able to approximate the underlying distribution even if it is entirely random or dependent on unobservable covariates. The asymptotic properties of the proposed estimator is discussed. This estimator is shown to minimize the Kullback-Leibler distance between the underlying distribution and the approximations. Monte Carlo simulations and experiments with real data demonstrate the efficacy and flexibility of the proposed deconvolution estimator. This method is applied to data from the U.S. Job Training Partnership Act (JTPA) program to estimate the distribution of its impact on individual earnings.

패널회귀모형에서 최대엔트로피 추정량에 관한 연구

횡단면 자료와 시계열 자료가 병합된 패널회귀모형을 다루는 대부분의 연구들에서 사용되고 있는 자료는 완전한 자료를 고려하고 있다. 그러나, 실제적으로 완전한 자료보다는 불완전한 자료가 많다. 이러한 상황을 고려하지 않고 통계적인 추론을 하게 되면 잘못된 결론이 도출될 수 있다. 따라서, 자료의 형태를 충분히 고려한 추정량을 바탕으로 자료를 분석해야 한다. 본 연구는 패널회귀모형에서 자료가 불완전 상태인 경우 최대 엔트로피 형식을 이용한 일반화최대엔트로피 추정량을 제안하고, 추정량들의 효율성을 모의실험을 통하여 비교하였다. 모의실험 결과, 일반화 최대엔트로피 추정량이 가장 안정적이고 효율적인 추정량임을 보여주었다. This paper considers a panel regression model with ill-posed data and proposes the generalized maximum entropy(GME) estimator of the unknown parameters. These are natural extensions from the biometries, statistics and econometrics literature. The performance of this estimator is investigated by using of Monte Carlo experiments. The results indicate that the GME method performs the best in estimating the unknown parameters.

Densities of states, moments, and maximally broken time-reversal symmetry

Power moments, modified moments, and optimized moments are powerful tools for solving microscopic models of macroscopic systems; however the expansion of the density of states as a continued fraction does not converge to the macroscopic limit point-wise in energy with increasing numbers of moments. In this work the moment problem is further constrained by minimal lifetimes or maximal breaking of time-reversal symmetry, to yield approximate densities of states with point-wise macroscopic limits. This is applied numerically to models with one and two finite bands with various singularities, as well as to a model with infinite band-width, and the results are compared with the maximum entropy approximation where possible.

Imposing parameter inequality restrictions using the principle of maximum entropy

In this paper, we propose a new way to impose parameter inequality constraints in a linear regression model using the principle of maximum entropy. We describe the generalized maximum entropy (GME) estimator and develop a method for imposing parameter inequality restrictions through the traditional GME parameter support matrix. In addition, we develop a more general parameter support matrix that allows us to impose a broader set of restrictions than is possible under the traditional formulation.

Efficient numerical approximation of maximum entropy estimates

A class of algorithms for approximation of the maximum entropy estimate of probability density functions on the basis of a finite number of sampled data is introduced. The algorithms are presented as a finite sequence in order of increasing accuracy and decreasing computational efficiency in which the last element of the sequence is the exact maximum entropy estimate. Numerical and applicative examples are reported.

Characteristic lengths and maximum entropy estimation from probe signals in the ellipsoidal bubble regime

The bubble size, surface and volume distributions in two and three phase flows are essential to determine energy and mass transfer processes. The traditional approaches commonly use a conditional probability density function of chord-lengths to calculate the bubble size distribution, when the bubble size, shape and velocity are known. However, the approach used in this paper obtains the above distributions from statistical relations, requiring only the moments inferred from the measurements given by a sampling probe. Using image analysis of bubbles injected in a water tank, and placing an ideal probe on the image, a sample of bubble diameter, shape factor and velocity angle are obtained. The samples of the bubble chord-length are synthetically generated from these variables. Thus, we propose a semi-parametric approach based on the maximum entropy (MaxEnt) distribution estimation subjected to a number of moment constraints avoiding the use of the complex backward transformation. Therefore, the method allows us to obtain the distributions in close form. The probability density functions of the most important length scales ( D, D 20, D 30, D 32), obtained applying the semi-parametric approach proposed here in the ellipsoidal bubble regime, are compared with experimental measurements.

Cost estimates for multiple pollutants: A maximum entropy approach

In this paper, we demonstrate the use of inequality restricted generalized maximum entropy estimation to calculate shadow prices of effluents in a distance function framework. We estimate the shadow prices of sulfur dioxide, nitrogen oxides, carbon dioxide and mercury emissions from a panel of 260 coal-fired U.S. electric power plants. This modeling procedure delivers results, which are consistent with existing abatement cost evidence produced with both deterministic approaches and forecasting models. Our results imply that it may not be possible to duplicate the remarkable success of the SO 2 trading system for the other pollutants because of their greater industry abatement cost homogeneity.

Computational complexity of queries based on itemsets

We investigate determining the exact bounds of the frequencies of conjunctions based on frequent sets. Our scenario is an important special case of some general probabilistic logic problems that are known to be intractable. We show that despite the limitations our problems are also intractable, namely, we show that checking whether the maximal consistent frequency of a query is larger than a given threshold is NP-complete and that evaluating the Maximum Entropy estimate of a query is PP-hard. We also prove that checking consistency is NP-complete.

Two alternative methods for solving ion transport equation with anisotropic approximation

By means of two alternative methods namely, the maximum entropy and Chapman–Enskog, flux limited approach the ion transport equation for slowing-down problem of low-energy light ions in solid has been solved explicitly. Maximum entropy technique yields approximate solutions in the form of locally Maxwellian distribution function, based on moments expansion truncated upon entropy maximization.The behavior of the approximate maximum entropy and the flux limited solutions have the same tendency. Knowing the distribution function obtained by flux limited approach, allows us to calculate directly the path length distributions of backscattered ions, and compared with that found by other theories such as Laplace-transform and double Legendre polynomial approximation. One can see that the flux limited approach is better than the previous method namely, (DPN) Laplace-transform.The results reported in this article provide further evidence of the usefulness of both maximum entropy and flux limited for obtaining the solution of ion transport equation in compact form.

Formulating informative, data-based priors for failure probability estimation in reliability analysis

Priors play an important role in the use of Bayesian methods in risk analysis, and using all available information to formulate an informative prior can lead to more accurate posterior inferences. This paper examines the practical implications of using five different methods for formulating an informative prior for a failure probability based on past data. These methods are the method of moments, maximum likelihood (ML) estimation, maximum entropy estimation, starting from a non-informative ‘pre-prior’, and fitting a prior based on confidence/credible interval matching. The priors resulting from the use of these different methods are compared qualitatively, and the posteriors are compared quantitatively based on a number of different scenarios of observed data used to update the priors. The results show that the amount of information assumed in the prior makes a critical difference in the accuracy of the posterior inferences. For situations in which the data used to formulate the informative prior is an accurate reflection of the data that is later observed, the ML approach yields the minimum variance posterior. However, the maximum entropy approach is more robust to differences between the data used to formulate the prior and the observed data because it maximizes the uncertainty in the prior subject to the constraints imposed by the past data.

Multilevel near optimal thresholding applied to watershed grouping

The major drawback of watershed transformation is over-segmentation. It also has a significant advantage: very good edge extraction. Thresholding methods usually utilize only global information such as an image histogram; however, they have the ability to group pixels into clusters by their value. The method presented in this paper combines the advantages of watershed segmentation and multilevel thresholding. This was achieved by modifying selected optimal thresholding methods so that they treat watersheds as a whole and using those methods in a multilevel thresholding algorithm for grouping watersheds. Otsu's, Kapur's, maximum entropy and step function approximation thresholding methods have been tested. The obtained results are presented and discussed.

Partially adaptive estimation via the maximum entropy densities

Adaptive estimation is frequently used when the error distribution is non-normal. We propose a partially adaptive estimator based on the maximum entropy estimate of the error distribution. Under the conditions specified in McDonald and Newey (1988), the proposed estimator is asymptotically normal and efficient for the slope parameters. We investigate the finite sample performance of the proposed method and compare it with existing methods. We also apply the estimator to real world data.

Maximum Entropy Models with Inequality Constraints: A Case Study on Text Categorization

Data sparseness or overfitting is a serious problem in natural language processing employing machine learning methods. This is still true even for the maximum entropy (ME) method, whose flexible modeling capability has alleviated data sparseness more successfully than the other probabilistic models in many NLP tasks. Although we usually estimate the model so that it completely satisfies the equality constraints on feature expectations with the ME method, complete satisfaction leads to undesirable overfitting, especially for sparse features, since the constraints derived from a limited amount of training data are always uncertain. To control overfitting in ME estimation, we propose the use of box-type inequality constraints, where equality can be violated up to certain predefined levels that reflect this uncertainty. The derived models, inequality ME models, in effect have regularized estimation with L1 norm penalties of bounded parameters. Most importantly, this regularized estimation enables the model parameters to become sparse. This can be thought of as automatic feature selection, which is expected to improve generalization performance further. We evaluate the inequality ME models on text categorization datasets, and demonstrate their advantages over standard ME estimation, similarly motivated Gaussian MAP estimation of ME models, and support vector machines (SVMs), which are one of the state-of-the-art methods for text categorization.