Problem Of Density Estimation Research Articles

Renormalization Group in Bayesian Statistical Inference

The prediction in Bayesian framework is extended from the point of view of renormalization group. For this purpose, we first make it clear that the advantages of Bayesian statistical inference can be understood by an adaptive property of a long-distance length scale. This suggests a close connection of Bayesian statistical inference to renormalization group. Next, we show that a cumulative entropic error can be rewritten as an effective action, which directly leads to a renormalization group equation in non-parametric Bayesian statistical inference. As a result, we introduce a scaling part to a prior distribution, and determine it so that we can obtain better prediction performance. We discuss how prediction performance improves, taking an example of a density estimation problem.

On an L 1 Consistent Wavelet Density Estimator for the Deconvolution Problem

The problem of wavelet density estimation is studied when the sample observations are contaminated with random noise. In particular, a truncated linear wavelet estimator based on Meyer-type wavelets is shown to be L 1 strongly consistent when the Fourier transform of the random noise has polynomial descent or exponential descent.

On minimax density estimation on \mathbb{R}}

The problem of density estimation on \mathbb{R}} on the basis of an independent sample X1,..., XN with common density f is discussed. The behaviour of the minimax Lp risk, 1≤p≤∞, is studied when f belongs to a Hölder class of regularity s on the real line. The lower bound for the minimax risk is given. We show that the linear estimator is not efficient in this setting and construct a wavelet adaptive estimator which attains (up to a logarithmic factor in N) the lower bounds involved. We show that the minimax risk depends on the parameter p when p<2+ 1/s.

Transition density estimation for stochastic differential equations via forward-reverse representations

The general reverse diffusion equations are derived and applied to the problem of transition density estimation of diffusion processes between two fixed states. For this problem we propose density estimation based on forward-reverse representations and show that this method allows essentially better results to be achieved than the usual kernel or projection estimation based on forward representations only.

Decision-Theoretic Bidding Based on Learned Density Models in Simultaneous, Interacting Auctions

Auctions are becoming an increasingly popular method for transacting business, especially over the Internet. This article presents a general approach to building autonomous bidding agents to bid in multiple simultaneous auctions for interacting goods. A core component of our approach learns a model of the empirical price dynamics based on past data and uses the model to analytically calculate, to the greatest extent possible, optimal bids. We introduce a new and general boosting-based algorithm for conditional density estimation problems of this kind, i.e., supervised learning problems in which the goal is to estimate the entire conditional distribution of the real-valued label. This approach is fully implemented as ATTac-2001, a top-scoring agent in the second Trading Agent Competition (TAC-01). We present experiments demonstrating the effectiveness of our boosting-based price predictor relative to several reasonable alternatives.

On confidence intervals for distribution function and density of ergodic diffusion process

In the problems of invariant distribution and density estimation of an ergodic diffusion process, the asymptotic variances of many estimators can be represented as some mathematical expectations with respect to the invariant law. Therefore, the construction of the confidence intervals requires the estimates of these quantities. The present work proposes estimators of them which are consistent and so it allows to construct the corresponding confidence intervals. Moreover, these quantities play an important role in the problems of asymptotically efficient estimation of the distribution function and density.

Distributed EM algorithms for density estimation and clustering in sensor networks

The paper considers the problem of density estimation and clustering in distributed sensor networks. It is assumed that each node in the network senses an environment that can be described as a mixture of some elementary conditions. The measurements are thus statistically modeled with a mixture of Gaussians, where each Gaussian component corresponds to one of the elementary conditions. The paper presents a distributed expectation-maximization (EM) algorithm for estimating the Gaussian components, which are common to the environment and sensor network as a whole, as well as the mixing probabilities that may vary from node to node. The algorithm produces an estimate (in terms of a Gaussian mixture approximation) of the density of the sensor data without requiring the data to be transmitted to and processed at a central location. Alternatively, the algorithm can be viewed as a distributed processing strategy for clustering the sensor data into components corresponding to predominant environmental features sensed by the network. The convergence of the distributed EM algorithm is investigated, and simulations demonstrate the potential of this approach to sensor network data analysis.

The Bahadur risk in probability density estimation

A nonparametric probability density estimation problem is studied for the Bahadur-type risk under the sup-norm losses. The risk is minimax over the Hölder classes of densities. The large sample limiting performance of this risk is found, and the links to asymptotic equivalent white-noise models are discussed.

ON A STRONGLY CONSISTENT WAVELET DENSITY ESTIMATOR FOR THE DECONVOLUTION PROBLEM

ABSTRACT The problem of wavelet density estimation is studied when the sample observations are contaminated with random noise. In this paper a linear wavelet estimator based on Meyer-type wavelets is shown to be strongly consistent when Fourier transform of random noise has polynomial descent or exponential descent.

Relative Expected Instantaneous Loss Bounds

In the literature a number of relative loss bounds have been shown for on-line learning algorithms. Here the relative loss is the total loss of the on-line algorithm in all trials minus the total loss of the best comparator that is chosen off-line. However, for many applications instantaneous loss bounds are more interesting where the learner first sees a batch of examples and then uses these examples to make a prediction on a new instance. We show relative expected instantaneous loss bounds for the case when the examples are i.i.d. with an unknown distribution. We bound the expected loss of the algorithm on the last example minus the expected loss of the best comparator on a random example. In particular, we study linear regression and density estimation problems and show how the leave-one-out loss can be used to prove instantaneous loss bounds for these cases. For linear regression we use an algorithm that is similar to a new on-line learning algorithm developed by Vovk. Recently a large number of relative total loss bounds have been shown that have the form O(lnT), where T is the number of trials/examples. Standard conversions of on-line algorithms to batch algorithms result in relative expected instantaneous loss bounds of the form O(lnTT). Our methods lead to O(1T) upper bounds. In many cases we give tight lower bounds.

On Density Estimation under Relative Entropy Loss Criterion

A density estimation problem under relative entropy loss criterion is considered. When the densities estimated constitute a d-dimensional smooth parameter family, an exact asymptotics of the minimax risk is established.

Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities

We study the rates of convergence of the maximum likelihood estimator (MLE) and posterior distribution in density estimation problems, where the densities are location or location-scale mixtures of normal distributions with the scale parameter lying between two positive numbers. The true density is also assumed to lie in this class with the true mixing distribution either compactly supported or having sub-Gaussian tails. We obtain bounds for Hellinger bracketing entropies for this class, and from these bounds, we deduce the convergence rates of (sieve) MLEs in Hellinger distance. The rate turns out to be $(log n)^\kappa /\sqrt{n}$, where $\kappa \ge 1$ is a constant that depends on the type of mixtures and the choice of the sieve. Next, we consider a Dirichlet mixture of normals as a prior on the unknown density. We estimate the prior probability of a certain Kullback-Leibler type neighborhood and then invoke a general theorem that computes the posterior convergence rate in terms the growth rate of the Hellinger entropy and the concentration rate of the prior. The posterior distribution is also seen to converge at the rate $(log n)^\kappa /\sqrt{n}$, where $\kappa$ now depends on the tail behavior of the base measure of the Dirichlet process.

Consistency of posterior distributions for neural networks

In this paper we show that the posterior distribution for feedforward neural networks is asymptotically consistent. This paper extends earlier results on universal approximation properties of neural networks to the Bayesian setting. The proof of consistency embeds the problem in a density estimation problem, then uses bounds on the bracketing entropy to show that the posterior is consistent over Hellinger neighborhoods. It then relates this result back to the regression setting. We show consistency in both the setting of the number of hidden nodes growing with the sample size, and in the case where the number of hidden nodes is treated as a parameter. Thus we provide a theoretical justification for using neural networks for nonparametric regression in a Bayesian framework.

Markov chain Monte Carlo in approximate Dirichlet and beta two-parameter process hierarchical models

SUMMARY We present some easy-to-construct random probability measures which approximate the Dirichlet process and an extension which we will call the beta two-parameter process. The nature of these constructions makes it simple to implement Markov chain Monte Carlo algorithms for fitting nonparametric hierarchical models and mixtures of nonparametric hierarchical models. For the Dirichlet process, we consider a truncation approximation as well as a weak limit approximation based on a mixture of Dirichlet processes. The same type of truncation approximation can also be applied to the beta two-parameter process. Both methods lead to posteriors which can be fitted using Markov chain Monte Carlo algorithms that take advantage of blocked coordinate updates. These algorithms promote rapid mixing of the Markov chain and can be readily applied to normal mean mixture models and to density estimation problems. We prefer the truncation approximations, since a simple device for monitoring the adequacy of the approximation can be easily computed from the output of the Gibbs sampler. Furthermore, for the Dirichlet process, the truncation approximation offers an exponentially higher degree of accuracy over the weak limit approximation for the same computational effort. We also find that a certain beta two-parameter process may be suitable for finite mixture modelling because the distinct number of sampled values from this process tends to match closely the number of components of the underlying mixture distribution.

The adaptive rate of convergence in a problem of pointwise density estimation

We estimate the common density function of n i.i.d. observations, at a fixed point, over Sobolev classes of functions having regularity β. We prove that the optimal rate of convergence cannot be attained in adaptive estimation, i.e. uniformly over β in some interval B n . A slower rate is shown to be adaptive.

A kurtosis-based dynamic approach to Gaussian mixture modeling

We address the problem of probability density function estimation using a Gaussian mixture model updated with the expectation-maximization (EM) algorithm. To deal with the case of an unknown number of mixing kernels, we define a new measure for Gaussian mixtures, called total kurtosis, which is based on the weighted sample kurtoses of the kernels. This measure provides an indication of how well the Gaussian mixture fits the data. Then we propose a new dynamic algorithm for Gaussian mixture density estimation which monitors the total kurtosis at each step of the EM algorithm in order to decide dynamically on the correct number of kernels and possibly escape from local maxima. We show the potential of our technique in approximating unknown densities through a series of examples with several density estimation problems.

Density Estimation under Constraints

We suggest a general method for tackling problems of density estimation under constraints. It is, in effect, a particular form of the weighted bootstrap, in which resampling weights are chosen so as to minimize distance from the empirical or uniform bootstrap distribution subject to the constraints being satisfied. A number of constraints are treated as examples. They include conditions on moments, quantiles, and entropy, the latter as a device for imposing qualitative conditions such as those of unimodality or “interestingness.” For example, without altering the data or the amount of smoothing, we may construct a density estimator that enjoys the same mean, median, and quartiles as the data. Different measures of distance·give rise to slightly different results.

Posterior consistency of Dirichlet mixtures in density estimation

A Dirichlet mixture of normal densities is a useful choice for a prior distribution on densities in the problem of Bayesian density estimation. In the recent years, efficient Markov chain Monte Carlo method for the computation of the posterior distribution has been developed. The method has been applied to data arising from different fields of interest. The important issue of consistency was however left open. In this paper, we settle this issue in affirmative.

Density estimation in the presence of noise

The problem of density estimation in the absence of noise has been widely studied and is well known. However if noise is added to the sample, the procedure must be modified by incorporating a deconvolution. In this paper we do so by using a procedure similar to empirical Bayes estimation which involves band-limited wavelets. Rates of mean square convergence are found under various hypotheses.

Self-organization in Cortical Maps &amp; EM-learning

Starting from the problem of density estimation, it is shown that Expectation Maximization (EM) learning can be considered a Hebbian mechanism. From this, it is possible to outline a theory of self-organization of cortical maps, which is based on a well-defined optimization process and preserves biologically desirable characteristics such as local computation and uniform treatment of input and lateral connections. A thalamocortical network is described that implements the theory in a fully distributed manner: it uses cortical dynamics for the E-step and Hebbian adaptation of cortico-cortical connections at steady state for the M-step.