Problem Of Density Estimation Research Articles

Dirichlet process mixtures under affine transformations of the data

Location-scale Dirichlet process mixtures of Gaussians (DPM-G) have proved extremely useful in dealing with density estimation and clustering problems in a wide range of domains. Motivated by an astronomical application, in this work we address the robustness of DPM-G models to affine transformations of the data, a natural requirement for any sensible statistical method for density estimation and clustering. First, we devise a coherent prior specification of the model which makes posterior inference invariant with respect to affine transformations of the data. Second, we formalise the notion of asymptotic robustness under data transformation and show that mild assumptions on the true data generating process are sufficient to ensure that DPM-G models feature such a property. Our investigation is supported by an extensive simulation study and illustrated by the analysis of an astronomical dataset consisting of physical measurements of stars in the field of the globular cluster NGC 2419.

Asymptotic Convergence of Soft-Constrained Neural Networks for Density Estimation

A soft-constrained neural network for density estimation (SC-NN-4pdf) has recently been introduced to tackle the issues arising from the application of neural networks to density estimation problems (in particular, the satisfaction of the second Kolmogorov axiom). Although the SC-NN-4pdf has been shown to outperform parametric and non-parametric approaches (from both the machine learning and the statistics areas) over a variety of univariate and multivariate density estimation tasks, no clear rationale behind its performance has been put forward so far. Neither has there been any analysis of the fundamental theoretical properties of the SC-NN-4pdf. This paper narrows the gaps, delivering a formal statement of the class of density functions that can be modeled to any degree of precision by SC-NN-4pdfs, as well as a proof of asymptotic convergence in probability of the SC-NN-4pdf training algorithm under mild conditions for a popular class of neural architectures. These properties of the SC-NN-4pdf lay the groundwork for understanding the strong estimation capabilities that SC-NN-4pdfs have only exhibited empirically so far.

Local differential privacy: Elbow effect in optimal density estimation and adaptation over Besov ellipsoids

We address the problem of non-parametric density estimation under the additional constraint that only privatised data are allowed to be published and available for inference. For this purpose, we adopt a recent generalisation of classical minimax theory to the framework of local $\alpha$-differential privacy and provide a lower bound on the rate of convergence over Besov spaces $\mathcal{B}^{s}_{pq}$ under mean integrated $\mathbb{L}^{r}$-risk. This lower bound is deteriorated compared to the standard setup without privacy, and reveals a twofold elbow effect. In order to fulfill the privacy requirement, we suggest adding suitably scaled Laplace noise to empirical wavelet coefficients. Upper bounds within (at most) a logarithmic factor are derived under the assumption that $\alpha$ stays bounded as $n$ increases: A linear but non-adaptive wavelet estimator is shown to attain the lower bound whenever $p\geq r$ but provides a slower rate of convergence otherwise. An adaptive non-linear wavelet estimator with appropriately chosen smoothing parameters and thresholding is shown to attain the lower bound within a logarithmic factor for all cases.

Boosting multiplicative model combination

AbstractIn this article, we define a new boosting‐type algorithm for multiplicative model combination using as loss function the Hyvärinen scoring rule. In particular, we focus on density estimation problems and the aim is to define a suitable estimator, using a multiplicative combination of elementary density functions, which correspond to simplified or partially specified probability models for the interest random phenomenon. The boosting algorithm provides a simple sequential procedure for updating the weights of the component density functions, until an optimality criterion is satisfied. An extension of this procedure can be useful for composite likelihood inference, in order to specify the weights of the component likelihood objects and, simultaneously, implement parameter estimation. Finally, three applications are presented. The first one regards prediction and inference for autoregressive models, the second one is the use of model pools for prediction in a time series framework, and the third one is the estimation of the covariance and the precision matrices of a multivariate Gaussian distribution. Empirical results on real‐world financial data are presented in challenging contexts, where we have to deal with a large dataset or with sparse matrices and a large number of unknown parameters.

Pointwise Optimality of Wavelet Density Estimation for Negatively Associated Biased Sample

This paper focuses on the density estimation problem that occurs when the sample is negatively associated and biased. We constructed a block thresholding wavelet estimator to recover the density function from the negatively associated biased sample. The pointwise optimality of this wavelet density estimation is shown as L p ( 1 ≤ p < ∞ ) risks over Besov space. To validate the effectiveness of the block thresholding wavelet method, we provide some examples and implement the numerical simulations. The results indicate that our block thresholding wavelet density estimator is superior in terms of the mean squared error (MSE) when comparing with the nonlinear wavelet density estimator.

A note on the applicability of the standard nonparametric maximum likelihood estimator for combined incident and prevalent cohort data

Nonparametric estimation of the survival function for either incident or prevalent cohort failure time data, exclusively, has been well studied in the literature; the Kaplan‐Meier (KM) estimator is routinely used for right‐censored incident cohort failure time data, whereas a modified form of the KM estimator, sometimes referred to as the Tsai–Jewell–Wang (TJW) estimator, is the default estimator used for prevalent cohort data with follow‐up. Often, failure time data comprise observations from a combination of incident and prevalent cohorts. In this note, we justify the use of the TJW estimator for a combined sample of incident and prevalent cohort data with follow‐up. We suggest how the TJW estimator forms the basis for density estimation and hypothesis testing problems, when incident and prevalent cohorts are combined.

Insect detection and density estimation based on a Ku‐band scanning entomological radar

Entomological radars need to address the problem of insect density estimation. Here, a Ku-band experimental entomological radar with two-dimensional scanning mode is used to collect insect echo data. The nearest neighbour method is used to associate the traces of the same insect. Then, the insects are classified according to their radar cross-sections (RCSs) and heights. Then, the corresponding radar-effective sampling volumes are calculated. So, we can get the density of insects with different RCSs at any height layer. Finally, the densities of insects with different RCSs at the same height layer are accumulated to obtain the total insect density at each height layer.

Fused Density Estimation: Theory and Methods

Summary We introduce a method for non-parametric density estimation on geometric networks. We define fused density estimators as solutions to a total variation regularized maximum likelihood density estimation problem. We provide theoretical support for fused density estimation by proving that the squared Hellinger rate of convergence for the estimator achieves the minimax bound over univariate densities of log-bounded variation. We reduce the original variational formulation to transform it into a tractable, finite dimensional quadratic program. Because random variables on geometric networks are simple generalizations of the univariate case, this method also provides a useful tool for univariate density estimation. Lastly, we apply this method and assess its performance on examples in the univariate and geometric network setting. We compare the performance of various optimization techniques to solve the problem and use these results to inform recommendations for the computation of fused density estimators.

Multivariate mixed kernel density estimators and their application in machine learning for classification of biological objects based on spectral measurements

A problem of non-parametric multivariate density estimation for machine learning and data augmentation is considered. A new mixed density estimation method based on calculating the convolution of independently obtained kernel density estimates for unknown distributions of informative features and a known (or independently estimated) density for non-informative interference occurring during measurements is proposed. Properties of the mixed density estimates obtained using this method are analyzed. The method is compared with a conventional Parzen-Rosenblatt window method applied directly to the training data. The equivalence of the mixed kernel density estimator and the data augmentation procedure based on the known (or estimated) statistical model of interference is theoretically and experimentally proven. The applicability of the mixed density estimators for training of machine learning algorithms for the classification of biological objects (elements of grain mixtures) based on spectral measurements in the visible and near-infrared regions is evaluated.

Fake uniformity in a shape inversion formula

We revisit a shape inversion formula derived by Panaretos in the context of a particle density estimation problem with unknown rotation of the particle. A distribution is presented which imitates, or 'fakes', the uniformity or Haar distribution that is part of that formula.

Pitman closeness domination in predictive density estimation for two-ordered normal means under $$\alpha$$-divergence loss

We consider Pitman closeness domination in predictive density estimation problems when the underlying loss metric is $$\alpha$$ -divergence, $$\{D(\alpha )\}$$ , a loss introduced by Csiszar (Stud Sci Math Hung 2:299–318, 1967). The underlying distributions considered are normal location-scale models, including the distribution of the observables, the distribution of the variable, whose density is to be predicted, and the estimated predictive density which will be taken to be of the plug-in type. The scales may be known or unknown. Chang and Strawderman (J Multivar Anal 128:1–9, 2014) have derived a general expression for the $$\alpha$$ -divergence loss in this setup, and have shown that it is a concave monotone function of quadratic loss, and also a function of the variances (predicand, and plug-in). We demonstrate $$\{D(\alpha )\}$$ Pitman closeness domination of certain plug-in predictive densities over others for the entire class of metrics simultaneously when modified Pitman closeness domination holds in the related problem of estimating the mean. We also establish $$\{D(\alpha )\}$$ Pitman closeness results for certain generalized Bayesian (best invariant) predictive density estimators. Examples of $$\{D(\alpha )\}$$ Pitman closeness domination presented relate to the problem of estimating the predictive density of the variable with the larger mean. We also consider the case of two-ordered normal means with a known covariance matrix.

Estimating a sharp convergence bound for randomized ensembles

When randomized ensembles such as bagging or random forests are used for binary classification, the prediction error of the ensemble tends to decrease and stabilize as the number of classifiers increases. However, the precise relationship between prediction error and ensemble size is unknown in practice. In the standard case when classifiers are aggregated by majority vote, the present work offers a way to quantify this convergence in terms of “algorithmic variance,” i.e. the variance of prediction error due only to the randomized training algorithm. Specifically, we study a theoretical upper bound on this variance, and show that it is sharp — in the sense that it is attained by a specific family of randomized classifiers. Next, we address the problem of estimating the unknown value of the bound, which leads to a unique twist on the classical problem of non-parametric density estimation. In particular, we develop an estimator for the bound and show that its MSE matches optimal non-parametric rates under certain conditions. (Concurrent with this work, some closely related results have also been considered in Cannings and Samworth (2017) and Lopes (2019).)

Training convolutional neural networks to estimate turbulent sub-grid scale reaction rates

This work presents a new approach for premixed turbulent combustion modeling based on convolutional neural networks (CNN).11Code and data for the deep learning in this work is available at https://gitlab.com/cerfacs/code-for-papers/2018/arXiv_1810.03691. We first propose a framework to reformulate the problem of subgrid flame surface density estimation as a machine learning task. Data needed to train the CNN is produced by direct numerical simulations (DNS) of a premixed turbulent flame stabilized in a slot-burner configuration. A CNN inspired from a U-Net architecture is designed and trained on the DNS fields to estimate subgrid-scale wrinkling. It is then tested on an unsteady turbulent flame where the mean inlet velocity is increased for a short time and the flame must react to a varying turbulent incoming flow. The CNN is found to efficiently extract the topological nature of the flame and predict subgrid-scale wrinkling, outperforming classical algebraic models.

Bayesian fractional posteriors

We consider the fractional posterior distribution that is obtained by updating a prior distribution via Bayes theorem with a fractional likelihood function, a usual likelihood function raised to a fractional power. First, we analyze the contraction property of the fractional posterior in a general misspecified framework. Our contraction results only require a prior mass condition on certain Kullback–Leibler (KL) neighborhood of the true parameter (or the KL divergence minimizer in the misspecified case), and obviate constructions of test functions and sieves commonly used in the literature for analyzing the contraction property of a regular posterior. We show through a counterexample that some condition controlling the complexity of the parameter space is necessary for the regular posterior to contract, rendering additional flexibility on the choice of the prior for the fractional posterior. Second, we derive a novel Bayesian oracle inequality based on a PAC-Bayes inequality in misspecified models. Our derivation reveals several advantages of averaging based Bayesian procedures over optimization based frequentist procedures. As an application of the Bayesian oracle inequality, we derive a sharp oracle inequality in multivariate convex regression problems. We also illustrate the theory in Gaussian process regression and density estimation problems.

Crowd density estimation based on classification activation map and patch density level

The task of crowd counting and density map estimation is riddled with many challenges, such as occlusions, non-uniform density, intra-scene and inter-scene variations in scale and perspective. Due to the development of deep learning and large crowd datasets in recent years, most crowd counting methods have achieved notable success. This paper aims to solve crowd density estimation problem for both sparse and dense conditions. To this end, we make two contributions: (1) a network named Patch Scale Discriminant Regression Network (PSDR). Given an input crowd image, it divides the image into patches and sends image patches of different density levels into different regression networks to get the corresponding density maps. It combines all patch density maps to predict the entire density map as the output. (2) A person classification activation map (CAM) method. CAM provides person location information and guides the generation of the entire density map in the final stage. Experiment confirms that CAM allows PSDR to gain another round of performance boost. For instance, on the SmartCity dataset, we achieve (8.6–1.1) MAE and (11.6–1.4) MSE. Our method combining above two methods performs better than state-of-the-art methods.

Density Estimation for RWRE

We consider the problem of nonparametric density estimation of a random environment from the observation of a single trajectory of a random walk in this environment. We build several density estimators using the beta-moments of this distribution. Then we apply the Goldenschluger-Lepski method to select an estimator satisfying an oracle type inequality. We obtain non-asymptotic bounds for the supremum norm of these estimators that hold when the RWRE is recurrent or transient to the right. A simulation study supports our theoretical findings.

Estimation of a functional single index model with dependent errors and unknown error density

The problem of error density estimation for a functional single index model with dependent errors is studied. A Bayesian method is utilized to simultaneously estimate the bandwidths in the kernel-form error density and regression function, under an autoregressive error structure. For estimating both the regression function and error density, empirical studies show that the functional single index model gives improved estimation and prediction accuracies than any nonparametric functional regression considered. Furthermore, estimation of error density facilitates the construction of prediction interval for the response variable.

An EM‐type approach for classification of bivariate MALDI‐MS data and identification of high fertility markers

AbstractDairy cows are responsible for a fair amount of gas emissions in the atmosphere (mainly methane, ammonia, and carbon dioxide), as well as waste outputs. Therefore, identifying high‐fertility breeding cows and increasing fertility rates can diminish pollution and help minimize the effect of global warming and improve the environmental impact of the farming system. As a step to achieve this goal, changes in the lipid composition of the bovine uterus exposed to greater (LF‐LCL group) or lower (SF‐SCL group) concentrations of progesterone during postovulation were investigated by matrix‐assisted laser desorption ionization mass spectrometry. Two measurements were made for each cow, and after preprocessing the data, the measurements available to analysis consist of relative intensities at significant 76 mass‐to‐charge ratio (m/z) values identifying specific ions in the spectra. Due to the small sample size, seven cows in the LF‐LCL group and 10 cows in the SF‐SCL group, the usual methods could not discriminate between groups. A model‐based approach was therefore proposed, and due to the discrete nature of the data, a truncated mixture of bivariate beta distributions was fitted to the data using an expectation–maximization algorithm. However, unlike the usual approach for mixture density estimation problems, to each 76 m/z value, we assign an unobserved label shared by all cows in the same group. The role of these labels is similar to the frailty effect in survival models in which all cows in a given group would share some random effect due to group effect. These labels will be used to identify m/z values, which could potentially account for different fertility rates.

Maximum-Likelihood Linear Transformation for Unsupervised Domain Adaptation in Speaker Verification

Recent advances in front-end factor analysis through development of i-Vectors have led to significant gains in speaker recognition technology. However, the problem of mismatch between the domains of system development and evaluation data remains a challenging one. This domain mismatch occurs primarily because of the variability in the sources of development and evaluation data. In this study, we propose a novel method of unsupervised probabilistic feature transformation (UPFT) to reduce this domain mismatch by transforming an out-of-domain development data toward in-domain development data. We formulate the alignment of two different domains as a probability density estimation problem. We first train a Gaussian mixture model (GMM) using the out-of-domain i-Vectors. Next, we employ an expectation–maximization (EM) algorithm to fit the means of the GMM to the in-domain i-Vectors by maximizing the overall likelihood. At the optimum, the two domains become closer to each other in the i-Vector space. While reaching the optimum through multiple iterations of the EM, we reparameterize the centroid locations using the following set of transformation parameters: rotation, translation, and scaling. These transformation parameters, which are obtained during the optimization process, are later used to transform the out-of-domain i-Vectors toward in-domain i-Vectors. We observe that such a transformation leads to an improvement in performance of the out-of-domain speaker recognition system. Our proposed method has an added advantage of being completely unsupervised, and thus does not rely on any tuning parameters. We conduct experiments on both 2013 domain adaptation challenge corpus as well as National Institute of Standards and Technology Speaker Recognition Evaluation (SRE)—2016 corpus. On both corpora, we obtain significant improvements using the proposed UPFT solution. Specifically for the SRE-2016 corpus, using a cosine distance scoring based system, we are able to recover almost 90% of the performance gap between an in-domain and out-of-domain system.

Statistical modeling and Gaussianization procedure based de-speckling algorithm for retinal OCT images

This paper presents a de-noising method for speckle noise removal called de-speckling, retinal optical coherence tomography image by combining the features of wavelet transform, statistical modeling, Bayesian estimators and Gaussianization procedure. De-speckling methods based on statistical modeling of wavelet coefficients depend on the correct estimation of probability density function (PDF). The density estimation problem has been solved by using the Gaussianization procedure. The dynamic range capability of Gaussian PDF has been used for sharp estimation of density. A Cauchy PDF is used for modeling wavelet coefficients and finding the cumulative distribution function (CDF) of wavelet coefficients. This CDF is used as the input to the Gaussianization procedure, where it equalizes with the Gaussian CDF to estimate the density function of wavelet coefficients. Finally, the wavelet coefficients are recovered using Bayesian minimum mean square error estimator. Both visual and quantitative comparisons are performed for demonstrating the prominence of the proposed method. From simulation result it is seen that the proposed method outperforms the state-of-the-art methods in terms of peak signal-to-noise ratio (PSNR), structural similarity, correlation coefficient (CoC) and edge preservation index (EPI). The proposed method has been achieved improvement of 5.13% in PSNR, 2.44% in SSIM, 2.11% in CoC and 3.45% in EPI over the well accepted existing method.