Problem Of Density Estimation Research Articles

Theoretical insights on the pre-image resolution in machine learning

While many nonlinear pattern recognition and data mining tasks rely on embedding the data into a latent space, one often needs to extract the patterns in the input space. Estimating the inverse of the nonlinear embedding is the so-called pre-image problem. Several strategies have been proposed to address the estimation of the pre-image; However, there are no theoretical results so far to understand the pre-image problem and its resolution. In this paper, we provide theoretical underpinnings of the resolution of the pre-image problem in Machine Learning. These theoretical results are on the gradient descent optimization, the fixed-point iteration algorithm and Newton’s method. We provide sufficient conditions on the convexity/nonconvexity of the pre-image problem. Moreover, we show that the fixed-point iteration is a Newton update and prove that it is a Majorize-Minimization (MM) algorithm where the surrogate function is a quadratic function. These theoretical results are derived for the wide classes of radial kernels and projective kernels. We also provide other insights by connecting the resolution of this problem to the gradient density estimation problem with the so-called mean shift algorithm.

Divergences based Bayesian inference with censored data

. In this work, we deal with some Bayesian inference problems in the presence of right censored data. First, we propose a dual ϕ−divergence Bayes type estimators for parametric models and we establish their asymptotic normality. To establish this result, we need a uniform strong law of large numbers that we prove as well. Then, we consider the problem of prior distributions construction for model selection using ϕ−divergences. Finally, we consider the problem of the predictive density estimation on the basis of ϕ−divergences. We apply an expansion result of the generalized Bayesian predictive density on two parametric models widely used in survival analysis, namely the Weibull and the inverse Weibull model, under right censoring. We also check the performances of our proposed methods through simulations and real data applications. The results of these studies show that our proposed dual ϕ−divergence Bayes type estimators are more robust than other Bayesian estimators. Moreover, the generalized Bayesian predictive density performs better than the classical estimative density especially for the inverse Weibull model.

Nonparametric density estimation over its unknown support for right censored data

The long-standing problem of nonparametric density estimation of the lifetime of interest over its unknown support, when only right-censored observations are available, is considered. Two cases, when the support of the lifetime of interest is a proper subset of the support of censoring variable and when the supports are the same, are explored. An orthogonal series density estimation, over a random interval defined by a specially chosen sequence of order statistics, is proposed and rates of the mean integrated squared error convergence are evaluated.

High-dimensional Bernstein–von Mises theorem for the Diaconis–Ylvisaker prior

We study the asymptotic normality of the posterior distribution of canonical parameter in the exponential family under the Diaconis–Ylvisaker prior which is a conjugate prior when the dimension of parameter space increases with the sample size. We prove under mild conditions on the true parameter value θ0 and hyperparameters of priors, the difference between the posterior distribution and a normal distribution centered at the maximum likelihood estimator, and variance equal to the inverse of the Fisher information matrix goes to 0 in the expected total variation distance. The proof assumes dimension of parameter space d grows linearly with sample size n only requiring d=o(n). En route, we derive a concentration inequality of the quadratic form of the maximum likelihood estimator without any specific assumption such as sub-Gaussianity. A specific illustration is provided for the Multinomial-Dirichlet model with an extension to the density estimation and Normal mean estimation problems.

An Update on Measurement Error Modeling

The issues caused by measurement errors have been recognized for almost 90 years, and research in this area has flourished since the 1980s. We review some of the classical methods in both density estimation and regression problems with measurement errors. In both problems, we consider when the original error-free model is parametric, nonparametric, and semiparametric, in combination with different error types. We also summarize and explain some new approaches, including recent developments and challenges in the high-dimensional setting.

Adaptive minimax density estimation on ℝd for Huber’s contamination model

Abstract We address the problem of adaptive minimax density estimation on $\mathbb{R}^{d}$ with $L_{p}$ loss functions under Huber’s contamination model. To investigate the contamination effect on the optimal estimation of the density, we first establish the minimax rate with the assumption that the density is in an anisotropic Nikol’skii class. We then develop a data-driven bandwidth selection procedure for kernel estimators, which can be viewed as a robust generalization of the Goldenshluger-Lepski method. We show that the proposed bandwidth selection rule can lead to the estimator being minimax adaptive to either the smoothness parameter or the contamination proportion. When both of them are unknown, we prove that finding any minimax-rate adaptive method is impossible. Extensions to smooth contamination cases are also discussed.

Exact Boundary Correction Methods for Multivariate Kernel Density Estimation

This paper develops a method to obtain multivariate kernel functions for density estimation problems in which the density function is defined on compact support. If domain-specific knowledge requires certain conditions to be satisfied at the boundary of the support of an unknown density, the proposed method incorporates the information contained in the boundary conditions into the kernel density estimators. The proposed method provides an exact kernel function that satisfies the boundary conditions, even for small samples. Existing methods primarily deal with a one-sided boundary in a one-dimensional problem. We consider density in a two-sided interval and extend it to a multi-dimensional problem.

Bayesian modeling via discrete nonparametric priors

The availability of complex-structured data has sparked new research directions in statistics and machine learning. Bayesian nonparametrics is at the forefront of this trend thanks to two crucial features: its coherent probabilistic framework, which naturally leads to principled prediction and uncertainty quantification, and its infinite-dimensionality, which exempts from parametric restrictions and ensures full modeling flexibility. In this paper, we provide a concise overview of Bayesian nonparametrics starting from its foundations and the Dirichlet process, the most popular nonparametric prior. We describe the use of the Dirichlet process in species discovery, density estimation, and clustering problems. Among the many generalizations of the Dirichlet process proposed in the literature, we single out the Pitman–Yor process, and compare it to the Dirichlet process. Their different features are showcased with real-data illustrations. Finally, we consider more complex data structures, which require dependent versions of these models. One of the most effective strategies to achieve this goal is represented by hierarchical constructions. We highlight the role of the dependence structure in the borrowing of information and illustrate its effectiveness on unbalanced datasets.

Predictive density estimators with integrated [formula omitted] loss

This paper addresses the problem of an efficient predictive density estimation for the density q(‖y−θ‖2) of Y based on X∼p(‖x−θ‖2) for y,x,θ∈Rd. The chosen criteria are integrated L1 loss given by L(θ,qˆ)=∫Rd|qˆ(y)−q(‖y−θ‖2)|dy, and the associated frequentist risk, for θ∈Θ. For absolutely continuous and strictly decreasing q, we establish the inevitability of scale expansion improvements qˆc(y;X)=1cdq(‖y−X‖2/c2) over the plug-in density qˆ1, for a subset of values c∈(1,c0). The finding is universal with respect to p,q, and d≥2, and extended to loss functions γ(L(θ,qˆ)) with strictly increasing γ. The finding is also extended to include scale expansion improvements of more general plug-in densities q(‖y−θˆ(X)‖2), when the parameter space Θ is a compact subset of Rd. Numerical analyses illustrative of the dominance findings are presented and commented upon. As a complement, we demonstrate that the unimodal assumption on q is necessary with a detailed analysis of cases where the distribution of Y|θ is uniformly distributed on a ball centered about θ. In such cases, we provide a univariate (d=1) example where the best equivariant estimator is a plug-in estimator, and we obtain cases (for d=1,3) where the plug-in density qˆ1 is optimal among all qˆc.

Automatic rape flower cluster counting method based on low-cost labelling and UAV-RGB images

BackgroundThe flowering period is a critical time for the growth of rape plants. Counting rape flower clusters can help farmers to predict the yield information of the corresponding rape fields. However, counting in-field is a time-consuming and labor-intensive task. To address this, we explored a deep learning counting method based on unmanned aircraft vehicle (UAV). The proposed method developed the in-field counting of rape flower clusters as a density estimation problem. It is different from the object detection method of counting the bounding boxes. The crucial step of the density map estimation using deep learning is to train a deep neural network that maps from an input image to the corresponding annotated density map.ResultsWe explored a rape flower cluster counting network series: RapeNet and RapeNet+. A rectangular box labeling-based rape flower clusters dataset (RFRB) and a centroid labeling-based rape flower clusters dataset (RFCP) were used for network model training. To verify the performance of RapeNet series, the paper compares the counting result with the real values of manual annotation. The average accuracy (Acc), relative root mean square error (rrMSE) and R^2 of the metrics are up to 0.9062, 12.03 and 0.9635 on the dataset RFRB, and 0.9538, 5.61 and 0.9826 on the dataset RFCP, respectively. The resolution has little influence for the proposed model. In addition, the visualization results have some interpretability.ConclusionsExtensive experimental results demonstrate that the RapeNet series outperforms other state-of-the-art counting approaches. The proposed method provides an important technical support for the crop counting statistics of rape flower clusters in field.

On robustness and local differential privacy

It is of soaring demand to develop statistical analysis tools that are robust against contamination as well as preserving individual data owners’ privacy. In spite of the fact that both topics host a rich body of literature, to the best of our knowledge, we are the first to systematically study the connections between the optimality under Huber’s contamination model and the local differential privacy (LDP) constraints. In this paper, we start with a general minimax lower bound result, which disentangles the costs of being robust against Huber contamination and preserving LDP. We further study four concrete examples: a two-point testing problem, a potentially diverging mean estimation problem, a nonparametric density estimation problem and a univariate median estimation problem. For each problem, we demonstrate procedures that are optimal in the presence of both contamination and LDP constraints, comment on the connections with the state-of-the-art methods that are only studied under either contamination or privacy constraints, and unveil the connections between robustness and LDP via partially answering whether LDP procedures are robust and whether robust procedures can be efficiently privatised. Overall, our work showcases a promising prospect of joint study for robustness and local differential privacy.

Semiparametric counterfactual density estimation

Abstract Causal effects are often characterized with averages, which can give an incomplete picture of the underlying counterfactual distributions. Here we consider estimating the entire counterfactual density and generic functionals thereof. We focus on two kinds of target parameters: density approximations and the distance between counterfactual densities. We study nonparametric efficiency bounds, giving results for smooth but otherwise generic models and distances. Importantly, we show how these bounds connect to means of particular nontrivial functions of counterfactuals, linking the problems of density and mean estimation. We propose doubly robust-style estimators, and study their rates of convergence, showing that they can be optimally efficient in large nonparametric models. We also give analogous methods for model selection and aggregation, when many models may be available and of interest. Our results all hold for generic models and distances, but we highlight results for L2 projections on linear models and Kullbach–Leibler projections on exponential families. Finally, we illustrate our method by estimating the density of the CD4 count among patients with HIV, had all been treated with combination therapy versus zidovudine alone, as well as a density effect. Our methods are implemented in the R package npcausal on GitHub.

The L2 convergence of stream data mining algorithms based on probabilistic neural networks

This paper concerns a new incremental approach to mining data streams. It is known that patterns in a data stream may evolve over time. In many cases, we need to track and analyze the nature of these changes. In the paper, the probabilistic neural networks are considered as basic models for tracking changes in data streams. We present globally convergent stream data mining algorithms applied to problems of regression, classification, and density estimation in a time-varying (drifting) environment. The algorithms are derived from the Parzen kernel-based probabilistic neural networks working in the online mode. For each problem, a theorem is presented ensuring the L2 convergence of the algorithm designed for tracking drifting regression, density, or discriminant functions. Illustrative examples explain in detail how to choose the bandwidth of the Parzen kernel and the learning rate of the online algorithm. The performance of all algorithms is shown in exemplary simulations. It should be noted that this paper is one of very few, in the existing literature, presenting mathematically justified stream data mining algorithms.

Stationary Density Estimation of Itô Diffusions Using Deep Learning

In this paper, we consider the density estimation problem associated with the stationary measure of ergodic Itô diffusions from a discrete-time series that approximate the solutions of the stochastic differential equations. To take advantage of the characterization of density function through the stationary solution of a parabolic-type Fokker–Planck PDE, we proceed as follows: First, we employ deep neural networks to approximate the drift and diffusion terms of the SDE by solving appropriate supervised learning tasks. Subsequently, we solve a steady-state Fokker–Planck equation associated with the estimated drift and diffusion coefficients with a neural-network–based least squares method. We establish the convergence of the proposed scheme under appropriate mathematical assumptions, accounting for the generalization errors induced by regressing the drift and diffusion coefficients and the PDE solvers. This theoretical study relies on a recent perturbation theory of Markov chain result that shows a linear dependence of the density estimation to the error in estimating the drift term and generalization error results of nonparametric regression and PDE regression solution obtained with neural-network models. We demonstrate the effectiveness of this method by numerical simulations of a two-dimensional Student t-distribution and a 20-dimensional Langevin dynamics.

Boundary-free estimators of the mean residual life function for data on general interval

We propose two new kernel-type estimators of the mean residual life function m X of bounded or half-bounded interval-supported distributions. Though not as severe as the boundary problems in kernel density estimation, eliminating the boundary bias problems that occur in the naive kernel estimator of the mean residual life function is needed. In this article, we utilize the property of bijective transformation. Furthermore, our proposed methods preserve the mean value property, which cannot be done by the naive kernel estimator. Some simulation results showing the estimators’ performances and real data analysis will be presented in the last part of this article.

Multivariate density estimation from privatised data: universal consistency and minimax rates

We revisit the classical problem of nonparametric density estimation but impose local differential privacy constraints. Under such constraints, the original multivariate data cannot be directly observed, and all estimators are functions of the randomised output of a suitable privacy mechanism. The statistician is free to choose the form of the privacy mechanism, and in this work we propose to add Laplace distributed noise to a discretisation of the location of an observed vector. Based on these randomised data, we propose a novel estimator of the density function, which can be viewed as a privatised version of the well-studied histogram density estimator. Our theoretical results include universal pointwise consistency and strong universal -consistency. In addition, a convergence rate for Lipschitz continuous functions is derived, which is complemented by a matching minimax lower bound. We illustrate the trade-off between data utility and privacy by means of a small simulation study.

Single Image Deraining With Continuous Rain Density Estimation

Single image deraining (SIDR) often suffers from over/under deraining due to the nonuniformity of rain densities and the variety of raindrop scales. In this paper, we propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">co</i> ntinuous <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">de</i> nsity-guided network (CODE-Net) for SIDR. Particularly, it is composed of a rain streak extractor and a denoiser, where the convolutional sparse coding (CSC) is exploited to filter out noises from the extracted rain streaks. Inspired by the reweighted iterative soft-threshold (ISTA) for CSC, we address the problem of continuous rain density estimation by learning the weights with channel attention blocks from sparse codes. We further develop a multiscale strategy to depict rain streaks appearing at different scales. Experiments on synthetic and real-world data demonstrate the superiority of our methods over recent state-of-the-arts, in terms of both quantitative and qualitative results. Additionally, instead of quantizing rain density with several levels, our CODE-Net can provide continuous-valued estimations of rain densities, which is more desirable in real applications.

Privacy by Projection: Federated Population Density Estimation by Projecting on Random Features.

We consider the problem of population density estimation based on location data crowdsourced from mobile devices, using kernel density estimation (KDE). In a conventional, centralized setting, KDE requires mobile users to upload their location data to a server, thus raising privacy concerns. Here, we propose a Federated KDE framework for estimating the user population density, which not only keeps location data on the devices but also provides probabilistic privacy guarantees against a malicious server that tries to infer users' location. Our approach Federated random Fourier feature (RFF) KDE leverages a random feature representation of the KDE solution, in which each user's information is irreversibly projected onto a small number of spatially delocalized basis functions, making precise localization impossible while still allowing population density estimation. We evaluate our method on both synthetic and real-world datasets, and we show that it achieves a better utility (estimation performance)-vs-privacy (distance between inferred and true locations) tradeoff, compared to state-of-the-art baselines (e.g., GeoInd). We also vary the number of basis functions per user, to further improve the privacy-utility trade-off, and we provide analytical bounds on localization as a function of areal unit size and kernel bandwidth.

Semi-Supervised Crowd Counting via Multiple Representation Learning.

There has been a growing interest in counting crowds through computer vision and machine learning techniques in recent years. Despite that significant progress has been made, most existing methods heavily rely on fully-supervised learning and require a lot of labeled data. To alleviate the reliance, we focus on the semi-supervised learning paradigm. Usually, crowd counting is converted to a density estimation problem. The model is trained to predict a density map and obtains the total count by accumulating densities over all the locations. In particular, we find that there could be multiple density map representations for a given image in a way that they differ in probability distribution forms but reach a consensus on their total counts. Therefore, we propose multiple representation learning to train several models. Each model focuses on a specific density representation and utilizes the count consistency between models to supervise unlabeled data. To bypass the explicit density regression problem, which makes a strong parametric assumption on the underlying density distribution, we propose an implicit density representation method based on the kernel mean embedding. Extensive experiments demonstrate that our approach outperforms state-of-the-art semi-supervised methods significantly.

Fast and fully-automated histograms for large-scale data sets

G-Enum histograms are a new fast and fully automated method for irregular histogram construction. By framing histogram construction as a density estimation problem and its automation as a model selection task, these histograms leverage the Minimum Description Length principle (MDL) to derive two different model selection criteria. Several proven theoretical results about these criteria give insights about their asymptotic behaviour and are used to speed up their optimisation. These insights, combined to a greedy search heuristic, are used to construct histograms in linearithmic time rather than the polynomial time incurred by previous works. The capabilities of the proposed MDL density estimation method are illustrated with reference to other fully automated methods in the literature, both on synthetic and large real-world data sets.