Nonparametric Density Estimation Research Articles

Dependencies Between Histogram Parameters and the Kernel Estimate of the Probability Density of a Multidimensional Random Variable

The dependencies between the sampling intervals of the domain of values of a multidimensional random variable and the blur coefficients of the kernel probability density estimate are determined. The results of an analysis of the asymptotic properties of a multidimensional nonparametric estimate of the probability density of the Rosenblatt–Parzen type and its modifications are investigated. Modification of the kernel probability density estimate is a smoothed multi-dimensional histogram. The formulas for calculating the optimal parameters of the kernel function blur coefficients and the lengths of the sampling intervals of the values of the components of a multidimensional random variable are analyzed. Blur coefficients are presented as the products of an indefinite parameter and the mean square deviations of random components. An indefinite parameter and the number of sampling intervals of multidimensional random variables are found from the condition of minimum mean square deviations of the considered probability density estimates. Based on the results obtained, the relationships between the blur coefficient parameters and the discretization procedure of the range of values of a multidimensional random variable are established. The discovered patterns are characterized by a set of constants. The constants under consideration are represented by the ratio of nonlinear functionals of the probability density and their second derivatives with respect to each random component. The values of the constants are characterized by the type of probability density and are independent of their parameters. From computational experimental data, we establish the relationships between the constants and the product of the counterexcess coefficients with the components of a multidimensional random variable. The results obtained make it possible to quickly determine the lengths of sampling intervals of the values of the components of a multidimensional random variable from the kernel function blur coefficients of a nonparametric probability density estimate.

Multi-index summer flow regime characterization to inform environmental flow contexts: A New England case study

Abstract Summer flow regime is an important determinant of the health and integrity of riverine ecosystems. In particular, summertime low flows and the range of variability impact the extent and connectivity of habitat, production in and composition of ecosystems, as well as water quality. Four streamflow-based metrics are used in the analysis and assessment of summertime hydrologic variability and trends: Lane Index, UK Institute of Hydrology Index, Base Flow Index, and 7-day Minimum Flow Index. In this study, historical streamflow records at 33 locations in New England (USA) were analyzed to understand the nature of trends as well as the extent of changes, individually and collectively, for the four indices. The main contribution of this paper is to develop a comprehensive characterization of the regional flow regime and trends therein. Two complementary statistical analyses applied in this study are: (a) quantile regression-based estimates of the spatial and temporal patterns of changes in flow indices variability over the past six decades and (b) interrelationship between flow indices based on non-parametric probability density estimation approach. The multi-index regional analysis revealed that flow variability (based on the Lane Index) within the summer season has increased significantly, with 25 out of the 33 streamgages showing increases, of which 12 are highly significant. Similar trends are mirrored in the UK Institute of Hydrology Index based analysis. Base Flow Index trends show decreases across 30 of the 33 streamgages at the median level, and 19 are highly significant. Trends are most pronounced at the upper quartile level, with 21 highly significant. Regional analysis of 7-day Minimum Flow Index reveal broad patterns towards increases at the three selected quantile levels ( τ = 0.25 , 0.5 , 0.75 ) with 29, 22, and 19 streamgages showing increases, of which 13, 10, and 6 are highly significant respectively. There is also diversity in trends across quantiles, wherein 17 locations show increases at all quantile levels, while 22 stations showed increases at the median level. The analysis of interrelationships between the four indices revealed inverse variations between Lane Index and 7-day Minimum Flow, and also a diversity of relationships ranging from weakly to strongly nonlinear. As such, our analysis underscores the importance of the use of multiple indices to fully assess variability and change of relevance to management and policy concerns, such as Maine’s water allocation rulemaking.

Probabilistic optimal power flow of AC/DC system with multiport current flow controller

To evaluate the impact of the randomness and correlation of photovoltaic (PV) and load on AC/DC systems with a multiport current flow controller (M-CFC), this paper proposes a probabilistic optimal power flow calculation for AC/DC systems based on a nonparametric kernel density estimation. First, according to the M-CFC model, the DC power flow calculation method with M-CFC was deduced, and its influence on line loss was analyzed. Second, a nonparametric kernel density estimation with an adaptive bandwidth was used to accurately describe the probability distribution of the PV and load, and correlation samples of the PV and load were obtained by the mixed copula function. Then an optimization model that considers system loss and static security was established, and a fast nondominated sorting genetic algorithm based on the elite strategy (NSGA-II) was used to calculate the multi-objective probability optimal power flow of the AC/DC system. Finally, a case study was performed on the modified IEEE39 bus system using measured PV and load data. We verified the nonparametric kernel density estimation with an adaptive bandwidth can better adapt to random component uncertainty, and M-CFC can improve the static security of the system.

Modal clustering asymptotics with applications to bandwidth selection

Density-based clustering relies on the idea of linking groups to some specific features of the probability distribution underlying the data. The reference to a true, yet unknown, population structure allows framing the clustering problem in a standard inferential setting, where the concept of ideal population clustering is defined as the partition induced by the true density function. The nonparametric formulation of this approach, known as modal clustering, draws a correspondence between the groups and the domains of attraction of the density modes. Operationally, a nonparametric density estimate is required and a proper selection of the amount of smoothing, governing the shape of the density and hence possibly the modal structure, is crucial to identify the final partition. In this work, we address the issue of density estimation for modal clustering from an asymptotic perspective. A natural and easy to interpret metric to measure the distance between density-based partitions is discussed, its asymptotic approximation explored, and employed to study the problem of bandwidth selection for nonparametric modal clustering.

Nonparametric estimation of the ability density in the Mixed-Effect Rasch Model

The Rasch model is widely used in the field of psychometrics when $n$ persons under test answer $m$ questions and the score, which describes the correctness of the answers, is given by a binary $n\times m$-matrix. We consider the Mixed-Effect Rasch Model, in which the persons are chosen randomly from a huge population. The goal is to estimate the ability density of this population under nonparametric constraints, which turns out to be a statistical linear inverse problem with an unknown but estimable operator. Based on our previous result on asymptotic equivalence to a two-layer Gaussian model, we construct an estimation procedure and study its asymptotic optimality properties as $n$ tends to infinity, as does $m$, but moderately with respect to $n$. Moreover numerical simulations are provided.

Posterior contraction and credible sets for filaments of regression functions

A filament consists of local maximizers of a smooth function $f$ when moving in a certain direction. A filamentary structure is an important feature of the shape of an object and is also considered as an important lower dimensional characterization of multivariate data. There have been some recent theoretical studies of filaments in the nonparametric kernel density estimation context. This paper supplements the current literature in two ways. First, we provide a Bayesian approach to the filament estimation in regression context and study the posterior contraction rates using a finite random series of B-splines basis. Compared with the kernel-estimation method, this has a theoretical advantage as the bias can be better controlled when the function is smoother, which allows obtaining better rates. Assuming that $f:\mathbb{R}^{2}\mapsto \mathbb{R}$ belongs to an isotropic Hölder class of order $\alpha \geq 4$, with the optimal choice of smoothing parameters, the posterior contraction rates for the filament points on some appropriately defined integral curves and for the Hausdorff distance of the filament are both $(n/\log n)^{(2-\alpha )/(2(1+\alpha ))}$. Secondly, we provide a way to construct a credible set with sufficient frequentist coverage for the filaments. We demonstrate the success of our proposed method in simulations and one application to earthquake data.

Conditional density estimation tools in python and R with applications to photometric redshifts and likelihood-free cosmological inference

It is well known in astronomy that propagating non-Gaussian prediction uncertainty in photometric redshift estimates is key to reducing bias in downstream cosmological analyses. Similarly, likelihood-free inference approaches, which are beginning to emerge as a tool for cosmological analysis, require a characterization of the full uncertainty landscape of the parameters of interest given observed data. However, most machine learning (ML) or training-based methods with open-source software target point prediction or classification, and hence fall short in quantifying uncertainty in complex regression and parameter inference settings. As an alternative to methods that focus on predicting the response (or parameters) $\mathbf{y}$ from features $\mathbf{x}$, we provide nonparametric conditional density estimation (CDE) tools for approximating and validating the entire probability density function (PDF) $\mathrm{p}(\mathbf{y}|\mathbf{x})$ of $\mathbf{y}$ given (i.e., conditional on) $\mathbf{x}$. As there is no one-size-fits-all CDE method, the goal of this work is to provide a comprehensive range of statistical tools and open-source software for nonparametric CDE and method assessment which can accommodate different types of settings and be easily fit to the problem at hand. Specifically, we introduce four CDE software packages in $\texttt{Python}$ and $\texttt{R}$ based on ML prediction methods adapted and optimized for CDE: $\texttt{NNKCDE}$, $\texttt{RFCDE}$, $\texttt{FlexCode}$, and $\texttt{DeepCDE}$. Furthermore, we present the $\texttt{cdetools}$ package, which includes functions for computing a CDE loss function for tuning and assessing the quality of individual PDFs, along with diagnostic functions. We provide sample code in $\texttt{Python}$ and $\texttt{R}$ as well as examples of applications to photometric redshift estimation and likelihood-free cosmological inference via CDE.

Estimating linear response statistics using orthogonal polynomials: An rkhs formulation

We study the problem of estimating linear response statistics under external perturbations using time series of unperturbed dynamics. Based on the fluctuation-dissipation theory, this problem is reformulated as an unsupervised learning task of estimating a density function. We consider a nonparametric density estimator formulated by the kernel embedding of distributions with kernels, constructed based on the classical orthogonal polynomials defined on non-compact domains. While the resulting representation is analogous to Polynomial Chaos Expansion (PCE), the connection to the reproducing kernel Hilbert space (RKHS) theory allows one to establish the uniform convergence of the estimator and to systematically address a practical question of identifying the PCE basis for a consistent estimation. We also provide practical conditions for the well-posedness of not only the estimator but also of the underlying response statistics. Finally, we provide a statistical error bound for the density estimation that accounts for the Monte-Carlo averaging over non-i.i.d time series and the biases due to a finite basis truncation. This error bound provides a means to understand the feasibility as well as limitation of the kernel embedding with Mercer-type kernels. Numerically, we verify the effectiveness of the estimator on two stochastic dynamics with known, yet, non-trivial equilibrium densities.

Density Forecasts in Panel Data Models: A Semiparametric Bayesian Perspective

This paper constructs individual-specific density forecasts for a panel of firms or households using a dynamic linear model with common and heterogeneous coefficients and cross-sectional heteroskedasticity. The panel considered in this paper features a large cross-sectional dimension N but short time series T. Due to the short T, traditional methods have difficulty in disentangling the heterogeneous parameters from the shocks, which contaminates the estimates of the heterogeneous parameters. To tackle this problem, I assume that there is an underlying distribution of heterogeneous parameters, model this distribution nonparametrically allowing for correlation between heterogeneous parameters and initial conditions as well as individual-specific regressors, and then estimate this distribution by pooling the information from the whole cross-section together. Theoretically, I prove that both the estimated common parameters and the estimated distribution of the heterogeneous parameters achieve posterior consistency, and that the density forecasts asymptotically converge to the oracle forecast. Methodologically, I develop a simulation-based posterior sampling algorithm specifically addressing the nonparametric density estimation of unobserved heterogeneous parameters. Monte Carlo simulations and an application to young firm dynamics demonstrate improvements in density forecasts relative to alternative approaches.

Gaussianization Machines for Non-Gaussian Function Estimation Models

A wide range of nonparametric function estimation models have been studied individually in the literature. Among them the homoscedastic nonparametric Gaussian regression is arguably the best known and understood. Inspired by the asymptotic equivalence theory, Brown, Cai and Zhou (Ann. Statist. 36 (2008) 2055–2084; Ann. Statist. 38 (2010) 2005–2046) and Brown et al. (Probab. Theory Related Fields 146 (2010) 401–433) developed a unified approach to turn a collection of non-Gaussian function estimation models into a standard Gaussian regression and any good Gaussian nonparametric regression method can then be used. These Gaussianization Machines have two key components, binning and transformation. When combined with BlockJS, a wavelet thresholding procedure for Gaussian regression, the procedures are computationally efficient with strong theoretical guarantees. Technical analysis given in Brown, Cai and Zhou (Ann. Statist. 36 (2008) 2055–2084; Ann. Statist. 38 (2010) 2005–2046) and Brown et al. (Probab. Theory Related Fields 146 (2010) 401–433) shows that the estimators attain the optimal rate of convergence adaptively over a large set of Besov spaces and across a collection of non-Gaussian function estimation models, including robust nonparametric regression, density estimation, and nonparametric regression in exponential families. The estimators are also spatially adaptive. The Gaussianization Machines significantly extend the flexibility and scope of the theories and methodologies originally developed for the conventional nonparametric Gaussian regression. This article aims to provide a concise account of the Gaussianization Machines developed in Brown, Cai and Zhou (Ann. Statist. 36 (2008) 2055–2084; Ann. Statist. 38 (2010) 2005–2046), Brown et al. (Probab. Theory Related Fields 146 (2010) 401–433).

A note on the behaviour of a kernel-smoothed kernel density estimator

Kernel density estimators have been studied in great detail. In this note a new family of kernels, depending on a parameter c, is obtained by kernel-smoothing an initial kernel density estimator. Under certain conditions, we show that nonparametric density estimators based on such kernels outperform the initial estimator in terms of minimized asymptotic mean integrated squared error and in kernel efficiency.

Assessing the impact of FDI on PM2.5 concentrations: A nonlinear panel data analysis for emerging economies

While emerging markets have obtained powerful growth for foreign direct investment (FDI) inflows, they are facing severe smog pollution, and this contradiction has become increasingly prominent since the financial crisis. Assessing the influence of FDI on pollutant emissions is of great significance for determining how to attract FDI to promote environmental sustainability. The present study simultaneously investigates the direct and indirect effects of FDI on PM2.5 contamination for emerging countries spanning the period 2010–2016. Due to the features of the nonlinear analysis, a generalized panel smooth transition regression (GPSTR) model was introduced, and cross-sectional dependence, heterogeneity, nonlinear unit root, nonlinear cointegration tests and non-parametric kernel density estimation were applied to achieve this goal. The results reveal that FDI directly contributes to decreasing PM2.5, but indirectly has on increasing PM2.5 emissions. The total effect of FDI on PM2.5 concentrations is proven to be negative, which confirms the pollution halo hypothesis. Moreover, the connection between FDI inflows and PM2.5 emissions displays a threshold and dynamic characteristic and is “S-shaped”. At lower levels of FDI, the inflows of FDI exert a positive effect on reducing PM2.5 concentrations, whereas when FDI exceeds the threshold of 23.2981, such influence is gradually weakened with an increase in its own accumulation. The study provides new assessments on FDI's contribution to pollutant emissions and evidence for environmental sustainability in the post-financial crisis era.

A Survey of Nonparametric Mixing Density Estimation via the Predictive Recursion Algorithm

Nonparametric estimation of a mixing density based on observations from the corresponding mixture is a challenging statistical problem. This paper surveys the literature on a fast, recursive estimator based on the predictive recursion algorithm. After introducing the algorithm and giving a few examples, I summarize the available asymptotic convergence theory, describe an important semiparametric extension, and highlight two interesting applications. I conclude with a discussion of several recent developments in this area and some open problems.

Fused Density Estimation: Theory and Methods

Summary We introduce a method for non-parametric density estimation on geometric networks. We define fused density estimators as solutions to a total variation regularized maximum likelihood density estimation problem. We provide theoretical support for fused density estimation by proving that the squared Hellinger rate of convergence for the estimator achieves the minimax bound over univariate densities of log-bounded variation. We reduce the original variational formulation to transform it into a tractable, finite dimensional quadratic program. Because random variables on geometric networks are simple generalizations of the univariate case, this method also provides a useful tool for univariate density estimation. Lastly, we apply this method and assess its performance on examples in the univariate and geometric network setting. We compare the performance of various optimization techniques to solve the problem and use these results to inform recommendations for the computation of fused density estimators.

Nonparametric Density Estimation Using Copula Transform, Bayesian Sequential Partitioning, and Diffusion-Based Kernel Estimator

Non-parametric density estimation methods are more flexible than parametric methods, due to the fact that they do not assume any specific shape or structure for the data. Most non-parametric methods, like Kernel estimation, require tuning of parameters to achieve good data smoothing, a non-trivial task, even in low dimensions. In higher dimensions, sparsity of data in local neighborhoods becomes a challenge even for non-parametric methods. In this paper, we use the copula transform and two efficient non-parametric methods to develop a new method for improved non-parametric density estimation in multivariate domain. After separation of marginal and joint densities using copula transform, a diffusion-based kernel estimator is employed to estimate the marginals. Next, Bayesian sequential partitioning (BSP) is used in the joint density estimation.

Optimal data-based binning for histograms and histogram-based probability density models

Histograms are convenient non-parametric density estimators, which continue to be used ubiquitously. Summary quantities estimated from histogram-based probability density models depend on the choice of the number of bins. We introduce a straightforward data-based method of determining the optimal number of bins in a uniform bin-width histogram. By assigning a multinomial likelihood and a non-informative prior, we derive the posterior probability for the number of bins in a piecewise-constant density model given the data. In addition, we estimate the mean and standard deviations of the resulting bin heights, examine the effects of small sample sizes and digitized data, and demonstrate the application to multi-dimensional histograms.

Bayesian nonparametric modelling of the link function in the single-index model using a Bernstein–Dirichlet process prior

ABSTRACTThis paper proposes the use of the Bernstein–Dirichlet process prior for a new nonparametric approach to estimating the link function in the single-index model (SIM). The Bernstein–Dirichlet process prior has so far mainly been used for nonparametric density estimation. Here we modify this approach to allow for an approximation of the unknown link function. Instead of the usual Gaussian distribution, the error term is assumed to be asymmetric Laplace distributed which increases the flexibility and robustness of the SIM. To automatically identify truly active predictors, spike-and-slab priors are used for Bayesian variable selection. Posterior computations are performed via a Metropolis-Hastings-within-Gibbs sampler using a truncation-based algorithm for stick-breaking priors. We compare the efficiency of the proposed approach with well-established techniques in an extensive simulation study and illustrate its practical performance by an application to nonparametric modelling of the power consumption in a sewage treatment plant.

Wind speed probability density estimation using root-transformed local linear regression

Despite the many attractive features of wind-based renewables, the intermittency of wind generation stands out as a critical challenge. An accurate and reliable nonparametric wind speed probability density model is proposed in this paper based on the application of local linear regression in tandem with a root transformation method, introduced here for the first time. The proposed root-transformed local linear regression approach provides a robust estimate of wind speed distribution and produces more accurate results than kernel density estimation models. The performance of the root-transformed local linear regression estimator is assessed via comparisons with three popular parametric models (Rayleigh, Weibull and Gaussian distributions), two additional parametric models (Birnbaum–Saunders and Nakagami distributions) recently suggested for wind speed probability density estimation, and two nonparametric kernel density estimation models using two standard goodness-of-fit hypothesis tests (Chi-squared and Kolmogorov–Smirnov), coefficient of determination and two error metrics (root mean square error and mean absolute error). Results confirm the accuracy of the proposed root-transformed local linear regression estimator for modelling wind speed probability density, with substantial improvements in all error metrics over parametric and kernel density estimation models.

Reduced bias nonparametric lifetime density and hazard estimation.

Kernel-based nonparametric hazard rate estimation is considered with a special class of infinite-order kernels that achieves favorable bias and mean square error properties. A fully automatic and adaptive implementation of a density and hazard rate estimator is proposed for randomly right censored data. Careful selection of the bandwidth in the proposed estimators yields estimates that are more efficient in terms of overall mean squared error performance, and in some cases achieves a nearly parametric convergence rate. Additionally, rapidly converging bandwidth estimates are presented for use in second-order kernels to supplement such kernel-based methods in hazard rate estimation. Simulations illustrate the improved accuracy of the proposed estimator against other nonparametric estimators of the density and hazard function. A real data application is also presented on survival data from 13,166 breast carcinoma patients.

Nonparametric estimation of probabilistic sensitivity measures

Computer experiments are becoming increasingly important in scientific investigations. In the presence of uncertainty, analysts employ probabilistic sensitivity methods to identify the key-drivers of change in the quantities of interest. Simulation complexity, large dimensionality and long running times may force analysts to make statistical inference at small sample sizes. Methods designed to estimate probabilistic sensitivity measures at relatively low computational costs are attracting increasing interest. We first, propose new estimators based on a one-sample design and building on the idea of placing piecewise constant Bayesian priors on the conditional distributions of the output given each input, after partitioning the input space. We then present two alternatives, based on Bayesian non-parametric density estimation, which bypass the need for predefined partitions. Quantification of uncertainty in the estimation process through is possible without requiring additional simulator evaluations via Bootstrap in the simplest proposal, or from the posterior distribution over the sensitivity measures, when the entire inferential procedure is Bayesian. The performance of the proposed methods is compared to that of traditional point estimators in a series of numerical experiments comprising synthetic but challenging simulators, as well as a realistic application.