Conditional Density Estimation Research Articles

Partition Weighted Approach For Estimating the Marginal Posterior Density With Applications

The computation of marginal posterior density in Bayesian analysis is essential in that it can provide complete information about parameters of interest. Furthermore, the marginal posterior density can be used for computing Bayes factors, posterior model probabilities, and diagnostic measures. The conditional marginal density estimator (CMDE) is theoretically the best for marginal density estimation but requires the closed-form expression of the conditional posterior density, which is often not available in many applications. We develop the partition weighted marginal density estimator (PWMDE) to realize the CMDE. This unbiased estimator requires only a single Markov chain Monte Carlo output from the joint posterior distribution and the known unnormalized posterior density. The theoretical properties and various applications of the PWMDE are examined in detail. The PWMDE method is also extended to the estimation of conditional posterior densities. We carry out simulation studies to investigate the empirical performance of the PWMDE and further demonstrate the desirable features of the proposed method with two real data sets from a study of dissociative identity disorder patients and a prostate cancer study, respectively. Supplementary materials for this article are available online.

Post-processing of NWP forecasts using ground or satellite-derived data through kernel conditional density estimation

Global horizontal irradiance (GHI) forecasts by numerical weather prediction (NWP) often contain model-led bias. There is thus a strong consensus on using post-processing techniques, such as model output statistics (MOS), to correct such errors. As opposed to the conventional parametric methods, this article considers a nonparametric approach for post-processing, namely, kernel conditional density estimation (KCDE). Essentially, KCDE constructs a relationship between the bias error (difference between the NWP-based GHI forecast and measurement) and NWP output variables, such as clear-sky index, zenith angle, air temperature, humidity, or surface pressure. Hence, when a new set of explanatory variables becomes available, the conditional expectation of the bias error can be estimated. Since the ground-based GHI measurements are not available everywhere, the possibility of using satellite-derived GHI data to correct NWP forecasts is also explored. In the case study, two years of GHI forecasts made using the North American Mesoscale forecast system are corrected using both ground-measured and satellite-derived GHI references. As compared to Lorenz's fourth-degree polynomial MOS, additional 10%–16% (using ground-measured GHI) and 5%–13% (using satellite-derived GHI) reductions in the forecast error are observed at 7 test stations across the continental United States.

ABC–CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations

ABSTRACTApproximate Bayesian computation (ABC) is typically used when the likelihood is either unavailable or intractable but where data can be simulated under different parameter settings using a forward model. Despite the recent interest in ABC, high-dimensional data and costly simulations still remain a bottleneck in some applications. There is also no consensus as to how to best assess the performance of such methods without knowing the true posterior. We show how a nonparametric conditional density estimation (CDE) framework, which we refer to as ABC–CDE, help address three nontrivial challenges in ABC: (i) how to efficiently estimate the posterior distribution with limited simulations and different types of data, (ii) how to tune and compare the performance of ABC and related methods in estimating the posterior itself, rather than just certain properties of the density, and (iii) how to efficiently choose among a large set of summary statistics based on a CDE surrogate loss. We provide theoretical and empirical evidence that justify ABC–CDE procedures that directly estimate and assess the posterior based on an initial ABC sample, and we describe settings where standard ABC and regression-based approaches are inadequate. Supplemental materials for this article are available online.

A novel probabilistic wind speed prediction approach using real time refined variational model decomposition and conditional kernel density estimation

Short-term wind speed prediction is an important task for the wind energy development. However, due to intermittency and uncertainty of wind resources, it is difficult to be achieved only by the deterministic prediction. Hence, there is a motivation to develop a novel method to further consider the uncertainty. The originality is to develop an innovative hybrid wind speed forecasting model based on empirical mode decomposition, variational mode decomposition, sample entropy and conditional kernel density estimation. More specifically, the original data are decomposed in real time by the refined variational mode decomposition, where the number of decomposition levels is adaptively determined by empirical mode decomposition and sample entropy. Then, conditional kernel density estimation with the bandwidth optimized by normal reference criterion is used to obtain the predictive probability density function for each subseries, by which the expectations and variances are derived. Finally, the deterministic prediction is produced by summarizing these expectations and the probabilistic prediction interval is obtained by the covariance and the Gaussian distribution assumption on target wind speed. It should be emphasized that some hypotheses are adopted in the proposed method: the correlations among different subseries can be characterized by the covariance and the target wind speed follows Gaussian distribution. Two case studies based on the measured data are used to evaluate the performance of the proposed method. The results show the improvement by the proposed method is up to 20% compared with empirical mode decomposition-based method. The main conclusion is that the proposed method may provide the better prediction over other models for the wind speed data with nonstationarity.

The Impact of Body Weight on Occupational Mobility and Career Development

This paper examines the relationship between individuals' weight and their employment decisions over the life cycle. I estimate a dynamic stochastic model of individuals' annual joint decisions of occupation, hours worked, and schooling. The model allows body weight to affect non-monetary costs, switching costs, and distribution of wages for each occupation; and also allows individuals' employment decisions to affect body weight. I use conditional density estimation to formulate the distributions of wages and body weight evolution. I find that heavier individuals face higher switching costs when transitioning into white collar occupations, earn lower returns to experience in white-collar occupations, and earn lower wages in socially intensive jobs. Simulating the model with estimated parameters, decreased occupational mobility accounts for 10 percent of the obesity wage gap. While contemporaneous wage penalties for body weight are small, the cost over the life cycle is substantial. An exogenous increase in initial body mass by 20 percent leads to a 10 percent decrease in wages over the life course.

Regional well-being in the OECD

This paper analyses convergence in well-being across 395 OECD regions in the period 2000–2014 using data from the Regional Well-being Dataset. It is widely known that well-being is a concept that goes far beyond income. However, whereas papers analysing convergence on income abound, the literature considering convergence in well-being is virtually nonexistent, especially at the regional level. Convergence is approached following the distribution dynamics technique and conditional density estimation (CDE), well-established data-driven methods that allow for the assessment of the shape and the time evolution of the kernel distribution of well-being. Moreover, the paper also assesses the role of a set of potential well-being determinants. Results show great disparities across the OECD regions, and no signs of convergence in the studied period. On the contrary, regions polarised into two clubs of low and high well-being, and country level factors explain in a large extent the observed tendencies.

Estimating quantiles in imperfect simulation models using conditional density estimation

In this article, we consider the problem of estimating quantiles related to the outcome of experiments with a technical system given the distribution of the input together with an (imperfect) simulation model of the technical system and (few) data points from the technical system. The distribution of the outcome of the technical system is estimated in a regression model, where the distribution of the residuals is estimated on the basis of a conditional density estimate. It is shown how Monte Carlo can be used to estimate quantiles of the outcome of the technical system on the basis of the above estimates, and the rate of convergence of the quantile estimate is analyzed. Under suitable assumptions, it is shown that this rate of convergence is faster than the rate of convergence of standard estimates which ignore either the (imperfect) simulation model or the data from the technical system; hence, it is crucial to combine both kinds of information. The results are illustrated by applying the estimates to simulated and real data.

Conditional kernel density estimation for some incomplete data models

A class of density estimators based on observed incomplete data are proposed. The method is to use a conditional kernel, defined as the expectation of a given kernel for the complete data conditioning on the observed data, to construct the density estimator. We study such kernel density estimators for several commonly used incomplete data models and establish their basic asymptotic properties. Some characteristics different from the classical kernel estimators are discovered. For instance, the asymptotic results of the proposed estimator do not depend on the choice of the kernel $k(\cdot )$. Simulation study is conducted to evaluate the performance of the estimator and compared with some exising methods.

Reference Health and the Demand for Medical Care

We propose that health in prior periods, termed reference health, is theoretically and empirically relevant to the demand for medical care. To address non‐normality in the distributions of medical care spending, consumption, and health, we use a conditional density estimator nested in a finite mixture framework. We find that reference health can help explain the variation in spending among individuals with the same contemporaneous health, particularly in the top tail of the spending distribution. We demonstrate that omitting reference health understates the potential cost‐savings of healthy aging initiatives by 50%.

Online estimation of discrete, continuous, and conditional joint densities using classifier chains

We address the problem of estimating discrete, continuous, and conditional joint densities online, i.e., the algorithm is only provided the current example and its current estimate for its update. The family of proposed online density estimators, estimation of densities online (EDO), uses classifier chains to model dependencies among features, where each classifier in the chain estimates the probability of one particular feature. Because a single chain may not provide a reliable estimate, we also consider ensembles of classifier chains and ensembles of weighted classifier chains. For all density estimators, we provide consistency proofs and propose algorithms to perform certain inference tasks. The empirical evaluation of the estimators is conducted in several experiments and on datasets of up to several millions of instances. In the discrete case, we compare our estimators to density estimates computed by Bayesian structure learners. In the continuous case, we compare them to a state-of-the-art online density estimator. Our experiments demonstrate that, even though designed to work online, EDO delivers estimators of competitive accuracy compared to other density estimators (batch Bayesian structure learners on discrete datasets and the state-of-the-art online density estimator on continuous datasets). Besides achieving similar performance in these cases, EDO is also able to estimate densities with mixed types of variables, i.e., discrete and continuous random variables.

Imitation of Demonstrations Using Bayesian Filtering With Nonparametric Data-Driven Models

This paper addresses the problem of learning dynamic models of hybrid systems from demonstrations and then the problem of imitation of those demonstrations by using Bayesian filtering. A linear programming-based approach is used to develop nonparametric kernel-based conditional density estimation technique to infer accurate and concise dynamic models of system evolution from data. The training data for these models have been acquired from demonstrations by teleoperation. The trained data-driven models for mode-dependent state evolution and state-dependent mode evolution are then used online for imitation of demonstrated tasks via particle filtering. The results of simulation and experimental validation with a hexapod robot are reported to establish generalization of the proposed learning and control algorithms.

Dynamic full Bayesian ensemble classifiers for small time series

Improving the reliability of small time series classifiers with continuous attributes is an important and challenging task. The information contained in small time series is not sufficient and a temporal dependency exists between data records, which makes it very difficult to optimize the fitting degree between the classifier and the data, and many mature techniques of non-time series data classifiers are not practical. We use a dynamic full Bayesian classifier to increase the amount of information provided by the attribute to the class, and realize the fusion of temporal and nonsequential information. By combining the conditional joint density estimation of attributes based on the multivariate Gaussian kernel function with a diagonal smoothing parameter matrix, the interval division of smoothing parameter values, the timing progressive classification accuracy criterion, the construction of the smoothing parameter configuration tree, classifier selection and averaging, etc, a dynamic full Bayesian ensemble classifier was established for small time series. Experiments were performed using small time series in macroeconomic analysis. The results show that the optimized dynamic full Bayesian ensemble classifiers have very good classification accuracy.

Household electricity demand forecasting using adaptive conditional density estimation

Large-scale deployment of advanced smart grid technologies bolsters load forecasting as a vital requirement for deregulated power systems. In this regard, providing an accurate short-term load forecasting (STLF) can facilitate demand response applications and real-time electricity dispatch. STLF is mainly influenced by meteorological conditions among which investigating the relationship between temperature and household total electricity consumption is notably important due to their strong correlation. Accordingly, in this paper, we estimate the total electricity consumption to explore the impact of temperature in terms of a non-linear relationship with electricity demand. We propose the adaptive conditional density estimation (ACDE) method on the basis of kernel density estimation (KDE) to enhance the load forecast accuracy. The aim of the suggested approach is to decompose and examine the mentioned relationship in the context of both temperature-related, and residual components of the total consumption. The performance of the model to forecast the electricity demand is evaluated using a comparison study. The results prove that an ACDE model can significantly improve the recognition capability of the temperature-related component of aggregated power. Finally, the efficacy of the ACDE method is examined via numerical analysis of real data.

Explanation of Variability and Removal of Confounding Factors from Data through Optimal Transport

A methodology based on the theory of optimal transport is developed to attribute variability in data sets to known and unknown factors and to remove such attributable components of the variability from the data. Denoting by x the quantities of interest and by z the explanatory factors, the procedure transforms x into filtered variables y through a z‐dependent map, so that the conditional probability distributions ρ(x|z) are pushed forward into a target distribution μ(y), independent of z. Among all maps and target distributions that achieve this goal, the procedure selects the one that minimally distorts the original data: the barycenter of the ρ(x|z). Connections are found to unsupervised learning and to fundamental problems in statistics such as conditional density estimation and sampling. Particularly simple instances of the methodology are shown to be equivalent to k‐means and principal component analysis. An application is shown to a time series of ground temperature hourly data across the United States.© 2017 Wiley Periodicals, Inc.

Process monitoring using probabilistic graphical models via nonparametric density estimation

Probabilistic graphical models like Bayesian networks have been widely used in process monitoring and fault diagnosis, however, their application is mostly limited to discrete variables or continuous Gaussian variables due to the difficult in estimation of multivariate joint density. In order to deal with the estimation problem of multivariate joint density for continuous variables, this paper decomposes the graphical model into hierarchical structure so that the problem of joint density estimation can be transferred to estimation of several conditional probability densities and low-dimensional probability densities. The conditional densities can be effectively estimated from data by a nonparametric kernel method and the low-dimensional densities can be estimated using the kernel density estimation (KDE). Based on the estimated densities, process faults can be detected by examining which probability is lower than the cutoff value. Application to the blast furnace ironmaking process is used to illustrate the advantages of the proposed method.

Photo-$z$ estimation: An example of nonparametric conditional density estimation under selection bias

Redshift is a key quantity for inferring cosmological model parameters. In photometric redshift estimation, cosmologists use the coarse data collected from the vast majority of galaxies to predict the redshift of individual galaxies. To properly quantify the uncertainty in the predictions, however, one needs to go beyond standard regression and instead estimate the full conditional density $f(z|\mathbf{x})$ of a galaxy’s redshift $z$ given its photometric covariates $\mathbf{x}$. The problem is further complicated by selection bias: usually only the rarest and brightest galaxies have known redshifts, and these galaxies have characteristics and measured covariates that do not necessarily match those of more numerous and dimmer galaxies of unknown redshift. Unfortunately, there is not much research on how to best estimate complex multivariate densities in such settings. Here we describe a general framework for properly constructing and assessing nonparametric conditional density estimators under selection bias, and for combining two or more estimators for optimal performance. We propose new improved photo-$z$ estimators and illustrate our methods on data from the Sloan Data Sky Survey and an application to galaxy–galaxy lensing. Although our main application is photo-$z$ estimation, our methods are relevant to any high-dimensional regression setting with complicated asymmetric and multimodal distributions in the response variable.

Hybrid methodology for modeling short-term wind power generation using conditional Kernel density estimation and singular spectrum analysis

Una parte fundamental del proceso de previsión probabilística de energía eólica es tener en cuenta las previsiones de la velocidad del viento. Para obtener pronósticos probabilísticos precisos de la producción eólica ha sido desarrollada una metodología híbrida utilizando técnicas no paramétricas conocidas como SSA (Análisis Singular Espectral) y Estimación Condicional de la Densidad por Kernel (CKDE). SSA es empleada para predecir la velocidad del viento y CKDE para obtener previsiones probabilísticas de la energía eólica, dado que la generación de la energía eólica tiene una relación no lineal con la velocidad del viento y ambas son variables aleatorias distribuidas que siguen una función de densidad conjunta. Haciendo uso de una base de datos brasilera horaria que incluye la velocidad del viento y la energía eólica es ilustrada dicha metodología. Una vez que las previsiones de la velocidad del viento son obtenidas, los correspondientes pronósticos probabilísticos de la generación de energía eólica son estimados para un horizonte de 24 horas. Los resultados obtenidos son comparados con otras metodologías existentes.

A unified framework for constructing, tuning and assessing photometric redshift density estimates in a selection bias setting.

Photometric redshift estimation is an indispensable tool of precision cosmology. One problem that plagues the use of this tool in the era of large-scale sky surveys is that the bright galaxies that are selected for spectroscopic observation do not have properties that match those of (far more numerous) dimmer galaxies; thus, ill-designed empirical methods that produce accurate and precise redshift estimates for the former generally will not produce good estimates for the latter. In this paper, we provide a principled framework for generating conditional density estimates (i.e. photometric redshift PDFs) that takes into account selection bias and the covariate shift that this bias induces. We base our approach on the assumption that the probability that astronomers label a galaxy (i.e. determine its spectroscopic redshift) depends only on its measured (photometric and perhaps other) properties n}{}boldsymbol {x} and not on its true redshift. With this assumption, we can explicitly write down risk functions that allow us to both tune and compare methods for estimating importance weights (i.e. the ratio of densities of unlabelled and labelled galaxies for different values of n}{}boldsymbol {x}) and conditional densities. We also provide a method for combining multiple conditional density estimates for the same galaxy into a single estimate with better properties. We apply our risk functions to an analysis of ≈106 galaxies, mostly observed by Sloan Digital Sky Survey, and demonstrate through multiple diagnostic tests that our method achieves good conditional density estimates for the unlabelled galaxies.

Conditional density estimation using the local Gaussian correlation

Let $$\mathbf {X} = (X_1,\ldots ,X_p)$$X=(X1,ź,Xp) be a stochastic vector having joint density function $$f_{\mathbf {X}}(\mathbf {x})$$fX(x) with partitions $$\mathbf {X}_1 = (X_1,\ldots ,X_k)$$X1=(X1,ź,Xk) and $$\mathbf {X}_2 = (X_{k+1},\ldots ,X_p)$$X2=(Xk+1,ź,Xp). A new method for estimating the conditional density function of $$\mathbf {X}_1$$X1 given $$\mathbf {X}_2$$X2 is presented. It is based on locally Gaussian approximations, but simplified in order to tackle the curse of dimensionality in multivariate applications, where both response and explanatory variables can be vectors. We compare our method to some available competitors, and the error of approximation is shown to be small in a series of examples using real and simulated data, and the estimator is shown to be particularly robust against noise caused by independent variables. We also present examples of practical applications of our conditional density estimator in the analysis of time series. Typical values for k in our examples are 1 and 2, and we include simulation experiments with values of p up to 6. Large sample theory is established under a strong mixing condition.

Smoothed kernel conditional density estimation

Given multiple Y observations for each observed X, we propose a conditional kernel density estimator that exploits smoothing of f(y|x) across x. We obtain large sample properties of the proposed estimator and present a practical cross validation bandwidth selector. An application to adult BMI densities conditional on age is provided.