Conditional Density Estimation Research Articles

Two generalized nonparametric methods for estimating like densities

This article presents two generalized nonparametric methods for estimating multiple, possibly like, densities. The first generalization contains the Nadaraya–Watson estimator, the Jones et al. (Biometrika 82(2):327–338, 1995) bias reduction estimator, and Ker (Stat Probab Lett 117:23–30, 2016) possibly similar estimator as special cases. The second generalization contains the Nadaraya–Watson estimator, Ker (2016) possibly similar estimator, and the conditional density estimator of Hall et al. (J Am Stat Assoc 99(468):1015–1026, 2004) as special cases. The generalizations do not require knowledge of the form or extent of likeness between the unknown densities; an attractive feature in empirical applications. Numerical simulations demonstrate that the two proposed generalizations lead to significant efficiency gains.

Going deep: models for continuous-time within-play valuation of game outcomes in American football with tracking data

AbstractContinuous-time assessments of game outcomes in sports have become increasingly common in the last decade. In American football, only discrete-time estimates of play value were possible, since the most advanced public football datasets were recorded at the play-by-play level. While measures such as expected points and win probability are useful for evaluating football plays and game situations, there has been no research into how these values change throughout the course of a play. In this work, we make two main contributions: First, we introduce a general framework for continuous-time within-play valuation in the National Football League using player-tracking data. Our modular framework incorporates several modular sub-models, to easily incorporate recent work involving player tracking data in football. Second, we use a long short-term memory recurrent neural network to construct a ball-carrier model to estimate how many yards the ball-carrier is expected to gain from their current position, conditional on the locations and trajectories of the ball-carrier, their teammates and opponents. Additionally, we demonstrate an extension with conditional density estimation so that the expectation of any measure of play value can be calculated in continuous-time, which was never before possible at such a granular level.

Predictive Density Aggregation

In this paper we propose a novel approach to obtain the predictive density of global GDP growth. It hinges upon a bottom-up probabilistic model that estimates and combines single countries’ predictive GDP growth densities, taking into account cross-country interdependencies. Speci?cally, we model non-parametrically the contemporaneous interdependencies across the United States, the euro area, and China via a conditional kernel density estimation of a joint distribution. Then, we characterize the potential ampli?cation e?ects stemming from other large economies in each region—also with kernel density estimations—and the reaction of all other economies with para-metric assumptions. Importantly, each economy’s predictive density also depends on a set of observable country-speci?c factors. Finally, the use of sampling techniques allows us to aggregate individual countries’ densities into a world aggregate while preserving the non-i.i.d. nature of the global GDP growth distribution. Out-of-sample metrics con?rm the accuracy of our approach.

Forecasting Flu Activity in the United States: Benchmarking an Endemic-Epidemic Beta Model.

Accurate prediction of flu activity enables health officials to plan disease prevention and allocate treatment resources. A promising forecasting approach is to adapt the well-established endemic-epidemic modeling framework to time series of infectious disease proportions. Using U.S. influenza-like illness surveillance data over 18 seasons, we assessed probabilistic forecasts of this new beta autoregressive model with proper scoring rules. Other readily available forecasting tools were used for comparison, including Prophet, (S)ARIMA and kernel conditional density estimation (KCDE). Short-term flu activity was equally well predicted up to four weeks ahead by the beta model with four autoregressive lags and by KCDE; however, the beta model runs much faster. Non-dynamic Prophet scored worst. Relative performance differed for seasonal peak prediction. Prophet produced the best peak intensity forecasts in seasons with standard epidemic curves; otherwise, KCDE outperformed all other methods. Peak timing was best predicted by SARIMA, KCDE or the beta model, depending on the season. The best overall performance when predicting peak timing and intensity was achieved by KCDE. Only KCDE and naive historical forecasts consistently outperformed the equal-bin reference approach for all test seasons. We conclude that the endemic-epidemic beta model is a performant and easy-to-implement tool to forecast flu activity a few weeks ahead. Real-time forecasting of the seasonal peak, however, should consider outputs of multiple models simultaneously, weighing their usefulness as the season progresses.

Probabilistic forecasts of time and energy flexibility in battery electric vehicle charging

Users charging the batteries of their electric vehicles in an uncoordinated manner can present energy systems with a challenge. One possible solution, smart charging, relies on the flexibility within each charging process and controls the charging process to optimize different objectives. Effective smart charging requires forecasts of energy requirements and parking duration at the charging station for each individual charging process. We use data from travel logs to create quantile forecasts of parking duration and energy requirements, approximated by upcoming trip distance. For this task, we apply quantile regression, multi-layer perceptrons with tilted loss function, and multivariate conditional kernel density estimators. The out-of-sample evaluation shows that the use of local information from the vehicle's travel data improves the forecasting accuracy by 13.7% for parking duration and 0.56% for trip distance compared to the data generated at the charging stations. In addition, the analysis of a case study shows that using probabilistic forecasts can control the interruption of charging processes more efficiently compared to point forecasts. Probabilistic forecasting leads up to 7.0% less interruptions, which can cause a restriction in drivers' mobility demand. The results show that charging station operators benefit from leveraging the driving patterns of electric vehicles. Thereby, smart charging and the application of the proposed models as benchmarks models for the related forecasting tasks is an improvement for the operators.

Frequency analysis of precipitation extremes under a changing climate: a case study in Heihe River basin, China

Abstract The stationary assumption for the traditional frequency analysis of precipitation extremes has been challenged due to natural climate variability or human intervention. To overcome this challenge, this paper, taking Heihe River basin as the case study, performed the frequency analysis by developing a nonstationary GEV model for those seasonal maximum daily precipitation (SMP) time series with nonstationary characteristics by employing the GEV conditional density estimation network. In addition, the confidence intervals (CIs) of estimated return levels were also investigated by using the residual bootstrap technique. Results showed that, 7 of 12 SMP series were nonstationary. The parameters in the nonstationary model were specified as functions of time varying or correlated climate indices varying covariates. The frequency analysis showed that the return levels varied linearly or nonlinearly with covariates. Precipitation extremes with the same magnitude in the study area were found to be occurring more frequently in the future. The CIs of such return levels increased with time passing, especially those from the more complex GEV11 model, embedding a nonlinear increasing trend in model scale parameters. It implied that the increase of model complexity is likely to result in the increase of uncertainty in estimates.

Conditional density estimation and simulation through optimal transport

A methodology to estimate from samples the probability density of a random variable x conditional to the values of a set of covariates $$\{z_{l}\}$$ is proposed. The methodology relies on a data-driven formulation of the Wasserstein barycenter, posed as a minimax problem in terms of the conditional map carrying each sample point to the barycenter and a potential characterizing the inverse of this map. This minimax problem is solved through the alternation of a flow developing the map in time and the maximization of the potential through an alternate projection procedure. The dependence on the covariates $$\{z_{l}\}$$ is formulated in terms of convex combinations, so that it can be applied to variables of nearly any type, including real, categorical and distributional. The methodology is illustrated through numerical examples on synthetic and real data. The real-world example chosen is meteorological, forecasting the temperature distribution at a given location as a function of time, and estimating the joint distribution at a location of the highest and lowest daily temperatures as a function of the date.

Conditional density estimation tools in python and R with applications to photometric redshifts and likelihood-free cosmological inference

It is well known in astronomy that propagating non-Gaussian prediction uncertainty in photometric redshift estimates is key to reducing bias in downstream cosmological analyses. Similarly, likelihood-free inference approaches, which are beginning to emerge as a tool for cosmological analysis, require a characterization of the full uncertainty landscape of the parameters of interest given observed data. However, most machine learning (ML) or training-based methods with open-source software target point prediction or classification, and hence fall short in quantifying uncertainty in complex regression and parameter inference settings. As an alternative to methods that focus on predicting the response (or parameters) $\mathbf{y}$ from features $\mathbf{x}$, we provide nonparametric conditional density estimation (CDE) tools for approximating and validating the entire probability density function (PDF) $\mathrm{p}(\mathbf{y}|\mathbf{x})$ of $\mathbf{y}$ given (i.e., conditional on) $\mathbf{x}$. As there is no one-size-fits-all CDE method, the goal of this work is to provide a comprehensive range of statistical tools and open-source software for nonparametric CDE and method assessment which can accommodate different types of settings and be easily fit to the problem at hand. Specifically, we introduce four CDE software packages in $\texttt{Python}$ and $\texttt{R}$ based on ML prediction methods adapted and optimized for CDE: $\texttt{NNKCDE}$, $\texttt{RFCDE}$, $\texttt{FlexCode}$, and $\texttt{DeepCDE}$. Furthermore, we present the $\texttt{cdetools}$ package, which includes functions for computing a CDE loss function for tuning and assessing the quality of individual PDFs, along with diagnostic functions. We provide sample code in $\texttt{Python}$ and $\texttt{R}$ as well as examples of applications to photometric redshift estimation and likelihood-free cosmological inference via CDE.

Conditional density estimation with covariate measurement error

We consider estimating the density of a response conditioning on an error-prone covariate. Motivated by two existing kernel density estimators in the absence of covariate measurement error, we propose a method to correct the existing estimators for measurement error. Asymptotic properties of the resultant estimators under different types of measurement error distributions are derived. Moreover, we adjust bandwidths readily available from existing bandwidth selection methods developed for error-free data to obtain bandwidths for the new estimators. Extensive simulation studies are carried out to compare the proposed estimators with naive estimators that ignore measurement error, which also provide empirical evidence for the effectiveness of the proposed bandwidth selection methods. A real-life data example is used to illustrate implementation of these methods under practical scenarios. An R package, lpme, is developed for implementing all considered methods, which we demonstrate via an R code example in Appendix B.2.

Predictive Density Aggregation: A Model for Global GDP Growth

In this paper we propose a novel approach to obtain the predictive density of global GDP growth. It hinges upon a bottom-up probabilistic model that estimates and combines single countries' predictive GDP growth densities, taking into account cross-country interdependencies. Specifically,we model non-parametrically the contemporaneous interdependencies across the United States,the euro area, and China via a conditional kernel density estimation of a joint distribution. Then, we characterize the potential amplification effects stemming from other large economies in each region-also with kernel density estimations-and the reaction of all other economies with para-metric assumptions. Importantly, each economy's predictive density also depends on a set of observable country-specific factors. Finally, the use of sampling techniques allows us to aggregate individual countries' densities into a world aggregate while preserving the non-i.i.d. nature of the global GDP growth distribution. Out-of-sample metrics con?rm the accuracy of our approach.

Packet Inter-Reception Time Conditional Density Estimation Based on Surrounding Traffic Distribution

Cooperation is an enabler for autonomous vehicles. A promising application of cooperative driving is high-density platooning, where trucks drive with low inter-vehicle distances. It aims at increasing the road and fuel efficiency whilst guaranteeing safety. The safe and efficient coordination of the control requires the regular and reliable exchange of V2V messages. The performance of the vehicular application has been shown to be strongly affected by the variation of the performances of the communications system. To be able to adapt their functional settings to these variations, vehicles need the ability to predict it. We present a prediction model for the packet inter-reception time platoon messages in an IEEE 802.11p network. This performance indicator is the subject of extensive research as it captures the irregularity of input for the control loop. The prediction model uses conditional density estimation based on the exponential distribution. We fit this model using a multi-layer perceptron regressor based on features representing the surrounding communication environment. The presented results are based on data collected during a full scale platooning simulations using ns-3 and SUMO. We compare different environment abstraction models and show the potential of on-line learning.

A Conditional Density Estimation Partition Model Using Logistic Gaussian Processes.

Conditional density estimation (density regression) estimates the distribution of a response variable y conditional on covariates x. Utilizing a partition model framework, a conditional density estimation method is proposed using logistic Gaussian processes. The partition is created using a Voronoi tessellation and is learned from the data using a reversible jump Markov chain Monte Carlo algorithm. The Markov chain Monte Carlo algorithm is made possible through a Laplace approximation on the latent variables of the logistic Gaussian process model. This approximation marginalizes the parameters in each partition element, allowing an efficient search of the posterior distribution of the tessellation. The method has desirable consistency properties. In simulation and applications, the model successfully estimates the partition structure and conditional distribution of y.

A novel wind speed prediction method based on robust local mean decomposition, group method of data handling and conditional kernel density estimation

Short-term wind speed prediction has a great practical significance in wind power exploitation and development. However, owing to strong nonstationarity and nonlinearity of natural wind, it is hard to obtain accurate and reliable prediction by traditional models. To this end, this paper develops an innovative hybrid model to reveal the different characteristics of wind speed data for obtaining the satisfactory forecasting capability. Besides the deterministic prediction, this method could also provide probabilistic prediction. Specifically, robust local mean decomposition is firstly implemented to address the nonstationarity and nonlinearity of wind speed time series by separating them into a series of different subseries with lower ones; Then, an unexplored group method of data handling neural network is developed to extract the nonlinear information in each subseries and subsequently perform the corresponding individual prediction; Furthermore, for the above modeling residuals, traditional conditional kernel density estimation with the embedded dimension determined by partial autocorrelation function is employed to describe the other potential linear and nonlinear characteristics in the form of conditional probability, where the bandwidth matrices are optimized by normal reference criterion. On this basis, the deterministic prediction with the single-value is generated through aggregate calculation. In order to further consider the uncertainty in the data, the probabilistic prediction using unbiased conditional kernel density estimation is performed and the corresponding result is exhibited by prediction intervals. Case studies based on three groups of the measured wind speed data are conducted to comprehensively illustrate the effectiveness of the proposed method. The final results indicate that the proposed method occupies greater accuracy and higher reliability in comparison with the other concerned models.

A new hybrid framework for probabilistic wind speed prediction using deep feature selection and multi-error modification

Wind speed prediction is critical for the wind power exploitation and management due to their strong positive correlation. However, since natural wind owns the high uncertainty and nonstationarity, reliable wind speed prediction is generally difficult to be realized. The objective of this study is to develop a new method for accurately explaining these characteristics and then providing reliable prediction. To this end, a novel probabilistic forecasting framework using enhanced variational mode decomposition, deep feature selection and multi-error modification is proposed. Concretely, the raw data is preprocessed firstly by the enhanced variational mode decomposition where its decomposition level number can be adaptively optimized by the combination of Hilbert transform and empirical model decomposition. Then, an innovative feature selection method which is hybrid of Kullback-Leibler divergence, Gram-Schmidt orthogonal and sample entropy is developed to conduct deep feature identification. Finally, after the deterministic prediction is performed by least square support vector machine, a post-processing multi-error modification is generated to implement the probabilistic prediction. In this method, four probabilistic models including kernel density estimation, univariate conditional kernel density estimation, generalized autoregressive conditional heteroscedasticity and their hybrid model are employed to capture different properties embedded in the error component. Four case studies based on the measured data are carried out. Systematic assessment results show the proposed method has well-pleasing forecasting capability and may be more suitable for the data with higher nonstationarity and non-Gaussianity. For example, the coverage width-based criterion of the proposed method in terms of data collection 4 is 0.261, while those from data collections 1, 2 and 3 are 0.343, 0.295 and 0.282, respectively.

Daily condition monitoring of grid-connected wind turbine via high-fidelity power curve and its comprehensive rating

With development of grid-connected wind power, data-driven wind turbine power curve (WTPC) becomes vital for many valuable applications whereas it is hugely affected by the potential outliers in supervisory control and data acquisition (SCADA) system. In this paper, high-fidelity WTPC modeling and its application on condition monitoring are deeply studied. Firstly, through irregular space-division and nonlinear space-mapping, stepwise data cleaning procedure is proposed. On this basis, the cleaned data are used for high-fidelity modeling where optimized least square support vector machine (LSSVM) is chosen for deterministic WTPC modeling and conditional kernel density estimation (CKDE) is selected for uncertainty modeling after comparison where the models are updated via the sliding time window mechanism. Using the consistency of historical data-driven models in a certain future time window, a daily condition monitoring system, including daily multi-index system and improved fuzzy comprehensive evaluation method (FCEM), is established to monitor future wind turbine condition. Finally, measured data from a wind farm in north China are acquired for validation. The results show that the outliers are effectively cleaned for high-fidelity WTPC modeling. The daily condition monitoring system including daily raw data preprocessing, multi-index system and FCEM algorithm can provide efficient daily rating score.

Bootstrap confidence intervals for conditional density function in Markov processes

Bootstrap confidence intervals for the conditional density function in Markov processes are established. For this purpose, recent bootstrap algorithms constructed for prediction intervals are reviewed and extended to this context. These methods are: model-free bootstrap, predictive model-free, limit model-free, bootstrap for nonparametric autoregressive models with predictive and fitted residuals, local bootstrap and bootstrap based on estimates of the transition density. However, in order to achieve a good coverage probability, the choice of an appropriate smoothing parameter for conditional density estimation turns out to be of utmost importance. In this sense, cross validation and plug-in bandwidth parameter selectors are considered, as well as a deterministic one. An extensive simulation study is carried out to show the empirical behavior of these methods and to compare them. Finally, the methods are illustrated by applying them to a real data set.

Comparative Analysis for Interval Modeling Algorithms of Wind Turbine Power Curve

Considering the stochastic uncertainty of wind power generation, joint estimation of wind turbine power curve (WTPC) in point and interval forms is carefully studied in this paper. Firstly, inducing factors of uncertain data and outliers types in measured data are analyzed to show necessity of uncertainty modeling of WTPC. Then, theoretical principles of different methods for WTPC modeling such as Gaussian Bin method, kernel density estimation (KDE), conditional kernel density estimation (CKDE), conditional probability via Copula, Gaussian process regression (GPR) and relevance vector machine (RVM), are seriously straightened up. Their theoretical essences are discussed and compared in depth. Subsequently, evaluation indexes for point and interval estimations of WTPC are defined to quantify modeling performance. Finally, using measured data from actual 1.5MW variable-speed variable-pitch (VSVP) wind turbines, simulations are executed to show diversities of these methods and their features are discussed from different views. Meanwhile, their potential application scenarios and usage strategies are carefully analyzed, including wind power prediction and operational performance evaluation of wind turbine in an infinite-time-horizon form or a sliding-window form. It is helpful to enhance development of WTPC modeling techniques.

CaDET: interpretable parametric conditional density estimation with decision trees and forests

We introduce CaDET, an algorithm for parametric Conditional Density Estimation (CDE) based on decision trees and random forests. CaDET uses the empirical cross entropy impurity criterion for tree growth, which incentivizes splits that improve predictive accuracy more than the regression criteria or estimated mean-integrated-square-error used in previous works. CaDET also admits more efficient training and query procedures than existing tree-based CDE approaches, and stores only a bounded amount of information at each tree leaf, by using sufficient statistics for all computations. Previous tree-based CDE techniques produce complicated uninterpretable distribution objects, whereas CaDET may be instantiated with easily interpretable distribution families, making every part of the model easy to understand. Our experimental evaluation on real datasets shows that CaDET usually learns more accurate, smaller, and more interpretable models, and is less prone to overfitting than existing tree-based CDE approaches.

Bernstein conditional density estimation with application to conditional distribution and regression functions

In this paper we propose a smooth nonparametric estimation for the conditional probability density function based on a Bernstein polynomial representation. Our estimator can be written as a finite mixture of beta densities with data-driven weights. Using the Bernstein estimator of the conditional density function, we derive new estimators for the distribution function and conditional mean. We establish the asymptotic properties of the proposed estimators, by proving their asymptotic normality and by providing their asymptotic bias and variance. Simulation results suggest that the proposed estimators can outperform the Nadaraya–Watson estimator and, in some specific setups, the local linear kernel estimators. Finally, we use our estimators for modeling the income in Italy, conditional on year from 1951 to 1998, and have another look at the well known Old Faithful Geyser data.

Maximum likelihood method for bandwidth selection in kernel conditional density estimate

This paper discusses the kernel estimator of conditional density. A significant problem of kernel smoothing is bandwidth selection. The problem consists in the fact that optimal bandwidth depends on the unknown conditional and marginal density. This is the reason why some data-driven method needs to be applied. In this paper, we suggest a method for bandwidth selection based on a classical maximum likelihood approach. We consider a slight modification of the original method—the maximum likelihood method with one observation being left out. Applied to two types of conditional density estimators—to the Nadaraya–Watson and local linear estimator, the proposed method is compared with other known methods in a simulation study. Our aim is to compare the methods from different points of view, concentrating on the accuracy of the estimated bandwidths, on the final model quality measure, and on the computational time.