Conditional Density Estimation Research Articles

A novel spatial prediction method for soil heavy metal based on unbiased conditional kernel density estimation

Soil contamination by heavy metals has emerged as a significant global problem. Accurately mapping the spatial distribution of soil heavy metal pollutant concentrations is indispensable for effective agriculture and environmental management. Nevertheless, challenges arise in obtaining comprehensive data at all desired locations, due to limitations in measuring equipment capacity and the associated capital costs. To obtain soil heavy metal maps efficiently and accurately, this paper proposes a nonparametric spatial prediction method, based on unbiased conditional kernel density estimation (UCKDE). The proposed method incorporates the advantages of both geostatistics and machine learning, including stability, adaptability, and the ability to account for various types of auxiliary information. Additionally, it can directly predict the probabilistic density function (PDF) of soil heavy metal content at the target location based on sampling data without complex parameter settings, providing both a deterministic single value and a probabilistic prediction interval. The proposed method and ordinary kriging (OK) were implemented for the spatial prediction of the six heavy metals (As, Cd, Cu, Hg, Mn, and Sb) with the greatest coefficients of variation (CV = 0.53, 1.14, 0.66, 1.05, 0.81 and 0.74, respectively) in Qingxi Town, Chongqing, China. The results showed that the predictive capability of the proposed method (with RMSE values of 5.82, 0.61, 14.76, 0.15, 383.84, and 0.85, respectively) is superior to that of OK (with RMSE values of 5.29, 0.87, 16.37, 0.22, 493.22, and 1.58, respectively) in most cases, particularly when the CV value is high. Besides, the prediction accuracy of the proposed method can be further enhanced by incorporating parent material, resulting in RMSE values of 3.02, 0.51, 8.98, 0.08, 194.16, and 0.56, respectively. The results affirm the reliability of the proposed method and suggest its effectiveness as a tool for soil heavy metal pollution prediction in practical applications.

Regression trees for fast and adaptive prediction intervals

In predictive modeling, quantifying prediction uncertainty is crucial for reliable decision-making. Traditional conformal inference methods provide marginally valid predictive regions but often produce non-adaptive intervals when naively applied to regression, potentially biasing applications. Recent advances using quantile regressors or conditional density estimators improve adaptability but are typically tied to specific prediction models, limiting their ability to quantify uncertainty around arbitrary models. Similarly, methods based on partitioning the feature space adopt sub-optimal strategies, failing to consistently measure predictive uncertainty across the feature space, especially in adversarial examples. This paper introduces a model-agnostic family of methods to calibrate prediction intervals for regression with local coverage guarantees. By leveraging regression trees and Random Forests, our approach constructs data-adaptive partitions of the feature space to approximate conditional coverage, enhancing the accuracy and scalability of prediction intervals. Our methods outperform established benchmarks on simulated and real-world datasets. They are implemented in the Python package clover, which integrates seamlessly with the scikit-learn interface for practical application.

Optimal place to apply post-processing in the deterministic photovoltaic power forecasting workflow

Is the post-processing of global horizontal irradiance (GHI) forecasts necessary for issuing good photovoltaic (PV) power forecasts? Whenever this question is raised, the instinctive supposition always seems to be “yes,” because GHI is the most important weather parameter governing the amount of PV power generated, and surely, the better the GHI forecasts are, the better the PV power forecasts should result. To attend to this question more scientifically and more formally, two classic deterministic-to-deterministic post-processing methods, namely, the model output statistics and kernel conditional density estimation, are applied at various stages of PV power forecasting, resulting in four distinct workflows. These different workflows are trained and tested on three PV plants in Hungary, using data from a four-year (2017–2020) period. Both ground-based GHI and satellite-retrieved GHI are used as the “truth” with which numerical weather prediction (NWP) GHI forecasts are post-processed. A very thorough deterministic forecast verification exercise is conducted following the best practices. It is found that contrary to the common supposition, post-processing GHI only leads to marginal, if that can be quantified at all, benefits, so long as the PV power forecasts are to be post-processed. This ought to be deemed as a very important finding, as it puts into question the “GHI forecasting + post-processing + irradiance-to-power conversion” workflow that has dominated solar forecasting for decades.

A hybrid residual correction method with auxiliary particle filter and conditional kernel density estimation for remaining useful life prediction

Predicting the remaining useful life (RUL) is crucial for prognosis and health management and provides an essential foundation for maintenance decisions. This study constructed a hybrid RUL prediction method based on residual correction to enhance prediction accuracy, minimize uncertainty, and mitigate errors caused by local fluctuations in the degeneration state within complex environments. This approach combined an auxiliary particle filter (APF), complementary ensemble empirical mode decomposition (CEEMD), and conditional kernel density estimation (CKDE). First, a Wiener degradation model was established, and the APF was used to acquire the initial state estimate of the equipment and residual difference between actual observations. Second, CEEMD was used to reconstruct the residual sequence and reduce interference. Subsequently, the nonparametric CKDE predicted the residual to correct the filter estimation. The corrected degeneration state value elucidated the local details of the system degradation process, and the point prediction of the RUL at the current time was obtained. The probability density distribution of the remaining life was estimated by substituting the modified degeneration state values into the kernel density estimation (KDE) model. Finally, the method was tested using an aircraft engine dataset. The results demonstrated the efficacy of the proposed method and its superiority over alternative approaches.

Flexible conditional density estimation for time series

Flexible conditional density estimation for time series

Generative Quantile Regression with Variability Penalty

Quantile regression and conditional density estimation can reveal structure that is missed by mean regression, such as multimodality and skewness. In this article, we introduce a deep learning generative model for joint quantile estimation called Penalized Generative Quantile Regression (PGQR). Our approach simultaneously generates samples from many random quantile levels, allowing us to infer the conditional distribution of a response variable given a set of covariates. Our method employs a novel variability penalty to avoid the problem of vanishing variability, or memorization, in deep generative models. Further, we introduce a new family of partial monotonic neural networks (PMNN) to circumvent the problem of crossing quantile curves. A major benefit of PGQR is that it can be fit using a single optimization, thus, bypassing the need to repeatedly train the model at multiple quantile levels or use computationally expensive cross-validation to tune the penalty parameter. We illustrate the efficacy of PGQR through extensive simulation studies and analysis of real datasets. Code to implement our method is available at https://github.com/shijiew97/PGQR.

Inventory, Dynamic Evolution, and Scenario Projections of Agricultural Carbon Emissions in Shandong Province, China

The reduction in agricultural carbon emissions (ACEs) in Shandong Province is essential to China’s carbon peak and carbon neutrality objectives. In this regard, we constructed an ACE inventory for Shandong Province at a resolution of 1 km × 1 km, integrating the emission factor method with geographic information system (GIS) technology. Building upon this, we explored the dynamic evolution patterns of ACEs using kernel density estimation and conditional probability density estimation. Additionally, long short-term memory networks were trained to predict ACEs under various scenarios. The results showed that: (1) ACEs in Shandong Province exhibited two stages of change, i.e., “rise and decline”. Notably, 64.39% of emissions originated from the planting industry. The distribution of emissions was closely correlated with regional agricultural production modes. Specifically, CO2 emissions were predominantly distributed in crop cultivation areas, while CH4 and N2O emissions were primarily distributed in livestock breeding areas. The uncertainty of the emission inventory ranged from −12.04% to 10.74%, mainly caused by emission factors. (2) The ACE intensity of various cities in Shandong Province is decreasing, indicating a decoupling between ACEs and agricultural economic growth. Furthermore, the emission disparities among different cities are diminishing, although significant spatial non-equilibrium still persists. (3) From 2022 to 2030, the ACEs in Shandong Province will show a continuous downward trend. By 2030, the projected values under the baseline scenario, low-carbon scenario I, and low-carbon scenario II will be 6301.74 × 104 tons, 5980.67 × 104 tons, and 5850.56 × 104 tons. The low-carbon scenario reveals greater potential for ACE reduction while achieving efficient rural economic development and urbanization simultaneously. This study not only advances the methodology of the ACE inventory but also provides quantitative references and scientific bases for promoting low-carbon, efficient, and sustainable regional agriculture.

The Quasar Catalogue for S-PLUS DR4 (QuCatS) and the estimation of photometric redshifts

ABSTRACT The advent of massive broad-band photometric surveys enabled photometric redshift estimates for unprecedented numbers of galaxies and quasars. These estimates can be improved using better algorithms or by obtaining complementary data such as narrow-band photometry, and broad-band photometry over an extended wavelength range. We investigate the impact of both approaches on photometric redshifts for quasars using data from Southern Photometric Local Universe Survey (S-PLUS) DR4, Galaxy Evolution Explorer (GALEX) DR6/7, and the unWISE catalog for the Wide-field Infrared Survey Explorer (WISE) in three machine learning methods: Random Forest, Flexible Conditional Density Estimation (FlexCoDE), and Bayesian Mixture Density Network (BMDN). Including narrow-band photometry improves the root-mean-square error by 11 per cent in comparison to a model trained with only broad-band photometry. Narrow-band information only provided an improvement of 3.8 per cent when GALEX and WISE colours were included. Thus, narrow bands play a more important role for objects that do not have GALEX or WISE counterparts, which respectively makes 92 per cent and 25 per cent of S-PLUS data considered here. Nevertheless, the inclusion of narrow-band information provided better estimates of the probability density functions obtained with FlexCoDE and BMDN. We publicly release a value-added catalogue of photometrically selected quasars with the photo-z predictions from all methods studied here. The catalogue provided with this work covers the S-PLUS DR4 area (∼3000 square degrees), containing 645 980, 244 912, 144 991 sources with the probability of being a quasar higher than, 80 per cent, 90 per cent, 95 per cent up to r < 21.3 and good photometry quality in the detection image. More quasar candidates can be retrieved from the S-PLUS data base by considering less restrictive selection criteria.

ConDENSE: Conditional Density Estimation for Time Series Anomaly Detection

In recent years deep learning methods, based on reconstruction errors, have facilitated huge improvements in unsupervised anomaly detection. These methods make the limiting assumption that the greater the distance between an observation and a prediction the lower the likelihood of that observation. In this paper we propose conDENSE, a novel anomaly detection algorithm, which does not use reconstruction errors but rather uses conditional density estimation in masked autoregressive flows. By directly estimating the likelihood of data, our model moves beyond approximating expected behaviour with a single point estimate, as is the case in reconstruction error models. We show how conditioning on a dense representation of the current trajectory, extracted from a variational autoencoder with a gated recurrent unit (GRU VAE), produces a model that is suitable for periodic datasets, while also improving performance on non-periodic datasets. Experiments on 31 time-series, including real-world anomaly detection benchmark datasets and synthetically generated data, show that the model can outperform state-of-the-art deep learning methods.

A selective review on conditional density estimation

A selective review on conditional density estimation

An Improved Ensemble-Strategy-Assisted Wind Speed Prediction Method for Railway Strong Wind Warnings

Reliable short-term wind speed prediction is one of the core technologies in the strong wind warning system for railway applications, which is of great significance for ensuring the safety of high-speed train operations and ancillary railway facilities. To improve forecasting accuracy, decomposition-based methods have attracted extensive attention due to their superior ability to address complex data characteristics (e.g., nonstationarity and nonlinearity). Currently, there are two pre-processing schemes for decomposition-based methods, i.e., one-time decomposition and real-time decomposition. In order to apply them better, this paper first expounds the difference between them, based on a combination of DWT (discrete wavelet transform) and CKDE (conditional kernel density estimation). The results show that although the one-time decomposition-based method has an unexceptionable accuracy, it only can provide offline prediction and thus may not be practical. The real-time decomposition-based method possesses stronger practicability and is able to provide online prediction, but it has limited accuracy. Then, an improved ensemble strategy is developed by optimizing the selection of appropriate decomposed components to conduct the prediction on the basis of real-time decomposition. This improved ensemble strategy provides an effective guidance for this selective combination, including taking historical information into consideration in the data. Finally, numerical examples and practicality analysis using two groups of measured wind speed data demonstrate that the proposed method is effective in providing high-precision online wind speed prediction. For example, compared with CKDE, the average degrees of improvement achieved by the proposed method in terms of MAE, RMSE, and MRPE, are 16.25%, 17.66%, and 16.93, respectively, while those compared with the traditional real-time decomposition method are 17.11%, 18.54%, and 16.84, respectively.

PATHFINDER: Designing Stimulus for Neuromodulation Through Data-Driven Inverse Estimation of Non-Linear Functions.

Using data-driven methods to design stimuli (e.g., electrical currents) which evoke desired neural responses in different neuron-types for applications in treating neural disorders. The problem of stimulus design is formulated as estimating the inverse of a many-to-one non-linear "forward" mapping, which takes as input the parameters of waveform and outputs the corresponding neural response, directly from the data. A novel optimization framework "PATHFINDER" is proposed in order to estimate the previously mentioned inverse mapping. A comparison with existing data-driven methods, namely conditional density estimation methods and numerical inversion of an estimated forward mapping is performed with different dataset sizes in toy examples and in detailed computational models of biological neurons. Using data from toy examples, as well as computational models of biological neurons, we show that PATHFINDER can outperform existing methods when the number of samples is low (i.e., a few hundred). Traditionally, the design of such stimuli has been model-driven and/or uses simplistic intuition, often aided by trial-and-error. Due to the inherent challenges in accurately modeling neural responses, as well as the sophistication of stimuli's effect on neural membrane potentials, data-driven approaches offer an attractive alternative. Our results suggest that PATHFINDER can be applied for optimizing stimulation parameters in experiments and treatments of neural disorders due to it requiring low number of data points.

Stratified learning: A general‐purpose statistical method for improved learning under covariate shift

AbstractWe propose a simple, statistically principled, and theoretically justified method to improve supervised learning when the training set is not representative, a situation known as covariate shift. We build upon a well‐established methodology in causal inference and show that the effects of covariate shift can be reduced or eliminated by conditioning on propensity scores. In practice, this is achieved by fitting learners within strata constructed by partitioning the data based on the estimated propensity scores, leading to approximately balanced covariates and much‐improved target prediction. We refer to the overall method as Stratified Learning, or StratLearn. We demonstrate the effectiveness of this general‐purpose method on two contemporary research questions in cosmology, outperforming state‐of‐the‐art importance weighting methods. We obtain the best‐reported AUC (0.958) on the updated “Supernovae photometric classification challenge,” and we improve upon existing conditional density estimation of galaxy redshift from Sloan Digital Sky Survey (SDSS) data.

New Feature Selection Approach for Photovoltaïc Power Forecasting Using KCDE

Feature selection helps improve the accuracy and computational time of solar forecasting. However, FS is often passed by or conducted with methods that do not suit the solar forecasting issue, such as filter or linear methods. In this study, we propose a wrapper method termed Sequential Forward Selection (SFS), with a Kernel Conditional Density Estimator (KCDE) named SFS-KCDE, as FS to forecast day-ahead regional PV power production in French Guiana. This method was compared to three other FS methods used in earlier studies: the Pearson correlation method, the RReliefF (RRF) method, and SFS using a linear regression. It has been shown that SFS-KCDE outperforms other FS methods, particularly for overcast sky conditions. Moreover, Wrapper methods show better forecasting performance than filter methods and should be used.

Conditional density estimation using population Monte Carlo based approximate Bayesian computation

Most statistical methods require likelihood evaluation to draw a statistical inference. However, in some situations, likelihood evaluation becomes difficult analytically or computationally. Different likelihood-free methods are available that eliminate the need to compute the likelihood function. Approximate Bayesian Computation (ABC) is a framework that implements likelihood-free inference and replaces the likelihood evaluation with simulations by using forward modeling. The goal of ABC methods is to approximate the posterior distribution. However, posterior approximation via ABC methods is still considerably expensive for high dimensions. ABC requires many simulations that become computationally infeasible for complex models. Here, a technique is proposed that combines a somewhat more efficient form of ABC (Population Monte Carlo, PMC) with a Conditional Density Estimation (CDE) approach. The proposed framework provides an estimation of the posterior distribution which is referred to as PMC-CDE. A simulation study is performed that provides empirical evidence to show the efficiency of PMC-CDE in terms of integrated squared error loss. Furthermore, real-life datasets manifest the application of the proposed method.

Multivariate probabilistic forecasting of intraday electricity prices using normalizing flows

Electricity is traded on various markets with different time horizons and regulations. Short-term intraday trading becomes increasingly important due to the higher penetration of renewables. In Germany, the intraday electricity price typically fluctuates around the day-ahead price of the European Power EXchange (EPEX) spot markets in a distinct hourly pattern. This work proposes a probabilistic modeling approach that models the intraday price difference to the day-ahead contracts. The model captures the emerging hourly pattern by considering the four 15min intervals in each day-ahead price interval as a four-dimensional joint probability distribution. The resulting nontrivial, multivariate price difference distribution is learned using a normalizing flow, i.e., a deep generative model that combines conditional multivariate density estimation and probabilistic regression. Furthermore, this work discusses the influence of different external impact factors based on literature insights and impact analysis using explainable artificial intelligence (XAI). The normalizing flow is compared to an informed selection of historical data and probabilistic forecasts using a Gaussian copula and a Gaussian regression model. Among the different models, the normalizing flow identifies the trends with the highest accuracy and has the narrowest prediction intervals. Both the XAI analysis and the empirical experiments highlight that the immediate history of the price difference realization and the increments of the day-ahead price have the most substantial impact on the price difference.

Analysis of ASEAN’s Stock Returns and/or Volatility Distribution under the Impact of the Chinese EPU: Evidence Based on Conditional Kernel Density Approach

Analysis of ASEAN’s Stock Returns and/or Volatility Distribution under the Impact of the Chinese EPU: Evidence Based on Conditional Kernel Density Approach-Economic Policy Uncertainty, Stock Market Returns/Volatility, Conditional Density Estimation, ASEAN5 Stock Markets

Amortized Bayesian inference on generative dynamical network models of epilepsy using deep neural density estimators

Whole-brain modeling of epilepsy combines personalized anatomical data with dynamical models of abnormal activities to generate spatio-temporal seizure patterns as observed in brain imaging data. Such a parametric simulator is equipped with a stochastic generative process, which itself provides the basis for inference and prediction of the local and global brain dynamics affected by disorders. However, the calculation of likelihood function at whole-brain scale is often intractable. Thus, likelihood-free algorithms are required to efficiently estimate the parameters pertaining to the hypothetical areas, ideally including the uncertainty. In this study, we introduce the simulation-based inference for the virtual epileptic patient model (SBI-VEP), enabling us to amortize the approximate posterior of the generative process from a low-dimensional representation of whole-brain epileptic patterns. The state-of-the-art deep learning algorithms for conditional density estimation are used to readily retrieve the statistical relationships between parameters and observations through a sequence of invertible transformations. We show that the SBI-VEP is able to efficiently estimate the posterior distribution of parameters linked to the extent of the epileptogenic and propagation zones from sparse intracranial electroencephalography recordings. The presented Bayesian methodology can deal with non-linear latent dynamics and parameter degeneracy, paving the way for fast and reliable inference on brain disorders from neuroimaging modalities.

Two-stage conditional density estimation based on Bernstein polynomials

Two-stage conditional probability density function estimators are proposed and studied. Specifically, the Nadaraya-Watson (NW) and local linear (LL) conditional distribution function estimators have been smoothed using Bernstein polynomials in the first stage. Second, the proposed estimators are obtained by differentiating NW and LL estimators. The asymptotic properties of these estimators are established such as asymptotic bias, variance, and normality. Finally, a simulation study is carried out to assess the relative advantage of our estimators compared to other competitors. In addition, the well-known Old Faithful Geyser data is analyzed using the proposed estimators.

On the central limit theorem for a conditional mode estimator in the single functional index modeling for functional time series data under random censorship

The main objective of this paper is to investigate the estimation of conditional density function based on the single-index model in the censorship model when the sample is considered as an dependent random variables. First of all, a kernel type estimator for the conditional density function (cond-df) is introduced. Afterwards, the asymptotic properties are stated when the observations are linked with a single-index structure. The pointwise almost complete convergence and the uniform almost complete convergence (with rate) of the kernel estimate of this model are established. As an application the conditional mode in functional single-index model is presented. Under general conditions, the asymptotic normality of the conditional density estimator is established. Simulation study is also presented to illustrate the validity and finite sample performance of the considered estimator. Finally, the estimation of the functional index via the pseudo-maximum likelihood method is discussed, but not tackled.