Conditional Density Estimation Research Articles

Generative Quantile Regression with Variability Penalty

Quantile regression and conditional density estimation can reveal structure that is missed by mean regression, such as multimodality and skewness. In this article, we introduce a deep learning generative model for joint quantile estimation called Penalized Generative Quantile Regression (PGQR). Our approach simultaneously generates samples from many random quantile levels, allowing us to infer the conditional distribution of a response variable given a set of covariates. Our method employs a novel variability penalty to avoid the problem of vanishing variability, or memorization, in deep generative models. Further, we introduce a new family of partial monotonic neural networks (PMNN) to circumvent the problem of crossing quantile curves. A major benefit of PGQR is that it can be fit using a single optimization, thus, bypassing the need to repeatedly train the model at multiple quantile levels or use computationally expensive cross-validation to tune the penalty parameter. We illustrate the efficacy of PGQR through extensive simulation studies and analysis of real datasets. Code to implement our method is available at https://github.com/shijiew97/PGQR.

Inventory, Dynamic Evolution, and Scenario Projections of Agricultural Carbon Emissions in Shandong Province, China

The reduction in agricultural carbon emissions (ACEs) in Shandong Province is essential to China’s carbon peak and carbon neutrality objectives. In this regard, we constructed an ACE inventory for Shandong Province at a resolution of 1 km × 1 km, integrating the emission factor method with geographic information system (GIS) technology. Building upon this, we explored the dynamic evolution patterns of ACEs using kernel density estimation and conditional probability density estimation. Additionally, long short-term memory networks were trained to predict ACEs under various scenarios. The results showed that: (1) ACEs in Shandong Province exhibited two stages of change, i.e., “rise and decline”. Notably, 64.39% of emissions originated from the planting industry. The distribution of emissions was closely correlated with regional agricultural production modes. Specifically, CO2 emissions were predominantly distributed in crop cultivation areas, while CH4 and N2O emissions were primarily distributed in livestock breeding areas. The uncertainty of the emission inventory ranged from −12.04% to 10.74%, mainly caused by emission factors. (2) The ACE intensity of various cities in Shandong Province is decreasing, indicating a decoupling between ACEs and agricultural economic growth. Furthermore, the emission disparities among different cities are diminishing, although significant spatial non-equilibrium still persists. (3) From 2022 to 2030, the ACEs in Shandong Province will show a continuous downward trend. By 2030, the projected values under the baseline scenario, low-carbon scenario I, and low-carbon scenario II will be 6301.74 × 104 tons, 5980.67 × 104 tons, and 5850.56 × 104 tons. The low-carbon scenario reveals greater potential for ACE reduction while achieving efficient rural economic development and urbanization simultaneously. This study not only advances the methodology of the ACE inventory but also provides quantitative references and scientific bases for promoting low-carbon, efficient, and sustainable regional agriculture.

The Quasar Catalogue for S-PLUS DR4 (QuCatS) and the estimation of photometric redshifts

ABSTRACT The advent of massive broad-band photometric surveys enabled photometric redshift estimates for unprecedented numbers of galaxies and quasars. These estimates can be improved using better algorithms or by obtaining complementary data such as narrow-band photometry, and broad-band photometry over an extended wavelength range. We investigate the impact of both approaches on photometric redshifts for quasars using data from Southern Photometric Local Universe Survey (S-PLUS) DR4, Galaxy Evolution Explorer (GALEX) DR6/7, and the unWISE catalog for the Wide-field Infrared Survey Explorer (WISE) in three machine learning methods: Random Forest, Flexible Conditional Density Estimation (FlexCoDE), and Bayesian Mixture Density Network (BMDN). Including narrow-band photometry improves the root-mean-square error by 11 per cent in comparison to a model trained with only broad-band photometry. Narrow-band information only provided an improvement of 3.8 per cent when GALEX and WISE colours were included. Thus, narrow bands play a more important role for objects that do not have GALEX or WISE counterparts, which respectively makes 92 per cent and 25 per cent of S-PLUS data considered here. Nevertheless, the inclusion of narrow-band information provided better estimates of the probability density functions obtained with FlexCoDE and BMDN. We publicly release a value-added catalogue of photometrically selected quasars with the photo-z predictions from all methods studied here. The catalogue provided with this work covers the S-PLUS DR4 area (∼3000 square degrees), containing 645 980, 244 912, 144 991 sources with the probability of being a quasar higher than, 80 per cent, 90 per cent, 95 per cent up to r < 21.3 and good photometry quality in the detection image. More quasar candidates can be retrieved from the S-PLUS data base by considering less restrictive selection criteria.

ConDENSE: Conditional Density Estimation for Time Series Anomaly Detection

In recent years deep learning methods, based on reconstruction errors, have facilitated huge improvements in unsupervised anomaly detection. These methods make the limiting assumption that the greater the distance between an observation and a prediction the lower the likelihood of that observation. In this paper we propose conDENSE, a novel anomaly detection algorithm, which does not use reconstruction errors but rather uses conditional density estimation in masked autoregressive flows. By directly estimating the likelihood of data, our model moves beyond approximating expected behaviour with a single point estimate, as is the case in reconstruction error models. We show how conditioning on a dense representation of the current trajectory, extracted from a variational autoencoder with a gated recurrent unit (GRU VAE), produces a model that is suitable for periodic datasets, while also improving performance on non-periodic datasets. Experiments on 31 time-series, including real-world anomaly detection benchmark datasets and synthetically generated data, show that the model can outperform state-of-the-art deep learning methods.

An Improved Ensemble-Strategy-Assisted Wind Speed Prediction Method for Railway Strong Wind Warnings

Reliable short-term wind speed prediction is one of the core technologies in the strong wind warning system for railway applications, which is of great significance for ensuring the safety of high-speed train operations and ancillary railway facilities. To improve forecasting accuracy, decomposition-based methods have attracted extensive attention due to their superior ability to address complex data characteristics (e.g., nonstationarity and nonlinearity). Currently, there are two pre-processing schemes for decomposition-based methods, i.e., one-time decomposition and real-time decomposition. In order to apply them better, this paper first expounds the difference between them, based on a combination of DWT (discrete wavelet transform) and CKDE (conditional kernel density estimation). The results show that although the one-time decomposition-based method has an unexceptionable accuracy, it only can provide offline prediction and thus may not be practical. The real-time decomposition-based method possesses stronger practicability and is able to provide online prediction, but it has limited accuracy. Then, an improved ensemble strategy is developed by optimizing the selection of appropriate decomposed components to conduct the prediction on the basis of real-time decomposition. This improved ensemble strategy provides an effective guidance for this selective combination, including taking historical information into consideration in the data. Finally, numerical examples and practicality analysis using two groups of measured wind speed data demonstrate that the proposed method is effective in providing high-precision online wind speed prediction. For example, compared with CKDE, the average degrees of improvement achieved by the proposed method in terms of MAE, RMSE, and MRPE, are 16.25%, 17.66%, and 16.93, respectively, while those compared with the traditional real-time decomposition method are 17.11%, 18.54%, and 16.84, respectively.

PATHFINDER: Designing Stimulus for Neuromodulation Through Data-Driven Inverse Estimation of Non-Linear Functions.

Using data-driven methods to design stimuli (e.g., electrical currents) which evoke desired neural responses in different neuron-types for applications in treating neural disorders. The problem of stimulus design is formulated as estimating the inverse of a many-to-one non-linear "forward" mapping, which takes as input the parameters of waveform and outputs the corresponding neural response, directly from the data. A novel optimization framework "PATHFINDER" is proposed in order to estimate the previously mentioned inverse mapping. A comparison with existing data-driven methods, namely conditional density estimation methods and numerical inversion of an estimated forward mapping is performed with different dataset sizes in toy examples and in detailed computational models of biological neurons. Using data from toy examples, as well as computational models of biological neurons, we show that PATHFINDER can outperform existing methods when the number of samples is low (i.e., a few hundred). Traditionally, the design of such stimuli has been model-driven and/or uses simplistic intuition, often aided by trial-and-error. Due to the inherent challenges in accurately modeling neural responses, as well as the sophistication of stimuli's effect on neural membrane potentials, data-driven approaches offer an attractive alternative. Our results suggest that PATHFINDER can be applied for optimizing stimulation parameters in experiments and treatments of neural disorders due to it requiring low number of data points.

Stratified learning: A general‐purpose statistical method for improved learning under covariate shift

AbstractWe propose a simple, statistically principled, and theoretically justified method to improve supervised learning when the training set is not representative, a situation known as covariate shift. We build upon a well‐established methodology in causal inference and show that the effects of covariate shift can be reduced or eliminated by conditioning on propensity scores. In practice, this is achieved by fitting learners within strata constructed by partitioning the data based on the estimated propensity scores, leading to approximately balanced covariates and much‐improved target prediction. We refer to the overall method as Stratified Learning, or StratLearn. We demonstrate the effectiveness of this general‐purpose method on two contemporary research questions in cosmology, outperforming state‐of‐the‐art importance weighting methods. We obtain the best‐reported AUC (0.958) on the updated “Supernovae photometric classification challenge,” and we improve upon existing conditional density estimation of galaxy redshift from Sloan Digital Sky Survey (SDSS) data.

New Feature Selection Approach for Photovoltaïc Power Forecasting Using KCDE

Feature selection helps improve the accuracy and computational time of solar forecasting. However, FS is often passed by or conducted with methods that do not suit the solar forecasting issue, such as filter or linear methods. In this study, we propose a wrapper method termed Sequential Forward Selection (SFS), with a Kernel Conditional Density Estimator (KCDE) named SFS-KCDE, as FS to forecast day-ahead regional PV power production in French Guiana. This method was compared to three other FS methods used in earlier studies: the Pearson correlation method, the RReliefF (RRF) method, and SFS using a linear regression. It has been shown that SFS-KCDE outperforms other FS methods, particularly for overcast sky conditions. Moreover, Wrapper methods show better forecasting performance than filter methods and should be used.

Conditional density estimation using population Monte Carlo based approximate Bayesian computation

Most statistical methods require likelihood evaluation to draw a statistical inference. However, in some situations, likelihood evaluation becomes difficult analytically or computationally. Different likelihood-free methods are available that eliminate the need to compute the likelihood function. Approximate Bayesian Computation (ABC) is a framework that implements likelihood-free inference and replaces the likelihood evaluation with simulations by using forward modeling. The goal of ABC methods is to approximate the posterior distribution. However, posterior approximation via ABC methods is still considerably expensive for high dimensions. ABC requires many simulations that become computationally infeasible for complex models. Here, a technique is proposed that combines a somewhat more efficient form of ABC (Population Monte Carlo, PMC) with a Conditional Density Estimation (CDE) approach. The proposed framework provides an estimation of the posterior distribution which is referred to as PMC-CDE. A simulation study is performed that provides empirical evidence to show the efficiency of PMC-CDE in terms of integrated squared error loss. Furthermore, real-life datasets manifest the application of the proposed method.

Multivariate probabilistic forecasting of intraday electricity prices using normalizing flows

Electricity is traded on various markets with different time horizons and regulations. Short-term intraday trading becomes increasingly important due to the higher penetration of renewables. In Germany, the intraday electricity price typically fluctuates around the day-ahead price of the European Power EXchange (EPEX) spot markets in a distinct hourly pattern. This work proposes a probabilistic modeling approach that models the intraday price difference to the day-ahead contracts. The model captures the emerging hourly pattern by considering the four 15min intervals in each day-ahead price interval as a four-dimensional joint probability distribution. The resulting nontrivial, multivariate price difference distribution is learned using a normalizing flow, i.e., a deep generative model that combines conditional multivariate density estimation and probabilistic regression. Furthermore, this work discusses the influence of different external impact factors based on literature insights and impact analysis using explainable artificial intelligence (XAI). The normalizing flow is compared to an informed selection of historical data and probabilistic forecasts using a Gaussian copula and a Gaussian regression model. Among the different models, the normalizing flow identifies the trends with the highest accuracy and has the narrowest prediction intervals. Both the XAI analysis and the empirical experiments highlight that the immediate history of the price difference realization and the increments of the day-ahead price have the most substantial impact on the price difference.

Analysis of ASEAN’s Stock Returns and/or Volatility Distribution under the Impact of the Chinese EPU: Evidence Based on Conditional Kernel Density Approach

Analysis of ASEAN’s Stock Returns and/or Volatility Distribution under the Impact of the Chinese EPU: Evidence Based on Conditional Kernel Density Approach-Economic Policy Uncertainty, Stock Market Returns/Volatility, Conditional Density Estimation, ASEAN5 Stock Markets

Amortized Bayesian inference on generative dynamical network models of epilepsy using deep neural density estimators

Whole-brain modeling of epilepsy combines personalized anatomical data with dynamical models of abnormal activities to generate spatio-temporal seizure patterns as observed in brain imaging data. Such a parametric simulator is equipped with a stochastic generative process, which itself provides the basis for inference and prediction of the local and global brain dynamics affected by disorders. However, the calculation of likelihood function at whole-brain scale is often intractable. Thus, likelihood-free algorithms are required to efficiently estimate the parameters pertaining to the hypothetical areas, ideally including the uncertainty. In this study, we introduce the simulation-based inference for the virtual epileptic patient model (SBI-VEP), enabling us to amortize the approximate posterior of the generative process from a low-dimensional representation of whole-brain epileptic patterns. The state-of-the-art deep learning algorithms for conditional density estimation are used to readily retrieve the statistical relationships between parameters and observations through a sequence of invertible transformations. We show that the SBI-VEP is able to efficiently estimate the posterior distribution of parameters linked to the extent of the epileptogenic and propagation zones from sparse intracranial electroencephalography recordings. The presented Bayesian methodology can deal with non-linear latent dynamics and parameter degeneracy, paving the way for fast and reliable inference on brain disorders from neuroimaging modalities.

Two-stage conditional density estimation based on Bernstein polynomials

Two-stage conditional probability density function estimators are proposed and studied. Specifically, the Nadaraya-Watson (NW) and local linear (LL) conditional distribution function estimators have been smoothed using Bernstein polynomials in the first stage. Second, the proposed estimators are obtained by differentiating NW and LL estimators. The asymptotic properties of these estimators are established such as asymptotic bias, variance, and normality. Finally, a simulation study is carried out to assess the relative advantage of our estimators compared to other competitors. In addition, the well-known Old Faithful Geyser data is analyzed using the proposed estimators.

A Novel Hybrid Model for Short-Term Traffic Flow Prediction Based on Extreme Learning Machine and Improved Kernel Density Estimation

Short-term traffic flow prediction is the basis of and ensures intelligent traffic control. However, the conventional models cannot make accurate predictions due to the strong nonlinearity and randomness in short-term traffic flow data. To this end, the authors of this paper developed a novel hybrid model based on extreme learning machine (ELM), adaptive kernel density estimation (AKDE), and conditional kernel density estimation (CKDE). Specifically, the ELM model was employed for nonlinear prediction. Then, AKDE was established to estimate the bandwidth of CKDE (i.e., AKDE-CKDE), which predicted the training residuals obtained by ELM. Finally, the predicted results of the two models were superimposed to derive the final prediction of the hybrid model. Two case studies based on measured data were conducted to evaluate the performance of the proposed method. The experimental results indicate that the proposed method can realize a significant improvement in terms of forecasting accuracy in comparison with the other concerned models. For instance, it performed better than the single ELM model, with an improvement in the evaluation criterion of a mean relative percentage error of 7.46%.

Asymptotic properties of a kernel etimate of the conditional mode in the presence of doubly censored data

We introduce a double kernel conditional mode estimator when the response variable is doubly censored. In this situation, the variable of interest may be right censored by a variable R or left censored by a variable L and L is not greater than R almost surely. Self- consistent estimators of R and L are extensively studied in literature. As they fit in the expression of the conditional density estimator, using their known properties, among other tools, we establish the uniform almost sure convergence of the conditional density estimator and its derivatives. Then, we deduce the consistency and the asymptotic normality of our conditional mode estimator. For that, we use the theory of Vapnik- ervonenkis classes of measurable functions. We also show the quality of this estimation and the Gaussian behavior of our proposed estimator through a simulation study in which, we consider various regression models with different sample sizes and censoring rates. Finally, we illustrate the performance of our estimator on a real data set which concerns the Stanford heart transplant data.

Classifying anomalies through outer density estimation

We propose a new model-agnostic search strategy for physics beyond the standard model (BSM) at the LHC, based on a novel application of neural density estimation to anomaly detection. Our approach, which we call Classifying Anomalies THrough Outer Density Estimation (CATHODE), assumes the BSM signal is localized in a signal region (defined e.g. using invariant mass). By training a conditional density estimator on a collection of additional features outside the signal region, interpolating it into the signal region, and sampling from it, we produce a collection of events that follow the background model. We can then train a classifier to distinguish the data from the events sampled from the background model, thereby approaching the optimal anomaly detector. Using the LHC Olympics R&D dataset, we demonstrate that CATHODE nearly saturates the best possible performance, and significantly outperforms other approaches that aim to enhance the bump hunt (CWoLa Hunting and ANODE). Finally, we demonstrate that CATHODE is very robust against correlations between the features and maintains nearly-optimal performance even in this more challenging setting.

Non-parametric discretization for probabilistic labeled data

• Weakly supervised non-parametric discretization. • Probabilistic labels • Class-conditioned kernel density estimation. • Exact/closed calculation of the points involved in the discretization policy. • High relative performance in retaining discriminative information under noise. Probabilistic label learning is a challenging task that arises from recent real-world problems within the weakly supervised classification framework. In this task algorithms have to deal with datasets where each instance has associated a set of probabilities belonging to different class labels. In this paper, we propose a supervised univariate non-parametric discretization algorithm based on kernel density estimation that can deal with probabilistic labeled data. The algorithm takes advantage of the estimation of the class conditional densities to produce different sets of cut points according to different smoothing parameters of the kernel. Then, the best set of cut points is selected according to a given supervised classification performance measure. The computational complexity is O ( N log N ) , where N is the number of instances. The proposal is tested on simulated probabilistic labeled data, which allows assessing the behavior with different noise degrees. The results show that the algorithm outperforms other discretization algorithms and is robust to different degrees of uncertainty.

Correction: Nonparametric conditional density estimation in a deep learning framework for short-term forecasting

Correction: Nonparametric conditional density estimation in a deep learning framework for short-term forecasting

Local linear approach: Conditional density estimate for functional and censored data

Abstract Let Y Y be a random real response, which is subject to right censoring by another random variable C C . In this paper, we study the nonparametric local linear estimation of the conditional density of a scalar response variable and when the covariable takes values in a semi-metric space. Our main aim is to prove under some regularity conditions both the pointwise and the uniform almost-sure consistencies with convergence rates of the conditional density estimator related by this estimation procedure.

Semi-supervised Conditional Density Estimation with Wasserstein Laplacian Regularisation

Conditional Density Estimation (CDE) has wide-reaching applicability to various real-world problems, such as spatial density estimation and environmental modelling. CDE estimates the probability density of a random variable rather than a single value and can thus model uncertainty and inverse problems. This task is inherently more complex than regression, and many algorithms suffer from overfitting, particularly when modelled with few labelled data points. For applications where unlabelled data is abundant but labelled data is scarce, we propose Wasserstein Laplacian Regularisation, a semi-supervised learning framework that allows CDE algorithms to leverage these unlabelled data. The framework minimises an objective function which ensures that the learned model is smooth along the manifold of the underlying data, as measured by Wasserstein distance. When applying our framework to Mixture Density Networks, the resulting semi-supervised algorithm can achieve similar performance to a supervised model with up to three times as many labelled data points on baseline datasets. We additionally apply our technique to the problem of remote sensing for chlorophyll-a estimation in inland waters.