Conditional Density Estimation Research Articles

A Deep Generative Approach to Conditional Sampling

We propose a deep generative approach to sampling from a conditional distribution based on a unified formulation of conditional distribution and generalized nonparametric regression function using the noise-outsourcing lemma. The proposed approach aims at learning a conditional generator, so that a random sample from the target conditional distribution can be obtained by transforming a sample drawn from a reference distribution. The conditional generator is estimated nonparametrically with neural networks by matching appropriate joint distributions using the Kullback-Liebler divergence. An appealing aspect of our method is that it allows either of or both the predictor and the response to be high-dimensional and can handle both continuous and discrete type predictors and responses. We show that the proposed method is consistent in the sense that the conditional generator converges in distribution to the underlying conditional distribution under mild conditions. Our numerical experiments with simulated and benchmark image data validate the proposed method and demonstrate that it outperforms several existing conditional density estimation methods. Supplementary materials for this article are available online.

Receding interval prediction of district heat load via finite difference multi-operating-domain dynamic modelling

To realize low carbon emission, district heat load (DHL) prediction is of great significance to the efficient operation of urban district heating system (DHS). Due to spatial topology of central heating network, weather conditions and heat users show great spatiotemporal dispersion, whose influence to DHL is very complex and its prediction is challengeable. To solve this problem, a multi-operating-domain dynamic modelling structure is proposed in this paper. Firstly, through detailed analysis from first-principle, external factors such as weather and user factors are quantitatively included to form candidate inputs set where the NCA (Neighbourhood Component Analysis) algorithm is used to extract feature inputs. Considering time-delays of feature inputs to DHL, AIC (Akaike Information Criterion) is applied to determine delay orders. Then, feature inputs, DHL output and their delay-orders are integrated to form a finite difference operating space, representing by a regression vector. After its high-dimensional clustering and hyperplane estimation, the whole space is convexly partitioned into multiple polyhedral domains. In each domain, using the finite difference input vector and output, multi-step receding prediction of heat load is realized using the Bi-LSTM (Bidirectional Long-short Temporal Memory) neural network. To further quantify prediction uncertainty, the nonparametric CKDE (Conditional Kernel Density Estimation) method are adopted to evaluate prediction error, where confidence boundaries can be obtained as interval prediction models. Finally, a DHS in Northwest China is selected for verification. The results show that the proposed method can accurately predict short-term DHL under the proposed multi-model structure. It is meaningful to achieve the purposes of on-demand heating, energy saving and environmental protection.

Via machinae: Searching for stellar streams using unsupervised machine learning

ABSTRACT We develop a new machine learning algorithm, via machinae, to identify cold stellar streams in data from the Gaia telescope. via machinae is based on ANODE, a general method that uses conditional density estimation and sideband interpolation to detect local overdensities in the data in a model agnostic way. By applying ANODE to the positions, proper motions, and photometry of stars observed by Gaia, via machinae obtains a collection of those stars deemed most likely to belong to a stellar stream. We further apply an automated line-finding method based on the Hough transform to search for line-like features in patches of the sky. In this paper, we describe the via machinae algorithm in detail and demonstrate our approach on the prominent stream GD-1. Though some parts of the algorithm are tuned to increase sensitivity to cold streams, the via machinae technique itself does not rely on astrophysical assumptions, such as the potential of the Milky Way or stellar isochrones. This flexibility suggests that it may have further applications in identifying other anomalous structures within the Gaia data set, for example debris flow and globular clusters.

A novel hybrid model of ARIMA‐MCC and CKDE‐GARCH for urban short‐term traffic flow prediction

The linearity and heteroscedasticity are the important characteristics of short-term traffic flow data. Generally, the autoregressive integrated moving average (ARIMA) model and the generalized autoregressive conditional heteroscedasticity (GARCH) model are respectively used to explain these characteristics. However, the prerequisite for the use of ARIMA is that the training residuals should follow the standard Gaussian distribution, which is hard to be satisfied in practice. Meanwhile, the variance in the GARCH model usually neglects the time-varying characteristic. To address these problems, this paper proposes an innovative method based on the combination of ARIMA, maximum correntropy criterion (MCC), conditional kernel density estimation (CKDE), and GARCH. Specifically, the MCC method is first employed to estimate the coefficients of the ARIMA model (i.e. ARIMA-MCC), by which the linear prediction is conducted. Then, the CKDE model is established to describe the training residuals obtained by ARIMA-MCC and estimate the time-varying variance in the GARCH model (i.e. GARCH-CKDE). Case studies based on four groups of data measured in the urban road are used to evaluate the performance of the proposed method. Compared with the traditional ARIMA model, the improvement by the proposed method reaches 14.45% in terms of the evaluation criterion of mean absolute percentage error.

Conditional sum-product networks: Modular probabilistic circuits via gate functions

While probabilistic graphical models are a central tool for reasoning under uncertainty in AI, they are in general not as expressive as deep neural models, and inference is notoriously hard and slow. In contrast, deep probabilistic models such as sum-product networks (SPNs) capture joint distributions and ensure tractable inference, but still lack the expressive power of intractable models based on deep neural networks. In this paper, we introduce conditional SPNs (CSPNs)—conditional density estimators for multivariate and potentially hybrid domains—and develop a structure-learning approach that derives both the structure and parameters of CSPNs from data. To harness the expressive power of deep neural networks (DNNs), we also show how to realize CSPNs by conditioning the parameters of vanilla SPNs on the input using DNNs as gate functions. In contrast to SPNs whose high-level structure can not be explicitly manipulated, CSPNs can naturally be used as tractable building blocks of deep probabilistic models whose modular structure maintains high-level interpretability. In experiments, we demonstrate that CSPNs are competitive with other probabilistic models and yield superior performance on structured prediction, conditional density estimation, auto-regressive image modeling, and multilabel image classification. In particular, we show that employing CSPNs as encoders and decoders within variational autoencoders can help to relax the commonly used mean field assumption and in turn improve performance.

Non parametric estimations of the conditional density and mode when the regressor and the response are curves

We develop new estimation results for the functional relationship between a regressor and a response which are functions indexed by time or by spatial locations. The regressor is assumed to belong to a semi-metric space whereas the responses belong to a Hilbert space First, we build a double-kernel estimator of the conditional density function, via a Nadaraya-Watson method. Then, we deduce a conditional mode estimator as the value that maximizes the conditional density estimator. Then, we establish the strong uniform consistencies, with rates, of the two constructed estimators. In this context, we wished to set up these preliminary results which will certainly motivate several works on this same subject.

Asymptotic properties of a nonparametric conditional density estimator in the local linear estimation for functional data via a functional single-index model

This paper deals with the conditional density estimator of a real response variable given a functional random variable (i.e., takes values in an infinite-dimensional space). Specifically, we focus on the functional index model, this approach represents a good compromise between nonparametric and parametric models. Then we give under general conditions and when the variables are independent, the quadratic error and asymptotic normality of estimator by local linear method, based on the single-index structure. Finally, we complete these theoretical advances by some simulation studies showing both the practical result of the local linear method and the good behaviour for finite sample sizes of the estimator and of the Monte Carlo methods to create functional pseudo-confidence area.

Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models

Mixture of experts (MoE) models are widely applied for conditional probability density estimation problems. We demonstrate the richness of the class of MoE models by proving denseness results in Lebesgue spaces, when inputs and outputs variables are both compactly supported. We further prove an almost uniform convergence result when the input is univariate. Auxiliary lemmas are proved regarding the richness of the soft-max gating function class, and their relationships to the class of Gaussian gating functions.

Kernel conditional density and mode estimation for psi-weakly dependent observations

In practical problems connected with forecasting, it sometimes happens that the classical regression estimation is not informative enough to make good predictions of a response variable. This occurs typically when multi-modality, asymmetry, or heteroscedastic noise characterizes the underlying distribution function. In this situation, conditional mode estimation may constitute an alternative method to prediction, because conditional density is more adequate to describe the association between an explanatory data vector and a target variable. In this paper we derive rates of convergence for kernel conditional density and mode functions estimators under psi-weak dependence condition. The asymptotic distribution of the mode function estimator is established and the accuracy of the proposed estimators is illustrated via a simulation study.

Complete parameter inference for GW150914 using deep learning

The LIGO and Virgo gravitational-wave observatories have detected many exciting events over the past 5 years. To infer the system parameters, iterative sampling algorithms such as MCMC are typically used with Bayes’ theorem to obtain posterior samples—by repeatedly generating waveforms and comparing to measured strain data. However, as the rate of detections grows with detector sensitivity, this poses a growing computational challenge. To confront this challenge, as well as that of fast multimessenger alerts, in this study we apply deep learning to learn non-iterative surrogate models for the Bayesian posterior. We train a neural-network conditional density estimator to model posterior probability distributions over the full 15-dimensional space of binary black hole system parameters, given detector strain data from multiple detectors. We use the method of normalizing flows—specifically, a neural spline flow—which allows for rapid sampling and density estimation. Training the network is likelihood-free, requiring samples from the data generative process, but no likelihood evaluations. Through training, the network learns a global set of posteriors: it can generate thousands of independent posterior samples per second for any strain data consistent with the training distribution. We demonstrate our method by performing inference on GW150914, and obtain results in close agreement with standard techniques.

On the conditional density estimation for continuous time processes with values in functional spaces

This paper is devoted to the conditional density estimation for continuous time processes with values in functional spaces. Under standard assumptions, we prove the uniform convergence of the conditional density estimator, from which we deduce the almost sure convergence of the conditional mode estimator. Then, the convergence of the conditional distribution function and the regression function estimators are established.

ADAPTIVE IMPORTANCE SAMPLING VIA AUTO-REGRESSIVE GENERATIVE MODELS AND GAUSSIAN PROCESSES.

The quality of importance distribution is vital to adaptive importance sampling, especially in high dimensional sampling spaces where the target distributions are sparse and hard to approximate. This requires that the proposal distributions are expressive and easily adaptable. Because of the need for weight calculation, point evaluation of the proposal distributions is also needed. The Gaussian process has been proven to be a highly expressive non-parametric model for conditional density estimation whose training process is also straightforward. In this paper, we introduce a class of adaptive importance sampling methods where the proposal distribution is constructed in a way that Gaussian processes are combined autoregressively. By numerical experiments of sampling from a high dimensional target distribution, we demonstrate that the method is accurate and efficient compared to existing methods.

Conditional Density Estimation of Service Metrics for Networked Services

We predict the conditional distributions of service metrics, such as response time or frame rate, from infrastructure measurements in a networked environment. From such distributions, key statistics of the service metrics, including mean, variance, or quantiles can be computed, which are essential for predicting SLA conformance and enabling service assurance. We present and assess two methods for prediction: (1) mixture models with Gaussian or Lognormal kernels, whose parameters are estimated using mixture density networks, a class of neural networks, and (2) histogram models, which require the target space to be discretized. We apply these methods to a VoD service and a KV store service running on our lab testbed. A comparative evaluation shows the relative effectiveness of the methods when applied to operational data. We find that both methods allow for accurate prediction. While mixture models provide a general and elegant solution, they incur a very high overhead related to hyper-parameter search and neural network training. Histogram models, on the other hand, allow for efficient training, but require adjustment to the specific use case.

Nonparametric conditional density estimation in a deep learning framework for short-term forecasting

Short-term forecasting is an important tool in understanding environmental processes. In this paper, we incorporate machine learning algorithms into a conditional distribution estimator for the purposes of forecasting tropical cyclone intensity. Many machine learning techniques give a single-point prediction of the conditional distribution of the target variable, which does not give a full accounting of the prediction variability. Conditional distribution estimation can provide extra insight on predicted response behavior, which could influence decision-making and policy. We propose a technique that simultaneously estimates the entire conditional distribution and flexibly allows for machine learning techniques to be incorporated. A smooth model is fit over both the target variable and covariates, and a logistic transformation is applied on the model output layer to produce an expression of the conditional density function. We provide two examples of machine learning models that can be used, polynomial regression and deep learning models. To achieve computational efficiency, we propose a case–control sampling approximation to the conditional distribution. A simulation study for four different data distributions highlights the effectiveness of our method compared to other machine learning-based conditional distribution estimation techniques. We then demonstrate the utility of our approach for forecasting purposes using tropical cyclone data from the Atlantic Seaboard. This paper gives a proof of concept for the promise of our method, further computational developments can fully unlock its insights in more complex forecasting and other applications.

Interval estimation of conditional probabilities in Bayesian Belief Network

An interval estimation of conditional probabilities in Bayesian belief network is considered, taking into account the uncertainty of the initial data. To account for errors in the values of functions and arguments, it is proposed to use confluent analysis methods. It is assumed that the conditional probability densities correspond to the normal distribution law. Formulas for interval estimation of conditional probability densities are given, taking into account the errors of their parameters. The results of mathematical modeling of obtaining point and interval estimates of conditional and a posteriori probabilities of the Bayesian belief network are shown.

An efficient estimation of failure probability in the presence of random and interval hybrid uncertainty

In the presence of random and interval hybrid uncertainty (RI-HU), the safety degree of the structure system can be quantified by the upper and lower bounds of failure probability. However, there is a lack of efficient methods for estimating failure probability under RI-HU in present. Therefore, a novel method is proposed in this paper. In the proposed method, the interval variables are extended to the random variables by assigning a priori probability density function, in which the conditional density estimation (CDE)–based method and conditional probability estimation (CPE)–based method are proposed, and the failure probability varying with the interval variables can be obtained by only one group Monte Carlo simulation (MCS). Since the computational complexity of CPE is much lower than that of CDE, the CPE-based method is mainly concerned. In the CPE-based method, the conditional failure probability on a realization of the extended interval vector is approximated by that on a differential region adjacent to the corresponding realization; then, the density function estimation required in the CDE can be avoided. In order to ensure the accuracy of the CPE, a strategy is proposed to adaptively select the differential region, in which the MCS can be combined with the CPE (CPE + MCS) and the adaptive Kriging can be nested into the CPE + MCS for improving the efficiency. To improve the efficiency further, the meta-model importance sampling nested Kriging is combined with the CPE-based method. The presented examples illustrate the superiority of the proposed method over the existing methods.

Asymptotic normality of conditional density and conditional mode in the functional single index model

The main objective of this paper is to investigate the nonparametric estimation of the conditional density of a scalar response variable Y, given the explanatory variable X taking value in a Hilbert space when the sample of observations is considered as an independent random variables with identical distribution (i.i.d) and are linked with a single functional index structure. First of all, a kernel type estimator for the conditional density function (cond-df) is introduced. Afterwards, the asymptotic properties are stated for a conditional density estimator when the observations are linked with a single-index structure from which one derives a central limit theorem (CLT) of the conditional density estimator to show the asymptotic normality of the kernel estimate of this model. As an application the conditional mode in functional single-index model is presented, and the asymptotic (1 – ζ) confidence interval of the conditional mode function is given for 0

Conditional Density Estimation, Latent Variable Discovery, and Optimal Transport

AbstractA framework is proposed that addresses both conditional density estimation and latent variable discovery. The objective function maximizes explanation of variability in the data, achieved through the optimal transport barycenter generalized to a collection of conditional distributions indexed by a covariate—either given or latent—in any suitable space. Theoretical results establish the existence of barycenters, a minimax formulation of optimal transport maps, and a general characterization of variability via the optimal transport cost. This framework leads to a family of nonparametric neural network‐based algorithms, the BaryNet, with a supervised version that estimates conditional distributions and an unsupervised version that assigns latent variables. The efficacy of BaryNets is demonstrated by tests on both artificial and real‐world data sets. A parallel drawn between autoencoders and the barycenter framework leads to the Barycentric autoencoder algorithm (BAE). © 2020 Wiley Periodicals LLC.

Conditional Kernel Density Estimation Considering Autocorrelation for Renewable Energy Probabilistic Modeling

Renewable energy is essential for energy security and global warming mitigation. However, renewable power generation is uncertain due to volatile weather conditions and complex equipment operations. It is therefore important to understand and characterize the uncertainty in renewable power generation to improve operational efficiency. In this paper, we proposed a novel conditional density estimation method to model the distribution of power generation under various weather conditions. It explicitly accounted for the temporal dependence in the data stream and used an iterative procedure to reduce the bias in conventional density estimation. Compared with existing literature, our approach is especially useful for the purpose of short-term modeling, where the temporal dependence plays a more significant role. We demonstrate our method and compare it with alternatives through real applications.

Evaluation of probabilistic photometric redshift estimation approaches for The Rubin Observatory Legacy Survey of Space and Time (LSST)

Abstract Many scientific investigations of photometric galaxy surveys require redshift estimates, whose uncertainty properties are best encapsulated by photometric redshift (photo-z) posterior probability density functions (PDFs). A plethora of photo-z PDF estimation methodologies abound, producing discrepant results with no consensus on a preferred approach. We present the results of a comprehensive experiment comparing twelve photo-z algorithms applied to mock data produced forLarge Synoptic Survey Telescope The Rubin Observatory Legacy Survey of Space and Time (lsst) Dark Energy Science Collaboration (desc). By supplying perfect prior information, in the form of the complete template library and a representative training set as inputs to each code, we demonstrate the impact of the assumptions underlying each technique on the output photo-z PDFs. In the absence of a notion of true, unbiased photo-z PDFs, we evaluate and interpret multiple metrics of the ensemble properties of the derived photo-z PDFs as well as traditional reductions to photo-z point estimates. We report systematic biases and overall over/under-breadth of the photo-z PDFs of many popular codes, which may indicate avenues for improvement in the algorithms or implementations. Furthermore, we raise attention to the limitations of established metrics for assessing photo-z PDF accuracy; though we identify the conditional density estimate (CDE) loss as a promising metric of photo-z PDF performance in the case where true redshifts are available but true photo-z PDFs are not, we emphasize the need for science-specific performance metrics.