Conditional Mean Estimator Research Articles

Cauchy difference priors for edge-preserving Bayesian inversion

Abstract We consider inverse problems in which the unknown target includes sharp edges, for example interfaces between different materials. Such problems are typical in image reconstruction, tomography, and other inverse problems algorithms. A common solution for edge-preserving inversion is to use total variation (TV) priors. However, as shown by Lassas and Siltanen 2004, TV-prior is not discretization-invariant: the edge-preserving property is lost when the computational mesh is made denser and denser. In this paper we propose another class of priors for edge-preserving Bayesian inversion, the Cauchy difference priors. We construct Cauchy priors starting from continuous one-dimensional Cauchy motion, and show that its discretized version, Cauchy random walk, can be used as a non-Gaussian prior for edge-preserving Bayesian inversion. We generalize the methodology to two-dimensional Cauchy fields, and briefly consider a generalization of the Cauchy priors to Lévy α-stable random field priors. We develop a suitable posterior distribution sampling algorithm for conditional mean estimates with single-component Metropolis–Hastings. We apply the methodology to one-dimensional deconvolution and two-dimensional X-ray tomography problems.

Calculating Degrees of Freedom in Multivariate Local Polynomial Regression

The matrix that transforms the response variable in a regression to its predicted value is commonly referred to as the hat matrix. The trace of the hat matrix is a standard metric for calculating degrees of freedom. The two prominent theoretical frameworks for studying hat matrices to calculate degrees of freedom in local polynomial regressions – ANOVA and non-ANOVA – abstract from both mixed data and the potential presence of irrelevant covariates, both of which dominate empirical applications. In the multivariate local polynomial setup with a mix of continuous and discrete covariates, which include some irrelevant covariates, we formulate asymptotic expressions for the trace of both the non-ANOVA and ANOVA-based hat matrices from the estimator of the unknown conditional mean. The asymptotic expression of the trace of the non-ANOVA hat matrix associated with the conditional mean estimator is equal up to a linear combination of kernel-dependent constants to that of the ANOVA-based hat matrix. Additionally, we document that the trace of the ANOVA-based hat matrix converges to 0 in any setting where the bandwidths diverge. This attrition outcome can occur in the presence of irrelevant continuous covariates or it can arise when the underlying data generating process is in fact of polynomial order.

Hyperpriors for Matérn fields with applications in Bayesian inversion

We introduce non-stationary Matérn field priors with stochastic partial differential equations, and construct correlation length-scaling with hyperpriors. We model both the hyperprior and the Matérn prior as continuous-parameter random fields. As hypermodels, we use Cauchy and Gaussian random fields, which we map suitably to a desired correlation length-scaling range. For computations, we discretise the models with finite difference methods. We consider the convergence of the discretised prior and posterior to the discretisation limit. We apply the developed methodology to certain interpolation, numerical differentiation and deconvolution problems, and show numerically that we can make Bayesian inversion which promotes competing constraints of smoothness and edge-preservation. For computing the conditional mean estimator of the posterior distribution, we use a combination of Gibbs and Metropolis-within-Gibbs sampling algorithms.

A frequentist approach to Bayesian asymptotics

Ergodic theorem shows that ergodic averages of the posterior draws converge in probability to the posterior mean under the stationarity assumption. The literature also shows that the posterior distribution is asymptotically normal when the sample size of the original data considered goes to infinity. To the best of our knowledge, there is little discussion on the large sample behaviour of the posterior mean. In this paper, we aim to fill this gap. In particular, we extend the posterior mean idea to the conditional mean case, which is conditioning on a given vector of summary statistics of the original data. We establish a new asymptotic theory for the conditional mean estimator for the case when both the sample size of the original data concerned and the number of Markov chain Monte Carlo iterations go to infinity. Simulation studies show that this conditional mean estimator has very good finite sample performance. In addition, we employ the conditional mean estimator to estimate a GARCH(1,1) model for S&P 500 stock returns and find that the conditional mean estimator performs better than quasi-maximum likelihood estimation in terms of out-of-sample forecasting.

Interactive nonparametric analysis of nonlinear systems

In this paper we outline an interactive nonparametric methodology designed to facilitate the statistical analysis of nonlinear systems. The approach exploits an ensemble of nonparametric techniques including conditional density function estimation, conditional distribution function estimation, conditional mean estimation (regression) and conditional quantile estimation (quantile regression). By exploiting recent developments in nonparametric methodology and also in open source interactive platforms for data visualization and statistical analysis, we are able to provide an approach that facilitates enhanced understanding of complex empirical phenomenon. We illustrate this approach by exploring the inherent complexity of the Southern Ocean system as a carbon sink, measured in terms of fugacity of carbon dioxide at sea surface temperature (f CO2), in relation to a number of oceanic state variables, all measured in situ during the annual South African National Antarctic Expedition (SANAE) austral summer trips from Cape Town to the Antarctic, and back, between 2010 and 2015.

Quantile Connectedness: Modelling Tail Behaviour in the Topology of Financial Networks

We develop a new technique to estimate vector autoregressions by quantile regression. A factor structure is used to remove cross-section correlation in the residuals such that the system can be estimated on an equation-by-equation basis using existing quantile regression toolboxes. We use our model to study credit risk spillovers among a panel of 18 sovereigns and their respective financial sectors between January 2006 and February 2012. We show that idiosyncratic credit risk shocks do not propagate strongly at the median but that powerful spillovers occur in both tails. Furthermore, rolling sample analysis reveals marked time-varying tail-dependence. These important features of credit risk transmission are obscured in models estimated using conventional conditional mean estimators.

Does Diversification Drive Down Risk-adjusted Returns? A Quantile Regression Approach

Abstract This study examines diversification-performance relationship in the U.S. property-liability insurance industry over the period of 1996–2010. Unlike prior studies that rely on the conditional mean estimation method, we employ quantile regression, which captures the heterogeneous effects of diversification on conditional return distribution. The results show that diversification does not necessarily drive down risk-adjusted returns and its effects vary along return distribution. We find that there is a diversification discount for firms in the lower levels of return distribution, whereas a diversification premium exists for firms in the upper levels of return distribution. We provide evidence that the relations between risk-adjusted returns and other explanatory variables are not constant, but vary over the quantiles of return distribution. Our results are robust to alternative measures of an insurer’s performance and product diversification.

Converting high-dimensional regression to high-dimensional conditional density estimation

There is a growing demand for nonparametric conditional density estimators (CDEs) in fields such as astronomy and economics. In astronomy, for example, one can dramatically improve estimates of the parameters that dictate the evolution of the Universe by working with full conditional densities instead of regression (i.e., conditional mean) estimates. More generally, standard regression falls short in any prediction problem where the distribution of the response is more complex with multi-modality, asymmetry or heteroscedastic noise. Nevertheless, much of the work on high-dimensional inference concerns regression and classification only, whereas research on density estimation has lagged behind. Here we propose FlexCode, a fully nonparametric approach to conditional density estimation that reformulates CDE as a non-parametric orthogonal series problem where the expansion coefficients are estimated by regression. By taking such an approach, one can efficiently estimate conditional densities and not just expectations in high dimensions by drawing upon the success in high-dimensional regression. Depending on the choice of regression procedure, our method can adapt to a variety of challenging high-dimensional settings with different structures in the data (e.g., a large number of irrelevant components and nonlinear manifold structure) as well as different data types (e.g., functional data, mixed data types and sample sets). We study the theoretical and empirical performance of our proposed method, and we compare our approach with traditional conditional density estimators on simulated as well as real-world data, such as photometric galaxy data, Twitter data, and line-of-sight velocities in a galaxy cluster.

Nonparametric Analysis of Complex Nonlinear Systems

In this paper we propose a nonparametric methodology designed to facilitate the statistical analysis of complex systems. The proposed approach exploits an ensemble of nonparametric techniques including conditional density function estimation, conditional distribution function estimation, conditional mean estimation (regression) and conditional quantile estimation (quantile regression). By exploiting recent developments in nonparametric methodology and also in open source interactive platforms for data visualization and statistical analysis, we are able to provide an approach that facilitates enhanced understanding of complex empirical phenomenon. We illustrate this approach by exploring the inherent complexity of the Southern Ocean system as a carbon sink, measured in terms of fugacity of carbon dioxide at sea surface temperature (f CO2), in relation to a number of oceanic state variables, all measured in situ during the annual South African National Antarctic Expedition (SANAE) austral summer trips from Cape Town to the Antarctic, and back, between 2010 and 2015.

Posterior consistency and convergence rates for Bayesian inversion with hypoelliptic operators

The Bayesian approach to inverse problems is studied in the case where the forward map is a linear hypoelliptic pseudodifferential operator and measurement error is additive white Gaussian noise. The measurement model for an unknown Gaussian random variable is , where A is a finitely many orders smoothing linear hypoelliptic operator and is the noise magnitude. The covariance operator CU of U is smoothing of order , self-adjoint, injective and elliptic pseudodifferential operator. If was taking values in L2 then in Gaussian case solving the conditional mean (and maximum a posteriori) estimate is linked to solving the minimisation problem However, Gaussian white noise does not take values in L2 but in where is big enough. A modification of the above approach to solve the inverse problem is presented, covering the case of white Gaussian measurement noise. Furthermore, the convergence of the conditional mean estimate to the correct solution as is proven in appropriate function spaces using microlocal analysis. Also the frequentist posterior contractions rates are studied.

Extension of de Bruijn's identity to dependent non-Gaussian noise channels

Abstract De Bruijn's identity relates two important concepts in information theory: Fisher information and differential entropy. Unlike the common practice in the literature, in this paper we consider general additive non-Gaussian noise channels where more realistically, the input signal and additive noise are not independently distributed. It is shown that, for general dependent signal and noise, the first derivative of the differential entropy is directly related to the conditional mean estimate of the input. Then, by using Gaussian and Farlie–Gumbel–Morgenstern copulas, special versions of the result are given in the respective case of additive normally distributed noise. The previous result on independent Gaussian noise channels is included as a special case. Illustrative examples are also provided.

Bayesian Conditional Mean Estimation in Log‐Normal Linear Regression Models with Finite Quadratic Expected Loss

AbstractLog‐normal linear regression models are popular in many fields of research. Bayesian estimation of the conditional mean of the dependent variable is problematic as many choices of the prior for the variance (on the log‐scale) lead to posterior distributions with no finite moments. We propose a generalized inverse Gaussian prior for this variance and derive the conditions on the prior parameters that yield posterior distributions of the conditional mean of the dependent variable with finite moments up to a pre‐specified order. The conditions depend on one of the three parameters of the suggested prior; the other two have an influence on inferences for small and medium sample sizes. A second goal of this paper is to discuss how to choose these parameters according to different criteria including the optimization of frequentist properties of posterior means.

Impact of price realization on India's tea export: Evidence from Quantile Autoregressive Distributed Lag Model

Th e quantile autoregressive distributed lag model of Galvao et al. (2013) was employed to assess the impact of price realization on India's tea export. Th e results of the QADL varied signifi cantly from the conditional mean estimates. It was found that the tea export from India had autoregressive impact, and that production and export price realization had asymmetric relationship with India's tea export that varied over quantiles.

Maximum a posteriori probability estimates in infinite-dimensional Bayesian inverse problems

A demanding challenge in Bayesian inversion is to efficiently characterize the posterior distribution. This task is problematic especially in high-dimensional non-Gaussian problems, where the structure of the posterior can be very chaotic and difficult to analyse. Current inverse problem literature often approaches the problem by considering suitable point estimators for the task. Typically the choice is made between the maximum a posteriori (MAP) or the conditional mean (CM) estimate. The benefits of either choice are not well-understood from the perspective of infinite-dimensional theory. Most importantly, there exists no general scheme regarding how to connect the topological description of a MAP estimate to a variational problem. The recent results by Dashti and others (Dashti et al 2013 Inverse Problems 29 095017) resolve this issue for nonlinear inverse problems in Gaussian framework. In this work we improve the current understanding by introducing a novel concept called the weak MAP (wMAP) estimate. We show that any MAP estimate in the sense of Dashti et al (2013 Inverse Problems 29 095017) is a wMAP estimate and, moreover, how the wMAP estimate connects to a variational formulation in general infinite-dimensional non-Gaussian problems. The variational formulation enables to study many properties of the infinite-dimensional MAP estimate that were earlier impossible to study. In a recent work by the authors (Burger and Lucka 2014 Maximum a posteriori estimates in linear inverse problems with logconcave priors are proper bayes estimators preprint) the MAP estimator was studied in the context of the Bayes cost method. Using Bregman distances, proper convex Bayes cost functions were introduced for which the MAP estimator is the Bayes estimator. Here, we generalize these results to the infinite-dimensional setting. Moreover, we discuss the implications of our results for some examples of prior models such as the Besov prior and hierarchical prior.

Stochastic Galerkin Finite Element Method with Local Conductivity Basis for Electrical Impedance Tomography

The objective of electrical impedance tomography is to deduce information about the conductivity inside a physical body from electrode measurements of current and voltage at the object boundary. In this work, the unknown conductivity is modeled as a random field parametrized by its values at a set of pixels. The uncertainty in the pixel values is propagated to the electrode measurements by numerically solving the forward problem of impedance tomography by a stochastic Galerkin finite element method in the framework of the complete electrode model. For a given set of electrode measurements, the stochastic forward solution is employed by approximately parametrizing the posterior probability density of the conductivity and contact resistances. Subsequently, the conductivity is reconstructed by computing the maximum a posteriori and conditional mean estimates as well as the posterior covariance. The functionality of this approach is demonstrated with experimental water tank data.

Comment

Comment

Maximum a posteriori estimates in linear inverse problems with log-concave priors are proper Bayes estimators

A frequent matter of debate in Bayesian inversion is the question of which of the two principal point-estimators, the maximum a posteriori (MAP) or the conditional mean (CM) estimate, is to be preferred. As the MAP estimate corresponds to the solution given by variational regularization techniques, this is also a constant matter of debate between the two research areas. Following a theoretical argument—the Bayes cost formalism—the CM estimate is classically preferred for being the Bayes estimator for the mean squared error cost, while the MAP estimate is classically discredited for being only asymptotically the Bayes estimator for the uniform cost function. In this article we present recent theoretical and computational observations that challenge this point of view, in particular for high-dimensional sparsity-promoting Bayesian inversion. Using Bregman distances, we present new, proper convex Bayes cost functions for which the MAP estimator is the Bayes estimator. We complement this finding with results that correct further common misconceptions about MAP estimates. In total, we aim to rehabilitate MAP estimates in linear inverse problems with log-concave priors as proper Bayes estimators.

Multivariate Cauchy Estimator with Scalar Measurement and Process Noises

The conditional mean estimator for a $n$-state linear system with additive Cauchy measurement and process noises is developed. It is shown that although the Cauchy densities, which model the initial state, the process noise and the measurement noise have undefined first moments and an infinite second moment, the probability density function conditioned on the measurement history does have a finite conditional mean and conditional variance. For the multivariable system state, the characteristic function of the unnormalized conditional probability density function is sequentially propagated through measurement updates and dynamic state propagation, while expressing the resulting characteristic function in a closed analytical form. Once the characteristic function of the unnormalized conditional probability density function is obtained, the probability density function of the measurement history, the conditional mean, and conditional variance are easily computed from the characteristic function and its continuous first and second derivatives, evaluated at the origin in the spectral variables' domain. These closed form expressions yield the sequential state estimator. A three-state dynamic system example demonstrates numerically the performance of the Cauchy estimator.

Likelihood Ratios and Inference for Poisson Channels

In recent years, infinite-dimensional methods have been introduced for Gaussian channel estimation. The aim of this paper is to study the application of similar methods to Poisson channels. In particular, we compute the noncausal conditional mean estimator of a Poisson channel using the likelihood ratio and the discrete Malliavin gradient. This algorithm is suitable for numerical implementation via the Monte-Carlo scheme. As an application, we provide a new proof of a very deep and remarkable formula in Information Theory obtained recently in the literature and relating the derivatives of the input-output mutual information of a general Poisson channel and the conditional mean estimator of the input regardless the distribution of the latter. The use of the aforementioned stochastic analysis techniques allows us to extend these results to more general channels such as mixed Gaussian-Poisson channels.

Factor analysis parameter estimation from incomplete data

An expectation–maximization (EM) algorithm for factor analysis parameter estimation when observations are missing is developed. In contrast to existing EM algorithms for this problem, the algorithm here is developed assuming the missing observations are not part of the complete data in the EM formulation. The resulting algorithm provides increased computational efficiency through sparse matrix operations. The algorithm is demonstrated on two sparse, high-dimensional data sets that are prohibitively large for existing algorithms: the Netflix movie recommendation data set and the Yahoo! musical item data set. The resulting factor models are applied to predict missing values using conditional mean estimation, achieving root mean square errors of 0.9001 and 24.08 on the Netflix and Yahoo! data sets, respectively.