Posterior Contraction Rates Research Articles

Adaptive Bayesian estimation of conditional discrete-continuous distributions with an application to stock market trading activity

We consider Bayesian nonparametric estimation of conditional discrete-continuous distributions. Our model is based on a mixture of normal distributions with covariate dependent mixing probabilities. We use continuous latent variables for modeling the discrete part of the distribution. The marginal distribution of covariates is not modeled. Under anisotropic smoothness conditions on the data generating conditional distribution and a possibly increasing number of the support points for the discrete part of the distribution, we show that the posterior in our model contracts at frequentist adaptive optimal rates up to a log factor. Our results also imply an upper bound on the posterior contraction rate for predictive distributions when the data follow an ergodic Markov process and our model is used for modeling the Markov transition distribution. The proposed model performs well in an application to stock market trading activity.

Multiscale Bayesian survival analysis

We consider Bayesian nonparametric inference in the right-censoring survival model, where modeling is made at the level of the hazard rate. We derive posterior limiting distributions for linear functionals of the hazard, and then for ‘many’ functionals simultaneously in appropriate multiscale spaces. As an application, we derive Bernstein–von Mises theorems for the cumulative hazard and survival functions, which lead to asymptotically efficient confidence bands for these quantities. Further, we show optimal posterior contraction rates for the hazard in terms of the supremum norm. In medical studies, a popular approach is to model hazards a priori as random histograms with possibly dependent heights. This and more general classes of arbitrarily smooth prior distributions are considered as applications of our theory. A sampler is provided for possibly dependent histogram posteriors. Its finite sample properties are investigated on both simulated and real data experiments.

Bayesian Modelling of Time-Varying Conditional Heteroscedasticity

Conditional heteroscedastic (CH) models are routinely used to analyze financial datasets. The classical models such as ARCH-GARCH with time-invariant coefficients are often inadequate to describe frequent changes over time due to market variability. However, we can achieve significantly better insight by considering the time-varying analogs of these models. In this paper, we propose a Bayesian approach to the estimation of such models and develop a computationally efficient MCMC algorithm based on Hamiltonian Monte Carlo (HMC) sampling. We also established posterior contraction rates with increasing sample size in terms of the average Hellinger metric. The performance of our method is compared with frequentist estimates and estimates from the time constant analogs. To conclude the paper we obtain time-varying parameter estimates for some popular Forex (currency conversion rate) and stock market datasets.

Bayesian factor-adjusted sparse regression

Many sparse regression methods rely on an assumption that the covariates are weakly correlated, which hardly holds in many economic and financial datasets. To relax this assumption, we model the strongly correlated covariates by a factor structure: strong correlations among covariates are modeled by common factors, while the remaining variations of covariates are modeled as idiosyncratic components. We then propose a factor-adjusted sparse regression model and develop a semi-Bayesian estimation method for it. Posterior contraction rate and model selection consistency are established by a non-asymptotic analysis. Experimental studies show that the proposed method outperforms its Lasso analogue, manifests insensitivity to overestimates of the number of common factors, pays a negligible price when covariates are uncorrelated, scales up well with increasing sample size, dimensionality and sparsity, and converges fast to the posterior distribution. An application to the U.S. bond risk premia lends further support to the proposed model and method.

On posterior contraction of parameters and interpretability in Bayesian mixture modeling

We study posterior contraction behaviors for parameters of interest in the context of Bayesian mixture modeling, where the number of mixing components is unknown while the model itself may or may not be correctly specified. Two representative types of prior specification will be considered: one requires explicitly a prior distribution on the number of mixture components, while the other places a nonparametric prior on the space of mixing distributions. The former is shown to yield an optimal rate of posterior contraction on the model parameters under minimal conditions, while the latter can be utilized to consistently recover the unknown number of mixture components, with the help of a fast probabilistic post-processing procedure. We then turn the study of these Bayesian procedures to the realistic settings of model misspecification. It will be shown that the modeling choice of kernel density functions plays perhaps the most impactful roles in determining the posterior contraction rates in the misspecified situations. Drawing on concrete posterior contraction rates established in this paper we wish to highlight some aspects about the interesting tradeoffs between model expressiveness and interpretability that a statistical modeler must negotiate in the rich world of mixture modeling.

Ultra high-dimensional multivariate posterior contraction rate under shrinkage priors

In recent years, shrinkage priors have received much attention in high-dimensional data analysis from a Bayesian perspective. Compared with widely used spike-and-slab priors, shrinkage priors have better computational efficiency. But the theoretical properties, especially posterior contraction rate, which is important in uncertainty quantification, are not established in many cases. In this paper, we apply global–local shrinkage priors to high-dimensional multivariate linear regression with unknown covariance matrix. We show that when the prior is highly concentrated near zero and has heavy tail, the posterior contraction rates for both coefficients matrix and covariance matrix are nearly optimal. Our results hold when number of features p grows much faster than the sample size n, which is of great interest in modern data analysis. We show that a class of readily implementable scale mixture of normal priors satisfies the conditions of the main theorem.

Posterior contraction rate of sparse latent feature models with application to proteomics

The Indian buffet process (IBP) and phylogenetic Indian buffet process (pIBP) can be used as prior models to infer latent features in a data set. The theoretical properties of these models are under-explored, however, especially in high dimensional settings. In this paper, we show that under mild sparsity condition, the posterior distribution of the latent feature matrix, generated via IBP or pIBP priors, converges to the true latent feature matrix asymptotically. We derive the posterior convergence rate, referred to as the contraction rate. We show that the convergence results remain valid even when the dimensionality of the latent feature matrix increases with the sample size, therefore making the posterior inference valid in high dimensional settings. We demonstrate the theoretical results using computer simulation, in which the parallel-tempering Markov chain Monte Carlo method is applied to overcome computational hurdles. The practical utility of the derived properties is demonstrated by inferring the latent features in a reverse phase protein arrays (RPPA) dataset under the IBP prior model.

Multivariate Gaussian RBF‐net for smooth function estimation and variable selection

AbstractNeural networks are routinely used for nonparametric regression modeling. The interest in these models is growing with ever‐increasing complexities in modern datasets. With modern technological advancements, the number of predictors frequently exceeds the sample size in many application areas. Thus, selecting important predictors from the huge pool is an extremely important task for judicious inference. This paper proposes a novel flexible class of single‐layer radial basis functions (RBF) networks. The proposed architecture can estimate smooth unknown regression functions and also perform variable selection. We primarily focus on Gaussian RBF‐net due to its attractive properties. The extensions to other choices of RBF are fairly straightforward. The proposed architecture is also shown to be effective in identifying relevant predictors in a low‐dimensional setting using the posterior samples without imposing any sparse estimation scheme. We develop an efficient Markov chain Monte Carlo algorithm to generate posterior samples of the parameters. We illustrate the proposed method's empirical efficacy through simulation experiments, both in high and low dimensional regression problems. The posterior contraction rate is established with respect to empirical distance assuming that the error variance is unknown, and the true function belongs to a Hölder ball. We illustrate our method in a Human Connectome Project dataset to predict vocabulary comprehension and to identify important edges of the structural connectome.

Neuronized Priors for Bayesian Sparse Linear Regression

Although Bayesian variable selection methods have been intensively studied, their routine use in practice has not caught up with their non-Bayesian counterparts such as Lasso, likely due to difficulties in both computations and flexibilities of prior choices. To ease these challenges, we propose the neuronized priors to unify and extend some popular shrinkage priors, such as Laplace, Cauchy, horseshoe, and spike-and-slab priors. A neuronized prior can be written as the product of a Gaussian weight variable and a scale variable transformed from Gaussian via an activation function. Compared with classic spike-and-slab priors, the neuronized priors achieve the same explicit variable selection without employing any latent indicator variables, which results in both more efficient and flexible posterior sampling and more effective posterior modal estimation. Theoretically, we provide specific conditions on the neuronized formulation to achieve the optimal posterior contraction rate, and show that a broadly applicable MCMC algorithm achieves an exponentially fast convergence rate under the neuronized formulation. We also examine various simulated and real data examples and demonstrate that using the neuronization representation is computationally more or comparably efficient than its standard counterpart in all well-known cases. An R package NPrior is provided for using neuronized priors in Bayesian linear regression.

Posterior contraction and credible regions for level sets

For a given function f on a multivariate domain, the level sets, given by {x:f(x)=c} for different values of c, provide important geometrical insights about the underlying function of interest. The distance on level sets of two functions may be measured by the Hausdorff metric or a metric based on the Lebesgue measure of a discrepancy, both of which can be linked with the L∞-distance on the underlying functions. In a Bayesian framework, we derive posterior contraction rates and optimal sized credible sets with assured frequentist coverage for level sets in some nonparametric settings by extending some univariate L∞-posterior contraction rates to the corresponding multivariate settings. For the multivariate Gaussian white noise model, adaptive Hausdorff and Lebesgue contraction rates for levels sets of the signal function and its mixed order partial derivatives are derived using a wavelet series prior on the function. Assuming a known smoothness level of the signal function, an optimal sized credible region for a level set with assured frequentist coverage is derived based on a multidimensional trigonometric series prior. For the nonparametric regression problem, adaptive rates for level sets of the function and its mixed partial derivatives are obtained using a multidimensional wavelet series prior. When the smoothness level is given, optimal sized credible regions with assured frequentist coverage are obtained using a finite random series prior based on tensor products of B-splines. We also derive Hausdorff and Lebesgue contraction rates of a multivariate density function under a known smoothness setting.

Bayesian estimation of sparse precision matrices in the presence of Gaussian measurement error

Estimation of sparse, high-dimensional precision matrices is an important and challenging problem. Existing methods all assume that observations can be made precisely but, in practice, this often is not the case; for example, the instruments used to measure the response may have limited precision. The present paper incorporates measurement error in the context of estimating a sparse, high-dimensional precision matrix. In particular, for a Gaussian graphical model with data corrupted by Gaussian measurement error with unknown variance, we establish a general result which gives sufficient conditions under which the posterior contraction rates that hold in the no-measurement-error case carry over to the measurement-error case. Interestingly, this result does not require that the measurement error variance be small. We apply our general result to several cases with well-known prior distributions for sparse precision matrices and also to a case with a newly-constructed prior for precision matrices with a sparse factor-loading form. Two different simulation studies highlight the empirical benefits of accounting for the measurement error as opposed to ignoring it, even when that measurement error is relatively small.

Oracle posterior contraction rates under hierarchical priors

We offer a general Bayes theoretic framework to derive posterior contraction rates under a hierarchical prior design: the first-step prior serves to assess the model selection uncertainty, and the second-step prior quantifies the prior belief on the strength of the signals within the model chosen from the first step. In particular, we establish non-asymptotic oracle posterior contraction rates under (i) a local Gaussianity condition on the log likelihood ratio of the statistical experiment, (ii) a local entropy condition on the dimensionality of the models, and (iii) a sufficient mass condition on the second-step prior near the best approximating signal for each model. The first-step prior can be designed generically. The posterior distribution enjoys Gaussian tail behavior and therefore the resulting posterior mean also satisfies an oracle inequality, automatically serving as an adaptive point estimator in a frequentist sense. Model mis-specification is allowed in these oracle rates. The local Gaussianity condition serves as a unified attempt of non-asymptotic Gaussian quantification of the experiments, and can be easily verified in various experiments considered in [GvdV07a] and beyond. The general results are applied in various problems including: (i) trace regression, (ii) shape-restricted isotonic/convex regression, (iii) high-dimensional partially linear regression, (iv) covariance matrix estimation in the sparse factor model, (v) detection of non-smooth polytopal image boundary, and (vi) intensity estimation in a Poisson point process model. These new results serve either as theoretical justification of practical prior proposals in the literature, or as an illustration of the generic construction scheme of a (nearly) minimax adaptive estimator for a complicated experiment.

Time-varying auto-regressive models for count time-series

Count-valued time series data are routinely collected in many application areas. We are particularly motivated to study the count time series of daily new cases, arising from the COVID-19 spread. First, we propose a Bayesian framework to study the time-varying semiparametric AR(p) model for the count and then extend it to a more sophisticated time-varying INGARCH model. We calculate posterior contraction rates of the proposed Bayesian methods with respect to the average Hellinger metric. Our proposed structures of the models are amenable to Hamiltonian Monte Carlo (HMC) sampling for efficient computation. We substantiate our methods by simulations that show superiority compared to some of the existing methods that closely fit this setting. Finally, we analyze the daily time series data of newly confirmed cases in NYC to study the spread of COVID for three months.

On the Beta Prime Prior for Scale Parameters in High-Dimensional Bayesian Regression Models

We study high-dimensional Bayesian linear regression with a general beta prime distribution for the scale parameter. Under the assumption of sparsity, we show that appropriate selection of the hyperparameters in the beta prime prior leads to the (near) minimax posterior contraction rate when $p \gg n$. For finite samples, we propose a data-adaptive method for estimating the hyperparameters based on marginal maximum likelihood (MML). This enables our prior to adapt to both sparse and dense settings, and under our proposed empirical Bayes procedure, the MML estimates are never at risk of collapsing to zero. We derive efficient Monte Carlo EM and variational EM algorithms for implementing our model, which are available in the R package NormalBetaPrime. Simulations and analysis of a gene expression data set illustrate our model's self-adaptivity to varying levels of sparsity and signal strengths.

Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior

Prior distributions for high-dimensional linear regression require specifying a joint distribution for the unobserved regression coefficients, which is inherently difficult. We instead propose a new class of shrinkage priors for linear regression via specifying a prior first on the model fit, in particular, the coefficient of determination, and then distributing through to the coefficients in a novel way. The proposed method compares favorably to previous approaches in terms of both concentration around the origin and tail behavior, which leads to improved performance both in posterior contraction and in empirical performance. The limiting behavior of the proposed prior is , both around the origin and in the tails. This behavior is optimal in the sense that it simultaneously lies on the boundary of being an improper prior both in the tails and around the origin. None of the existing shrinkage priors obtain this behavior in both regions simultaneously. We also demonstrate that our proposed prior leads to the same near-minimax posterior contraction rate as the spike-and-slab prior. Supplementary materials for this article are available online.

Posterior contraction in sparse generalized linear models

Summary We study posterior contraction rates in sparse high-dimensional generalized linear models using priors incorporating sparsity. A mixture of a point mass at zero and a continuous distribution is used as the prior distribution on regression coefficients. In addition to the usual posterior, the fractional posterior, which is obtained by applying Bayes theorem with a fractional power of the likelihood, is also considered. The latter allows uniformity in posterior contraction over a larger subset of the parameter space. In our set-up, the link function of the generalized linear model need not be canonical. We show that Bayesian methods achieve convergence properties analogous to lasso-type procedures. Our results can be used to derive posterior contraction rates in many generalized linear models including logistic, Poisson regression and others.

Bayesian MIDAS penalized regressions: Estimation, selection, and prediction

We propose a new approach to mixed-frequency regressions in a high-dimensional environment that resorts to Group Lasso penalization and Bayesian estimation and inference. In particular, to improve the prediction properties of the model and its sparse recovery ability, we consider a Group Lasso with a spike-and-slab prior. Penalty hyper-parameters governing the model shrinkage are automatically tuned via an adaptive MCMC algorithm. We establish good frequentist asymptotic properties of the posterior prediction error, we recover the optimal posterior contraction rate, and we show optimality of the posterior predictive density. Simulations show that the proposed models have good selection and forecasting performance in small samples, even when the design matrix presents cross-correlation. When applied to forecasting U.S. GDP, our penalized regressions can outperform many strong competitors. Results suggest that financial variables may have some, although very limited, short-term predictive content.

Nonparametric Bayesian estimation of a Hölder continuous diffusion coefficient

We consider a nonparametric Bayesian approach to estimate the diffusion coefficient of a stochastic differential equation given discrete time observations over a fixed time interval. As a prior on the diffusion coefficient, we employ a histogram-type prior with piecewise constant realisations on bins forming a partition of the time interval. Specifically, these constants are realizations of independent inverse Gamma distributed randoma variables. We justify our approach by deriving the rate at which the corresponding posterior distribution asymptotically concentrates around the data-generating diffusion coefficient. This posterior contraction rate turns out to be optimal for estimation of a H\"older-continuous diffusion coefficient with smoothness parameter $0<\lambda\leq 1.$ Our approach is straightforward to implement, as the posterior distributions turn out to be inverse Gamma again, and leads to good practical results in a wide range of simulation examples. Finally, we apply our method on exchange rate data sets.

Optimal Bayesian estimation for random dot product graphs

Summary We propose and prove the optimality of a Bayesian approach for estimating the latent positions in random dot product graphs, which we call posterior spectral embedding. Unlike classical spectral-based adjacency, or Laplacian spectral embedding, posterior spectral embedding is a fully likelihood-based graph estimation method that takes advantage of the Bernoulli likelihood information of the observed adjacency matrix. We develop a minimax lower bound for estimating the latent positions, and show that posterior spectral embedding achieves this lower bound in the following two senses: it both results in a minimax-optimal posterior contraction rate and yields a point estimator achieving the minimax risk asymptotically. The convergence results are subsequently applied to clustering in stochastic block models with positive semidefinite block probability matrices, strengthening an existing result concerning the number of misclustered vertices. We also study a spectral-based Gaussian spectral embedding as a natural Bayesian analogue of adjacency spectral embedding, but the resulting posterior contraction rate is suboptimal by an extra logarithmic factor. The practical performance of the proposed methodology is illustrated through extensive synthetic examples and the analysis of Wikipedia graph data.

Posterior contraction rates for support boundary recovery

Given a sample of a Poisson point process with intensity λf(x,y)=n1(f(x)≤y), we study recovery of the boundary function f from a nonparametric Bayes perspective. Because of the irregularity of this model, the analysis is non-standard. We establish a general result for the posterior contraction rate with respect to the L1-norm based on entropy and one-sided small probability bounds. From this, specific posterior contraction results are derived for Gaussian process priors and priors based on random wavelet series.