Shrinkage Priors Research Articles

Bayesian approaches to variable selection: a comparative study from practical perspectives.

In many clinical studies, researchers are interested in parsimonious models that simultaneously achieve consistent variable selection and optimal prediction. The resulting parsimonious models will facilitate meaningful biological interpretation and scientific findings. Variable selection via Bayesian inference has been receiving significant advancement in recent years. Despite its increasing popularity, there is limited practical guidance for implementing these Bayesian approaches and evaluating their comparative performance in clinical datasets. In this paper, we review several commonly used Bayesian approaches to variable selection, with emphasis on application and implementation through R software. These approaches can be roughly categorized into four classes: namely the Bayesian model selection, spike-and-slab priors, shrinkage priors, and the hybrid of both. To evaluate their variable selection performance under various scenarios, we compare these four classes of approaches using real and simulated datasets. These results provide practical guidance to researchers who are interested in applying Bayesian approaches for the purpose of variable selection.

Intergenerational Associations Between Maternal Diet and Childhood Adiposity: A Bayesian Regularized Mediation Analysis

Growing evidence supports a positive association between childhood obesity and chronic diseases in later life. It is also suggested that childhood obesity is more prevalent for children born from pregnancies complicated by metabolic disorders such as gestational diabetes, and can be related to maternal dietary factors during gestation. Extending conventional analyses that report only the marginal associations within non-causal mediation frameworks, we present mediation analysis in the case of multiple exposures and multiple mediators using a regularized two-stage approach. By placing shrinkage priors on each parameter relating to direct and indirect effects, a parsimonious model can be obtained, and consequently, the most relevant pathways will be selected to inform the development of efficient prevention programs. We apply this method to data from the Danish site of the Diabetes & Women’s Health Study, Danish National Birth Cohort (DNBC), and find 6 significant maternal risk factors either directly or indirectly affecting childhood body mass index z score at age 7. Simulations with data-generating mechanisms similar to the DNBC data demonstrate good performance of the proposed model.

Adaptive Bayesian Spectral Analysis of High-Dimensional Nonstationary Time Series

This article introduces a nonparametric approach to spectral analysis of a high-dimensional multivariate nonstationary time series. The procedure is based on a novel frequency-domain factor model that provides a flexible yet parsimonious representation of spectral matrices from a large number of simultaneously observed time series. Real and imaginary parts of the factor loading matrices are modeled independently using a prior that is formulated from the tensor product of penalized splines and multiplicative gamma process shrinkage priors, allowing for infinitely many factors with loadings increasingly shrunk toward zero as the column index increases. Formulated in a fully Bayesian framework, the time series is adaptively partitioned into approximately stationary segments, where both the number and locations of partition points are assumed unknown. Stochastic approximation Monte Carlo techniques are used to accommodate the unknown number of segments, and a conditional Whittle likelihood-based Gibbs sampler is developed for efficient sampling within segments. By averaging over the distribution of partitions, the proposed method can approximate both abrupt and slowly varying changes in spectral matrices. Performance of the proposed model is evaluated by extensive simulations and demonstrated through the analysis of high-density electroencephalography. Supplementary materials for this article are available online.

Minnesota-type adaptive hierarchical priors for large Bayesian VARs

Large Bayesian VARs with stochastic volatility are increasingly used in empirical macroeconomics. The key to making these highly parameterized VARs useful is the use of shrinkage priors. We develop a family of priors that captures the best features of two prominent classes of shrinkage priors: adaptive hierarchical priors and Minnesota priors. Like adaptive hierarchical priors, these new priors ensure that only ‘small’ coefficients are strongly shrunk to zero, while ‘large’ coefficients remain intact. At the same time, these new priors can also incorporate many useful features of the Minnesota priors such as cross-variable shrinkage and shrinking coefficients on higher lags more aggressively. We introduce a fast posterior sampler to estimate BVARs with this family of priors—for a BVAR with 25 variables and 4 lags, obtaining 10,000 posterior draws takes about 3 min on a standard desktop computer. In a forecasting exercise, we show that these new priors outperform both adaptive hierarchical priors and Minnesota priors.

Fast Markov Chain Monte Carlo for High-Dimensional Bayesian Regression Models With Shrinkage Priors

In the past decade, many Bayesian shrinkage models have been developed for linear regression problems where the number of covariates, p, is large. Computation of the intractable posterior is often done with three-block Gibbs samplers (3BG), based on representing the shrinkage priors as scale mixtures of Normal distributions. An alternative computing tool is a state of the art Hamiltonian Monte Carlo (HMC) method, which can be easily implemented in the Stan software. However, we found both existing methods to be inefficient and often impractical for large p problems. Following the general idea of Rajaratnam et al., we propose two-block Gibbs samplers (2BG) for three commonly used shrinkage models, namely, the Bayesian group lasso, the Bayesian sparse group lasso, and the Bayesian fused lasso models. We demonstrate with simulated and real data examples that the Markov chains underlying 2BG’s converge much faster than that of 3BG’s, and no worse than that of HMC. At the same time, the computing costs of 2BG’s per iteration are as low as that of 3BG’s, and can be several orders of magnitude lower than that of HMC. As a result, the newly proposed 2BG is the only practical computing solution to do Bayesian shrinkage analysis for datasets with large p. Further, we provide theoretical justifications for the superior performance of 2BG’s. We establish geometric ergodicity of Markov chains associated with the 2BG for each of the three Bayesian shrinkage models. We also prove, for most cases of the Bayesian group lasso and the Bayesian sparse group lasso model, the Markov operators for the 2BG chains are trace-class. Whereas for all cases of all three Bayesian shrinkage models, the Markov operator for the 3BG chains is not even Hilbert–Schmidt. Supplementary materials for this article are available online.

Dynamic shrinkage in time-varying parameter stochastic volatility in mean models.

SummarySuccessful forecasting models strike a balance between parsimony and flexibility. This is often achieved by employing suitable shrinkage priors that penalize model complexity but also reward model fit. In this article, we modify the stochastic volatility in mean (SVM) model by introducing state‐of‐the‐art shrinkage techniques that allow for time variation in the degree of shrinkage. Using a real‐time inflation forecast exercise, we show that employing more flexible prior distributions on several key parameters sometimes improves forecast performance for the United States, the United Kingdom, and the euro area (EA). Comparing in‐sample results reveals that our proposed model yields qualitatively similar insights to the original version of the model.

Shrinkage in the Time-Varying Parameter Model Framework Using the R Package shrinkTVP

Time-varying parameter (TVP) models are widely used in time series analysis to flexibly deal with processes which gradually change over time. However, the risk of overfitting in TVP models is well known. This issue can be dealt with using appropriate global-local shrinkage priors, which pull time-varying parameters towards static ones. In this paper, we introduce the R package shrinkTVP (Knaus, Bitto-Nemling, Cadonna, and FrühwirthSchnatter 2021), which provides a fully Bayesian implementation of shrinkage priors for TVP models, taking advantage of recent developments in the literature, in particular those of Bitto and Frühwirth-Schnatter (2019) and Cadonna, Frühwirth-Schnatter, and Knaus (2020). The package shrinkTVP allows for posterior simulation of the parameters through an efficient Markov Chain Monte Carlo scheme. Moreover, summary and visualization methods, as well as the possibility of assessing predictive performance through log-predictive density scores, are provided. The computationally intensive tasks have been implemented in C++ and interfaced with R. The paper includes a brief overview of the models and shrinkage priors implemented in the package. Furthermore, core functionalities are illustrated, both with simulated and real data.

Sparse Modeling for Cross-Industry Returns Predictability

We develop a new variational Bayes estimation method for large-dimensional sparse vector autoregressive models with exogenous predictors. Unlike existing Markov chain Monte Carlo (MCMC) and variational Bayes (VB) algorithms, our approach is not based on a structural form representation of the model, but directly shrinkage the lead-lag cross sectional interdependencies in the transition matrix. We compare the performance of our approach against state-of-the-art MCMC and VB methods, both in simulation and when forecasting a large cross section of industry portfolios spanning almost a hundred years of monthly data. The main results provide robust evidence that by directly shrinking weak cross sectional signals one can substantially improve both the statistical and economic out-of-sample performance of multivariate models for return predictability. This result holds across a variety of alternative shrinkage priors, such as Bayesian adaptive lasso, normal-gamma and horseshoe priors.

Shrinkage Priors for High-Dimensional Demand Estimation

Estimating demand for wide assortments of differentiated goods requires the specification of a demand system that is sufficiently flexible. However, flexible models are highly parameterized so estimation requires appropriate forms of regularization to avoid overfitting. In this paper, we study the specification of Bayesian shrinkage priors for price elasticity parameters within a log-linear demand system where the number of price elasticity parameters grows quadratically in the number of goods. Traditional regularized estimators shrink regression coefficients towards zero which can be at odds with many economic properties of price effects. We propose a hierarchical extension of the class of global-local priors commonly used in regression modeling to allow the direction and rate of shrinkage to depend on a product classification tree. We use both simulated data and retail scanner data to show that, in the absence of a strong signal in the data, estimates of price elasticities and demand predictions can be improved by imposing shrinkage to higher-level group effects rather than zero.

Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior

Prior distributions for high-dimensional linear regression require specifying a joint distribution for the unobserved regression coefficients, which is inherently difficult. We instead propose a new class of shrinkage priors for linear regression via specifying a prior first on the model fit, in particular, the coefficient of determination, and then distributing through to the coefficients in a novel way. The proposed method compares favorably to previous approaches in terms of both concentration around the origin and tail behavior, which leads to improved performance both in posterior contraction and in empirical performance. The limiting behavior of the proposed prior is , both around the origin and in the tails. This behavior is optimal in the sense that it simultaneously lies on the boundary of being an improper prior both in the tails and around the origin. None of the existing shrinkage priors obtain this behavior in both regions simultaneously. We also demonstrate that our proposed prior leads to the same near-minimax posterior contraction rate as the spike-and-slab prior. Supplementary materials for this article are available online.

Funding Liquidity Risk in Emerging Markets: Is COVID-19 Any Different?

We construct synthetic spreads representing funding liquidity risk in BRICS economies and examine whether this risk transmits from the developed world through market channels. Applying a Bayesian Trend Filtering model on daily data encompassing three crises, we arrive at two salient conclusions: (a) the credit market and stock market channels in BRICS economies are not risk transmitters, and (b) the debt market is an intermittent carrier in a handful of countries. Deeper analysis of the results' temporal patterns shows key central banking decisions to precede or coincide with spillover attenuation, which, too, progressively waned with each crisis. In fact, with the understandable exception of China, spillover phenomenon is inert during the COVID-19 pandemic.

Regularizing Bayesian predictive regressions

Regularizing Bayesian predictive regressions provides a framework for prior sensitivity analysis via the regularization path. We jointly regularize both expectations and covariance matrices using a pair of shrinkage priors. Our methodology applies directly to vector autoregressions and seemingly unrelated regressions (SUR). By exploiting a duality between penalties and priors, we reinterpret two classic macrofinance studies: equity premium predictability and macroforecastability of bond risk premia. We find those plausible prior specifications for predictability for excess S&P 500 returns exist, using predictors as book-to-market ratios, consumption–wealth ratio, and T-bill rates. We evaluate our forecasts using a market-timing strategy and show how ours outperforms buy-and-hold. We also predict multiple bond excess returns involving a high-dimensional set of macroeconomic fundamentals with a regularized SUR model. We find the predictions from latent factor models such as PCA are sensitive to prior specifications. Finally, we conclude with directions for future research.

Efficient weighting methods for genomic best linear-unbiased prediction (BLUP) adapted to the genetic architectures of quantitative traits.

Genomic best linear-unbiased prediction (GBLUP) assumes equal variance for all marker effects, which is suitable for traits that conform to the infinitesimal model. For traits controlled by major genes, Bayesian methods with shrinkage priors or genome-wide association study (GWAS) methods can be used to identify causal variants effectively. The information from Bayesian/GWAS methods can be used to construct the weighted genomic relationship matrix (G). However, it remains unclear which methods perform best for traits varying in genetic architecture. Therefore, we developed several methods to optimize the performance of weighted GBLUP and compare them with other available methods using simulated and real data sets. First, two types of methods (marker effects with local shrinkage or normal prior) were used to obtain test statistics and estimates for each marker effect. Second, three weighted G matrices were constructed based on the marker information from the first step: (1) the genomic-feature-weighted G, (2) the estimated marker-variance-weighted G, and (3) the absolute value of the estimated marker-effect-weighted G. Following the above process, six different weighted GBLUP methods (local shrinkage/normal-prior GF/EV/AEWGBLUP) were proposed for genomic prediction. Analyses with both simulated and real data demonstrated that these options offer flexibility for optimizing the weighted GBLUP for traits with a broad spectrum of genetic architectures. The advantage of weighting methods over GBLUP in terms of accuracy was trait dependant, ranging from 14.8% to marginal for simulated traits and from 44% to marginal for real traits. Local-shrinkage prior EVWGBLUP is superior for traits mainly controlled by loci of a large effect. Normal-prior AEWGBLUP performs well for traits mainly controlled by loci of moderate effect. For traits controlled by some loci with large effects (explain 25-50% genetic variance) and a range of loci with small effects, GFWGBLUP has advantages. In conclusion, the optimal weighted GBLUP method for genomic selection should take both the genetic architecture and number of QTLs of traits into consideration carefully.

Joint Bayesian Estimation of Voxel Activation and Inter-regional Connectivity in fMRI Experiments.

Brain activation and connectivity analyses in task-based functional magnetic resonance imaging (fMRI) experiments with multiple subjects are currently at the forefront of data-driven neuroscience. In such experiments, interest often lies in understanding activation of brain voxels due to external stimuli and strong association or connectivity between the measurements on a set of pre-specified groups of brain voxels, also known as regions of interest (ROI). This article proposes a joint Bayesian additive mixed modeling framework that simultaneously assesses brain activation and connectivity patterns from multiple subjects. In particular, fMRI measurements from each individual obtained in the form of a multi-dimensional array/tensor at each time are regressed on functions of the stimuli. We impose a low-rank parallel factorization decomposition on the tensor regression coefficients corresponding to the stimuli to achieve parsimony. Multiway stick-breaking shrinkage priors are employed to infer activation patterns and associated uncertainties in each voxel. Further, the model introduces region-specific random effects which are jointly modeled with a Bayesian Gaussian graphical prior to account for the connectivity among pairs of ROIs. Empirical investigations under various simulation studies demonstrate the effectiveness of the method as a tool to simultaneously assess brain activation and connectivity. The method is then applied to a multi-subject fMRI dataset from a balloon-analog risk-taking experiment, showing the effectiveness of the model in providing interpretable joint inference on voxel-level activations and inter-regional connectivity associated with how the brain processes risk. The proposed method is also validated through simulation studies and comparisons to other methods used within the neuroscience community.

The adaptive normal-hypergeometric-inverted-beta priors for sparse signals

We investigate the estimation of high-dimensional normal mean under sparsity. Most shrinkage priors in the literature are based on certain assumptions of sparsity levels and signal sizes. Violation of these assumptions can lead to unsatisfactory estimation. In this paper, we propose a new class of flexible priors, the adaptive normal-hypergeometric-inverted-Beta (ANHIB) priors, which generalize several popular shrinkage priors without requiring prior knowledge of data sparsity levels and signal sizes, and thus can be used as good default priors in a large variety of situations. We show that the ANHIB estimators provide strong suppression to noises and little shrinkage to large signals, and have consistently superior estimation performance under various sparsity levels and signal sizes.

Over-Fitting in The TVP Model: A Comparison of Shrinkage Priors in Inflation Forecasting

The study is empirically motivated to analyze the performance of new class of Bayesian shrinkage priors that are powerful in reducing time-varying parameters to static ones to avoid over-fitting problem in time-varying parameter models. We utilized newly improved shrinkage priors in a non-centered parameterization form following Bitto and Fruhwirth-Schnatter (2019) who introduced variance selection using normal-gamma prior which nests the Bayesian LASSO prior of Belmonte et al. (2014) and spike and slab prior as in Schnatter and Wagner (2010). Therefore, the priors are able to discriminate time-varying coefficients from the static ones and the coefficients that can be shrunk to zero. In the empirical exercise, we estimated the generalized Phillip curve for three inflation-targeting countries and produced evidence of significant time-variation in most of the predictors. We compare the performance of the priors in density forecast of inflation allowing for constant and stochastic volatility in the model estimation. Evidence from the estimates of the log predictive density score shows that the hierarchical Normal Gamma shrinkage prior produces the best results for Canada and South Africa whilst the Normal Bayesian LASSO produces the best results for New Zealand, and that adding stochastic volatility improves the performance of models in density forecast.

Bayesian Estimation of Correlation Matrices of Longitudinal Data

Estimation of correlation matrices is a challenging problem due to the notorious positive-definiteness constraint and high-dimensionality. Reparameterizing Cholesky factors of correlation matrices in terms of angles or hyperspherical coordinates where the angles vary freely in the range [0,π) has become popular in the last two decades. However, it has not been used in Bayesian estimation of correlation matrices perhaps due to lack of clear statistical relevance and suitable priors for the angles. In this paper, we show for the first time that for longitudinal data these angles are the inverse cosine of the semi-partial correlations (SPCs). This simple connection makes it possible to introduce physically meaningful selection and shrinkage priors on the angles or correlation matrices with emphasis on selection (sparsity) and shrinking towards longitudinal structure. Our method deals effectively with the positive-definiteness constraint in posterior computation. We compare the performance of our Bayesian estimation based on angles with some recent methods based on partial autocorrelations through simulation and apply the method to a data related to clinical trial on smoking.

Infinite Mixtures of Infinite Factor Analysers

Factor-analytic Gaussian mixture models are often employed as a model-based approach to clustering high-dimensional data. Typically, the numbers of clusters and latent factors must be specified in advance of model fitting, and remain fixed. The pair which optimises some model selection criterion is then chosen. For computational reasons, models in which the number of latent factors differ across clusters are rarely considered. Here the infinite mixture of infinite factor analysers (IMIFA) model is introduced. IMIFA employs a Pitman-Yor process prior to facilitate automatic inference of the number of clusters using the stick-breaking construction and a slice sampler. Furthermore, IMIFA employs multiplicative gamma process shrinkage priors to allow cluster-specific numbers of factors, automatically inferred via an adaptive Gibbs sampler. IMIFA is presented as the flagship of a family of factor-analytic mixture models, providing flexible approaches to clustering high-dimensional data. Applications to a benchmark data set, metabolomic spectral data, and a manifold learning handwritten digit example illustrate the IMIFA model and its advantageous features. These include obviating the need for model selection criteria, reducing the computational burden associated with the search of the model space, improving clustering performance by allowing cluster-specific numbers of factors, and quantifying uncertainty in the numbers of clusters and cluster-specific factors.

Function-on-Function Kriging, With Applications to Three-Dimensional Printing of Aortic Tissues

ABSTRACT Three-dimensional printed medical prototypes, which use synthetic metamaterials to mimic biological tissue, are becoming increasingly important in urgent surgical applications. However, the mimicking of tissue mechanical properties via three-dimensional printed metamaterial can be difficult and time-consuming, due to the functional nature of both inputs (metamaterial structure) and outputs (mechanical response curve). To deal with this, we propose a novel function-on-function kriging model for efficient emulation and tissue-mimicking optimization. For functional inputs, a key novelty of our model is the spectral-distance (SpeD) correlation function, which captures important spectral differences between two functional inputs. Dependencies for functional outputs are then modeled via a co-kriging framework. We further adopt shrinkage priors on both the input spectra and the output co-kriging covariance matrix, which allows the emulator to learn and incorporate important physics (e.g., dominant input frequencies, output curve properties). Finally, we demonstrate the effectiveness of the proposed SpeD emulator in a real-world study on mimicking human aortic tissue, and show that it can provide quicker and more accurate tissue-mimicking performance compared to existing methods in the medical literature.

BGVAR: Bayesian Global Vector Autoregressions with Shrinkage Priors in R

This document introduces the R library BGVAR to estimate Bayesian global vector autoregressions (GVAR) with shrinkage priors and stochastic volatility. The Bayesian treatment of GVARs allows us to include large information sets by mitigating issues related to overfitting. This improves inference and often leads to better out-of-sample forecasts. Computational efficiency is achieved by using C++ to considerably speed up time-consuming functions. To maximize usability, the package includes numerous functions for carrying out structural inference and forecasting. These include generalized and structural impulse response functions, forecast error variance and historical decompositions as well as conditional forecasts.