Scale Mixtures Of Normal Distributions Research Articles

Integrated deviance information criterion for spatial autoregressive models with heteroskedasticity

In this study, we introduce the integrated deviance information criterion (DIC) for nested and non-nested model selection problems in heteroskedastic spatial autoregressive models. In a Bayesian estimation setting, we assume that the idiosyncratic error terms of our spatial autoregressive model have a scale mixture of normal distributions, where the scale mixture variables are latent variables that induce heteroskedasticity. We first derive the integrated likelihood function by analytically integrating out the scale mixture variables from the complete-data likelihood function. We then use the integrated likelihood function to formulate the integrated DIC measure. We investigate the finite sample performance of the integrated DIC in selecting the true model in a simulation study. The simulation results show that the integrated DIC performs satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, in a spatially augmented economic growth model, we use the integrated DIC to choose the spatial weights matrix that leads to better predictive accuracy.

Bayesian boundary trend filtering

Estimating boundary curves has many applications such as economics, climate science, and medicine. Bayesian trend filtering has been developed as one of locally adaptive smoothing methods to estimate the non-stationary trend of data. This paper develops a Bayesian trend filtering for estimating the boundary trend. To this end, the truncated multivariate normal working likelihood and global-local shrinkage priors based on the scale mixtures of normal distribution are introduced. In particular, well-known horseshoe prior for difference leads to locally adaptive shrinkage estimation for boundary trend. However, the full conditional distributions of the Gibbs sampler involve high-dimensional truncated multivariate normal distribution. To overcome the difficulty of sampling, an approximation of truncated multivariate normal distribution is employed. Using the approximation, the proposed models lead to an efficient Gibbs sampling algorithm via the Pólya-Gamma data augmentation. The proposed method is also extended by considering a nearly isotonic constraint. The performance of the proposed method is illustrated through some numerical experiments and real data examples.

The skewness of mean–variance normal mixtures

Mean–variance mixtures of normal distributions are very flexible: they model many nonnormal features, such as skewness, kurtosis and multimodality. Special cases include generalized asymmetric Laplace distributions, mixtures of two normal distributions with proportional covariance matrices, scale mixtures of normal distributions and normal distributions. This paper investigates the skewness of multivariate mean–variance normal mixtures. The special case of mixtures of two normal distributions with proportional covariance matrices is treated in greater detail. The paper derives the analytical forms of prominent measures of multivariate skewness and applies them to model-based clustering, normalizing linear transformations, projection pursuit and normality testing. The practical relevance of the theoretical results is assessed with both real and simulated data.

Information criteria for matrix exponential spatial specifications

In this study, we suggest using information criteria for nested and non-nested model selection problems for the matrix exponential spatial specifications (MESS) under both homoskedasticity and heteroskedasticity. To this end, we consider the deviance information criterion, the Akaike information criterion and the Bayesian information criterion in a Bayesian setting. In the heteroskedastic case, we assume that the error terms have a scale mixture of normal distributions, where the scale mixture variables are latent variables that lead to different distributions. We demonstrate how the integrated likelihood function can be obtained analytically by integrating out the scale mixture variables from the complete-data likelihood function, and how this integrated likelihood function can be used to formulate the information criteria. We investigate the finite sample performance of these criteria in selecting the true model in a simulation study. The results show that these criteria perform satisfactorily and can be useful for selecting the correct model in specification search exercises. Finally, we apply the proposed information criteria to a spatially augmented growth model and a carbon emission model to show their usefulness for both nested and non-nested model selection problems.

Regression modeling of censored data based on compound scale mixtures of normal distributions

Regression modeling of censored data based on compound scale mixtures of normal distributions

Sparse Online Variational Bayesian Regression

This work considers variational Bayesian inference as an inexpensive and scalable alternative to a fully Bayesian approach in the context of sparsity-promoting priors. In particular, the priors considered arise from scale mixtures of Normal distributions with a generalized inverse Gaussian mixing distribution. This includes the variational Bayesian LASSO as an inexpensive and scalable alternative to the Bayesian LASSO introduced in [65]. It also includes a family of priors which more strongly promote sparsity. For linear models the method requires only the iterative solution of deterministic least squares problems. Furthermore, for p unknown covariates the method can be implemented exactly online with a cost of $O(p^3)$ in computation and $O(p^2)$ in memory per iteration -- in other words, the cost per iteration is independent of n, and in principle infinite data can be considered. For large $p$ an approximation is able to achieve promising results for a cost of $O(p)$ per iteration, in both computation and memory. Strategies for hyper-parameter tuning are also considered. The method is implemented for real and simulated data. It is shown that the performance in terms of variable selection and uncertainty quantification of the variational Bayesian LASSO can be comparable to the Bayesian LASSO for problems which are tractable with that method, and for a fraction of the cost. The present method comfortably handles $n = 65536$, $p = 131073$ on a laptop in less than 30 minutes, and $n = 10^5$, $p = 2.1 \times 10^6$ overnight.

Robust mixture regression modeling based on two-piece scale mixtures of normal distributions

Robust mixture regression modeling based on two-piece scale mixtures of normal distributions

Fast inference for robust nonlinear mixed-effects models

The interest for nonlinear mixed-effects models comes from application areas as pharmacokinetics, growth curves and HIV viral dynamics. However, the modeling procedure usually leads to many difficulties, as the inclusion of random effects, the estimation process and the model sensitivity to atypical or nonnormal data. The scale mixture of normal distributions include heavy-tailed models, as the Student-t, slash and contaminated normal distributions, and provide competitive alternatives to the usual models, enabling the obtention of robust estimates against outlying observations. Our proposal is to compare two estimation methods in nonlinear mixed-effects models where the random components follow a multivariate scale mixture of normal distributions. For this purpose, a Monte Carlo expectation-maximization algorithm (MCEM) and an efficient likelihood-based approximate method are developed. Results show that the approximate method is much faster and enables a fairly efficient likelihood maximization, although a slightly larger bias may be produced, especially for the fixed-effects parameters. A discussion on the robustness aspects of the proposed models are also provided. Two real nonlinear applications are discussed and a brief simulation study is presented.

Variable Selection for Robust Mixture Regression Model with Skew Scale Mixtures of Normal Distributions

In this paper, we propose a robust mixture regression model based on the skew scale mixtures of normal distributions (RMR-SSMN) which can accommodate asymmetric, heavy-tailed and contaminated data better. For the variable selection problem, the penalized likelihood approach with a new combined penalty function which balances the SCAD and l2 penalty is proposed. The adjusted EM algorithm is presented to get parameter estimates of RMR-SSMN models at a faster convergence rate. As simulations show, our mixture models are more robust than general FMR models and the new combined penalty function outperforms SCAD for variable selection. Finally, the proposed methodology and algorithm are applied to a real data set and achieve reasonable results.

Semiparametric inference for the scale-mixture of normal partial linear regression model with censored data

In the censored data exploration, the classical linear regression model which assumes normally distributed random errors is perhaps one of the commonly used frameworks. However, practical studies have often criticized the classical linear regression model because of its sensitivity to departure from the normality and partial nonlinearity. This paper proposes to solve these potential issues simultaneously in the context of the partial linear regression model by assuming that the random errors follow a scale-mixture of normal (SMN) family of distributions. The postulated method allows us to model data with great flexibility, accommodating heavy tails and outliers. By implementing the B-spline approximation and using the convenient hierarchical representation of the SMN distributions, a computationally analytical EM-type algorithm is developed for obtaining maximum likelihood (ML) parameter estimates. Various simulation studies are conducted to investigate the finite sample properties, as well as the robustness of the model in dealing with the heavy tails distributed datasets. Real-world data examples are finally analyzed for illustrating the usefulness of the proposed methodology.

Approximate Inferences for Nonlinear Mixed Effects Models with Scale Mixtures of Skew-Normal Distributions

Nonlinear mixed effects models have received a great deal of attention in the statistical literature in recent years because of their flexibility in handling longitudinal studies, including human immunodeficiency virus viral dynamics, pharmacokinetic analyses, and studies of growth and decay. A standard assumption in nonlinear mixed effects models for continuous responses is that the random effects and the within-subject errors are normally distributed, making the model sensitive to outliers. We present a novel class of asymmetric nonlinear mixed effects models that provides efficient parameters estimation in the analysis of longitudinal data. We assume that, marginally, the random effects follow a multivariate scale mixtures of skew--normal distribution and that the random errors follow a symmetric scale mixtures of normal distribution, providing an appealing robust alternative to the usual normal distribution. We propose an approximate method for maximum likelihood estimation based on an EM-type algorithm that produces approximate maximum likelihood estimates and significantly reduces the numerical difficulties associated with the exact maximum likelihood estimation. Techniques for prediction of future responses under this class of distributions are also briefly discussed. The methodology is illustrated through an application to Theophylline kinetics data and through some simulating studies.

Estimation and Influence Diagnostics for the Multivariate Linear Regression Models with Skew Scale Mixtures of Normal Distributions

In this paper, we present recent results in the context of multivariate linear regression models considering that random errors follow multivariate skew scale mixtures of normal distributions. This class of distributions includes the scale mixtures of multivariate normal distributions, as special cases, and provides flexibility in capturing a wide variety of asymmetric behaviors. We implemented the algorithm ECM (Expectation/Conditional Maximization) and we obtained closed-form expressions for all the estimators of the parameters of the proposed model. Inspired by the ECM algorithm, we have developed an influence diagnostics for detecting influential observations to investigate the sensitivity of the maximum likelihood estimators. To examine the performance and the usefulness of the proposed methodology, we present simulation studies and analyze a real dataset.

Bayesian sparse convex clustering via global-local shrinkage priors

Sparse convex clustering is to group observations and conduct variable selection simultaneously in the framework of convex clustering. Although a weighted L_1 norm is usually employed for the regularization term in sparse convex clustering, its use increases the dependence on the data and reduces the estimation accuracy if the sample size is not sufficient. To tackle these problems, this paper proposes a Bayesian sparse convex clustering method based on the ideas of Bayesian lasso and global-local shrinkage priors. We introduce Gibbs sampling algorithms for our method using scale mixtures of normal distributions. The effectiveness of the proposed methods is shown in simulation studies and a real data analysis.

Fast Markov Chain Monte Carlo for High-Dimensional Bayesian Regression Models With Shrinkage Priors

In the past decade, many Bayesian shrinkage models have been developed for linear regression problems where the number of covariates, p, is large. Computation of the intractable posterior is often done with three-block Gibbs samplers (3BG), based on representing the shrinkage priors as scale mixtures of Normal distributions. An alternative computing tool is a state of the art Hamiltonian Monte Carlo (HMC) method, which can be easily implemented in the Stan software. However, we found both existing methods to be inefficient and often impractical for large p problems. Following the general idea of Rajaratnam et al., we propose two-block Gibbs samplers (2BG) for three commonly used shrinkage models, namely, the Bayesian group lasso, the Bayesian sparse group lasso, and the Bayesian fused lasso models. We demonstrate with simulated and real data examples that the Markov chains underlying 2BG’s converge much faster than that of 3BG’s, and no worse than that of HMC. At the same time, the computing costs of 2BG’s per iteration are as low as that of 3BG’s, and can be several orders of magnitude lower than that of HMC. As a result, the newly proposed 2BG is the only practical computing solution to do Bayesian shrinkage analysis for datasets with large p. Further, we provide theoretical justifications for the superior performance of 2BG’s. We establish geometric ergodicity of Markov chains associated with the 2BG for each of the three Bayesian shrinkage models. We also prove, for most cases of the Bayesian group lasso and the Bayesian sparse group lasso model, the Markov operators for the 2BG chains are trace-class. Whereas for all cases of all three Bayesian shrinkage models, the Markov operator for the 3BG chains is not even Hilbert–Schmidt. Supplementary materials for this article are available online.

Mixture of linear experts model for censored data: A novel approach with scale-mixture of normal distributions

Mixture of linear experts (MoE) model is one of the widespread statistical frameworks for modeling, classification, and clustering of data. Built on the normality assumption of the error terms for mathematical and computational convenience, the classical MoE model has two challenges: (1) it is sensitive to atypical observations and outliers, and (2) it might produce misleading inferential results for censored data. The aim is then to resolve these two challenges, simultaneously, by proposing a robust MoE model for model-based clustering and discriminant censored data with the scale-mixture of normal (SMN) class of distributions for the unobserved error terms. An analytical expectation–maximization (EM) type algorithm is developed in order to obtain the maximum likelihood parameter estimates. Simulation studies are carried out to examine the performance, effectiveness, and robustness of the proposed methodology. Finally, a real dataset is used to illustrate the superiority of the new model.

Parseval’s identity and optimal transport maps

Recent findings for optimal transport maps between distribution functions sharing the same copula show that componentwise the solution is the optimal map between the marginal distributions. This is an important discovery since in the multivariate setting optimal maps are difficult to find and only known in a few special cases. In this paper, we extend the result on common copulas by showing that orthonormal transformations of variables sharing a common copula also have a known optimal map. We illustrate this by establishing optimal maps between members of a class of scale mixture of normal distributions.

Linear mixed models based on skew scale mixtures of normal distributions

Scale mixtures of normal distributions are useful for statistical procedures involving symmetric and heavy-tailed data. Ferreira, Lachos, and Bolfarine (2016) defined a multivariate skewed version of these distributions that offers much-needed flexibility by combining both skewness and heavy tails. In this work, we develop a linear mixed model based on skew scale mixtures of normal distributions, with emphasis on the skew Student-t normal, skew–slash and skew–contaminated normal distributions. Using the hierarchical structure of the model, we develop maximum likelihood estimation of the model parameters via the expectation-maximization (EM) algorithm. In addition, the standard errors are obtained via the approximate information matrix and the local influence analysis is explored under some perturbation schemes. To examine the performance and the usefulness of the proposed method, we present simulation studies and analyze a real dataset.

Linear censored regression models with skew scale mixtures of normal distributions

A special source of difficulty in the statistical analysis is the possibility that some subjects may not have a complete observation of the response variable. Such incomplete observation of the response variable is called censoring. Censorship can occur for a variety of reasons, including limitations of measurement equipment, design of the experiment, and non-occurrence of the event of interest until the end of the study. In the presence of censoring, the dependence of the response variable on the explanatory variables can be explored through regression analysis. In this paper, we propose to examine the censorship problem in context of the class of asymmetric, i.e., we have proposed a linear regression model with censored responses based on skew scale mixtures of normal distributions. We develop a Monte Carlo EM (MCEM) algorithm to perform maximum likelihood inference of the parameters in the proposed linear censored regression models with skew scale mixtures of normal distributions. The MCEM algorithm has been discussed with an emphasis on the skew-normal, skew Student-t-normal, skew-slash and skew-contaminated normal distributions. To examine the performance of the proposed method, we present some simulation studies and analyze a real dataset.

Spearman rank correlation of the bivariate Student t and scale mixtures of normal distributions

We derive an expression for the Spearman rank correlation of bivariate scale mixtures of normals (SMN) and we show that within this class, for any value of the correlation parameter, the Spearman rank correlation of the normal is the greatest in absolute value. We then provide expressions for the symmetric generalized hyperbolic, the Bessel, and the Laplace distributions. We further derive an expression for the Spearman rank correlation of the Student t distribution in terms of an easily computable one-dimensional integral, and we also consider the special case of the Cauchy. Finally, we show how our results can be used in a rank-based estimation of the parameters of the Student t distribution.

WEIGHTED CROSS VALIDATION IN THE SELECTION OF ROBUST REGRESSION MODEL WITH CHANGE-POINT FOR TELEVISION RATING FORECAST

The paper proposes a weighted cross-validation (WCV) algorithm to select a linear regression model with change-point under a scale mixtures of normal (SMN) distribution that yields the best prediction results. SMN distributions are used to construct robust regression models to the influence of outliers on the parameter estimation process. Thus, we relaxed the usual assumption of normality of the regression models and considered that the random errors follow a SMN distribution, specifically the Student-t distribution. In addition, we consider the fact that the parameters of the regression model can change from a specific and unknown point, called change-point. In this context, the estimations of the model parameters, which include the change-point, are obtained via the EM-type algorithm (Expectation-Maximization). The WCV method is used in the selection of the model that presents greater robustness and that offers a smaller prediction error, considering that the weighting values come from step E of the EM-type algorithm. Finally, numerical examples considering simulated and real data (data from television audiences) are presented to illustrate the proposed methodology.