Scale Mixtures Of Normals Research Articles

Data Augmentation for Bayesian Deep Learning

Deep Learning (DL) methods have emerged as one of the most powerful tools for functional approximation and prediction. While the representation properties of DL have been well studied, uncertainty quantification remains challenging and largely unexplored. Data augmentation techniques are a natural approach to provide uncertainty quantification and to incorporate stochastic Monte Carlo search into stochastic gradient descent (SGD) methods. The purpose of our paper is to show that training DL architectures with data augmentation leads to efficiency gains. We use the theory of scale mixtures of normals to derive data augmentation strategies for deep learning. This allows variants of the expectation-maximization and MCMC algorithms to be brought to bear on these high dimensional nonlinear deep learning models. To demonstrate our methodology, we develop data augmentation algorithms for a variety of commonly used activation functions: logit, ReLU, leaky ReLU and SVM. Our methodology is compared to traditional stochastic gradient descent with back-propagation. Our optimization procedure leads to a version of iteratively re-weighted least squares and can be implemented at scale with accelerated linear algebra methods providing substantial improvement in speed. We illustrate our methodology on a number of standard datasets. Finally, we conclude with directions for future research.

Mixture Representations for Generalized Burr, Snedecor–Fisher and Generalized Student Distributions with Related Results

In this paper, the representability of the generalized Student’s distribution as uniform and normal-scale mixtures is considered. It is also shown that the generalized Burr and the Snedecor–Fisher distributions can be represented as the scale mixtures of uniform, folded normal, exponential, Weibull or Fréchet distributions. New multiplication-type theorems are proven for these and related distributions. The relation between the generalized Student and generalized Burr distribution is studied. It is shown that the Snedecor–Fisher distribution is a special case of the generalized Burr distribution. Based on these mixture representations, some limit theorems are proven for random sums in which the symmetric and asymmetric generalized Student or symmetric and asymmetric two-sided generalized Burr distributions are limit laws. Also, limit theorems are proven for maximum and minimum random sums and absolute values of random sums in which the generalized Burr distributions are limit laws.

Analytic and Asymptotic Properties of the Generalized Student and Generalized Lomax Distributions

Analytic and asymptotic properties of the generalized Student and generalized Lomax distributions are discussed, with the main focus on the representation of these distributions as scale mixtures of the laws that appear as limit distributions in classical limit theorems of probability theory, such as the normal, folded normal, exponential, Weibull, and Fréchet distributions. These representations result in the possibility of proving some limit theorems for statistics constructed from samples with random sizes in which the generalized Student and generalized Lomax distributions are limit laws. An overview of known properties of the generalized Student distribution is given, and some simple bounds for its tail probabilities are presented. An analog of the ‘multiplication theorem’ is proved, and the identifiability of scale mixtures of generalized Student distributions is considered. The normal scale mixture representation for the generalized Student distribution is discussed, and the properties of the mixing distribution in this representation are studied. Some simple general inequalities are proved that relate the tails of the scale mixture with that of the mixing distribution. It is proved that for some values of the parameters, the generalized Student distribution is infinitely divisible and admits a representation as a scale mixture of Laplace distributions. Necessary and sufficient conditions are presented that provide the convergence of the distributions of sums of a random number of independent random variables with finite variances and other statistics constructed from samples with random sizes to the generalized Student distribution. As an example, the convergence of the distributions of sample quantiles in samples with random sizes is considered. The generalized Lomax distribution is defined as the distribution of the absolute value of the random variable with the generalized Student distribution. It is shown that the generalized Lomax distribution can be represented as a scale mixture of folded normal distributions. The convergence of the distributions of maximum and minimum random sums to the generalized Lomax distribution is considered. It is demonstrated that the generalized Lomax distribution can be represented as a scale mixture of Weibull distributions or that of Fréchet distributions. As a consequence, it is demonstrated that the generalized Lomax distribution can be limiting for extreme statistics in samples with random size. The convergence of the distributions of mixed geometric random sums to the generalized Lomax distribution is considered, and the corresponding extension of the famous Rényi theorem is proved. The law of large numbers for mixed Poisson random sums is presented, in which the limit random variable has a generalized Lomax distribution.

Bayesian P-Splines Applied to Semiparametric Models with Errors Following a Scale Mixture of Normals

Bayesian P-Splines Applied to Semiparametric Models with Errors Following a Scale Mixture of Normals

Discrete mixtures of normals pseudo maximum likelihood estimators of structural vector autoregressions

Likelihood inference in structural vector autoregressions with independent non-Gaussian shocks leads to parametric identification and efficient estimation at the risk of inconsistencies under distributional misspecification. We prove that autoregressive coefficients and (scaled) impact multipliers remain consistent, but the drifts and shocks’ standard deviations are generally inconsistent. Nevertheless, we show consistency when the non-Gaussian log-likelihood uses a discrete scale mixture of normals in the symmetric case, or an unrestricted finite mixture more generally, and compare the efficiency of these estimators to other consistent two-step proposals, including our own. Finally, our empirical application looks at dynamic linkages between three popular volatility indices.

Dimension-wise scaled normal mixtures with application to finance and biometry

We propose the family of dimension-wise scaled normal mixtures (DSNMs) to model the joint distribution of a d-variate random variable with real-valued components. Each member of the family generalizes the multivariate normal (MN) distribution in two directions. Firstly, the DSNM has a more general type of symmetry with respect to the elliptical symmetry of the MN distribution. Secondly, the univariate marginals have similar heavy-tailed normal scale mixture distributions with (possibly) different tailedness parameters; as a consequence of practical interest, the DSNM allows for a different excess kurtosis on each dimension. These peculiarities are obtained in an MN scale mixture framework by introducing a d-variate mixing random variable with independent and similar components acting separately for each dimension. We examine several properties of DSNMs, such as the joint density function, hierarchical and stochastic representations, relations with other families of symmetric distributions, type of symmetry, marginal and conditional distributions, no correlation implying independence, moment generating function, and moments of practical interest. For illustrative purposes, we describe two members of the DSNM family obtained in the case of components of the mixing random vector being either uniform or shifted exponential; these are examples of mixing distributions that guarantee a closed-form expression for the joint density of the DSNM. For the two DSNMs analyzed in detail, we introduce parsimony by allowing the d tailedness parameters to be tied across dimensions, and describe algorithms, based on the expectation–maximization (EM) principle, to estimate the parameters by maximum likelihood. We use real data from the financial and biometrical fields to appreciate the advantages of our DSNMs over other symmetric heavy-tailed distributions available in the literature.

Log-regularly varying scale mixture of normals for robust regression

Linear regression that employs the assumption of normality for the error distribution may lead to an undesirable posterior inference of regression coefficients due to potential outliers. A finite mixture of two components, one with thin and one with heavy tails, is considered as the error distribution in this study. For the heavily-tailed component, the novel class of distributions is introduced; their densities are log-regularly varying and have heavier tails than the Cauchy distribution. Yet, they are expressed as a scale mixture of normals which enables the efficient posterior inference when using a Gibbs sampler. The robustness of the posterior distributions is proved under the proposed models using a minimal set of assumptions, which justifies the use of shrinkage priors with unbounded densities for the coefficient vector in the presence of outliers. An extensive comparison with the existing methods via simulation study shows the improved performance of the proposed model in point and interval estimation, as well as its computational efficiency. Further, the posterior robustness of the proposed method is confirmed in an empirical study with shrinkage priors for regression coefficients.

On shrinkage estimation of a spherically symmetric distribution for balanced loss functions

We consider the problem of estimating the mean vector θ of a d-dimensional spherically symmetric distributed X based on balanced loss functions of the forms: (i) ωρ(‖δ−δ0‖2)+(1−ω)ρ(‖δ−θ‖2) and (ii) ℓω‖δ−δ0‖2+(1−ω)‖δ−θ‖2, where δ0 is a target estimator, and where ρ and ℓ are increasing and concave functions. For d≥4 and the target estimator δ0(X)=X, we provide Baranchik-type estimators that dominate δ0(X)=X and are minimax. The findings represent extensions of those of Marchand & Strawderman (2020) in two directions: (a) from scale mixture of normals to the spherical class of distributions with Lebesgue densities, and (b) from completely monotone to concave ρ′ and ℓ′.

Simulation Study for Penalized Bayesian Elastic Net Quantile Regression

Bayesian regression analysis has great importance in recent years, especially in the Regularization method, Such as ridge, Lasso, adaptive lasso, elastic net methods, where choosing the prior distribution of the interested parameter is the main idea in the Bayesian regression analysis. By penalizing the Bayesian regression model, the variance of the estimators are reduced notable and the bias is getting smaller. The tradeoff between the bias and variance of the penalized Bayesian regression estimator consequently produce more interpretable model with more prediction accuracy. In this paper, we proposed new hierarchical model for the Bayesian quantile regression by employing the scale mixture of normals mixing with truncated gamma distribution that stated by (Li and Lin, 2010) as Laplace prior distribution. Therefore, new Gibbs sampling algorithms are introduced. A comparison has made with classical quantile regression model and with lasso quantile regression model by conducting simulations studies. Our model is comparable and gives better results.

Semiparametric inference for the scale-mixture of normal partial linear regression model with censored data

In the censored data exploration, the classical linear regression model which assumes normally distributed random errors is perhaps one of the commonly used frameworks. However, practical studies have often criticized the classical linear regression model because of its sensitivity to departure from the normality and partial nonlinearity. This paper proposes to solve these potential issues simultaneously in the context of the partial linear regression model by assuming that the random errors follow a scale-mixture of normal (SMN) family of distributions. The postulated method allows us to model data with great flexibility, accommodating heavy tails and outliers. By implementing the B-spline approximation and using the convenient hierarchical representation of the SMN distributions, a computationally analytical EM-type algorithm is developed for obtaining maximum likelihood (ML) parameter estimates. Various simulation studies are conducted to investigate the finite sample properties, as well as the robustness of the model in dealing with the heavy tails distributed datasets. Real-world data examples are finally analyzed for illustrating the usefulness of the proposed methodology.

Mixture of linear experts model for censored data: A novel approach with scale-mixture of normal distributions

Mixture of linear experts (MoE) model is one of the widespread statistical frameworks for modeling, classification, and clustering of data. Built on the normality assumption of the error terms for mathematical and computational convenience, the classical MoE model has two challenges: (1) it is sensitive to atypical observations and outliers, and (2) it might produce misleading inferential results for censored data. The aim is then to resolve these two challenges, simultaneously, by proposing a robust MoE model for model-based clustering and discriminant censored data with the scale-mixture of normal (SMN) class of distributions for the unobserved error terms. An analytical expectation–maximization (EM) type algorithm is developed in order to obtain the maximum likelihood parameter estimates. Simulation studies are carried out to examine the performance, effectiveness, and robustness of the proposed methodology. Finally, a real dataset is used to illustrate the superiority of the new model.

Two new matrix-variate distributions with application in model-based clustering

Two matrix-variate distributions, both elliptical heavy-tailed generalization of the matrix-variate normal distribution, are introduced. They belong to the normal scale mixture family, and are respectively obtained by choosing a convenient shifted exponential or uniform as mixing distribution. Moreover, they have a closed-form for the probability density function that is characterized by only one additional parameter, with respect to the nested matrix-variate normal, governing the tail-weight. Both distributions are then used for model-based clustering via finite mixture models. The resulting mixtures, being able to handle data with atypical observations in a better way than the matrix-variate normal mixture, can avoid the disruption of the true underlying group structure. Different EM-based algorithms are implemented for parameter estimation and tested in terms of computational times and parameter recovery. Furthermore, these mixture models are fitted to simulated and real datasets, and their fitting and clustering performances are analyzed and compared to those obtained by other well-established competitors.

Sparsity via new Bayesian Lasso

Lasso estimate as the posterior mode assuming that the parameter has prior density as double exponential distribution [1]. In this paper, we proposed Scale Mixture of Normals mixing with Rayleigh (SMNR) density on their variances to represent the double exponential distribution. Hierarchical model formulation presented with Gibbs sampler under SMNR as alternative Bayesian analysis of minimization problem of classical lasso. We conducted two simulation examples to explore path solution of the Ridge, Lasso, Bayesian Lasso, and New Bayesian Lasso (R, L, BL, NBL) regression methods through the prediction accuracy using the bias of the estimates with different sample sizes, bias indicates that the lasso regression perform well, followed by the NBL. The Median Mean Absolute Deviations (MMAD) used to compared the perform of the regression methods using real data, MMAD indicates that the proposed method (NBL) perform better than the others.

Bayesian extensions on Lasso and adaptive Lasso Tobit regressions

Since lasso method launched, a lot of applications and extensions were run on it which made it to become deeply widely used in various discipline. In this paper, we proposed the Scale Mixture of Normals mixing with Rayleigh (SMNR) distribution on their variances to represent the double exponential distribution. Hierarchical model formula have derived with Gibbs sampler for SMNR. The proposed models; Bayesian Tobit Adaptive Lasso (BTAL) and Bayesian Tobit Lasso (BTL) models are illustrated using simulation example and a real data example through the prediction accuracy using the estimated relative efficiency with different sample. This is the first work that discussed regularization regression models under SMNR.

Spearman rank correlation of the bivariate Student t and scale mixtures of normal distributions

We derive an expression for the Spearman rank correlation of bivariate scale mixtures of normals (SMN) and we show that within this class, for any value of the correlation parameter, the Spearman rank correlation of the normal is the greatest in absolute value. We then provide expressions for the symmetric generalized hyperbolic, the Bessel, and the Laplace distributions. We further derive an expression for the Spearman rank correlation of the Student t distribution in terms of an easily computable one-dimensional integral, and we also consider the special case of the Cauchy. Finally, we show how our results can be used in a rank-based estimation of the parameters of the Student t distribution.

WEIGHTED CROSS VALIDATION IN THE SELECTION OF ROBUST REGRESSION MODEL WITH CHANGE-POINT FOR TELEVISION RATING FORECAST

The paper proposes a weighted cross-validation (WCV) algorithm to select a linear regression model with change-point under a scale mixtures of normal (SMN) distribution that yields the best prediction results. SMN distributions are used to construct robust regression models to the influence of outliers on the parameter estimation process. Thus, we relaxed the usual assumption of normality of the regression models and considered that the random errors follow a SMN distribution, specifically the Student-t distribution. In addition, we consider the fact that the parameters of the regression model can change from a specific and unknown point, called change-point. In this context, the estimations of the model parameters, which include the change-point, are obtained via the EM-type algorithm (Expectation-Maximization). The WCV method is used in the selection of the model that presents greater robustness and that offers a smaller prediction error, considering that the weighting values come from step E of the EM-type algorithm. Finally, numerical examples considering simulated and real data (data from television audiences) are presented to illustrate the proposed methodology.

Multivariate Scale-Mixed Stable Distributions and Related Limit Theorems

In the paper, multivariate probability distributions are considered that are representable as scale mixtures of multivariate stable distributions. Multivariate analogs of the Mittag–Leffler distribution are introduced. Some properties of these distributions are discussed. The main focus is on the representations of the corresponding random vectors as products of independent random variables and vectors. In these products, relations are traced of the distributions of the involved terms with popular probability distributions. As examples of distributions of the class of scale mixtures of multivariate stable distributions, multivariate generalized Linnik distributions and multivariate generalized Mittag–Leffler distributions are considered in detail. Their relations with multivariate ‘ordinary’ Linnik distributions, multivariate normal, stable and Laplace laws as well as with univariate Mittag–Leffler and generalized Mittag–Leffler distributions are discussed. Limit theorems are proved presenting necessary and sufficient conditions for the convergence of the distributions of random sequences with independent random indices (including sums of a random number of random vectors and multivariate statistics constructed from samples with random sizes) to scale mixtures of multivariate elliptically contoured stable distributions. The property of scale-mixed multivariate elliptically contoured stable distributions to be both scale mixtures of a non-trivial multivariate stable distribution and a normal scale mixture is used to obtain necessary and sufficient conditions for the convergence of the distributions of random sums of random vectors with covariance matrices to the multivariate generalized Linnik distribution.

Bayesian robust estimation of partially functional linear regression models using heavy-tailed distributions

Functional linear regression (FLR) is a popular method that studies the relationship between a scalar response and a functional predictor. A common estimation procedure for the FLR model is using maximum likelihood by assuming normal distributions for measurement errors; however this method may make inferences vulnerable to the presence of outliers. In this article, we introduce a robust estimation method of partially functional linear model by considering a class of scale mixtures of normal (SMN) distributions for measurement errors. Due to intractable closed form of likelihood function with the SMN distributions, a Bayesian framework is adopted and an MCMC algorithm is developed to carry out posterior inference on model parameters. The finite sample performance of our proposed method is evaluated by using some simulation studies and a real dataset.

On Mixture Representations for the Generalized Linnik Distribution and Their Applications in Limit Theorems

We present new mixture representations for the generalized Linnik distribution in terms of normal, Laplace, and generalized Mittag–Leffler laws. In particular, we prove that the generalized Linnik distribution is a normal scale mixture with the generalized Mittag–Leffler mixing distribution. Based on these representations, we prove some limit theorems for a wide class of statistics constructed from samples with random sized including, e.g., random sums of independent random variables with finite variances, in which the generalized Linnik distribution plays the role of the limit law. Thus we demonstrate that the scheme of geometric (or, in general, negative binomial) summation is by far not the only asymptotic setting (even for sums of independent random variables) in which the generalized Linnik law appears as the limit distribution.

On shrinkage estimation for balanced loss functions

The estimation of a multivariate mean θ is considered under natural modifications of balanced loss functions of the form: (i) ωρ(‖δ−δ0‖2)+(1−ω)ρ(‖δ−θ‖2), and (ii) ℓω‖δ−δ0‖2+(1−ω)‖δ−θ‖2, where δ0 is a target estimator of γ(θ). After briefly reviewing known results for original balanced loss with identity ρ or ℓ, we provide, for increasing and concave ρ and ℓ which also satisfy a completely monotone property, Baranchik-type estimators of θ which dominate the benchmark δ0(X)=X for X either distributed as multivariate normal or as a scale mixture of normals. Implications are given with respect to model robustness and simultaneous dominance with respect to either ρ or ℓ.