Gaussian Scale Mixture Research Articles

Efficient Modeling of Spatial Extremes over Large Geographical Domains

Various natural phenomena exhibit spatial extremal dependence at short spatial distances. However, existing models proposed in the spatial extremes literature often assume that extremal dependence persists across the entire domain. This is a strong limitation when modeling extremes over large geographical domains, and yet it has been mostly overlooked in the literature. We here develop a more realistic Bayesian framework based on a novel Gaussian scale mixture model, with the Gaussian process component defined by a stochastic partial differential equation yielding a sparse precision matrix, and the random scale component modeled as a low-rank Pareto-tailed or Weibull-tailed spatial process determined by compactly-supported basis functions. We show that our proposed model is approximately tail-stationary and that it can capture a wide range of extremal dependence structures. Its inherently sparse structure allows fast Bayesian computations in high spatial dimensions based on a customized Markov chain Monte Carlo algorithm prioritizing calibration in the tail. We fit our model to analyze heavy monsoon rainfall data in Bangladesh. Our study shows that our model outperforms natural competitors and that it fits precipitation extremes well. We finally use the fitted model to draw inference on long-term return levels for marginal precipitation and spatial aggregates.

Bayesian inversion with Student’s t priors based on Gaussian scale mixtures

Bayesian inversion with Student’s t priors based on Gaussian scale mixtures

A dynamic scale-mixture model of motion in natural scenes.

Some of the most important tasks of visual and motor systems involve estimating the motion of objects and tracking them over time. Such systems evolved to meet the behavioral needs of the organism in its natural environment, and may therefore be adapted to the statistics of motion it is likely to encounter. By tracking the movement of individual points in movies of natural scenes, we begin to identify common properties of natural motion across scenes. As expected, objects in natural scenes move in a persistent fashion, with velocity correlations lasting hundreds of milliseconds. More subtly, but crucially, we find that the observed velocity distributions are heavy-tailed and can be modeled as a Gaussian scale-mixture. Extending this model to the time domain leads to a dynamic scale-mixture model, consisting of a Gaussian process multiplied by a positive scalar quantity with its own independent dynamics. Dynamic scaling of velocity arises naturally as a consequence of changes in object distance from the observer, and may approximate the effects of changes in other parameters governing the motion in a given scene. This modeling and estimation framework has implications for the neurobiology of sensory and motor systems, which need to cope with these fluctuations in scale in order to represent motion efficiently and drive fast and accurate tracking behavior.

Joint Learning of Full-Structure Noise in Hierarchical Bayesian Regression Models.

We consider the reconstruction of brain activity from electroencephalography (EEG). This inverse problem can be formulated as a linear regression with independent Gaussian scale mixture priors for both the source and noise components. Crucial factors influencing the accuracy of the source estimation are not only the noise level but also its correlation structure, but existing approaches have not addressed the estimation of noise covariance matrices with full structure. To address this shortcoming, we develop hierarchical Bayesian (type-II maximum likelihood) models for observations with latent variables for source and noise, which are estimated jointly from data. As an extension to classical sparse Bayesian learning (SBL), where across-sensor observations are assumed to be independent and identically distributed, we consider Gaussian noise with full covariance structure. Using the majorization-maximization framework and Riemannian geometry, we derive an efficient algorithm for updating the noise covariance along the manifold of positive definite matrices. We demonstrate that our algorithm has guaranteed and fast convergence and validate it in simulations and with real MEG data. Our results demonstrate that the novel framework significantly improves upon state-of-the-art techniques in the real-world scenario where the noise is indeed non-diagonal and full-structured. Our method has applications in many domains beyond biomagnetic inverse problems.

Defect detection enhancement using Gaussian Sparse Models

Detecting defects is critical in industrial fabrication, such as pipe welding, where radiography testing (RT) is the gold standard as a non-destructive method for monitoring weld quality and weld corrosion. The extraction of seamless information from radiographic images is critical for this approach. Image processing techniques can improve the quality of radiographic images by enhancing image contrast, especially in flawed regions. In this study, a method based on Gaussian mixture models was implemented and applied to radiographs of welded objects to improve visualization and detectability. In the Sparse Coding and Gaussian Scale Mixture method (SSC-GSM), the local image patches are described as a mixture of Gaussian distributions. Given the different levels of noise in the individual images, the background was determined and subtracted from each original image. The results show that the proposed techniques can maintain and improve the edge information in radiographs and defective regions. The density of pixels along the analyzed profile lines has yielded enhancement by a factor of approximately two for the reconstructed images as compared to the original images. We tested the SSC-GSM method on GD-Xray database with 68 radiographs of the weld. In total, the SSC-GSM technique can improve the contrast of radiography image by back ground removal method and show the defects better that exist in weld radiographs, compared to traditional method for the detection of weld defects.

Robust learning of parsimonious deep neural networks

We propose a simultaneous learning and pruning algorithm capable of identifying and eliminating irrelevant structures in a neural network during the early stages of training. Simultaneous learning and pruning presents serious challenges such as premature pruning of units that can lead to poor performance. Our method is capable of overcoming such challenges and is robust, i.e., it gives consistent pruning levels and prediction accuracy regardless of weight initialization or the size of the starting network. Thus, it allows for substantial computational cost savings during training, besides that of inference, and it can enable the training of very deep networks when transfer learning to obtain fully trained deep networks that can be pruned after training is not possible. Our approach is based on variational inference principles using Gaussian scale mixture priors on the neural network weights. The variational posterior distribution of Bernoulli random variables multiplying the units/filters is learned, similarly to adaptive dropout. We construct a novel hyper-prior distribution over the prior parameters to impose properties crucial for their optimal selection and the overall robustness of our algorithm. It is shown in the context of our algorithm that the parameters of the posterior distributions practically converge to either 0 or 1, establishing a deterministic final network. Convergence is proved analytically based on dynamical systems theory and from the theoretical results, practical pruning conditions are established. The proposed algorithm is evaluated on the MNIST, CIFAR-10 and ImageNet data sets and the commonly used fully connected, convolutional and residual architectures LeNet, VGG16 and ResNet. The simulations show that our method typically achieves better pruning levels while maintaining test-accuracy on par with state-of the-art methods for structured pruning in a manner robust with respect to network initialization and initial size.

Graph laplacian regularization with sparse coding in secure image restoration and representation for Internet of Things

Image restoration (IR) attempts to recreate the original (ideal) scene from a degraded observation. The goal of image restoration is to avert the process of image deterioration that happens during image acquisition and processing. Blurring and noise are two common types of picture capture degradation. When the blurring function is unknown, the image restoration issue is referred to as blind image restoration. Existing methods of image restoration do not provide efficient results due to time complexity, computational complexity, large-scale scaling factor and limited input. Therefore, in order to overcome those problems, an Improved Graph Laplacian Regularization with Sparse Coding by integrating Internet of Things (IoT) is developed in this research. The process of removing de-noised portions of image to texture layer and cartoon layer is carried out by Morphological Component Analysis (MCA). An improved Graph Laplacian regularized method and Simultaneous Sparse Coding with Gaussian Scale Mixture (SSC-GSM) methods is performed on texture layer, cartoon layer and restored image. A Levin’s dataset and a real-time dataset such as industrial manufacturing products are analyzed in this research. The proposed Graph Laplacian algorithm and sparse coding model are better efficient in identifying sharper texture information and getting a restored image. The proposed method proves that it has better performance in terms of Peak Signal-to-Noise Ratio (PSNR) of 35.82 and Structural Similarity Index Measure (SSIM) of 0.94 than the existing methods.

Deep Gaussian Scale Mixture Prior for Image Reconstruction.

Image reconstruction from partial observations has attracted increasing attention. Conventional image reconstruction methods with hand-crafted priors often fail to recover fine image details due to the poor representation capability of the hand-crafted priors. Deep learning methods attack this problem by directly learning mapping functions between the observations and the targeted images can achieve much better results. However, most powerful deep networks lack transparency and are nontrivial to design heuristically. This paper proposes a novel image reconstruction method based on the Maximum a Posterior (MAP) estimation framework using learned Gaussian Scale Mixture (GSM) prior. Unlike existing unfolding methods that only estimate the image means (i.e., the denoising prior) but neglected the variances, we propose characterizing images by the GSM models with learned means and variances through a deep network. Furthermore, to learn the long-range dependencies of images, we develop an enhanced variant based on the Swin Transformer for learning GSM models. All parameters of the MAP estimator and the deep network are jointly optimized through end-to-end training. Extensive simulation and real data experimental results on spectral compressive imaging and image super-resolution demonstrate that the proposed method outperforms existing state-of-the-art methods.

Survey of Denoising, Segmentation and Classification of Pancreatic Cancer Imaging

Pancreatic cancer is one of the most serious problems that has taken many lives worldwide. The diagnostic procedure using the traditional approaches was manual by visually analyzing the large volumes of the dataset, making it time-consuming and prone to subjective errors. Hence the need for the computer-aided diagnosis system (CADs) emerged that comprises the machine and deep learning approaches for denoising, segmentation and classification of pancreatic cancer. There are different modalities used for the diagnosis of pancreatic cancer, such as Positron Emission Tomography/Computed Tomography (PET/CT), Magnetic Resonance Imaging (MRI), Multiparametric-MRI (Mp-MRI), Radiomics and Radio-genomics. Although these modalities gave remarkable results in diagnosis on the basis of different criteria. CT is the most commonly used modality that produces detailed and fine contrast images of internal organs of the body. However, it may also contain a certain amount of gaussian and rician noise that is necessary to be preprocessed before segmentation of the required region of interest (ROI) from the images and classification of cancer. This paper analyzes different methodologies used for the complete diagnosis of pancreatic cancer, including the denoising, segmentation and classification, along with the challenges and future scope for the diagnosis of pancreatic cancer. Various filters are used for denoising and image smoothening and filters as gaussian scale mixture process, non-local means, median filter, adaptive filter and average filter have been used more for better results. In terms of segmentation, atlas based region-growing method proved to give better results as compared to the state of the art whereas, for the classification, deep learning approaches outperformed other methodologies to classify the images as cancerous and non- cancerous. These methodologies have proved that CAD systems have become a better solution to the ongoing research proposals for the detection of pancreatic cancer worldwide.

Joint Activity Detection and Channel Estimation for Intelligent-Reflecting-Surface-Assisted Wireless IoT Networks

Intelligent reflecting surfaces (IRSs) provide the ability to tune the wireless environments by introducing passive scattering elements between the base station (BS) or access points (APs) and users, and it is a promising technology for next generation wireless networks. In this paper, we consider joint activity detection and channel estimation in both centralized and distributed IRS assisted IoT networks. We determine the explicit distribution for the equivalent channel coefficients from the BS or APs to the users. It is shown that this distribution can be equivalently represented with a Gaussian scale mixture (GSM) model. With generalized approximate message passing (GAMP) algorithm, the received pilot signals at the BS or APs are decoupled into scalar Gaussian noise corrupted versions of the effective channel coefficients. Subsequently, minimum mean square error (MMSE) estimate of the effective channel coefficients and threshold detection rules are acquired. Finally, the optimal fusion rule is used to obtain the activity detection results of each user. As an additional low-complexity approach, we approximate the equivalent channel coefficients with a Gaussian approximation (GA) model. We derive the theoretical mean square error for the MMSE estimate of the effective channel coefficients using the GA model and the state evolution equations of the GAMP algorithm, under the assumption that the direct links from the BS to the users can be neglected due to unfavorable propagation conditions. Numerical results show that the Gaussian approximation for the distribution of the equivalent channel coefficients is accurate even for moderate number of elements at the IRS. And the theoretical mean square error for the MMSE estimate of the effective channel coefficients under the assumption that direct link is negligible is shown to overlap with the corresponding numerical simulations.

Identification of MISO Hammerstein system using sparse multiple kernel-based hierarchical mixture prior and variational Bayesian inference

The Hammerstein model is a cascade composition of a static memoryless nonlinear function followed by a linear time-invariant dynamical subsystem, which is capable of modeling a wide range of nonlinear dynamical systems. Model structural parameter selection (including the model order and the nonlinearity order) and sparse representation of the static nonlinear function are two issues that receive increasing interests in Hammerstein system identification. In this paper, a novel Bayesian sparse multiple kernel-based identification method (BSMKM) for multiple-input single-output (MISO) Hammerstein system is proposed to handle those issues, where the basis-function model and the finite impulse response model are used to represent the nonlinear part and the linear part respectively. Firstly, in order to jointly realize the model parameter estimation, the sparse representation of static nonlinear function (the nonlinearity order selection can also be realized indirectly) and the model order selection of linear dynamical system, a hierarchical prior distribution is constructed based on Gaussian scale mixture model and sparse multiple kernel, which can characterize both inter-group sparsity and intra-group correlation structure. Then, a full Bayesian method based on variational Bayesian inference is proposed to estimate all unknown model parameters, including finite impulse response coefficients, hyperparameters and noise variance. Finally, the performance of the proposed BSMKM identification method is evaluated by numerical experiments using both simulation and real data.

Stochastic interpolation of sparsely sampled time series by a superstatistical random process and its synthesis in Fourier and wavelet space

We present a novel method for stochastic interpolation of sparsely sampled time signals based on a superstatistical random process generated from a multivariate Gaussian scale mixture. In comparison to other stochastic interpolation methods such as Gaussian process regression, our method possesses strong multifractal properties and is thus applicable to a broad range of real-world time series, e.g. from solar wind or atmospheric turbulence. Furthermore, we provide a sampling algorithm in terms of a mixing procedure that consists of generating a -dimensional field , where each Gaussian component is synthesized with identical underlying noise but different covariance function parameterized by a log-normally distributed parameter ξ. Due to the Gaussianity of each component , we can exploit standard sampling algorithms such as Fourier or wavelet methods and, most importantly, methods to constrain the process on the sparse measurement points. The scale mixture u(t) is then initialized by assigning each point in time t a and therefore a specific value from , where the time-dependent parameter follows a log-normal process with a large correlation time scale compared to the correlation time of . We juxtapose Fourier and wavelet methods and show that a multiwavelet-based hierarchical approximation of the interpolating paths, which produce a sparse covariance structure, provide an adequate method to locally interpolate large and sparse datasets.

GSM Filter: A Learning Model of Measurement Noise Distributions for Bayesian Filtering

In this paper, a new universal Bayesian filter with a Gaussian scale mixture (GSM) model, called the Gaussian scale mixture (GSM) filter, for adaptively learning any measurement noise distribution of a state space system is presented. It is demonstrated that the discrete measurement noise distribution of the system takes on a Gaussian sum form when described by a GSM model. It implies that, as the Gaussian sum model does, the GSM model can also be used to approximate any measurement distribution as closely as desired, whatever the probability density function (pdf) of the measurement noise is. If its covariance is known as a priori or can be estimated, it is found that the scale parameter and matrix of the GSM model at each time step can be achieved adaptively to make the following Bayesian inference much simpler. Further analyses show that the GSM model's auxiliary random variable and the system's state posterior at each time step can be inferred by a variational Bayesian (VB) approach adaptively. At each time step, the posterior distribution of system states shown is a Gaussian whatever the measurement noise pdf of the system is. As a result, the Bayesian filtering for nonlinear systems with any measurement noise pdf can be done by our GSM filter, almost as simple as the general Gaussian filters for filtering a nonlinear Gaussian system. Both simulations and experiments are conducted to verify the effectiveness and correctness of the analytical results. The results on the performances of our algorithms fully outperforming those of the existing state of the art Bayesian filters are given.

Informative Bayesian Neural Network Priors for Weak Signals

Encoding domain knowledge into the prior over the high-dimensional weight space of a neural network is challenging but essential in applications with limited data and weak signals. Two types of domain knowledge are commonly available in scientific applications: 1. feature sparsity (fraction of features deemed relevant); 2. signal-to-noise ratio, quantified, for instance, as the proportion of variance explained. We show how to encode both types of domain knowledge into the widely used Gaussian scale mixture priors with Automatic Relevance Determination. Specifically, we propose a new joint prior over the local (i.e., feature-specific) scale parameters that encodes knowledge about feature sparsity, and a Stein gradient optimization to tune the hyperparameters in such a way that the distribution induced on the model’s proportion of variance explained matches the prior distribution. We show empirically that the new prior improves prediction accuracy compared to existing neural network priors on publicly available datasets and in a genetics application where signals are weak and sparse, often outperforming even computationally intensive cross-validation for hyperparameter tuning.

Superstatistical Wind Fields from Pointwise Atmospheric Turbulence Measurements

Accurate models of turbulent wind fields have become increasingly important in the atmospheric sciences, e.g., for the determination of spatiotemporal correlations in wind parks, the estimation of individual loads on turbine rotor and blades, or the modeling of particle-turbulence interaction in atmospheric clouds or pollutant distributions in urban settings. Because of the difficult task of resolving the fields across a broad range of scales, one oftentimes has to invoke stochastic wind field models that fulfill specific, empirically observed, properties. Whereas commonly used Gaussian random field models solely control second-order statistics (i.e., velocity correlation tensors or kinetic energy spectra), we explicitly show that our extended model emulates the effects of higher-order statistics as well. Most importantly, the empirically observed phenomenon of small-scale intermittency, which can be regarded as one of the key features of atmospheric turbulent flows, is reproduced with a very high level of accuracy and at considerably low computational cost. Our method is based on a multipoint statistical description of turbulent velocity fields that consists of a superposition of multivariate Gaussian statistics with fluctuating covariances. We propose a new and efficient sampling algorithm for this Gaussian scale mixture and demonstrate how such “superstatistical” wind fields can be constrained on a certain number of real-world measurement data points from a meteorological mast array.Received 12 April 2022Revised 26 July 2022Accepted 22 August 2022DOI:https://doi.org/10.1103/PRXEnergy.1.023006Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.Published by the American Physical SocietyPhysics Subject Headings (PhySH)Research AreasAtmospheric scienceBoundary layersShear flowsStochastic processesStructure & turbulence of boundary layersSustainabilityTurbulenceWind energyEnergy Science & TechnologyNonlinear DynamicsFluid DynamicsStatistical Physics

CLT for linear spectral statistics of high-dimensional sample covariance matrices in elliptical distributions

In this paper, we establish a new central limit theorem for the linear spectral statistics of high-dimensional sample covariance matrices. The underlying population belongs to the family of elliptical distributions, and the dimension of the population is allowed to grow to infinity, in proportion to the sample size. As an application, we construct confidence intervals for the model parameters of a Gaussian scale mixture.

Image Deblurring Algorithm Based on the Gaussian-Scale Mixture Expert Field Model

With the proliferation of portable digital products, image quality degradation has received a lot of attention. As the most common phenomenon in image degradation, the issue of image deblurring is the focus of much attention. Blind motion blur removal is the main target of this paper. The heavy-tailed distribution is the most dominant statistical feature of natural images. However, most image deblurring methods use a gradient prior with fixed parameters to recover a clear image, which leads to loss of details in the recovered clear image and does not consider the higher order prior of the natural image. Therefore, this paper proposes a new regularized image recovery model based on the Gaussian-scale mixture expert field(GSM-FOE) model. First, the GSM-FoE model learns filters and corresponding parameters with higher order prior information of images by training images in a natural image library; second, these learning results are used to guide the image recovery process. The GSM-FoE model and gradient-fidelity based image recovery model is proposed, which can be used with an iterative re-weighted least squares (IRLS) method. Experiments demonstrate that the suggested recovery approach is simple to use and successful at reducing blur and noise, as well as suppressing ringing effects while preserving image information. Moreover, the image restoration method performs well for large blurring kernels. The results fully reflect the effectiveness and robustness of the proposed method for complex noise scenarios. The quality of the generated images is significantly better than that of several classical methods.

Bayesian Correlation Filter Learning With Gaussian Scale Mixture Model for Visual Tracking

Correlation filters (CF), a popular tool for visual tracking, suffer from unwanted boundary effects due to the periodic assumption needed for FFT implementation. To address this issue, spatially regularized discriminative correlation filters (SRDCF) have been proposed by introducing a weighting matrix to the regularization term. However, the existing design of spatial weighting matrix is often heuristic and non-adaptive. Inspired by recent advances in joint discrimination and reliability learning for correlation tracking, we propose a principled Bayesian correlation filter learning method using Gaussian scale mixture (GSM) model. The key idea is to decompose each CF coefficient into the product of a positive scalar multiplier and a Gaussian random variable. Treating positive multipliers as weighting coefficients, GSM-based modeling of CFs leads to a spatially adaptive regularization strategy with improved capability of handling various appearance-related uncertainty factors (e.g., scale variation, out-of-plane rotation, and motion blur). Moreover, by imposing a sparse prior over the multipliers, we can jointly learn multipliers and CFs under a unified Bayesian estimation framework. Structured GSM model allows us to better exploit the spatial correlations among CFs and further improve the tracking performance. Experimental results on OTB-2013, OTB-2015, Temple Color-128, VOT-2016, and VOT-2017 show that our tracking method performs favorably when compared with current state-of-the-art methods.

Correlation filters based on spatial-temporal Gaussion scale mixture modelling for visual tracking

Correlation Filters (CFs) have shown outstanding performance in tracking, but are subject to unwanted boundary effects. Spatial regularization (SR) is widely used as an efficient method to alleviate the boundary effects. However, spatial regularization is almost handcrafted and fixed during tracking process, which cannot handle the diversity of objects and the complexity of motion. Furthermore, the rich spatio-temporal correlations among multiple targets of interest cannot be fully exploited. Herein, we propose a spatio-temporal Gaussian scale mixture model (ST-GSM) for correlation-filter-based visual tracking. In our Gaussian scale mixture (GSM) model, each correlation filter coefficient is decomposed into the product of a positive scalar multiplier with sparsity and a Gaussian random variable. The reliable components of the Gaussian random variable can be adaptively selected based on the positive multipliers, aiming at alleviating the notorious boundary effects. To exploit the temporal consistency between adjacent frames, nonzero-means GSM models are developed to characterize the temporal correlations. Specifically, the filter coefficient obtained in the previous frame is used as the mean prior for the current frame. The spatial correlations among filter coefficients have been considered in the structured GSM model, thereby further improving the tracking performance. Experimental results show that the proposed model can significantly improve the performance of CF-based trackers.

A type-II maximum-likelihood approach to Gaussian scale mixture-based sparse regression Kriging

In this paper, a novel sparse regression Kriging method termed SRK is proposed, putting an emphasis on efficiently identifying an adaptive overall trend. The main idea underlying SRK is that, by applying a Cholesky decomposition on the correlation matrix, a general Gaussian scale mixture prior- based sparse Bayesian learning scheme can be naturally incorporated into Gaussian process regression, thus facilitating determination of the adaptive trend and correlation functions in an iterative manner. In particular, two sparsity-inducing distributions including Laplacian and Student’s T are implemented as special cases of the Gaussian scale mixture prior, and it is found that their influence to SRK just differs in the estimating formula of a common hyper-parameter. Metamodeling experiments are performed on practical engineering design problems with very limited training data points. Results demonstrate that the Laplacian-based SRK is not only more sensible than T-based SRK, but also achieves comparable or even better performance than benchmark approaches in terms of either computational cost or prediction precision.