Separable Covariance Structure Research Articles

Data-driven optimal shrinkage of singular values under high-dimensional noise with separable covariance structure with application

We develop a data-driven optimal shrinkage algorithm, named extended OptShrink (eOptShrink), for matrix denoising with high-dimensional noise and a separable covariance structure. This noise is colored and dependent across samples. The algorithm leverages the asymptotic behavior of singular values and vectors of the noisy data's random matrix. Our theory includes the sticking property of non-outlier singular values, delocalization of weak signal singular vectors, and the spectral behavior of outlier singular values and vectors. We introduce three estimators: a novel rank estimator, an estimator for the spectral distribution of the pure noise matrix, and the optimal shrinker eOptShrink. Notably, eOptShrink does not require estimating the noise's separable covariance structure. We provide a theoretical guarantee for these estimators with a convergence rate. Through numerical simulations and comparisons with state-of-the-art optimal shrinkage algorithms, we demonstrate eOptShrink's application in extracting maternal and fetal electrocardiograms from single-channel trans-abdominal maternal electrocardiograms.

Decorrelated nearest shrunken centroids for tensor data

AbstractThe nearest shrunken centroids (NSC) method is an efficient and accurate classifier. However, it is incapable of modelling correlation among predictors. Moreover, many contemporary datasets have tensor predictors that cannot be directly handled by NSC. We tackle these challenges by proposing a new distance‐based classifier, tensor decorrelated NSC (TDNSC). TDNSC leverages the popular separable covariance structure on tensor data to decorrelate data and allow easy application of NSC afterwards. Unlike existing tensor classifiers that often rely on complicated iterative algorithms, TDNSC has analytical solutions. The theoretical properties and empirical results suggest that TDNSC is a promising method for tensor classification.

Generalized kernel distance covariance in high dimensions: non-null CLTs and power universality

Abstract Distance covariance is a popular dependence measure for two random vectors $X$ and $Y$ of possibly different dimensions and types. Recent years have witnessed concentrated efforts in the literature to understand the distributional properties of the sample distance covariance in a high-dimensional setting, with an exclusive emphasis on the null case that $X$ and $Y$ are independent. This paper derives the first non-null central limit theorem for the sample distance covariance, and the more general sample (Hilbert–Schmidt) kernel distance covariance in high dimensions, in the distributional class of $(X,Y)$ with a separable covariance structure. The new non-null central limit theorem yields an asymptotically exact first-order power formula for the widely used generalized kernel distance correlation test of independence between $X$ and $Y$. The power formula in particular unveils an interesting universality phenomenon: the power of the generalized kernel distance correlation test is completely determined by $n\cdot \operatorname{dCor}^{2}(X,Y)/\sqrt{2}$ in the high-dimensional limit, regardless of a wide range of choices of the kernels and bandwidth parameters. Furthermore, this separation rate is also shown to be optimal in a minimax sense. The key step in the proof of the non-null central limit theorem is a precise expansion of the mean and variance of the sample distance covariance in high dimensions, which shows, among other things, that the non-null Gaussian approximation of the sample distance covariance involves a rather subtle interplay between the dimension-to-sample ratio and the dependence between $X$ and $Y$.

Modeling and Learning on High-Dimensional Matrix-Variate Sequences

We propose a new matrix factor model, named RaDFaM, which is strictly derived from the general rank decomposition and assumes a high-dimensional vector factor model structure for each basis vector. RaDFaM contributes a novel class of low-rank latent structures that trade off between signal intensity and dimension reduction from a tensor subspace perspective. Based on the intrinsic separable covariance structure of RaDFaM, for a collection of matrix-valued observations, we derive a new class of PCA variants for estimating loading matrices, and sequentially the latent factor matrices. The peak signal-to-noise ratio of RaDFaM is proved to be superior in the category of PCA-type estimators. We also establish an asymptotic theory including the consistency, convergence rates, and asymptotic distributions for components in the signal part. Numerically, we demonstrate the performance of RaDFaM in applications such as matrix reconstruction, supervised learning, and clustering, on uncorrelated and correlated data, respectively. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Tensor generalized canonical correlation analysis

Regularized Generalized Canonical Correlation Analysis (RGCCA) is a general statistical framework for multiblock data analysis. RGCCA enables deciphering relationships between several sets of variables and subsumes many well-known multivariate analysis methods as special cases. However, RGCCA only deals with vector-valued blocks, disregarding their possible higher-order structures. This paper presents Tensor GCCA (TGCCA), a new method for analyzing higher-order tensors with canonical vectors admitting an orthogonal rank-R CP decomposition. Moreover, two algorithms for TGCCA, based on whether a separable covariance structure is imposed or not, are presented along with convergence guarantees. The efficiency and usefulness of TGCCA are evaluated on simulated and real data and compared favorably to state-of-the-art approaches.

Model-Based Tensor Low-Rank Clustering

ABSTRACT Tensors have become prevalent in business applications and scientific studies. It is of great interest to analyze and understand the heterogeneity in tensor-variate observations. We propose a novel tensor low-rank mixture model (TLMM) to conduct efficient estimation and clustering on tensors. The model combines the Tucker low-rank structure in mean contrasts and the separable covariance structure to achieve parsimonious and interpretable modeling. To implement efficient computation under this model, we develop a low-rank enhanced expectation-maximization (LEEM) algorithm. The pseudo E-step and the pseudo M-step are carefully designed to incorporate variable selection and efficient parameter estimation. Numerical results in extensive experiments demonstrate the encouraging performance of the proposed method compared to popular vector and tensor methods. Supplementary materials for this article are available online.

Regularized matrix data clustering and its application to image analysis

We propose a novel regularized mixture model for clustering matrix-valued data. The proposed method assumes a separable covariance structure for each cluster and imposes a sparsity structure (eg, low rankness, spatial sparsity) for the mean signal of each cluster. We formulate the problem as a finite mixture model of matrix-normal distributions with regularization terms, and then develop an expectation maximization type of algorithm for efficient computation. In theory, we show that the proposed estimators are strongly consistent for various choices of penalty functions. Simulation and two applications on brain signal studies confirm the excellent performance of the proposed method including a better prediction accuracy than the competitors and the scientific interpretability of thesolution.

A test of weak separability for multi-way functional data, with application to brain connectivity studies

This paper concerns the modeling of multi-way functional data where double or multiple indices are involved. We introduce a concept of weak separability. The weakly separable structure supports the use of factorization methods that decompose the signal into its spatial and temporal components. The analysis reveals interesting connections to the usual strongly separable covariance structure, and provides insights into tensor methods for multi-way functional data. We propose a formal test for the weak separability hypothesis, where the asymptotic null distribution of the test statistic is a chi-square type mixture. The method is applied to study brain functional connectivity derived from source localized magnetoencephalography signals during motor tasks.

Estimators Comparison of Separable Covariance Structure with One Component as Compound Symmetry Matrix

The maximum likelihood estimation (MLE) of separable covariance structure with one component as compound symmetry matrix has been widely studied in the literature. Nevertheless, the proposed estimates are not given in explicit form and can be determined only numerically. In this paper we give an alternative form of MLE and we show that this new algorithm is much quicker than the algorithms given in the literature.\\ Another estimator of covariance structure can be found by minimizing the entropy loss function. In this paper we give three methods of finding the best approximation of separable covariance structure with one component as compound symmetry matrix and we compare the quickness of proposed algorithms.\\ We conduct simulation studies to compare statistical properties of MLEs and entropy loss estimators (ELEs), such us biasedness, variability and loss. Another estimator of covariance structure can be found by minimizing the entropy loss function. In this paper we give three methods of finding the best approximation of separable covariance structure with one component as compound symmetry matrix and we compare the quickness of proposed algorithms. We conduct simulation studies to compare statistical properties of MLEs and entropy loss estimators (ELEs), such us biasedness and variability.

Separable covariance models for health care quality measures across years and topics.

Public quality reports for Medicare Advantage health plans include 11 measures of patient experiences reported in the annual Consumer Assessment of Healthcare Providers and Systems surveys. Computing summaries at the health plan level (of multiple measures in multiple years) yields an array-structured random variable. To summarize associations among measures and years, we model the variance-covariance matrix governing the plan-level vectors of yearly quality measures as a Kronecker product of an across-measure matrix and an across-year matrix, or a sum of such Kronecker products. This approach extends separable covariance structure to Fay-Herriot models. In addition, we develop linear combinations of Kronecker products similar to principal components for array random variables. To each Kronecker-product term, we apply post hoc analyses suited to the corresponding dimension of the cross-classification: 1-way factor analysis for the across-measure factor and time-series analysis to the across-year factor. These methods draw out key patterns of variation in the quality measures over time and suggest new strategies for reporting quality information to consumers.

Bayesian Spectral Modeling for Multivariate Spatial Distributions of Elemental Concentrations in Soil

Recent technological advances have enabled researchers in a variety of fields to collect accurately geocoded data for several variables simultaneously. In many cases it may be most appropriate to jointly model these multivariate spatial processes without constraints on their conditional relationships. When data have been collected on a regular lattice, the multivariate conditionally autoregressive (MCAR) models are a common choice. However, inference from these MCAR models relies heavily on the pre-specified neighborhood structure and often assumes a separable covariance structure. Here, we present a multivariate spatial model using a spectral analysis approach that enables inference on the conditional relationships between the variables that does not rely on a pre-specified neighborhood structure, is non-separable, and is computationally efficient. Covariance and cross-covariance functions are defined in the spectral domain to obtain computational efficiency. The resulting pseudo posterior inference on the correlation matrix allows for quantification of the conditional dependencies. A comparison is made with an MCAR model that is shown to be highly sensitive to the choice of neighborhood. The approaches are illustrated for the toxic element arsenic and four other soil elements whose relative concentrations were measured on a microscale spatial lattice. Understanding conditional relationships between arsenic and other soil elements provides insights for mitigating pervasive arsenic poisoning in drinking water in southern Asia and elsewhere.

Gaussian tree constraints applied to acoustic linguistic functional data

Evolutionary models of languages are usually considered to take the form of trees. With the development of so-called tree constraints the plausibility of the tree model assumptions can be assessed by checking whether the moments of observed variables lie within regions consistent with Gaussian latent tree models. In our linguistic application, the data set comprises acoustic samples (audio recordings) from speakers of five Romance languages or dialects. The aim is to assess these functional data for compatibility with a hereditary tree model at the language level. A novel combination of canonical function analysis (CFA) with a separable covariance structure produces a representative basis for the data. The separable-CFA basis is formed of components which emphasize language differences whilst maintaining the integrity of the observational language-groupings. A previously unexploited Gaussian tree constraint is then applied to component-by-component projections of the data to investigate adherence to an evolutionary tree. The results highlight some aspects of Romance language speech that appear compatible with an evolutionary tree model but indicate that it would be inappropriate to model all features as such.

Score test for a separable covariance structure with the first component as compound symmetric correlation matrix

Likelihood ratio tests (LRTs) for separability of a covariance structure for doubly multivariate data are widely studied in the literature. There are three types of LRT: biased tests based on an asymptotic chi-square null distribution; unbiased/unmodified tests based on an empirical null distribution; and unbiased/modified tests with a test statistic modified to follow a theoretical chi-square null distribution. The Rao’s score test (RST) statistic, an alternative for both biased and unbiased/unmodified versions of the corresponding LRT test statistics, is derived for a common case. In this paper the separability of a covariance structure with the first component as a compound symmetric correlation matrix under the assumption of multivariate normality is tested. For this purpose Monte Carlo simulation studies, which compare the biased LRT to biased RST, and unbiased/unmodified LRT to unbiased/unmodified RST, are conducted. It is shown that the RSTs outperform their corresponding LRTs in the sense of empirical Type I error as well as empirical null distribution. Moreover, since the RST does not require estimation of a general variance–covariance matrix (the alternative hypothesis), RST can be performed for small sample sizes, where the variance–covariance matrix could not be estimated for the corresponding LRT, making the LRT infeasible. Three examples are presented to illustrate and compare statistical inference based on LRT and RST.

Emulation of Multivariate Simulators Using Thin-Plate Splines with Application to Atmospheric Dispersion

It is often desirable to build a statistical emulator of a complex computer simulator in order to perform analysis which would otherwise be computationally infeasible. We propose methodology to model multivariate output from a computer simulator taking into account output structure in the responses. The utility of this approach is demonstrated by applying it to a chemical and biological hazard prediction model. Predicting the hazard area that results from an accidental or deliberate chemical or biological release is imperative in civil and military planning and also in emergency response. The hazard area resulting from such a release is highly structured in space and we therefore propose the use of a thin-plate spline to capture the spatial structure and fit a Gaussian process emulator to the coefficients of the resultant basis functions. We compare and contrast four different techniques for emulating multivariate output: dimension-reduction using (i) a fully Bayesian approach with a principal component basis, (ii) a fully Bayesian approach with a thin-plate spline basis, assuming that the basis coefficients are independent, and (iii) a "plug-in" Bayesian approach with a thin-plate spline basis and a separable covariance structure; and (iv) a functional data modeling approach using a tensor-product (separable) Gaussian process. We develop methodology for the two thin-plate spline emulators and demonstrate that these emulators significantly outperform the principal component emulator. Further, the separable thin-plate spline emulator, which accounts for the dependence between basis coefficients, provides substantially more realistic quantification of uncertainty, and is also computationally more tractable, allowing fast emulation. For high resolution output data, it also offers substantial predictive and computational advantages over the tensor-product Gaussian process emulator.

A semiparametric spatio-temporal model for solar irradiance data

We evaluate semiparametric spatio-temporal models for global horizontal irradiance at high spatial and temporal resolution. These models represent the spatial domain as a lattice and are capable of predicting irradiance at lattice points, given data measured at other lattice points. Using data from a 1.2 MW PV plant located in Lanai, Hawaii, we show that a semiparametric model can be more accurate than simple interpolation between sensor locations. We investigate spatio-temporal models with separable and nonseparable covariance structures and find no evidence to support assuming a separable covariance structure. Our results indicate a promising approach for modeling irradiance at high spatial resolution consistent with available ground-based measurements. Such modeling may find application in design, valuation, and operation of fleets of utility-scale photovoltaic power systems.

Multivariate Gaussian Process Emulators With Nonseparable Covariance Structures

The Gaussian process regression model is a popular type of “emulator” used as a fast surrogate for computationally expensive simulators (deterministic computer models). For simulators with multivariate output, common practice is to specify a separable covariance structure for the Gaussian process. Though computationally convenient, this can be too restrictive, leading to poor performance of the emulator, particularly when the different simulator outputs represent different physical quantities. Also, treating the simulator outputs as independent can lead to inappropriate representations of joint uncertainty. We develop nonseparable covariance structures for Gaussian process emulators, based on the linear model of coregionalization and convolution methods. Using two case studies, we compare the performance of these covariance structures both with standard separable covariance structures and with emulators that assume independence between the outputs. In each case study, we find that only emulators with nonseparable covariances structures have sufficient flexibility both to give good predictions and to represent joint uncertainty about the simulator outputs appropriately. This article has supplementary material online.

Separable covariance arrays via the Tucker product, with applications to multivariate relational data

Modern datasets are often in the form of matrices or arrays, potentially having correlations along each set of data indices. For example, data involving repeated measurements of several variables over time may exhibit temporal correlation as well as correlation among the variables. A possible model for matrix-valued data is the class of matrix normal distributions, which is parametrized by two covariance matrices, one for each index set of the data. In this article we discuss an extension of the matrix normal model to accommodate multidimensional data arrays, or tensors. We show how a particular array-matrix product can be used to generate the class of array normal distributions having separable covariance structure. We derive some properties of these covariance structures and the corresponding array normal distributions, and show how the array-matrix product can be used to define a semi-conjugate prior distribution and calculate the corresponding posterior distribution. We illustrate the methodology in an analysis of multivariate longitudinal network data which take the form of a four-way array.

Principal components analysis for sparsely observed correlated functional data using a kernel smoothing approach

We consider the problem of functional principal component analysis for correlated functional data. In particular, we focus on a separable covariance structure and consider irregularly and possibly sparsely observed sample trajectories. By observing that under the sparse measurements setting, the empirical covariance of pre-smoothed sample trajectories is a highly biased estimator along the diagonal, we propose to modify the empirical covariance by estimating the diagonal and off-diagonal parts of the covariance kernel separately. We prove that under a separable covariance structure, this method can consistently estimate the eigenfunctions of the covariance kernel. We also quantify the role of the correlation in the L2 risk of the estimator, and show that under a weak correlation regime, the risk achieves the optimal nonparametric rate when the number of measurements per curve is bounded.