Large Covariance Matrix Research Articles

Comparison of linear shrinkage estimators of a large covariance matrix in normal and non-normal distributions

The problem of estimating the large covariance matrix of both normal and non-normal distributions is addressed. In convex combinations of the sample covariance matrix and a positive definite target matrix, the optimal weight is estimated by exact or approximate unbiased estimators of the numerator and denominator of the optimal weight in normal or non-normal cases. A spherical and a diagonal matrices are two typical examples of target matrices, and the corresponding single shrinkage estimators are provided. A double shrinkage estimator which shrinks the sample covariance matrix toward the two target matrices is also suggested. The performances of single and double shrinkage estimators are numerically investigated through simulation and empirical studies.

Efficient estimation of approximate factor models via penalized maximum likelihood

We study an approximate factor model in the presence of both cross sectional dependence and heteroskedasticity. For efficient estimations it is essential to estimate a large error covariance matrix. We estimate the common factors and factor loadings based on maximizing a Gaussian quasi-likelihood, through penalizing a large covariance sparse matrix. The weighted ℓ1 penalization is employed. While the principal components (PC) based methods estimate the covariance matrices and individual factors and loadings separately, they require consistent estimation of residual terms. In contrast, the penalized maximum likelihood method (PML) estimates the factor loading parameters and the error covariance matrix jointly. In the numerical studies, we compare PML with the regular PC method, the generalized PC method (Choi 2012) combined with the thresholded covariance matrix estimator (Fan et al. 2013), as well as several related methods, on their estimation and forecast performances. Our numerical studies show that the proposed method performs well in the presence of cross-sectional dependence and heteroskedasticity.

Numerical Methods of Karhunen–Loève Expansion for Spatial Data

Abstract With the development of technology, a large amount of spatial data are usually observed in many applications. These massive spatial data impose a challenge to the traditional spatial data analysis primarily because of the large covariance matrix. One way to overcome the computation burden is to utilize a low rank model. The optimal low rank model is provided by the Karhunen–Loève (KL) expansion of the spatial process. However, the inference and prediction of the spatial data require an efficient algorithm for the KL expansion. In this paper, we compare four algorithms that have been proposed to numerically obtain the KL expansion. It is found that the Gaussian quadrature method outperforms the others for spatial processes.

Large Covariance Matrix Estimation by Composite Minimization

The present thesis concerns large covariance matrix estimation via composite minimization under the assumption of low rank plus sparse structure. Existing methods like POET (Principal Orthogonal complEment Thresholding) perform estimation by extracting principal components and then applying a soft thresholding algorithm. In contrast, our method recovers the low rank plus sparse decomposition of the covariance matrix by least squares minimization under nuclear norm plus $l_1$ norm penalization. This non-smooth convex minimization procedure is based on semidefinite programming and subdifferential methods, resulting in two separable problems solved by a singular value thresholding plus soft thresholding algorithm. The most recent estimator in literature is called LOREC (Low Rank and sparsE Covariance estimator) and provides non-asymptotic error rates as well as identifiability conditions in the context of algebraic geometry. Our work shows that the unshrinkage of the estimated eigenvalues of the low rank component improves the performance of LOREC considerably. The same method also recovers covariance structures with very spiked latent eigenvalues like in the POET setting, thus overcoming the necessary condition $p\leq n$. In addition, it is proved that our method recovers structures with intermediate degrees of spikiness, obtaining a loss which is bounded accordingly. Then, an ad hoc model selection criterion which detects the optimal point in terms of composite penalty is proposed. Empirical results coming from a wide original simulation study where various low rank plus sparse settings are simulated according to different parameter values are described outlining in detail the improvements upon existing methods. Two real data-sets are finally explored highlighting the usefulness of our method in practical applications.

Review on statistical methods for large spatial Gaussian data

The Gaussian geostatistical model has been widely used for modeling spatial data. However, this model suffers from a severe difficulty in computation because inference requires to invert a large covariance matrix in evaluating log-likelihood. In addressing this computational challenge, three strategies have been employed: likelihood approximation, lower dimensional space approximation, and Markov random field approximation. In this paper, we reviewed statistical approaches attacking the computational challenge. As an illustration, we also applied integrated nested Laplace approximation (INLA) technology, one of Markov approximation approach, to real data to provide an example of its use in practice dealing with large spatial data.

Testing the Diagonality of a Large Covariance Matrix in a Regression Setting

In multivariate analysis, the covariance matrix associated with a set of variables of interest (namely response variables) commonly contains valuable information about the dataset. When the dimension of response variables is considerably larger than the sample size, it is a nontrivial task to assess whether there are linear relationships between the variables. It is even more challenging to determine whether a set of explanatory variables can explain those relationships. To this end, we develop a bias-corrected test to examine the significance of the off-diagonal elements of the residual covariance matrix after adjusting for the contribution from explanatory variables. We show that the resulting test is asymptotically normal. Monte Carlo studies and a numerical example are presented to illustrate the performance of the proposed test.

Multi-antenna spectrum sensing by exploiting spatio-temporal correlation

In this paper, we propose a novel mechanism for spectrum sensing that leads us to exploit the spatio-temporal correlation present in the received signal at a multi-antenna receiver. For the proposed mechanism, we formulate the spectrum sensing scheme by adopting the generalized likelihood ratio test (GLRT). However, the GLRT degenerates in the case of limited sample support. To circumvent this problem, several extensions are proposed that bring robustness to the GLRT in the case of high dimensionality and small sample size. In order to achieve these sample-efficient detection schemes, we modify the GLRT-based detector by exploiting the covariance structure and factoring the large spatio-temporal covariance matrix into spatial and temporal covariance matrices. The performance of the proposed detectors is evaluated by means of numerical simulations, showing important advantages over existing detectors.

Sequential Sufficient Dimension Reduction for Large p, Small n Problems

Summary We propose a new and simple framework for dimension reduction in the large p, small n setting. The framework decomposes the data into pieces, thereby enabling existing approaches for n>p to be adapted to n < p problems. Estimating a large covariance matrix, which is a very difficult task, is avoided. We propose two separate paths to implement the framework. Our paths provide sufficient procedures for identifying informative variables via a sequential approach. We illustrate the paths by using sufficient dimension reduction approaches, but the paths are very general. Empirical evidence demonstrates the efficacy of our paths. Additional simulations and applications are given in an on-line supplementary file.

On the strong convergence of the optimal linear shrinkage estimator for large dimensional covariance matrix

In this work we construct an optimal linear shrinkage estimator for the covariance matrix in high dimensions. The recent results from the random matrix theory allow us to find the asymptotic deterministic equivalents of the optimal shrinkage intensities and estimate them consistently. The developed distribution-free estimators obey almost surely the smallest Frobenius loss over all linear shrinkage estimators for the covariance matrix. The case we consider includes the number of variables $p\rightarrow\infty$ and the sample size $n\rightarrow\infty$ so that $p/n\rightarrow c\in (0, +\infty)$. Additionally, we prove that the Frobenius norm of the sample covariance matrix tends almost surely to a deterministic quantity which can be consistently estimated.

Realistic action recognition via sparsely-constructed Gaussian processes

Realistic action recognition has been one of the most challenging research topics in computer vision. The existing methods are commonly based on non-probabilistic classification, predicting category labels but not providing an estimation of uncertainty. In this paper, we propose a probabilistic framework using Gaussian processes (GPs), which can tackle regression problems with explicit uncertain models, for action recognition. A major challenge for GPs when applied to large-scale realistic data is that a large covariance matrix needs to be inverted during inference. Additionally, from the manifold perspective, the intrinsic structure of the data space is only constrained by a local neighborhood and data relationships with far-distance usually can be ignored. Thus, we design our GPs covariance matrix via the proposed ℓ1 construction and a local approximation (LA) covariance weight updating method, which are demonstrated to be robust to data noise, automatically sparse and adaptive to the neighborhood. Extensive experiments on four realistic datasets, i.e., UCF YouTube, UCF Sports, Hollywood2 and HMDB51, show the competitive results of ℓ1-GPs compared with state-of-the-art methods on action recognition tasks.

Eigenvector dynamics under free addition

We investigate the evolution of a given eigenvector of a symmetric (deterministic or random) matrix under the addition of a matrix in the Gaussian orthogonal ensemble. We quantify the overlap between this single vector with the eigenvectors of the initial matrix and identify precisely a "Cauchy flight" regime. In particular, we compute the local density of this vector in the eigenvalues space of the initial matrix. Our results are obtained in a non-perturbative setting and are derived using the ideas of [O. Ledoit and S. Péché, Eigenvectors of some large sample covariance matrix ensembles, Probab. Theory Related Fields151 (2011) 233]. Finally, we give a robust derivation of a result obtained in [R. Allez and J.-P. Bouchaud, Eigenvector dynamics: General theory and some applications, Phys. Rev. E86 (2012) 046202] to study eigenspace dynamics in a semi-perturbative regime.

Testing the Diagonality of a Large Covariance Matrix in a Regression Setting

Testing the Diagonality of a Large Covariance Matrix in a Regression Setting

An empirical estimator for the sparsity of a large covariance matrix under multivariate normal assumptions

Large covariance or correlation matrix is frequently assumed to be sparse in that a number of the off-diagonal elements of the matrix are zero. This paper focuses on estimating the sparsity of a large population covariance matrix using a sample correlation matrix under multivariate normal assumptions. We show that sparsity of a population covariance matrix can be well estimated by thresholding the sample correlation matrix. We then propose an empirical estimator for the sparsity and show that it is closely related to the thresholding methods. Upper bounds for the estimation error of the empirical estimator are given under mild conditions. Simulation shows that the empirical estimator can have smaller mean absolute errors than its main competitors. Furthermore, when the dimension of the covariance matrix is very large, we propose a generalized empirical estimator using simple random sampling. It is shown that the generalized empirical estimator can still estimate the sparsity well while the computation complexity can be greatly reduced.

Bayesian site selection for fast Gaussian process regression

Gaussian Process (GP) regression is a popular method in the field of machine learning and computer experiment designs; however, its ability to handle large data sets is hindered by the computational difficulty in inverting a large covariance matrix. Likelihood approximation methods were developed as a fast GP approximation, thereby reducing the computation cost of GP regression by utilizing a much smaller set of unobserved latent variables called pseudo points. This article reports a further improvement to the likelihood approximation methods by simultaneously deciding both the number and locations of the pseudo points. The proposed approach is a Bayesian site selection method where both the number and locations of the pseudo inputs are parameters in the model, and the Bayesian model is solved using a reversible jump Markov chain Monte Carlo technique. Through a number of simulated and real data sets, it is demonstrated that with appropriate priors chosen, the Bayesian site selection method can produce a good balance between computation time and prediction accuracy: it is fast enough to handle large data sets that a full GP is unable to handle, and it improves, quite often remarkably, the prediction accuracy, compared with the existing likelihood approximations.

A local moment estimator of the spectrum of a large dimensional covariance matrix

This paper considers the problem of estimating the population spectral distribution from a sample covariance matrix in large dimensional situations. We generalize the contour-integral based method in Mestre (2008) and present a local moment estimation procedure. Compared with the original one, the new proce- dure can be applied successfully to models where the asymptotic clusters of sample eigenvalues generated by dierent population eigenvalues are not all separate. The proposed estimates are proved to be consistent. Numerical results illustrate the im- plementation of the estimation procedure and demonstrate its eciency in various cases.

Algorithm for the Characterization of the Cross-Correlation Structure in Multivariate Time Series

In this paper, it is shown that the block circulant matrix decomposition technique makes the multivariate singular spectrum analysis (M-SSA) a well suited tool for detecting of changes of the correlation structure in non-stationary multivariate time series in the presence of high observational noise levels. The major drawback of M-SSA, that it operates on a large covariance matrix and becomes computationally expensive, can be avoided by reordering the Toeplitz-block covariance matrix into a block Toeplitz matrix, embedding this into a block circulant matrix and efficiently block-diagonalizing this by the means of the Fast Fourier Transform (FFT) using the well known algorithm. The overall degree of synchronization among multiple-channel signals is defined by the synchronization index (the S-estimator) of the rearranged and truncated eigenvalue spectrum. Throughout the experiment, the high capability of the proposed algorithm to detect the lag-synchronized state under the influence of strong noise is validated with simulated data—a network of time series generated by autoregressive models (AR) and a network of coupled chaotic Roessler oscillators.

Convergence rates of eigenvector empirical spectral distribution of large dimensional sample covariance matrix

The eigenvector Empirical Spectral Distribution (VESD) is adopted to investigate the limiting behavior of eigenvectors and eigenvalues of covariance matrices. In this paper, we shall show that the Kolmogorov distance between the expected VESD of sample covariance matrix and the Mar\v{c}enko-Pastur distribution function is of order $O(N^{-1/2})$. Given that data dimension $n$ to sample size $N$ ratio is bounded between 0 and 1, this convergence rate is established under finite 10th moment condition of the underlying distribution. It is also shown that, for any fixed $\eta>0$, the convergence rates of VESD are $O(N^{-1/4})$ in probability and $O(N^{-1/4+\eta})$ almost surely, requiring finite 8th moment of the underlying distribution.

Consistency of large dimensional sample covariance matrix under weak dependence

Convergence rates for banded and tapered estimates of large dimensional covariance matrices are known when the vector observations are independent and identically distributed. We investigate the case where the independence does not hold. Our models can accommodate suitable patterned cross covariance matrices. These estimators remain consistent in the operator norm with appropriate rates of convergence under suitable class of models.

Estimation of the population spectral distribution from a large dimensional sample covariance matrix

This paper introduces a new method to estimate the spectral distribution of a population covariance matrix from high-dimensional data. The method is founded on a meaningful generalization of the seminal Marčenko–Pastur equation, originally defined in the complex plane, to the real line. Beyond its easy implementation and the established asymptotic consistency, the new estimator outperforms two existing estimators from the literature in almost all the situations tested in a simulation experiment. An application to the analysis of the correlation matrix of S&P 500 daily stock returns is also given.

A Resampling-Based Stochastic Approximation Method for Analysis of Large Geostatistical Data

The Gaussian geostatistical model has been widely used in modeling of spatial data. However, it is challenging to computationally implement this method because it requires the inversion of a large covariance matrix, particularly when there is a large number of observations. This article proposes a resampling-based stochastic approximation method to address this challenge. At each iteration of the proposed method, a small subsample is drawn from the full dataset, and then the current estimate of the parameters is updated accordingly under the framework of stochastic approximation. Since the proposed method makes use of only a small proportion of the data at each iteration, it avoids inverting large covariance matrices and thus is scalable to large datasets. The proposed method also leads to a general parameter estimation approach, maximum mean log-likelihood estimation, which includes the popular maximum (log)-likelihood estimation (MLE) approach as a special case and is expected to play an important role in analyzing large datasets. Under mild conditions, it is shown that the estimator resulting from the proposed method converges in probability to a set of parameter values of equivalent Gaussian probability measures, and that the estimator is asymptotically normally distributed. To the best of the authors’ knowledge, the present study is the first one on asymptotic normality under infill asymptotics for general covariance functions. The proposed method is illustrated with large datasets, both simulated and real. Supplementary materials for this article are available online.