High-dimensional Covariance Matrices Research Articles

Fast hybrid Bayesian integrative learning of multiple gene regulatory networks for type 1 diabetes.

Motivated by the study of the molecular mechanism underlying type 1 diabetes with gene expression data collected from both patients and healthy controls at multiple time points, we propose a hybrid Bayesian method for jointly estimating multiple dependent Gaussian graphical models with data observed under distinct conditions, which avoids inversion of high-dimensional covariance matrices and thus can be executed very fast. We prove the consistency of the proposed method under mild conditions. The numerical results indicate the superiority of the proposed method over existing ones in both estimation accuracy and computational efficiency. Extension of the proposed method to joint estimation of multiple mixed graphical models is straightforward.

A scalable sparse Cholesky based approach for learning high-dimensional covariance matrices in ordered data

Covariance estimation for high-dimensional datasets is a fundamental problem in machine learning, and has numerous applications. In these high-dimensional settings the number of features or variables p is typically larger than the sample size n. A popular way of tackling this challenge is to induce sparsity in the covariance matrix, its inverse or a relevant transformation. In many applications, the data come with a natural ordering. In such settings, methods inducing sparsity in the Cholesky parameter of the inverse covariance matrix can be quite useful. Such methods are also better positioned to yield a positive definite estimate of the covariance matrix, a critical requirement for several downstream applications. Despite some important advances in this area, a principled approach to general sparse-Cholesky based covariance estimation with both statistical and algorithmic convergence safeguards has been elusive. In particular, the two popular likelihood based methods proposed in the literature either do not lead to a well-defined estimator in high-dimensional settings, or consider only a restrictive class of models. In this paper, we propose a principled and general method for sparse-Cholesky based covariance estimation that aims to overcome some of the shortcomings of current methods, but retains their respective strengths. We obtain a jointly convex formulation for our objective function, and show that it leads to rigorous convergence guarantees and well-defined estimators, even when $$p > n$$ . Very importantly, the approach always leads to a positive definite and symmetric estimator of the covariance matrix. We establish both high-dimensional estimation and selection consistency, and also demonstrate excellent finite sample performance on simulated/real data.

Learning algorithms to evaluate forensic glass evidence

Glass fragments are often compared in the course of a forensic evaluation using their chemical composition determined with technologies such as LA-ICP-MS. At present forensic scientists advocate the use of two comparison criteria based on univariate intervals around all mean elemental concentrations for fragments originating from a known piece of broken glass. The main drawback of this approach is that it does not consider the dependencies between concentrations. Further, when the elemental concentrations are more variable within panes, it becomes harder to reject the null hypothesis of no difference between fragments. In the legal context higher variance would tend to incriminate the defendant because the intervals would tend to be wider. We demonstrate that a score-based approach to assess the probative value of evidence in glass comparisons outperforms the two standard interval methods and other methods proposed in the literature, at least in terms of minimizing classification error in the glass fragment sources we analyzed. We use machine learning algorithms to construct a similarity score between pairs of glass fragments. The learning algorithms exploit the dependencies among elemental concentrations and result in an empirical class probability; so, we can report the degree of similarity between two fragments. Our group is in the process of assembling the first glass composition database with enough information within and between glass samples to permit computing well-conditioned estimates of high-dimensional covariance matrices. These data will be available to anyone who wishes to carry out research in this area.

Tests for high-dimensional covariance matrices

In this paper, we propose some new tests for high-dimensional covariance matrices that are applicable to generally distributed populations with finite fourth moments. The proposed test statistics are the maximum of the likelihood ratio test statistic and the statistic based on the Frobenius norm. The advantage of the new tests is the good performance in terms of power for both the traditional case, in which the dimension is much smaller than the sample size, and the high-dimensional case, in which the dimension is large compared to the sample size. In the one-sample case, the new test is proposed for testing the hypothesis that the high-dimensional covariance matrix equals an identity matrix. In the two-sample case, the new test is developed for testing the equality of two high-dimensional covariance matrices. By using the random matrix theory, the asymptotic distributions of the proposed new tests are derived under the assumption that the dimension and the sample size proportionally tend toward infinity. Finally, numerical studies are conducted to investigate the finite sample performance of the proposed new tests.

A hybrid of Bayesian approach based global search with clustering aided local refinement

We propose a global optimization algorithm in which the P-algorithm is hybridized with a clustering aided local search. The proposed algorithm is free from the computational burden typical for the known implementations of Bayesian algorithms, i.e. the inversion of high dimensional covariance matrices, and the solution of inner non-convex optimization problems. Clustering aided switching between the global and local search rationally combines advantages of both strategies: the basins of potential global minima indicated by the clustering are excluded from the global search, and the potential minima are refined by a local search method. We complete the paper with numerical experiments that illustrate the performance achieved.

Scalable inference for space‐time Gaussian Cox processes

The log‐Gaussian Cox process is a flexible and popular stochastic process for modeling point patterns exhibiting spatial and space‐time dependence. Model fitting requires approximation of stochastic integrals which is implemented through discretization over the domain of interest. With fine scale discretization, inference based on Markov chain Monte Carlo is computationally burdensome because of the cost of matrix decompositions and storage, such as the Cholesky, for high dimensional covariance matrices associated with latent Gaussian variables. This article addresses these computational bottlenecks by combining two recent developments: (i) a data augmentation strategy that has been proposed for space‐time Gaussian Cox processes that is based on exact Bayesian inference and does not require fine grid approximations for infinite dimensional integrals, and (ii) a recently developed family of sparsity‐inducing Gaussian processes, called nearest‐neighbor Gaussian processes, to avoid expensive matrix computations. Our inference is delivered within the fully model‐based Bayesian paradigm and does not sacrifice the richness of traditional log‐Gaussian Cox processes. We apply our method to crime event data in San Francisco and investigate the recovery of the intensity surface.

Equality tests of high-dimensional covariance matrices under the strongly spiked eigenvalue model

We consider the equality test of high-dimensional covariance matrices under the strongly spiked eigenvalue (SSE) model. We find the difference of covariance matrices by dividing high-dimensional eigenspaces into the first eigenspace and the others. We create a new test procedure on the basis of those high-dimensional eigenstructures. We precisely study the influence of spiked eigenvalues on a test statistic and consider its bias correction so that the proposed test procedure has a consistency property for the size. We also show that the proposed test procedure has preferable properties for the power. We discuss the performance of the test procedure by simulations. We give a demonstration in actual data analyses using microarray data sets.

High-dimensional covariance matrices in elliptical distributions with application to spherical test

This paper discusses fluctuations of linear spectral statistics of high-dimensional sample covariance matrices when the underlying population follows an elliptical distribution. Such population often possesses high order correlations among their coordinates, which have great impact on the asymptotic behaviors of linear spectral statistics. Taking such kind of dependency into consideration, we establish a new central limit theorem for the linear spectral statistics in this paper for a class of elliptical populations. This general theoretical result has wide applications and, as an example, it is then applied to test the sphericity of elliptical populations.

Invariant test based on the modified correction to LRT for the equality of two high-dimensional covariance matrices

In this paper, we propose an invariant test based on the modified correction to the likelihood ratio test (LRT) of the equality of two high-dimensional covariance matrices. It is well-known that the classical log-LRT is not well defined when the dimension is larger than or equal to one of the sample sizes. Or even the log-LRT is well-defined, it is usually perceived as a bad statistic in the high-dimensional cases because of their low powers under some alternatives. In this paper, we will justify the usefulness of the modified log-LRT, and an invariant test that works well in cases where the dimension is larger than the sample sizes. Besides, the test is established under the weakest conditions on the dimensions and the moments of the samples. The asymptotic distribution of the proposed test statistic is also obtained under the null hypothesis. What is more, we also propose a lite version of the modified LRT in the paper. A simulation study and a real data analysis show that the performances of the two proposed statistics are invariant under affine transformations.

Testing homogeneity of high-dimensional covariance matrices

Testing homogeneity of high-dimensional covariance matrices

Confidence Sets for Spectral Projectors of Covariance Matrices

A sample X1,...,Xn consisting of independent identically distributed vectors in ℝp with zero mean and a covariance matrix Σ is considered. The recovery of spectral projectors of high-dimensional covariance matrices from a sample of observations is a key problem in statistics arising in numerous applications. In their 2015 work, V. Koltchinskii and K. Lounici obtained nonasymptotic bounds for the Frobenius norm $$\parallel {P_r} - {\hat P_r}{\parallel _2}$$ of the distance between sample and true projectors and studied asymptotic behavior for large samples. More specifically, asymptotic confidence sets for the true projector Pr were constructed assuming that the moment characteristics of the observations are known. This paper describes a bootstrap procedure for constructing confidence sets for the spectral projector Pr of the covariance matrix Σ from given data. This approach does not use the asymptotical distribution of $$\parallel {P_r} - {\hat P_r}{\parallel _2}$$ and does not require the computation of its moment characteristics. The performance of the bootstrap approximation procedure is analyzed.

High-dimensional covariance forecasting based on principal component analysis of high-frequency data

This study provides a new approach to forecasting high-dimensional covariance matrices based on a principal component analysis (PCA) of high-frequency data, by which realized eigenvalues could be estimated and modeled. Our method can avoid the so-called "curse of dimensionality" and handle the case that the number of assets is time-varying. In particular, we propose four (V)HAR-type dynamic models for predicting high-dimensional covariance matrices. All of them can well characterize the long memory behavior of realized eigenvalue series and be easily estimated by OLS. The empirical evidence shows that they outperform the competing models without consideration of long memory behavior in terms of in-sample fitting, out-of-sample prediction, and out-of-sample portfolio allocation.

Estimation of two high-dimensional covariance matrices and the spectrum of their ratio

Let Sp,1, Sp,2 be two independent p×p sample covariance matrices with degrees of freedom n1 and n2, respectively, whose corresponding population covariance matrices are Σp,1 and Σp,2, respectively. Knowing Sp,1, Sp,2, this article proposes a class of estimators for the spectrum (eigenvalues) of the matrix Σp,2Σp,1−1 as well as the pair of the whole matrices (Σp,1,Σp,2). The estimators are created based on Random Matrix Theory. Under mild conditions, our estimator for the spectrum of Σp,2Σp,1−1 is shown to be weakly consistent and the estimator for (Σp,1,Σp,2) is shown to be optimal in the sense of minimizing the asymptotic loss within the class of equivariant estimators as n1,n2,p→∞ with p∕n1→c1∈(0,1), p∕n2→c2∈(0,1)∪(1,∞). Also, our estimators are easy to implement. Even when p is 1000, our estimators can be computed in seconds using a personallaptop.

A test of sphericity for high-dimensional data and its application for detection of divergently spiked noise

In this article, we consider a test of the sphericity for high-dimensional covariance matrices. We produce a test statistic by using the extended cross-data-matrix (ECDM) methodology. We show that the ECDM test statistic is based on an unbiased estimator of a sphericity measure. In addition, the ECDM test statistic enjoys consistency properties and the asymptotic normality in high-dimensional settings. We propose a new test procedure based on the ECDM test statistic and evaluate its asymptotic size and power theoretically and numerically. We give a two-stage sampling scheme so that the test procedure can ensure a prespecified level both for the size and power. We apply the test procedure to detect divergently spiked noise in high-dimensional statistical analysis. We analyze gene expression data by the proposed test procedure.

Robust dependence modeling for high-dimensional covariance matrices with financial applications

A very important problem in finance is the construction of portfolios of assets that balance risk and reward in an optimal way. A critical issue in portfolio development is how to address data outliers that reflect very unusual, generally non-recurring, market conditions. Should we allow these to have a significant impact on our estimation and portfolio construction process or should they be considered separately as evidence of a regime shift and/or be used to adjust baseline results? In financial asset allocation, a fundamental step is often a mean-variance optimization problem that makes use of the location vector and dispersion matrix of the financial assets. In this paper, we introduce a new high- dimensional covariance estimator that is much less sensitive to outliers compared to its classical counterparts. We then apply this estimator to the active asset allocation application, and show that our proposed new estimator delivers better results compared to many existing asset allocation methods. An important bonus is that on our examples, the method has a smaller proportion of stock weights greater than 10% and, in many cases, a higher alpha. Covariance estimation is more challenging than mean estimation and only locally and not globally optimal solutions are available. Our proposed new robust covariance estimator uses a regular vine dependence structure and only pairwise robust partial correlation estimators. The resulting robust covariance estimator delivers high performance for identifying outliers for large high dimensional datasets, has a high breakdown point, and is positive definite. When the full vine structure is not available, we propose using a minimal spanning tree algorithm to replace missing vine structure.

Online Data Thinning via Multi-Subspace Tracking.

In an era of ubiquitous large-scale streaming data, the availability of data far exceeds the capacity of expert human analysts. In many settings, such data is either discarded or stored unprocessed in data centers. This paper proposes a method of online data thinning, in which large-scale streaming datasets are winnowed to preserve unique, anomalous, or salient elements for timely expert analysis. At the heart of this proposed approach is an online anomaly detection method based on dynamic, low-rank Gaussian mixture models. Specifically, the high-dimensional covariance matrices associated with the Gaussian components are associated with low-rank models. According to this model, most observations lie near a union of subspaces. The low-rank modeling mitigates the curse of dimensionality associated with anomaly detection for high-dimensional data, and recent advances in subspace clustering and subspace tracking allow the proposed method to adapt to dynamic environments. Furthermore, the proposed method allows subsampling, is robust to missing data, and uses a mini-batch online optimization approach. The resulting algorithms are scalable, efficient, and are capable of operating in real time. Experiments on wide-area motion imagery and e-mail databases illustrate the efficacy of the proposed approach.

Robust estimation of high-dimensional covariance and precision matrices.

High-dimensional data are often most plausibly generated from distributions with complex structure and leptokurtosis in some or all components. Covariance and precision matrices provide a useful summary of such structure, yet the performance of popular matrix estimators typically hinges upon a sub-Gaussianity assumption. This paper presents robust matrix estimators whose performance is guaranteed for a much richer class of distributions. The proposed estimators, under a bounded fourth moment assumption, achieve the same minimax convergence rates as do existing methods under a sub-Gaussianity assumption. Consistency of the proposed estimators is also established under the weak assumption of bounded 2 + ε moments for ε ∈ (0, 2). The associated convergence rates depend on ε.

Forecasting Large Stochastic Covariance Matrices

Portfolio selection based on high dimensional covariance matrices is a key challenge in data-rich environments with the curse of dimensionality severely affecting most of the available covariance models. We challenge several multivariate Dynamic Conditional Correlation (DCC)-type and Stochastic Volatility (SV)-type models to obtain minimum variance and mean-variance portfolios with up to 1000 assets. We conclude that, in a realistic context in which transaction costs are taken into account, although DCC-type models lead to portfolios with lower variance, modeling the covariance matrices as latent Wishart processes with a shrinkage towards the diagonal covariance matrix delivers more stable optimal portfolios with lower turnover and higher information ratios. Our results reconcile previous findings in the portfolio selection literature as those claiming for equicorrelations, a smooth dynamic evolution of correlations or correlations close to zero.

Applications of Peter Hall's martingale limit theory to estimating and testing high dimensional covariance matrices

Applications of Peter Hall's martingale limit theory to estimating and testing high dimensional covariance matrices

Factor State-Space Models for High-Dimensional Realized Covariance Matrices of Asset Returns

We propose a dynamic factor state-space model for high-dimensional covariance matrices of asset returns. It uses observed risk factors and assumes that the latent covariance matrix of assets and factors is observed through their realized covariance matrix with a Wishart measurement density. The imposed strict factor structure allows for dynamics in the covariances of the factors and the residual components as well as in the factor loadings. The model structure facilitates inference using simple Bayesian MCMC procedures making the approach scalable w.r.t. the number of assets. An empirical application shows that the model performs very well in-and-out-of-sample.