High-dimensional Covariance Matrix Research Articles

Estimation and optimal structure selection of high-dimensional Toeplitz covariance matrix

The estimation of structured covariance matrix arises in many fields. An appropriate covariance structure not only improves the accuracy of covariance estimation but also increases the efficiency of mean parameter estimators in statistical models. In this paper, a novel statistical method is proposed, which selects the optimal Toeplitz covariance structure and estimates the covariance matrix, simultaneously. An entropy loss function with nonconvex penalty is employed as a matrix-discrepancy measure, under which the optimal selection of sparse or nearly sparse Toeplitz structure and the parameter estimators of covariance matrix are made, simultaneously, through its minimization. The cases of both low-dimensional (p≤n) and high-dimensional (p>n) covariance matrix estimation are considered. The resulting Toeplitz structured covariance estimators are guaranteed to be positive definite and consistent. Asymptotic properties are investigated and simulation studies are conducted, showing that very high accurate Toeplitz covariance structure estimation is made. The proposed method is then applied to practical data analysis, which demonstrates its good performance in covariance estimation in practice.

Nowcasting Monthly GDP with Big Data: A Model Averaging Approach

AbstractGross domestic product (GDP) is the most comprehensive and authoritative measure of economic activity. The macroeconomic literature has focused on nowcasting and forecasting this measure at the monthly frequency, using related high-frequency indicators. We address the issue of estimating monthly GDP using a large-dimensional set of monthly indicators, by pooling the disaggregate estimates arising from simple and feasible bivariate models that consider one indicator at a time in conjunction to GDP. Our base model handles mixed-frequency data and ragged-edge data structure with any pattern of missingness. Our methodology enables to distil the common component of the available economic indicators, so that the monthly GDP estimates arise from the projection of the quarterly figures on the space spanned by the common component. The weights used for the combination reflect the ability to nowcast quarterly GDP and are obtained as a function of the regularized estimator of the high-dimensional covariance matrix of the nowcasting errors. A recursive nowcasting and forecasting experiment with data on euro area GDP illustrates that the optimal weights adapt to the information set available in real time and vary according to the phase of the business cycle.

A New Principle for Tuning-Free Huber Regression

The robustification parameter, which balances bias and robustness, has played a critical role in the construction of sub-Gaussian estimators for heavy-tailed and/or skewed data. Although it can be tuned by cross-validation in traditional practice, in large scale statistical problems such as high dimensional covariance matrix estimation and large scale multiple testing, the number of robustification parameters scales with the dimensionality so that cross-validation can be computationally prohibitive. In this paper, we propose a new data-driven principle to choose the robustification parameter for Huber-type sub-Gaussian estimators in three fundamental problems: mean estimation, linear regression, and sparse regression in high dimensions. Our proposal is guided by the non-asymptotic deviation analysis, and is conceptually different from cross-validation which relies on the mean squared error to assess the fit. Extensive numerical experiments and real data analysis further illustrate the efficacy of the proposed methods

Prediction in Locally Stationary Time Series

We develop an estimator for the high-dimensional covariance matrix of a locally stationary process with a smoothly varying trend and use this statistic to derive consistent predictors in nonstationary time series. In contrast to the currently available methods for this problem the predictor developed here does not rely on fitting an autoregressive model and does not require a vanishing trend. The finite sample properties of the new methodology are illustrated by means of a simulation study and a financial indices study. Supplementary materials for this article are available online.

Model averaging estimation for high-dimensional covariance matrices with a network structure.

In this paper, we develop a model averaging method to estimate a high-dimensional covariance matrix, where the candidate models are constructed by different orders of polynomial functions. We propose a Mallows-type model averaging criterion and select the weights by minimizing this criterion, which is an unbiased estimator of the expected in-sample squared error plus a constant. Then, we prove the asymptotic optimality of the resulting model average covariance estimators. Finally, we conduct numerical simulations and a case study on Chinese airport network structure data to demonstrate the usefulness of the proposed approaches.

PolSAR Ship Detection Based on Neighborhood Polarimetric Covariance Matrix

The detection of small ships in polarimetric synthetic aperture radar (PolSAR) images is still a topic for further investigation. Recently, patch detection techniques, such as superpixel-level detection, have stimulated wide interest because they can use the information contained in similarities among neighboring pixels. In this article, we propose a novel neighborhood polarimetric covariance matrix (NPCM) to detect the small ships in PolSAR images, leading to a significant improvement in the separability between ship targets and sea clutter. The NPCM utilizes the spatial correlation between neighborhood pixels and maps the representation for a given pixel into a high-dimensional covariance matrix by embedding spatial and polarization information. Using the NPCM formalism, we apply a standard whitening filter, similar to the polarimetric whitening filter (PWF). We show how the inclusion of neighborhood information improves the performance compared with the traditional polarimetric covariance matrix. However, this is at the expense of a higher computation cost. The theory is validated via the simulated and measured data under different sea states and using different radar platforms.

Large-scale minimum variance portfolio allocation using double regularization

Estimation of time-varying covariances is a crucial input in minimum variance (MV) portfolio allocations. Rolling window-based sample estimates are widely used for this purpose, but they usually suffer from two major issues when applied to a moderately large number of assets: the “curse of dimensionality” and “temporal instability.” Here, we propose a double-regularized estimator for a high dimensional covariance matrix in which we impose both a temporal and cross-sectional sparsity regularization on the sample-based estimates to simultaneously mitigate these two issues. We investigate the performance of our proposed covariance estimator for MV portfolio construction using Monte Carlo experiments and empirical examples. We find that the resulting MV portfolio strikes a good balance between risk and turnover reduction, and produces more accurate equivalent returns after transaction costs are taken into account when compared to four other MV strategies.

An empirical evaluation of large dynamic covariance models in portfolio value-at-risk estimation

The estimation of portfolio value-at-risk (VaR) requires a good estimate of the covariance matrix. As it is well known that a sample covariance matrix based on some historical rolling window is noisy and is a poor estimate for the high-dimensional population covariance matrix, to estimate the conditional portfolio VaR we develop a framework using the dynamic conditional covariance model, within which various de-noising tools are employed for the estimation of the unconditional covariance target. Various de-noising treatments in our study include shrinkage methods, random matrix theory methods and regularization methods. We validate the model empirically by using various coverage tests and loss function measures and discover that the choice of de-noising treatments for the covariance target plays a critical role in measuring the accuracy of the dynamic portfolio VaR estimates. In our large-scale empirical evaluation of de-noising tools, the regularization methods seem to produce the poorest VaR estimates under various coverage tests and loss function measures, while the shrinkage methods and the random matrix theory methods produce comparable results.

Nowcasting Monthly GDP with Big Data: A Model Averaging Approach

Gross domestic product (GDP) is the most comprehensive and authoritative measure of economic activity. The macroeconomic literature has focused on nowcasting and forecasting this measure at the monthly frequency, using related high frequency indicators. We address the issue of estimating monthly gross domestic product using a large dimensional set of monthly indicators, by pooling the disaggregate estimates arising from simple and feasible bivariate models that consider one indicator at a time in conjunction to GDP. Our base model handles mixed frequency data and ragged-edge data structure with any pattern of missingness. Our methodology enables to distill the common component of the available economic indicators, so that the monthly GDP estimates arise from the projection of the quarterly figures on the space spanned by the common component. The weights used for the combination reflect the ability to nowcast quarterly GDP and are obtained as a function of the regularized estimator of the high-dimensional covariance matrix of the nowcasting errors. A recursive nowcasting and forecasting experiment illustrates that the optimal weights adapt to the information set available in real time and vary according to the phase of the business cycle.

Phase-I monitoring of high-dimensional covariance matrix using an adaptive thresholding LASSO rule

High-dimensional variability monitoring and diagnosing is of great prominence for the quality improvement and cost reduction. Most of the existing control charts are mainly based on the assumption that the in-control covariance matrix is known in prior. This paper proposes a new control chart for monitoring of variability of high-dimensional process under the sparsity conditions. The proposed control chart uses the adaptive thresholding LASSO rule for estimating the unknown covariance matrix. To evaluate the performance of the proposed chart, named as T-COV, the signal probability was estimated under several patterns of out-of-control conditions and compared with the conditional entropy (CE) control chart. This paper uses the process of spur gear production as a real-world example to illustrate the operating procedures of the T-COV chart. The results of the simulation and real studies have revealed the advantage of the T-COV chart over the CE in quickly capturing and precisely diagnosing the deviation in the covariance matrix of the high-dimensional processes.

Graph Laplacian for image anomaly detection

Reed–Xiaoli detector (RXD) is recognized as the benchmark algorithm for image anomaly detection; however, it presents known limitations, namely the dependence over the image following a multivariate Gaussian model, the estimation and inversion of a high-dimensional covariance matrix, and the inability to effectively include spatial awareness in its evaluation. In this work, a novel graph-based solution to the image anomaly detection problem is proposed; leveraging the graph Fourier transform, we are able to overcome some of RXD’s limitations while reducing computational cost at the same time. Tests over both hyperspectral and medical images, using both synthetic and real anomalies, prove the proposed technique is able to obtain significant gains over performance by other algorithms in the state of the art.

Right Singular Vector Projection Graphs: Fast High Dimensional Covariance Matrix Estimation under Latent Confounding

SummaryWe consider the problem of estimating a high dimensional p × p covariance matrix Σ, given n observations of confounded data with covariance Σ+ΓΓT, where Γ is an unknown p × q matrix of latent factor loadings. We propose a simple and scalable estimator based on the projection onto the right singular vectors of the observed data matrix, which we call right singular vector projection (RSVP). Our theoretical analysis of this method reveals that, in contrast with approaches based on the removal of principal components, RSVP can cope well with settings where the smallest eigenvalue of ΓTΓ is relatively close to the largest eigenvalue of Σ, as well as when the eigenvalues of ΓTΓ are diverging fast. RSVP does not require knowledge or estimation of the number of latent factors q, but it recovers Σ only up to an unknown positive scale factor. We argue that this suffices in many applications, e.g. if an estimate of the correlation matrix is desired. We also show that, by using subsampling, we can further improve the performance of the method. We demonstrate the favourable performance of RSVP through simulation experiments and an analysis of gene expression data sets collated by the GTEX consortium.

Improved Large Dynamic Covariance Matrix Estimation With Graphical Lasso and Its Application in Portfolio Selection

The estimation of the large and high-dimensional covariance matrix and precision matrix is a fundamental problem in modern multivariate analysis. It has been widely applied in economics, finance, biology, social networks and health sciences. However, the traditional sample estimators perform poorly for large and high-dimensional data. There are many approaches to improve the covariance matrix estimation. The large dynamic conditional correlation model based on the nonlinear shrinkage and its application in portfolio selection attract increasing attention. In the estimation of the unconditional covariance matrix, the graphical lasso is more robust than the nonlinear shrinkage model, and the leptokurtic and fat tail characteristics of the asset returns are also more obvious. This article proposes improved large dynamic covariance matrix estimation based on the graphical lasso models under the multivariate normal distribution (glasso) and t distribution (tlasso), and the corresponding dynamic conditional correlation glasso and tlasso approaches are developed. To verify the effectiveness and robustness of the proposed methods, we conduct simulations and then apply the models to the classic Markowitz portfolio selection problem. Simulations and empirical results show that the combined dynamic conditional correlation glasso and tlasso approaches outperform the current dynamic covariance matrix estimators.

HYPOTHESIS TESTING ON LINEAR STRUCTURES OF HIGH DIMENSIONAL COVARIANCE MATRIX.

This paper is concerned with test of significance on high dimensional covariance structures, and aims to develop a unified framework for testing commonly-used linear covariance structures. We first construct a consistent estimator for parameters involved in the linear covariance structure, and then develop two tests for the linear covariance structures based on entropy loss and quadratic loss used for covariance matrix estimation. To study the asymptotic properties of the proposed tests, we study related high dimensional random matrix theory, and establish several highly useful asymptotic results. With the aid of these asymptotic results, we derive the limiting distributions of these two tests under the null and alternative hypotheses. We further show that the quadratic loss based test is asymptotically unbiased. We conduct Monte Carlo simulation study to examine the finite sample performance of the two tests. Our simulation results show that the limiting null distributions approximate their null distributions quite well, and the corresponding asymptotic critical values keep Type I error rate very well. Our numerical comparison implies that the proposed tests outperform existing ones in terms of controlling Type I error rate and power. Our simulation indicates that the test based on quadratic loss seems to have better power than the test based on entropy loss.

Functional test for high-dimensional covariance matrix, with application to mitochondrial calcium concentration

In this paper, we present a novel method to test equality of covariance matrices of two high-dimensional samples. The methodology applies the idea of functional data analysis into high-dimensional data study. Asymptotic results of the proposed method are established. Some simulation studies are conducted to investigate the finite sample performance of the proposed method. We illustrate our testing procedures on a mitochondrial calcium concentration data for testing equality of covariance matrices.

Tests for high-dimensional covariance matrices

In this paper, we propose some new tests for high-dimensional covariance matrices that are applicable to generally distributed populations with finite fourth moments. The proposed test statistics are the maximum of the likelihood ratio test statistic and the statistic based on the Frobenius norm. The advantage of the new tests is the good performance in terms of power for both the traditional case, in which the dimension is much smaller than the sample size, and the high-dimensional case, in which the dimension is large compared to the sample size. In the one-sample case, the new test is proposed for testing the hypothesis that the high-dimensional covariance matrix equals an identity matrix. In the two-sample case, the new test is developed for testing the equality of two high-dimensional covariance matrices. By using the random matrix theory, the asymptotic distributions of the proposed new tests are derived under the assumption that the dimension and the sample size proportionally tend toward infinity. Finally, numerical studies are conducted to investigate the finite sample performance of the proposed new tests.

A nonparametric test for block-diagonal covariance structure in high dimension and small samples

In this paper we consider the problem of hypothesis testing for block-diagonal structure of high-dimensional covariance matrix. We develop a bias correction to the existing scalar transform invariant test statistic that is constructed based on an empirical distance between the full and a block diagonal covariance matrix, without requiring any specific parametric distribution such as the normality assumption. Under the high-dimensional null hypothesis and the scenario of the alternatives, which allows power evaluations, we derive the asymptotic distribution of the proposed test statistic without specifying an explicit relationship between the dimension and the sample size. Monte Carlo simulation studies demonstrate that it has good size and power in a wide range of settings. A real data example is also considered to illustrate the efficacy of the approach.

Optimal Shrinkage Covariance Matrix Estimation Under Random Sampling From Elliptical Distributions

This paper considers the problem of estimating a high-dimensional (HD) covariance matrix when the sample size is smaller, or not much larger, than the dimensionality of the data, which could potentially be very large. We develop a regularized sample covariance matrix (RSCM) estimator which can be applied in commonly occurring sparse data problems. The proposed RSCM estimator is based on estimators of the unknown optimal (oracle) shrinkage parameters that yield the minimum mean squared error (MMSE) between the RSCM and the true covariance matrix when the data is sampled from an unspecified elliptically symmetric distribution. We propose two variants of the RSCM estimator which differ in the approach in which they estimate the underlying sphericity parameter involved in the theoretical optimal shrinkage parameter. The performance of the proposed RSCM estimators are evaluated with numerical simulation studies. In particular when the sample sizes are low, the proposed RSCM estimators often show a significant improvement over the conventional RSCM estimator by Ledoit and Wolf (2004). We further evaluate the performance of the proposed estimators in classification and portfolio optimization problems with real data wherein the proposed methods are able to outperform the benchmark methods.

Pre-averaging estimate of high dimensional integrated covariance matrix with noisy and asynchronous high-frequency data

With rapid development of the global market, the number of financial securities has significantly grown, which greatly challenges the measuring of financial quantities. Among others, the estimation of covariance matrix which plays an important role in risk management becomes no longer accurate. In this paper, we consider the estimation of integrated covariance matrix of semi-martingales under framework of high dimension by using high frequency data. We assume that the multivariate asset prices are observed asynchronously and all the observed prices are contaminated by microstructure noise. We employ the pre-averaging method to remove the microstructure noise and the generalized synchronization method to deal with the non-synchronicity. Moreover, to avoid the inconsistency in the high-dimensional covariance matrix estimation, we propose a regularized estimate. The consistency under matrix [Formula: see text]-norm is established. Compared to existing results, our estimator improves the accuracy of the estimation. Finally, we assess the theoretical results via some simulation studies.

Testing identity of high-dimensional covariance matrix

ABSTRACTTwo new statistics are proposed for testing the identity of high-dimensional covariance matrix. Applying the large dimensional random matrix theory, we study the asymptotic distributions of our proposed statistics under the situation that the dimension p and the sample size n tend to infinity proportionally. The proposed tests can accommodate the situation that the data dimension is much larger than the sample size, and the situation that the population distribution is non-Gaussian. The numerical studies demonstrate that the proposed tests have good performance on the empirical powers for a wide range of dimensions and sample sizes.