High-dimensional Covariance Matrix Estimation Research Articles

High-Dimensional Covariance Matrix Estimation: Shrinkage Toward a Diagonal Target

High-Dimensional Covariance Matrix Estimation: Shrinkage Toward a Diagonal Target

A Best Linear Empirical Bayes Method for High-Dimensional Covariance Matrix Estimation

Covariance matrix estimation plays a significant role in both in the theory and practice of portfolio analysis and risk management. This paper deals with the available data prior to developing a factor model to enhance covariance matrix estimation. Our work has two main outcomes. First, for a general linear model with unknown prior parameters, a class of best linear empirical Bayes estimators is established through two kinds of architectures to improve the estimation accuracy by utilizing additional data prior. The theoretical results indicate two key points: the proposed estimators are equivalent to the linear minimum mean-square error estimator when complete or sufficient partial data prior are provided; and the proposed estimators perform better than the optimal weighted least squares method, which ignores the data prior in each situation. Second, the proposed estimators are used for calculating a high-dimensional covariance matrix through factor models. The numerical example and the simulation results verify the effectiveness of our methods.

High-Dimensional Covariance Estimation via Constrained Lq-Type Regularization

High-dimensional covariance matrix estimation is one of the fundamental and important problems in multivariate analysis and has a wide range of applications in many fields. In practice, it is common that a covariance matrix is composed of a low-rank matrix and a sparse matrix. In this paper we estimate the covariance matrix by solving a constrained Lq-type regularized optimization problem. We establish the first-order optimality conditions for this problem by using proximal mapping and the subspace method. The proposed stationary point degenerates to the first-order stationary points of the unconstrained Lq regularized sparse or low-rank optimization problems. A smoothing alternating updating method is proposed to find an estimator for the covariance matrix. We establish the convergence of the proposed calculation method. The numerical simulation results show the effectiveness of the proposed approach for high-dimensional covariance estimation.

Sparsistent filtering of comovement networks from high-dimensional data

Network filtering is a technique to isolate core subnetworks of large and complex interconnected systems, which has recently found many applications in financial, biological, physical and technological networks among others. We introduce a new technique to filter large dimensional networks arising out of dynamical behavior of the constituent nodes, exploiting their spectral properties. As opposed to the well known network filters that rely on preserving key topological properties of the realized network, our method treats the spectrum as the fundamental object and preserves spectral properties. Applying asymptotic theory of high-dimensional covariance matrix estimation, we show that the proposed filter can be tuned to interpolate between zero filtering to maximal filtering that induces sparsity via thresholding, while having the least spectral distance from a consistent (non-)linear shrinkage estimator. We demonstrate the application of our proposed filter by applying it to covariance networks constructed from financial data, to extract core subnetworks embedded in full networks.

Empirical Analysis of the Impact of High and New Technology on the Export of Traditional Cultural Industry Based on Random Matrix Theory

Accurate estimation of covariance matrix is the basis of effective development of high-dimensional optimal portfolio. Random matrix theory provides an effective means to improve the estimation of high-dimensional covariance matrix. Based on the empirical research on the export excess return rate of traditional cultural industry, the result is a combination with higher precision and lower risk. Based on relevant data, this paper constructs a stochastic matrix theoretical model to analyze the impact of human capital, new fixed asset investment, independent innovation and technology purchase, financing sources, and other factors on high-tech industry export. Based on the results of empirical analysis, some policy suggestions are put forward, such as increasing r&d investment, improving enterprises’ innovation ability, and introducing core scientific and technological talents to promote the export scale growth of high-tech industries. Model comparative analysis is made on the influence of traditional industries’ concentrated export, foreign trade environment, and cost advantage on the export of high-tech products. Through comparative analysis of the influencing factors of the export of high-tech products in the two periods before and after the export crisis of traditional culture, the new changes of the influencing factors of the export of high-tech products are analyzed. In addition, through the change of the internal and external environment of the current high-tech product export, the reasons for the new change of the influencing factors are discussed.

High-dimensional correlation matrix estimation for general continuous data with Bagging technique

High-dimensional covariance matrix estimation plays a central role in multivariate statistical analysis. It is well-known that the sample covariance matrix is singular when the sample size is smaller than the dimension of the variable, but the covariance estimate must be positive-definite. This motivates some modifications of the sample covariance matrix to preserve its efficient estimation of pairwise covariance. In this paper, we modify the sample correlation matrix using the Bagging technique. The proposed Bagging estimator is flexible for general continuous data. Under some mild conditions, we show theoretically that the Bagging estimator can ensure positive-definiteness with probability one in finite samples. We also prove the consistency of the bootstrap estimator of Pearson correlation and the consistency of our Bagging estimator when the dimension p is fixed. Simulation results and a real application are provided to demonstrate that our method strikes a better balance between RMSE and likelihood, and is more robust, than other existing estimators.

A New Principle for Tuning-Free Huber Regression

The robustification parameter, which balances bias and robustness, has played a critical role in the construction of sub-Gaussian estimators for heavy-tailed and/or skewed data. Although it can be tuned by cross-validation in traditional practice, in large scale statistical problems such as high dimensional covariance matrix estimation and large scale multiple testing, the number of robustification parameters scales with the dimensionality so that cross-validation can be computationally prohibitive. In this paper, we propose a new data-driven principle to choose the robustification parameter for Huber-type sub-Gaussian estimators in three fundamental problems: mean estimation, linear regression, and sparse regression in high dimensions. Our proposal is guided by the non-asymptotic deviation analysis, and is conceptually different from cross-validation which relies on the mean squared error to assess the fit. Extensive numerical experiments and real data analysis further illustrate the efficacy of the proposed methods

Right Singular Vector Projection Graphs: Fast High Dimensional Covariance Matrix Estimation under Latent Confounding

SummaryWe consider the problem of estimating a high dimensional p × p covariance matrix Σ, given n observations of confounded data with covariance Σ+ΓΓT, where Γ is an unknown p × q matrix of latent factor loadings. We propose a simple and scalable estimator based on the projection onto the right singular vectors of the observed data matrix, which we call right singular vector projection (RSVP). Our theoretical analysis of this method reveals that, in contrast with approaches based on the removal of principal components, RSVP can cope well with settings where the smallest eigenvalue of ΓTΓ is relatively close to the largest eigenvalue of Σ, as well as when the eigenvalues of ΓTΓ are diverging fast. RSVP does not require knowledge or estimation of the number of latent factors q, but it recovers Σ only up to an unknown positive scale factor. We argue that this suffices in many applications, e.g. if an estimate of the correlation matrix is desired. We also show that, by using subsampling, we can further improve the performance of the method. We demonstrate the favourable performance of RSVP through simulation experiments and an analysis of gene expression data sets collated by the GTEX consortium.

Pre-averaging estimate of high dimensional integrated covariance matrix with noisy and asynchronous high-frequency data

With rapid development of the global market, the number of financial securities has significantly grown, which greatly challenges the measuring of financial quantities. Among others, the estimation of covariance matrix which plays an important role in risk management becomes no longer accurate. In this paper, we consider the estimation of integrated covariance matrix of semi-martingales under framework of high dimension by using high frequency data. We assume that the multivariate asset prices are observed asynchronously and all the observed prices are contaminated by microstructure noise. We employ the pre-averaging method to remove the microstructure noise and the generalized synchronization method to deal with the non-synchronicity. Moreover, to avoid the inconsistency in the high-dimensional covariance matrix estimation, we propose a regularized estimate. The consistency under matrix [Formula: see text]-norm is established. Compared to existing results, our estimator improves the accuracy of the estimation. Finally, we assess the theoretical results via some simulation studies.

Tail sums of Wishart and Gaussian eigenvalues beyond the bulk edge.

Consider the classical Gaussian unitary ensemble of size N and the real white Wishart ensemble with N variables and n degrees of freedom. In the limits of large N and n, with positive ratio γ in the Wishart case, the expected number of eigenvalues that exit the upper bulk edge is less than one, approaching 0.031 and 0.170 respectively, the latter number being independent of γ. These statements are consequences of quantitative bounds on tail sums of eigenvalues outside the bulk which are established here for applications in high dimensional covariance matrix estimation.

Variable selection in multivariate linear models with high-dimensional covariance matrix estimation

In this paper, we propose a novel variable selection approach in the framework of multivariate linear models taking into account the dependence that may exist between the responses. It consists in estimating beforehand the covariance matrix Σ of the responses and to plug this estimator in a Lasso criterion, in order to obtain a sparse estimator of the coefficient matrix. The properties of our approach are investigated both from a theoretical and a numerical point of view. More precisely, we give general conditions that the estimators of the covariance matrix and its inverse have to satisfy in order to recover the positions of the null and non null entries of the coefficient matrix when the size of Σ is not fixed and can tend to infinity. We prove that these conditions are satisfied in the particular case of some Toeplitz matrices. Our approach is implemented in the R package MultiVarSel available from the Comprehensive R Archive Network (CRAN) and is very attractive since it benefits from a low computational load. We also assess the performance of our methodology using synthetic data and compare it with alternative approaches. Our numerical experiments show that including the estimation of the covariance matrix in the Lasso criterion dramatically improves the variable selection performance in many cases.

Structural Variability from Noisy Tomographic Projections.

In cryo-electron microscopy, the three-dimensional (3D) electric potentials of an ensemble of molecules are projected along arbitrary viewing directions to yield noisy two-dimensional images. The volume maps representing these potentials typically exhibit a great deal of structural variability, which is described by their 3D covariance matrix. Typically, this covariance matrix is approximately low rank and can be used to cluster the volumes or estimate the intrinsic geometry of the conformation space. We formulate the estimation of this covariance matrix as a linear inverse problem, yielding a consistent least-squares estimator. For n images of size N-by-N pixels, we propose an algorithm for calculating this covariance estimator with computational complexity , where the condition number κ is empirically in the range 10-200. Its efficiency relies on the observation that the normal equations are equivalent to a deconvolution problem in six dimensions. This is then solved by the conjugate gradient method with an appropriate circulant preconditioner. The result is the first computationally efficient algorithm for consistent estimation of the 3D covariance from noisy projections. It also compares favorably in runtime with respect to previously proposed nonconsistent estimators. Motivated by the recent success of eigenvalue shrinkage procedures for high-dimensional covariance matrix estimation, we incorporate a shrinkage procedure that improves accuracy at lower signal-to-noise ratios. We evaluate our methods on simulated datasets and achieve classification results comparable to state-of-the-art methods in shorter running time. We also present results on clustering volumes in an experimental dataset, illustrating the power of the proposed algorithm for practical determination of structural variability.

High dimensional covariance matrix estimation by penalizing the matrix-logarithm transformed likelihood

It is well known that when the dimension of the data becomes very large, the sample covariance matrix S will not be a good estimator of the population covariance matrix Σ. Using such estimator, one typical consequence is that the estimated eigenvalues from S will be distorted. Many existing methods tried to solve the problem, and examples of which include regularizing Σ by thresholding or banding. In this paper, we estimate Σ by maximizing the likelihood using a new penalization on the matrix logarithm of Σ (denoted by A) of the form: ‖A−mI‖F2=∑i(log(di)−m)2, where di is the ith eigenvalue of Σ. This penalty aims at shrinking the estimated eigenvalues of A toward the mean eigenvalue m. The merits of our method are that it guarantees Σ to be non-negative definite and is computational efficient. The simulation study and applications on portfolio optimization and classification of genomic data show that the proposed method outperforms existing methods.

High dimensional covariance matrix estimation using multi-factor models from incomplete information

Covariance matrix plays an important role in risk management, asset pricing, and portfolio allocation. Covariance matrix estimation becomes challenging when the dimensionality is comparable or much larger than the sample size. A widely used approach for reducing dimensionality is based on multi-factor models. Although it has been well studied and quite successful in many applications, the quality of the estimated covariance matrix is often degraded due to a nontrivial amount of missing data in the factor matrix for both technical and cost reasons. Since the factor matrix is only approximately low rank or even has full rank, existing matrix completion algorithms are not applicable. We consider a new matrix completion paradigm using the factor models directly and apply the alternating direction method of multipliers for the recovery. Numerical experiments show that the nuclear-norm matrix completion approaches are not suitable but our proposed models and algorithms are promising.

High-dimensional covariance matrix estimation with missing observations

In this paper, we study the problem of high-dimensional covariance matrix estimation with missing observations. We propose a simple procedure computationally tractable in high-dimension and that does not require imputation of the missing data. We establish non-asymptotic sparsity oracle inequalities for the estimation of the covariance matrix involving the Frobenius and the spectral norms which are valid for any setting of the sample size, probability of a missing observation and the dimensionality of the covariance matrix. We further establish minimax lower bounds showing that our rates are minimax optimal up to a logarithmic factor.

High-dimensional covariance matrix estimation in approximate factor models

The variance covariance matrix plays a central role in the inferential theories of high dimensional factor models in finance and economics. Popular regularization methods of directly exploiting sparsity are not directly applicable to many financial problems. Classical methods of estimating the covariance matrices are based on the strict factor models, assuming independent idiosyncratic components. This assumption, however, is restrictive in practical applications. By assuming sparse error covariance matrix, we allow the presence of the cross-sectional correlation even after taking out common factors, and it enables us to combine the merits of both methods. We estimate the sparse covariance using the adaptive thresholding technique as in Cai and Liu (2011), taking into account the fact that direct observations of the idiosyncratic components are unavailable. The impact of high dimensionality on the covariance matrix estimation based on the factor structure is then studied.

Group Lasso Estimation of High-dimensional Covariance Matrices

In this paper, we consider the Group Lasso estimator of the covariance matrix of a stochastic process corrupted by an additive noise. We propose to estimate the covariance matrix in a high-dimensional setting under the assumption that the process has a sparse representation in a large dictionary of basis functions. Using a matrix regression model, we propose a new methodology for high-dimensional covariance matrix estimation based on empirical contrast regularization by a group Lasso penalty. Using such a penalty, the method selects a sparse set of basis functions in the dictionary used to approximate the process, leading to an approximation of the covariance matrix into a low dimensional space. Consistency of the estimator is studied in Frobenius and operator norms and an application to sparse PCA is proposed.