High-dimensional Covariance Matrix Research Articles

PolSAR Ship Detection Based on Azimuth Sublook Polarimetric Covariance Matrix

Polarimetric synthetic aperture radar (PolSAR) is an important tool for marine remote sensing. In the past two decades, subaperture decomposition is considered as an alternative method to extract ship targets, especially in complex conditions where the detection based on intensity performs not well. The introduction of polarization information can enhance the detection ability of radar. The combination of sublook techniques and polarization will lead to more precise target extraction. The first methods jointly using information of polarization and sublook is the polarimetric internal Hermitian product (Pol-IHP), which nowadays has been corrected as polarimetric Hermitian internal product (Pol-HIP). However, it is limited by the number of sublooks and recently developed as generalized multi-sublooks correlation (GMC). In this study, a novel sublook polarimetric covariance matrix (SPCM) is established, based on which a sublook-polarimetric whitening filter (Sub-PWF) is further proposed for ship detection. The SPCM represents a given pixel using a high-dimensional covariance matrix which utilizes the spectral correlation in imaging progress and the scattering characteristics of different polarization channels. The superiority of Sub-PWF based on SPCM is validated via the measured single look complex (SLC) PolSAR data from RadarSAT-2 (RS-2) and GaoFen-3 (GF-3). Results show this new method can improve the target to clutter ratio (TCR) and reduce the clutter fluctuation. The ROC curves in both small and large scenes proves that Sub-PWF can reach ideal detection performance.

Covariance function versus covariance matrix estimation in efficient semi-parametric regression for longitudinal data analysis

Improving estimation efficiency for regression coefficients is an important issue in the analysis of longitudinal data, which involves estimating the covariance matrix of the within-subject errors. In the balanced or nearly balanced setting, we can also regard the covariance matrix of the dependent errors as the bivariate covariance function evaluated at specific time points. In this paper, we compare the performance of the proposed regularized-covariance-function-based estimator and the conventional high-dimensional covariance matrix estimator of the within-subject errors. It shows that when the number p of the time points in each subject is large enough compared to the number n of the subjects, i.e., p≫n1/4logn, the estimation errors of the high-dimensional covariance matrix will be accumulated, therefore the error bound of the proposed regularized-covariance-function-based estimator will be smaller than that of the high-dimensional covariance matrix estimator in Frobenius norm. We also assess the performance of these two estimators for the incomplete longitudinal data. All the comparisons and theoretical results are illustrated using both simulated and real data.

A Permutation Test for High-Dimensional Covariance Matrix

In the case of “big p and small n”, classical statistical methods and theories are difficult to apply to high-dimensional data problems. This article considers testing the equality of two sample covariance matrices. When both the dimension p and the sample size n tend to infinity, a permutation method is proposed. The new test method eliminates the limitation of sample distribution and dimension. Numerical research shows that the proposed test method has good results in both normal and non-Gaussian distributions under high-dimensional data.

Sphericity and Identity Test for High-dimensional Covariance Matrix Using Random Matrix Theory

This paper addresses the issue of testing sphericity and identity of high-dimensional population covariance matrix when the data dimension exceeds the sample size. The central limit theorem of the first four moments of eigenvalues of sample covariance matrix is derived using random matrix theory for generally distributed populations. Further, some desirable asymptotic properties of the proposed test statistics are provided under the null hypothesis as data dimension and sample size both tend to infinity. Simulations show that the proposed tests have a greater power than existing methods for the spiked covariance model.

Ensemble sparse estimation of covariance structure for exploring genetic disease data

High-dimensional data often occur nowadays in various areas, such as genetic and microarray data. The covariance matrix is of fundamental importance in analyzing the relationship between multivariate variables. A powerful tool for estimating a covariance matrix is the modified Cholesky decomposition, which allows for unconstrained estimation and guarantees the positive definiteness of the estimate. However, it requires a pre-specified ordering of variables before analysis, which is often not available in the real data. Hence, an ensemble Cholesky-based sparse estimation is proposed for a high-dimensional covariance matrix by adopting the model averaging idea. The asymptotically theoretical convergence rate is established under some regularity conditions. The merits of the proposed model are illustrated by the numerical study and two genetic disease data.

Global one-sample tests for high-dimensional covariance matrices

Testing high-dimensional covariance matrix plays an important role in multivariate statistical analysis. Many statisticians used the statistics based on to test with being the covariance matrix. However, none have proposed a statistic based on for this purpose. In fact, neither of the two tests is superior to the other based on their powers because they target different kinds of dense alternatives. Furthermore, some maximum-type tests were proposed to accommodate sparse alternatives. By using the advantages of these tests, we propose two new tests for one-sample covariance matrix when the sample size and data dimension increase proportionally. One is suitable for dense alternatives, the other is powerful against a wide range of situations, such as dense, sparse or a mixture of both alternatives. Extensive simulation results show that our proposed tests maintain high powers against various alternatives while the existing tests fail in at least one situation.

Estimation and optimal structure selection of high-dimensional Toeplitz covariance matrix

The estimation of structured covariance matrix arises in many fields. An appropriate covariance structure not only improves the accuracy of covariance estimation but also increases the efficiency of mean parameter estimators in statistical models. In this paper, a novel statistical method is proposed, which selects the optimal Toeplitz covariance structure and estimates the covariance matrix, simultaneously. An entropy loss function with nonconvex penalty is employed as a matrix-discrepancy measure, under which the optimal selection of sparse or nearly sparse Toeplitz structure and the parameter estimators of covariance matrix are made, simultaneously, through its minimization. The cases of both low-dimensional (p≤n) and high-dimensional (p>n) covariance matrix estimation are considered. The resulting Toeplitz structured covariance estimators are guaranteed to be positive definite and consistent. Asymptotic properties are investigated and simulation studies are conducted, showing that very high accurate Toeplitz covariance structure estimation is made. The proposed method is then applied to practical data analysis, which demonstrates its good performance in covariance estimation in practice.

Nowcasting Monthly GDP with Big Data: A Model Averaging Approach

AbstractGross domestic product (GDP) is the most comprehensive and authoritative measure of economic activity. The macroeconomic literature has focused on nowcasting and forecasting this measure at the monthly frequency, using related high-frequency indicators. We address the issue of estimating monthly GDP using a large-dimensional set of monthly indicators, by pooling the disaggregate estimates arising from simple and feasible bivariate models that consider one indicator at a time in conjunction to GDP. Our base model handles mixed-frequency data and ragged-edge data structure with any pattern of missingness. Our methodology enables to distil the common component of the available economic indicators, so that the monthly GDP estimates arise from the projection of the quarterly figures on the space spanned by the common component. The weights used for the combination reflect the ability to nowcast quarterly GDP and are obtained as a function of the regularized estimator of the high-dimensional covariance matrix of the nowcasting errors. A recursive nowcasting and forecasting experiment with data on euro area GDP illustrates that the optimal weights adapt to the information set available in real time and vary according to the phase of the business cycle.

A New Principle for Tuning-Free Huber Regression

The robustification parameter, which balances bias and robustness, has played a critical role in the construction of sub-Gaussian estimators for heavy-tailed and/or skewed data. Although it can be tuned by cross-validation in traditional practice, in large scale statistical problems such as high dimensional covariance matrix estimation and large scale multiple testing, the number of robustification parameters scales with the dimensionality so that cross-validation can be computationally prohibitive. In this paper, we propose a new data-driven principle to choose the robustification parameter for Huber-type sub-Gaussian estimators in three fundamental problems: mean estimation, linear regression, and sparse regression in high dimensions. Our proposal is guided by the non-asymptotic deviation analysis, and is conceptually different from cross-validation which relies on the mean squared error to assess the fit. Extensive numerical experiments and real data analysis further illustrate the efficacy of the proposed methods

Prediction in Locally Stationary Time Series

We develop an estimator for the high-dimensional covariance matrix of a locally stationary process with a smoothly varying trend and use this statistic to derive consistent predictors in nonstationary time series. In contrast to the currently available methods for this problem the predictor developed here does not rely on fitting an autoregressive model and does not require a vanishing trend. The finite sample properties of the new methodology are illustrated by means of a simulation study and a financial indices study. Supplementary materials for this article are available online.

Model averaging estimation for high-dimensional covariance matrices with a network structure.

In this paper, we develop a model averaging method to estimate a high-dimensional covariance matrix, where the candidate models are constructed by different orders of polynomial functions. We propose a Mallows-type model averaging criterion and select the weights by minimizing this criterion, which is an unbiased estimator of the expected in-sample squared error plus a constant. Then, we prove the asymptotic optimality of the resulting model average covariance estimators. Finally, we conduct numerical simulations and a case study on Chinese airport network structure data to demonstrate the usefulness of the proposed approaches.

PolSAR Ship Detection Based on Neighborhood Polarimetric Covariance Matrix

The detection of small ships in polarimetric synthetic aperture radar (PolSAR) images is still a topic for further investigation. Recently, patch detection techniques, such as superpixel-level detection, have stimulated wide interest because they can use the information contained in similarities among neighboring pixels. In this article, we propose a novel neighborhood polarimetric covariance matrix (NPCM) to detect the small ships in PolSAR images, leading to a significant improvement in the separability between ship targets and sea clutter. The NPCM utilizes the spatial correlation between neighborhood pixels and maps the representation for a given pixel into a high-dimensional covariance matrix by embedding spatial and polarization information. Using the NPCM formalism, we apply a standard whitening filter, similar to the polarimetric whitening filter (PWF). We show how the inclusion of neighborhood information improves the performance compared with the traditional polarimetric covariance matrix. However, this is at the expense of a higher computation cost. The theory is validated via the simulated and measured data under different sea states and using different radar platforms.

Large-scale minimum variance portfolio allocation using double regularization

Estimation of time-varying covariances is a crucial input in minimum variance (MV) portfolio allocations. Rolling window-based sample estimates are widely used for this purpose, but they usually suffer from two major issues when applied to a moderately large number of assets: the “curse of dimensionality” and “temporal instability.” Here, we propose a double-regularized estimator for a high dimensional covariance matrix in which we impose both a temporal and cross-sectional sparsity regularization on the sample-based estimates to simultaneously mitigate these two issues. We investigate the performance of our proposed covariance estimator for MV portfolio construction using Monte Carlo experiments and empirical examples. We find that the resulting MV portfolio strikes a good balance between risk and turnover reduction, and produces more accurate equivalent returns after transaction costs are taken into account when compared to four other MV strategies.

An empirical evaluation of large dynamic covariance models in portfolio value-at-risk estimation

The estimation of portfolio value-at-risk (VaR) requires a good estimate of the covariance matrix. As it is well known that a sample covariance matrix based on some historical rolling window is noisy and is a poor estimate for the high-dimensional population covariance matrix, to estimate the conditional portfolio VaR we develop a framework using the dynamic conditional covariance model, within which various de-noising tools are employed for the estimation of the unconditional covariance target. Various de-noising treatments in our study include shrinkage methods, random matrix theory methods and regularization methods. We validate the model empirically by using various coverage tests and loss function measures and discover that the choice of de-noising treatments for the covariance target plays a critical role in measuring the accuracy of the dynamic portfolio VaR estimates. In our large-scale empirical evaluation of de-noising tools, the regularization methods seem to produce the poorest VaR estimates under various coverage tests and loss function measures, while the shrinkage methods and the random matrix theory methods produce comparable results.

Nowcasting Monthly GDP with Big Data: A Model Averaging Approach

Gross domestic product (GDP) is the most comprehensive and authoritative measure of economic activity. The macroeconomic literature has focused on nowcasting and forecasting this measure at the monthly frequency, using related high frequency indicators. We address the issue of estimating monthly gross domestic product using a large dimensional set of monthly indicators, by pooling the disaggregate estimates arising from simple and feasible bivariate models that consider one indicator at a time in conjunction to GDP. Our base model handles mixed frequency data and ragged-edge data structure with any pattern of missingness. Our methodology enables to distill the common component of the available economic indicators, so that the monthly GDP estimates arise from the projection of the quarterly figures on the space spanned by the common component. The weights used for the combination reflect the ability to nowcast quarterly GDP and are obtained as a function of the regularized estimator of the high-dimensional covariance matrix of the nowcasting errors. A recursive nowcasting and forecasting experiment illustrates that the optimal weights adapt to the information set available in real time and vary according to the phase of the business cycle.

Phase-I monitoring of high-dimensional covariance matrix using an adaptive thresholding LASSO rule

High-dimensional variability monitoring and diagnosing is of great prominence for the quality improvement and cost reduction. Most of the existing control charts are mainly based on the assumption that the in-control covariance matrix is known in prior. This paper proposes a new control chart for monitoring of variability of high-dimensional process under the sparsity conditions. The proposed control chart uses the adaptive thresholding LASSO rule for estimating the unknown covariance matrix. To evaluate the performance of the proposed chart, named as T-COV, the signal probability was estimated under several patterns of out-of-control conditions and compared with the conditional entropy (CE) control chart. This paper uses the process of spur gear production as a real-world example to illustrate the operating procedures of the T-COV chart. The results of the simulation and real studies have revealed the advantage of the T-COV chart over the CE in quickly capturing and precisely diagnosing the deviation in the covariance matrix of the high-dimensional processes.

Graph Laplacian for image anomaly detection

Reed–Xiaoli detector (RXD) is recognized as the benchmark algorithm for image anomaly detection; however, it presents known limitations, namely the dependence over the image following a multivariate Gaussian model, the estimation and inversion of a high-dimensional covariance matrix, and the inability to effectively include spatial awareness in its evaluation. In this work, a novel graph-based solution to the image anomaly detection problem is proposed; leveraging the graph Fourier transform, we are able to overcome some of RXD’s limitations while reducing computational cost at the same time. Tests over both hyperspectral and medical images, using both synthetic and real anomalies, prove the proposed technique is able to obtain significant gains over performance by other algorithms in the state of the art.

Right Singular Vector Projection Graphs: Fast High Dimensional Covariance Matrix Estimation under Latent Confounding

SummaryWe consider the problem of estimating a high dimensional p × p covariance matrix Σ, given n observations of confounded data with covariance Σ+ΓΓT, where Γ is an unknown p × q matrix of latent factor loadings. We propose a simple and scalable estimator based on the projection onto the right singular vectors of the observed data matrix, which we call right singular vector projection (RSVP). Our theoretical analysis of this method reveals that, in contrast with approaches based on the removal of principal components, RSVP can cope well with settings where the smallest eigenvalue of ΓTΓ is relatively close to the largest eigenvalue of Σ, as well as when the eigenvalues of ΓTΓ are diverging fast. RSVP does not require knowledge or estimation of the number of latent factors q, but it recovers Σ only up to an unknown positive scale factor. We argue that this suffices in many applications, e.g. if an estimate of the correlation matrix is desired. We also show that, by using subsampling, we can further improve the performance of the method. We demonstrate the favourable performance of RSVP through simulation experiments and an analysis of gene expression data sets collated by the GTEX consortium.

Improved Large Dynamic Covariance Matrix Estimation With Graphical Lasso and Its Application in Portfolio Selection

The estimation of the large and high-dimensional covariance matrix and precision matrix is a fundamental problem in modern multivariate analysis. It has been widely applied in economics, finance, biology, social networks and health sciences. However, the traditional sample estimators perform poorly for large and high-dimensional data. There are many approaches to improve the covariance matrix estimation. The large dynamic conditional correlation model based on the nonlinear shrinkage and its application in portfolio selection attract increasing attention. In the estimation of the unconditional covariance matrix, the graphical lasso is more robust than the nonlinear shrinkage model, and the leptokurtic and fat tail characteristics of the asset returns are also more obvious. This article proposes improved large dynamic covariance matrix estimation based on the graphical lasso models under the multivariate normal distribution (glasso) and t distribution (tlasso), and the corresponding dynamic conditional correlation glasso and tlasso approaches are developed. To verify the effectiveness and robustness of the proposed methods, we conduct simulations and then apply the models to the classic Markowitz portfolio selection problem. Simulations and empirical results show that the combined dynamic conditional correlation glasso and tlasso approaches outperform the current dynamic covariance matrix estimators.

HYPOTHESIS TESTING ON LINEAR STRUCTURES OF HIGH DIMENSIONAL COVARIANCE MATRIX.

This paper is concerned with test of significance on high dimensional covariance structures, and aims to develop a unified framework for testing commonly-used linear covariance structures. We first construct a consistent estimator for parameters involved in the linear covariance structure, and then develop two tests for the linear covariance structures based on entropy loss and quadratic loss used for covariance matrix estimation. To study the asymptotic properties of the proposed tests, we study related high dimensional random matrix theory, and establish several highly useful asymptotic results. With the aid of these asymptotic results, we derive the limiting distributions of these two tests under the null and alternative hypotheses. We further show that the quadratic loss based test is asymptotically unbiased. We conduct Monte Carlo simulation study to examine the finite sample performance of the two tests. Our simulation results show that the limiting null distributions approximate their null distributions quite well, and the corresponding asymptotic critical values keep Type I error rate very well. Our numerical comparison implies that the proposed tests outperform existing ones in terms of controlling Type I error rate and power. Our simulation indicates that the test based on quadratic loss seems to have better power than the test based on entropy loss.