Sparse Covariance Matrix Research Articles

Estimation of sparse covariance matrix via non-convex regularization

Estimation of high-dimensional sparse covariance matrix is one of the fundamental and important problems in multivariate analysis and has a wide range of applications in many fields. This paper presents a novel method for sparse covariance matrix estimation via solving a non-convex regularization optimization problem. We establish the asymptotic properties of the proposed estimator and develop a multi-stage convex relaxation method to find an effective estimator. The multi-stage convex relaxation method guarantees any accumulation point of the sequence generated is a first-order stationary point of the non-convex optimization. Moreover, the error bounds of the first two stage estimators of the multi-stage convex relaxation method are derived under some regularity conditions. The numerical results show that our estimator outperforms the state-of-the-art estimators and has a high degree of sparsity on the premise of its effectiveness.

Joint regression analysis of multiple traits based on genetic relationships.

Polygenic scores (PGSs) are widely available and employed in genomic data analyses for predicting and understanding genetic architectures. Existing approaches either require information on SNP level, do not infer clusters of traits sharing genetic characteristic, or do not have any immediate predictive properties. Here, we present geneJAM, which is a novel clustering and estimation method using PGSs for inferring a genetic relationship among multiple, simultaneously measured and potentially correlated traits in a multivariate GWAS.Using graphical lasso, we estimate a sparse covariance matrix of the PGSs and obtain clusters of traits sharing genetic characteristics. We use the clusters to specify the structure of the error covariance matrix of a generalized least squares (GLS) model and use the feasible GLS estimator for estimating a linear regression model with a certain unknown degree of correlation between the residuals.The method suits many biology studies well with traits embedded in some genetic functioning groups and facilitates development of the PGS research. We compare the method with fully parametric techniques on simulated data and illustrate the utility of the methods by examining a heterogeneous stock mouse data set from the Wellcome Trust Centre for Human Genetics. We demonstrate that the method successfully identifies clusters of traits and increases precision, power, and computational efficiency. GeneJAM is implemented in R and available at: https://github.com/abuchardt/geneJAM.

Differentially Private Sparse Covariance Matrix Estimation under Lower-Bounded Moment Assumption

This paper investigates the problem of sparse covariance matrix estimation while the sampling set contains sensitive information, and both the differentially private algorithm and locally differentially private algorithm are adopted to preserve privacy. It is worth noting that the requirement of the distribution assumption in our work is only the existing bounded 4+ε(ε>0) moment. Meanwhile, we reduce the error bounds by modifying the threshold of the existing differentially private algorithms. Finally, the numerical simulations and results from a real data application are presented to support our theoretical claims.

Improvement of Classification Performance in High-Dimension Low-Sample-Size Modeling by Sparse Functional Connectivity States in Subjects with Attention Deficit-Hyperactivity Disorder and Healthy Controls

Background: The precise identification of attention deficit-hyperactivity disorder (ADHD) is one of the challenging clinical processes. Disorganizations in functional neural networks revealed via functional magnetic resonance imaging have recently been contributing. Machine learning approaches, particularly classification methods, have commonly been employed as a framework for diverse data analysis, indicating promising medical diagnosis results. However, as the neuroimaging data are high-dimensional with a low sample size (the current dataset), this study aimed to evaluate the classification performance of the models by considering the specific contribution of the sparsity of data matrices. Methods: This cross-sectional study analyzed the preprocessed data from the 2011 ADHD-200 Global Competition. A total of 768 and 171 data items were considered training and test, respectively. The diagnosis status was used as a response variable. Age, gender, hand dominance, and activity relationship between 116 brain regions derived from inverse covariance matrix and inverse sparse covariance matrix were used as predictive variables. Accordingly, this study compared the performance of three models, namely support vector machine (SVM), distance-weighted discrimination (DWD), and data maximum dispersion classifier (DMDC) for ADHD categorization. Results: The highest value for the total accuracy was reported for the SVM model on the sparse covariance matrix. Moreover, the highest values for the balanced classification rate (BCR) (59%) and sensitivity (64%) were reported for DMDC on the sparse covariance matrix. The best level of specificity (99%) was obtained from DWD using the sparse covariance matrix. The highest levels of the values (i.e., total accuracy and BCR) were achieved through the model fitting on the sparse matrices. Among the six models, the DMDC model on sparse covariance matrix was the most optimal algorithm due to the superiority of the two indices (i.e., accuracy: 60% and BCR: 60%) and the favorable balance between sensitivity and specificity values. Conclusions: Among the current studied three models, DMDC performance, applying the sparse data, remarkably improved the results of classification processes. Based on the present findings, the neuronal connectivity among subcortical structures comprising parts of the basal ganglia and cerebellum provides a distinction between ADHD subjects and healthy controls.

Regularized Multi-Output Gaussian Convolution Process With Domain Adaptation.

Multi-output Gaussian process (MGP) has been attracting increasing attention as a transfer learning method to model multiple outputs. Despite its high flexibility and generality, MGP still faces two critical challenges when applied to transfer learning. The first one is negative transfer, which occurs when there exists no shared information among the outputs. The second challenge is the input domain inconsistency, which is commonly studied in transfer learning yet not explored in MGP. In this paper, we propose a regularized MGP modeling framework with domain adaptation to overcome these challenges. More specifically, a sparse covariance matrix of MGP is constructed using convolution process, where penalization terms are added to adaptively select the most informative outputs for knowledge transfer. To deal with the domain inconsistency, a domain adaptation method is proposed by marginalizing inconsistent features and expanding missing features to align the input domains among different outputs. Statistical properties of the proposed method are provided to guarantee the performance practically and asymptotically. The proposed framework outperforms state-of-the-art benchmarks in comprehensive simulation studies and one real case study of a ceramic manufacturing process. The results demonstrate the effectiveness of our method in dealing with both the negative transfer and the domain inconsistency.

Tyler’s and Maronna’s M-estimators: Non-asymptotic concentration results

Tyler’s and Maronna’s M-estimators, as well as their regularized variants, are popular robust methods to estimate the scatter or covariance matrix of a multivariate distribution. In this work, we study the non-asymptotic behavior of these estimators, for data sampled from a distribution that satisfies one of the following properties: (1) independent sub-Gaussian entries, up to a linear transformation; (2) log-concave distributions; (3) distributions satisfying a convex concentration property. Our main contribution is the derivation of tight non-asymptotic concentration bounds of these M-estimators around a suitably scaled version of the data sample covariance matrix. Prior to our work, non-asymptotic bounds were derived only for Elliptical and Gaussian distributions. Our proof uses a variety of tools from non asymptotic random matrix theory and high dimensional geometry. Finally, we illustrate the utility of our results on two examples of practical interest: sparse covariance and sparse precision matrix estimation.

Low complexity sparse beamspace DOA estimation via single measurement vectors for uniform circular array

In this paper, we present a low complexity sparse beamspace direction-of-arrival (DOA) estimation method for uniform circular array (UCA). In the proposed method, we firstly use the beamspace transformation (BT) to transform the signal model of UCA in element-space domain to that of virtual uniform linear array (ULA) in beamspace domain. Subsequently, by applying the vectoring operator on the virtual ULA-like array signal model, a novel dimension-reduction sparse beamspace signal model is derived based on Khatri-Rao (KR) product, the observation data of which is represented by the single measurement vectors (SMVs) via vectorization of sparse covariance matrix. And then, the DOA estimation is formulated as a convex optimization problem by following the concept of a sparse-signal-representation (SSR) of the SMVs. Finally, simulations are carried out to validate the effectiveness of the proposed method. The results show that without knowledge of the number of signals, the proposed method not only has higher DOA resolution than the subspace-based methods in low signal-to-noise ratio (SNR), but also has far lower computational complexity than other sparse-like DOA estimation methods.

Source localization utilizing weighted power iterative compensation via acoustic vector hydrophone array

To address the source localization issue in the condition of low signal-to-noise ratio (SNR) or closely-spaced targets, a sparse signal power estimation method based on sparse covariance matrix fitting criterion (SCMFC) is proposed, which uses the weighted power compensation strategy. The analytical expression of the sparse signal power is derived by utilizing the Frobenius norm property, and then the compensation weight with a user parameter q is designed to further enforce the sparsity of signal power in spatial domain. Furthermore, the signal power compensation method is given. Moreover, a theoretical guidance how to automatically update q is also presented according to the Bayesian information criterion (BIC). Extensive numerical simulation and experimental results verify the superiority of the proposed method compared to some existing methods.

Sparse covariance estimation in logit mixture models

Summary This paper introduces a new data-driven methodology for estimating sparse covariance matrices of the random coefficients in logit mixture models. Researchers typically specify covariance matrices in logit mixture models under one of two extreme assumptions: either an unrestricted full covariance matrix (allowing correlations between all random coefficients), or a restricted diagonal matrix (allowing no correlations at all). Our objective is to find optimal subsets of correlated coefficients for which we estimate covariances. We propose a new estimator, called MISC (mixed integer sparse covariance), that uses a mixed-integer optimization (MIO) program to find an optimal block diagonal structure specification for the covariance matrix, corresponding to subsets of correlated coefficients, for any desired sparsity level using Markov Chain Monte Carlo (MCMC) posterior draws from the unrestricted full covariance matrix. The optimal sparsity level of the covariance matrix is determined using out-of-sample validation. We demonstrate the ability of MISC to correctly recover the true covariance structure from synthetic data. In an empirical illustration using a stated preference survey on modes of transportation, we use MISC to obtain a sparse covariance matrix indicating how preferences for attributes are related to one another.

Differentially private high dimensional sparse covariance matrix estimation

In this paper, we study the problem of estimating the covariance matrix under differential privacy, where the underlying covariance matrix is assumed to be sparse and of high dimensions. We propose a new method, called DP-Thresholding, to achieve a non-trivial ℓ2-norm based error bound whose dependence on the dimension drops to logarithmic instead of polynomial, it is significantly better than the existing ones, which add noise directly to the empirical covariance matrix. We also extend the ℓ2-norm based error bound to a general ℓw-norm based one for any 1≤w≤∞, and show that they share the same upper bound asymptotically. Our approach can be easily extended to local differential privacy. Experiments on the synthetic datasets show results that are consistent with theoretical claims.

A Factor-Based Estimation of Integrated Covariance Matrix With Noisy High-Frequency Data

This article studies a high-dimensional factor model with sparse idiosyncratic covariance matrix in continuous time, using asynchronous high-frequency financial data contaminated by microstructure noise. We focus on consistent estimations of the number of common factors, the integrated covariance matrix and its inverse, based on the flat-top realized kernels introduced by Varneskov. Simulation results illustrate the satisfactory performance of our estimators in finite samples. We apply our methodology to the high-frequency price data on a large number of stocks traded in Shanghai and Shenzhen stock exchanges, and demonstrate its value for capturing time-varying covariations and portfolio allocation.

Long-Term Channel Statistic Estimation for Highly-Mobile Hybrid MmWave Multi-User MIMO Systems

Channel estimation is crucial to beamforming techniques in directional millimetre wave (mmWave) communications, which is generally designed based on channel state information static. However, due to the Doppler effect caused by the mobility of users, such as unmanned aerial vehicles, high-speed trains and autonomous vehicles, the mmWave channel is changing rapidly. Spatial channel covariance, defined by long-term statistic information of channels, is a promising solution to reduce channel estimation frequency and can be used to design hybrid precoders. In this paper, we first proposed a highly mobile hybrid mmWave multi-user (MU) multiple input multiple output (MIMO) system based on transition probabilities which can represent moving action of the MU. Secondly, we investigate compressive sensing based spatial channel covariance estimation based on the proposed dynamic system. We then propose a dynamic covariance forward-backward pursuit (DCFBP) algorithm which introduces forward and backward mechanisms to reconstruct the Hermitian sparse covariance matrix. We further explore the constructed MU sensing matrix quality for conventional sparse Bayesian learning (SBL) framework. The updated sparse Bayesian learning (Updated-SBL) algorithm is developed to reduce the total squared coherence of a constructed sensing matrix with updated receive precoder. Numerical analysis demonstrates the proposed DCFBP method outperforms the benchmark methods. The total squared coherence of the proposed Updated-SBL algorithm is dramatically reduced. and the superiority of this algorithm is validated compared with other benchmark methods with comparable computation complexity.

On variable ordination of Cholesky‐based estimation for a sparse covariance matrix

AbstractEstimation of a large sparse covariance matrix is of great importance for statistical analysis, especially in high‐dimensional settings. The traditional approach such as the sample covariance matrix performs poorly due to the high dimensionality. The modified Cholesky decomposition (MCD) is a commonly used method for sparse covariance matrix estimation. However, the MCD method relies on the order of variables, which often is not available or cannot be pre‐determined in practice. In this work, we solve this order issue by obtaining a set of covariance matrix estimates based on assuming different orders of variables used in the MCD. Then we consider an ensemble estimator as the “centre” of such a set of covariance matrix estimates with respect to the Frobenius norm. Our proposed method not only ensures that the estimator is positive definite, but also captures the underlying sparse structure of the covariance matrix. Under some regularity conditions, we establish both algorithmic and asymptotic convergence of the proposed method. Its merits are illustrated via simulation studies and a practical example using data from a prostate cancer study.

Statistical and computational limits for sparse matrix detection

This paper investigates the fundamental limits for detecting a high-dimensional sparse matrix contaminated by white Gaussian noise from both the statistical and computational perspectives. We consider $p\times p$ matrices whose rows and columns are individually $k$-sparse. We provide a tight characterization of the statistical and computational limits for sparse matrix detection, which precisely describe when achieving optimal detection is easy, hard or impossible, respectively. Although the sparse matrices considered in this paper have no apparent submatrix structure and the corresponding estimation problem has no computational issue at all, the detection problem has a surprising computational barrier when the sparsity level $k$ exceeds the cubic root of the matrix size $p$: attaining the optimal detection boundary is computationally at least as hard as solving the planted clique problem. The same statistical and computational limits also hold in the sparse covariance matrix model, where each variable is correlated with at most $k$ others. A key step in the construction of the statistically optimal test is a structural property of sparse matrices, which can be of independent interest.

Tight lower bound of sparse covariance matrix estimation in the local differential privacy model

In this paper, we study the sparse covariance matrix estimation problem in the local differential privacy model, and give a lower bound of Ω(s2log⁡pnϵ2) on the ϵ non-interactive private minimax risk in the metric of squared spectral norm, where s is the row sparsity of the underlying covariance matrix, n is the sample size, and p is the dimensionality of the data. We show that the lower bound is actually tight, as it matches a previous upper bound. Our main technique for achieving this lower bound is a general framework, called General Private Assouad Lemma, which is a considerable generalization of the previous private Assouad lemma and can be used as a general method for bounding the private minimax risk of matrix-related estimation problems.

Iterative Sparse Covariance Matrix Fitting Direction of Arrival Estimation Method Based on Vector Hydrophone Array

Aiming at the direction of arrival (DOA) estimation of coherent signals in vector hydrophone array, an iterative sparse covariance matrix fitting algorithm is proposed. Based on the fitting criterion of weighted covariance matrix, the objective function of sparse signal power is constructed, and the recursive formula of sparse signal power iteration updating is deduced by using the properties of Frobenius norm. The present algorithm uses the idea of iterative reconstruction to calculate the power of signals on discrete grids, so that the estimated power is more accurate, and thus more accurate DOA estimation can be obtained. The theoretical analysis shows that the power of the signal at the grid point solved by the present algorithm is preprocessed by a filter, which allows signals in specified directions to pass through and attenuate signals in other directions, and has low sensitivity to the correlation of signals. The simulation results show that the average error estimated by the present method is 39.4% of the multi-signal classification high resolution method and 73.7% of the iterative adaptive sparse signal representation method when the signal-to-noise ratio is 15 dB and the non-coherent signal. Moreover, the average error estimated by the present method is 12.9% of the iterative adaptive sparse signal representation method in the case of coherent signal. Therefore, the present algorithm effectively improves the accuracy of target DOA estimation when applying to DOA estimation with highly correlated targets.

Robust sparse covariance estimation by thresholding Tyler’s M-estimator

Estimating a high-dimensional sparse covariance matrix from a limited number of samples is a fundamental task in contemporary data analysis. Most proposals to date, however, are not robust to outliers or heavy tails. Toward bridging this gap, in this work we consider estimating a sparse shape matrix from $n$ samples following a possibly heavy-tailed elliptical distribution. We propose estimators based on thresholding either Tyler’s M-estimator or its regularized variant. We prove that in the joint limit as the dimension $p$ and the sample size $n$ tend to infinity with $p/n\to\gamma>0$, our estimators are minimax rate optimal. Results on simulated data support our theoretical analysis.

Statistical analysis of sparse approximate factor models

We consider the problem of estimating sparse approximate factor models. In a first step, we jointly estimate the factor loading parameters and the error - or idiosyncratic - covariance matrix based on the Gaussian quasi-maximum likelihood method. Conditionally on these first step estimators, using the SCAD, MCP and Lasso regularisers, we obtain a sparse error covariance matrix based on a Gaussian QML and, as an alternative criterion, a least squares loss function. Under suitable regularity conditions, we derive error bounds for the regularised idiosyncratic factor model matrix for both Gaussian QML and least squares losses. Moreover, we establish the support recovery property, including the case when the regulariser is non-convex. These theoretical results are supported by empirical studies.

Sparse covariance matrix estimation in high-dimensional deconvolution

We study the estimation of the covariance matrix $\Sigma$ of a $p$-dimensional normal random vector based on $n$ independent observations corrupted by additive noise. Only a general nonparametric assumption is imposed on the distribution of the noise without any sparsity constraint on its covariance matrix. In this high-dimensional semiparametric deconvolution problem, we propose spectral thresholding estimators that are adaptive to the sparsity of $\Sigma$. We establish an oracle inequality for these estimators under model miss-specification and derive non-asymptotic minimax convergence rates that are shown to be logarithmic in $n/\log p$. We also discuss the estimation of low-rank matrices based on indirect observations as well as the generalization to elliptical distributions. The finite sample performance of the threshold estimators is illustrated in a numerical example.

Computing the Moments of the Complex Gaussian: Full and Sparse Covariance Matrix

Given a multivariate complex centered Gaussian vector Z = ( Z 1 , ⋯ , Z p ) with non-singular covariance matrix Σ , we derive sufficient conditions on the nullity of the complex moments and we give a closed-form expression for the non-null complex moments. We present conditions for the factorisation of the complex moments. Computational consequences of these results are discussed.