Consistent estimation of low-rank spatial covariance matrix: A penalized random effects approach

In this letter, we develop a low-complexity algorithm named maximum-likelihood estimation of low-rank Toeplitz covariance matrix (MELT) to solve the maximum-likelihood estimation of a low-rank Toeplitz covariance matrix. Our derivation of MELT is based on the technique of majorization–minimization (MM), in which we design and optimize a novel tight upper-bound function. MELT is an iterative algorithm, and its each iterative step is a closed-form update, which can be implemented efficiently by fast Fourier transforms. As MELT is based on MM, it enjoys nice properties such as monotonicity and guaranteed convergence to a stationary point. Finally, we numerically show that the performance of MELT is much better than some of the algorithms currently available in the literature.

We consider ridge-type estimation of the multivariate normal distribution’s covariance matrix and its inverse, the precision matrix. While several ridge-type covariance and precision matrix estimators have been presented in the literature, their respective inverses are often not considered as precision and covariance matrix estimators even though their estimands are one-to-one related through the matrix inverse. We study which estimator is to be preferred in what case. Hereto we compare the ridge-type covariance matrix estimators and their properties to that of the inverse of the ridge-type precision matrix estimators, and vice versa. The comparison, in which we take all ridge-type estimators along, is limited to a specific case that is illustrative of the difference between the covariance and precision matrix estimators. The comparison addresses the estimators’ estimating equation, analytic expression, analytic properties like positive definiteness and penalization limit, mean squared error, consistency, Bayesian formulation, and their loss and potential for marginal and partial correlation screening.

Bayesian inference for adaptive low rank and sparse matrix estimation

Journal Article Nonparametric estimation of large covariance matrices of longitudinal data Get access Wei Biao Wu, Wei Biao Wu Search for other works by this author on: Oxford Academic Google Scholar Mohsen Pourahmadi Mohsen Pourahmadi Search for other works by this author on: Oxford Academic Google Scholar Biometrika, Volume 90, Issue 4, December 2003, Pages 831–844, https://doi.org/10.1093/biomet/90.4.831 Published: 01 December 2003

This paper introduces a new method for deriving covariance matrix estimators that are decision-theoretically optimal within a class of nonlinear shrinkage estimators. The key is to employ large-dimensional asymptotics: the matrix dimension and the sample size go to infinity together, with their ratio converging to a finite, nonzero limit. As the main focus, we apply this method to Stein’s loss. Compared to the estimator of Stein (Estimation of a covariance matrix (1975); J. Math. Sci. 34 (1986) 1373–1403), ours has five theoretical advantages: (1) it asymptotically minimizes the loss itself, instead of an estimator of the expected loss; (2) it does not necessitate post-processing via an ad hoc algorithm (called “isotonization”) to restore the positivity or the ordering of the covariance matrix eigenvalues; (3) it does not ignore any terms in the function to be minimized; (4) it does not require normality; and (5) it is not limited to applications where the sample size exceeds the dimension. In addition to these theoretical advantages, our estimator also improves upon Stein’s estimator in terms of finite-sample performance, as evidenced via extensive Monte Carlo simulations. To further demonstrate the effectiveness of our method, we show that some previously suggested estimators of the covariance matrix and its inverse are decision-theoretically optimal in the large-dimensional asymptotic limit with respect to the Frobenius loss function.

This paper presents a novel low-rank matrix recovery approach that jointly performs measurement collection and matrix estimation to improve the overall sample efficiency under unknown rank information. It builds on a key observation that the minimum number of measurements needed for matrix rank estimation can be much less than that for matrix recovery. Such a gap in measurement requirements is first delineated in closes form through empirical quantification. Then, capitalizing on this quantified gap on measurements, a two-step procedure is developed for adaptive measurement collection. The actual rank of the matrix is estimated in the first step, which informs the number of measurements to be collected in the second step for low-rank matrix recovery. Simulations corroborate that our approach can considerably reduce the total number of required measurements for matrix recovery in practice. The improvement in sample efficiency is particularly pronounced for large-scale applications where low-rank matrix estimation is of great relevance.

We consider a problem involving estimation of a high-dimensional covariance matrix that is the sum of a diagonal matrix and a low-rank matrix, and making a decision based on the resulting estimate. Such problems arise, for example, in portfolio management, where a common approach employs principal component analysis (PCA) to estimate factors used in constructing the low-rank term of the covariance matrix. The decision problem is typically treated separately, with the estimated covariance matrix taken to be an input to an optimization problem. We propose directed PCA, an efficient algorithm that takes the decision objective into account when estimating the covariance matrix. Directed PCA effectively adjusts factors that would be produced by PCA so that they better guide the specific decision at hand. We demonstrate through computational studies that directed PCA yields significant benefit, and we prove theoretical results establishing that the degree of improvement over conventional PCA can be arbitrarily large.

We consider the robust PCA problem of recovering a low-rank matrix corrupted by Gaussian noise and large elementlevel outliers. Motivated by the sparse estimation literature, we consider outlier rejection schemes that apply hard or soft thresholding, respectively, to the elements of the data matrix to efficiently estimate the sparse component and then apply an SVD on the residual matrix to estimate the underlying low-rank component. We analyze the performance of the low-rank matrix thus estimated by comparing the asymptotic accuracy of the recovered subspace to that of an oracle estimator that replaces the a priori known outlier-corrupted entries of the data matrix with zeros and then applies an SVD to estimate the low-rank component. Our analysis reveals that, in the "not-too-sparse" outlier regime, both hard and soft thresholding asymptotically attain oracle performance. However, in the dense outlier regime, while hard thresholding again achieves oracle performance, soft thresholding does not and the estimated subspace is asymptotically orthogonal to the true underlying subspace. We validate our theoretical predictions with numerical simulations.

This article is an extended version of previous work of Lesieur et al (2015 IEEE Int. Symp. on Information Theory Proc. pp 1635–9 and 2015 53rd Annual Allerton Conf. on Communication, Control and Computing (IEEE) pp 680–7) on low-rank matrix estimation in the presence of constraints on the factors into which the matrix is factorized. Low-rank matrix factorization is one of the basic methods used in data analysis for unsupervised learning of relevant features and other types of dimensionality reduction. We present a framework to study the constrained low-rank matrix estimation for a general prior on the factors, and a general output channel through which the matrix is observed. We draw a parallel with the study of vector-spin glass models—presenting a unifying way to study a number of problems considered previously in separate statistical physics works. We present a number of applications for the problem in data analysis. We derive in detail a general form of the low-rank approximate message passing (Low-RAMP) algorithm, that is known in statistical physics as the TAP equations. We thus unify the derivation of the TAP equations for models as different as the Sherrington–Kirkpatrick model, the restricted Boltzmann machine, the Hopfield model or vector (xy, Heisenberg and other) spin glasses. The state evolution of the Low-RAMP algorithm is also derived, and is equivalent to the replica symmetric solution for the large class of vector-spin glass models. In the section devoted to result we study in detail phase diagrams and phase transitions for the Bayes-optimal inference in low-rank matrix estimation. We present a typology of phase transitions and their relation to performance of algorithms such as the Low-RAMP or commonly used spectral methods.

In heteroskedastic regression models, the least squares (OLS) covariance matrix estimator is inconsistent and inference is not reliable. To deal with inconsistency one can estimate the regression coefficients by OLS, and then implement a heteroskedasticity consistent covariance matrix (HCCM) estimator. Unfortunately the HCCM estimator is biased. The bias is reduced by implementing a robust regression, and by using the robust residuals to compute the HCCM estimator (RHCCM). A Monte-Carlo study analyzes the behavior of RHCCM and of other HCCM estimators, in the presence of systematic and random heteroskedasticity, and of outliers in the explanatory variables.

This study develops cluster robust inference methods for panel quantile regression (QR) models with individual fixed effects, allowing for temporal correlation within each individual. The conventional QR standard errors can seriously underestimate the uncertainty of estimators and, therefore, overestimate the significance of effects, when outcomes are serially correlated. Thus, we propose a clustered covariance matrix (CCM) estimator to solve this problem. The CCM estimator is an extension of the heteroskedasticity and autocorrelation consistent covariance matrix estimator for QR models with fixed effects. The autocovariance element in the CCM estimator can be substantially biased, due to the incidental parameter problem. Thus, we develop a bias‐correction method for the CCM estimator. We derive an optimal bandwidth formula that minimizes the asymptotic mean squared errors, and propose a data‐driven bandwidth selection rule. We also propose two cluster robust tests, and establish their asymptotic properties. We then illustrate the practical usefulness of the proposed methods using an empirical application.

Covariance matrix estimation is one of the most important problems in statistics. To accommodate the complexity of modern datasets, it is desired to have estimation procedures that not only can incorporate the structural assumptions of covariance matrices, but are also robust to outliers from arbitrary sources. In this paper, we define a new concept called matrix depth and then propose a robust covariance matrix estimator by maximizing the empirical depth function. The proposed estimator is shown to achieve minimax optimal rate under Huber’s $\\varepsilon$-contamination model for estimating covariance/scatter matrices with various structures including bandedness and sparsity.

GTRI is developing an 8-channel X-band experimental radar for a small unmanned aircraft system (SUAS) for adaptive multi-channel, MIMO, and waveform diversity test bed studies. New adaptive algorithms, one of which is covered in this paper, are also part of the test bed. Estimation of the statistical covariance matrix forms a central role in radar detection and adaptive beamforming algorithm. For example, the optimal (adaptive) linear combiner (beamformer) weights for a radar sensor array are expressed in terms of the inverse of the multi-channel (MC) covariance matrix for MIMO problems. Rather than form an estimate of the covariance matrix directly from the available data and inverting (sample matrix inversion [SMI]), an alternative direct estimate of the inverse may be obtained by forming parametric MC linear prediction estimates and then expressing the inverse in terms of these parametric MC estimates. The resulting parametric inverse estimate will be more accurate than inverting the estimate of the covariance matrix, leading to greatly improved detection performance over conventional methods of covariance estimation and inversion. This paper reveals the structure of the inverse of the covariance matrix for one parametric technique. The inverse structure involves products of triangular block MC Toeplitz matrices, which leads to a fast computational solution. Performance improvements over classical sample covariance matrix estimation are illustrated.

Transformation of multiple-trait records that undergo sequential selection can be used with derivative-free algorithms to maximize the restricted likelihood in estimation of covariance matrices as with derivative methods. Data transformation with appropriate parts of the Choleski decomposition of the current estimate of the residual covariance matrix results in mixed-model equations that are easily modified from round to round for calculation of the logarithm of the likelihood. The residual sum of squares is the same for transformed and untransformed analyses. Most importantly, the logarithm of the determinant of the untransformed coefficient matrix is an easily determined function of the Choleski decomposition of the residual covariance matrix and the determinant of the transformed coefficient matrix. Thus, the logarithm of the likelihood for any combination of covariance matrices can be determined from the transformed equations. Advantages of transformation are 1) the multiple-trait mixed-model equations are easy to set up, 2) the least squares part of the equations does not change from round to round, 3) right-hand sides change from round to round by constant multipliers, and 4) less memory is required. An example showed only a slight advantage of the transformation compared with no transformation in terms of solution time for each round (1 to 5%).

To address the severe performance degradation of radar weak target detection induced by dense cluster targets and sea-spike interference in nonhomogeneous sea clutter environments, this paper proposes an enhanced Adaptive Normalized Matched Filter algorithm based on iterative weighting and sample purification (IWP-ANMF). The proposed algorithm establishes a closed-loop iterative detection framework capable of highly sensitive discrimination of anomalous data within the reference window—particularly cluster targets and strong discrete sea spikes that severely distort covariance matrix features—identifying them as “contaminated samples.” During each iteration, target-likelihood statistics are calculated for all reference samples based on the current covariance matrix estimate. Subsequently, an adaptive deep-notch suppression strategy is applied to contaminated samples, such as cluster targets, according to their statistical characteristics, thereby progressively purifying the sample covariance matrix (SCM) estimation. Theoretically, this iterative procedure is rigorously proven to converge to the optimal solution of a robust weighted covariance matrix estimation problem. Comprehensive validations using both Monte Carlo simulations and measured K-distributed sea clutter data demonstrate that, compared to classical ANMF and Generalized Inner Product (GIP) approaches, the proposed algorithm exhibits outstanding robustness and detection performance when confronted with heterogeneous contamination scenarios, especially high-density cluster targets. This method effectively eliminates the blind-zone expansion and performance deterioration caused by the wideband masking of cluster targets, significantly enhancing weak target detection capabilities under complex maritime conditions.