Sample Covariance Estimator Research Articles

Doubly Sparse estimator for high-dimensional covariance matrices

The classical sample covariance estimator lacks desirable properties such as consistency and suffers from eigenvalue spreading in high-dimensional settings. Improved estimators have been proposed that shrink sample eigenvalues but retain the eigenvectors of the sample covariance estimator. In high dimensions, however, sample eigenvectors are generally strongly inconsistent, rendering eigenvalue shrinkage estimators suboptimal. A Doubly Sparse Covariance Estimator (DSCE) is developed that goes beyond mere eigenvalue shrinkage: a covariance matrix is decomposed into a signal part, where sparse eigenvectors are estimated via truncation, and an idiosyncratic part, estimated via thresholding. It is shown that accurate estimation is possible if the leading eigenvectors are sufficiently sparse affecting proportionately less than p of the variables. DSCE fills the gap for empirical applications that fall in-between fully sparse settings and conditionally sparse settings: DSCE takes advantage of conditional sparsity implied by factor models while allowing only a subset of variables to load on factors, which relaxes pervasiveness assumption of traditional factor models. An empirical application to the constituents of the S&P 1500 illustrates that DSCE-based portfolios outperform competing methods in terms of Sharpe ratio, maximum drawdown, and cumulative return for monthly and daily data.

The empirical performance of option implied volatility surface-driven optimal portfolios

We apply a two-step strategy to forecast the dynamics of the volatility surface implicit in option prices to all American-style options written on the stocks that have entered the Dow Jones Industrial Average Index between 2004 and 2016. We explore whether the implied volatilities extracted through the two-step approach help improve the out-of-sample performance of minimum-variance portfolios. We find that, by using option-implied volatilities in estimating the covariance matrix, the ex-post volatility of the minimum-variance portfolio is lower when compared with the equal-weighted portfolio and a minimum-variance portfolio simply derived from the historical, sample covariance matrix estimator. Moreover, over most of our 13-year sample, the realized Sharpe, Sortino and information ratios increase when the sample covariance matrix estimator is replaced with its implied counterpart. However, the benefits of using option-implied information are countered by an increase in portfolio turnover that may imply higher (implicit) transaction costs. We also apply shrinkage methods to both the sample covariance estimator and the implied covariance estimator and note that they often lead to significant improvements in portfolio performance.

Structured channel covariance estimation from limited samples for large antenna arrays

In massive multiuser multiple antenna systems, the knowledge of the users’ channel covariance matrix is crucial for minimum mean square error channel estimation in the uplink and it plays an important role in several multiuser beamforming schemes in the downlink. Due to the large number of base station antennas, accurate covariance estimation is challenging especially in the case where the number of samples is limited and thus comparable to the channel vector dimension. As a result, the standard sample covariance estimator may yield a too large estimation error which in turn may yield significant system performance degradation with respect to the ideal channel covariance knowledge case. To address such problem, we propose a method based on a parametric representation of the channel angular scattering function. The proposed parametric representation includes a discrete specular component which is addressed using the well-known MUltiple SIgnal Classification (MUSIC) method, and a diffuse scattering component, which is modeled as the superposition of suitable dictionary functions. To obtain the representation parameters, we propose two methods, where the first solves a nonnegative least-squares problem and the second maximizes the likelihood function using expectation–maximization. Our simulation results show that the proposed methods outperform the state of the art with respect to various estimation quality metrics and different sample sizes.

Difference-frequency MUSIC for beamforming

A beamforming method is derived to estimate direction-of-arrivals (DOAs) for high-frequency sources. The algorithm uses multiple frequencies and their difference frequencies (DFs). Due to the DFs below the aliasing frequency, the method mitigates spatial aliasing. DF-processing has been applied previously to the conventional beamforming [Abadi et al., J.Acoust. Soc. Am., 132(5):3018–3029, 2012]. The proposed method uses a classical beamformer, MUSIC, which is a subspace method. The MUSIC shows sharp peaks in the beamforming spectrum and high-resolution. The DF-MUSIC integrates data samples into a sample covariance matrix and estimates DOAs. We propose three ways to treat multiple time and DF samples. The time-DF-MUSIC obtains MUSIC spectra at each DF and averages the spectra over time to get final DOAs. The frequency-DF-MUSIC obtains MUSIC spectra at each time snapshot and averages the spectra over frequency to get final DOAs. The time-frequency-DF-MUSIC considers all time and frequency jointly and improves the DOA estimation.

Adaptive Brain Activity Detection in Structured Interference and Partially Homogeneous Locally Correlated Disturbance.

In this paper, we aim to address the problem of subspace detection in the presence of locally-correlated complex Gaussian noise and interference. For applications like brain activity detection using functional magnetic resonance imaging (fMRI) data where the noise is possibly locally correlated, using the sample covariance estimator is not a suitable choice due to significant dependency of its accuracy on the number of observations. In this study, we take advantage of an assumed banded structure in the covariance matrix to model the local dependence in the noise and propose a new covariance estimation approach. In particular, we use the idea of factorizing the joint likelihood function into a few conditional likelihood terms and maximizing each term independently of the others. This process leads to an explicit estimator for banded covariance matrices which requires fewer observations to achieve the same accuracy as the sample covariance. This estimate is then fed into an adaptive matched filter, two-step Rao and two-step Wald tests for detection. Simulation results reveal the superiority of the proposed methods over well known classical detectors. Finally, the proposed methods are applied to functional magnetic resonance imaging (fMRI) data to localize neural activities in the brain. The proposed method can offer better activation maps in terms of accuracy and spatial smoothness. The proposed methods can be seen as alternatives for standard detection approaches which are not perfectly aligned with the properties of fMRI data.

Robust functional principal component analysis via a functional pairwise spatial sign operator.

Functional principal component analysis (FPCA) has been widely used to capture major modes of variation and reduce dimensions in functional data analysis. However, standard FPCA based on the sample covariance estimator does not work well if the data exhibits heavy-tailedness or outliers. To address this challenge, a new robust FPCA approach based on a functional pairwise spatial sign (PASS) operator, termed PASS FPCA, is introduced. We propose robust estimation procedures for eigenfunctions and eigenvalues. Theoretical properties of the PASS operator are established, showing that it adopts the same eigenfunctions as the standard covariance operator and also allows recovering ratios between eigenvalues. We also extend the proposed procedure to handle functional data measured with noise. Compared to existing robust FPCA approaches, the proposed PASS FPCA requires weaker distributional assumptions to conserve the eigenspace of the covariance function. Specifically, existing work are often built upon a class of functional elliptical distributions, which requires inherently symmetry. In contrast, we introduce a class of distributions called the weakly functional coordinate symmetry (weakly FCS), which allows for severe asymmetry and is much more flexible than the functional elliptical distribution family. The robustness of the PASS FPCA is demonstrated via extensive simulation studies, especially its advantages in scenarios with nonelliptical distributions. The proposed method was motivated by and applied to analysis of accelerometry data from the Objective Physical Activity and Cardiovascular Health Study, a large-scale epidemiological study to investigate the relationship between objectively measured physical activity and cardiovascular health among olderwomen.

Charles Stein and invariance: Beginning with the Hunt–Stein theorem

When statistical decision theory was emerging as a promising new paradigm, Charles Stein was to play a major role in the development of minimax theory for invariant statistical problems. In some of his earliest work with Gil Hunt, he set out to prove that, in problems where invariant procedures have constant risk, any best invariant test would be minimax among all tests. Although finding it not quite true in general, this led to the legendary Hunt–Stein theorem, which established the result under restrictive conditions on the underlying group of transformations. In decision problems invariant under such suitable groups, an overall minimax test was guaranteed to reside within the class of invariant procedures where it would typically be much easier to find. But when it did not seem possible to establish this result for invariance under the full linear group, he instead turned to prove its impossibility with counterexamples such as the nonminimaxity of the usual sample covariance estimator where the full linear group was just too big for the Hunt–Stein theorem to apply. Further explorations of invariance such as the sometimes problematic inference under a fiducial distribution, or the characterization of a best invariant procedure as a formal Bayes procedure under a right Haar prior, are further examples of the far reaching influence of Stein’s contributions to invariance theory.

Cross-validated covariance estimators for high-dimensional minimum-variance portfolios

The global minimum-variance portfolio is a typical choice for investors because of its simplicity and broad applicability. Although it requires only one input, namely the covariance matrix of asset returns, estimating the optimal solution remains a challenge. In the presence of high-dimensionality in the data, the sample covariance estimator becomes ill-conditioned and leads to suboptimal portfolios out-of-sample. To address this issue, we review recently proposed efficient estimation methods for the covariance matrix and extend the literature by suggesting a multi-fold cross-validation technique for selecting the necessary tuning parameters within each method. Conducting an extensive empirical analysis with four datasets based on the S&P 500, we show that the data-driven choice of specific tuning parameters with the proposed cross-validation improves the out-of-sample performance of the global minimum-variance portfolio. In addition, we identify estimators that are strongly influenced by the choice of the tuning parameter and detect a clear relationship between the selection criterion within the cross-validation and the evaluated performance measure.

Deep Networks for Direction-of-Arrival Estimation in Low SNR

In this work, we consider direction-of-arrival (DoA) estimation in the presence of extreme noise using Deep Learning (DL). In particular, we introduce a Convolutional Neural Network (CNN) that is trained from mutli-channel data of the true array manifold matrix and is able to predict angular directions using the sample covariance estimate. We model the problem as a multi-label classification task and train a CNN in the low-SNR regime to predict DoAs across all SNRs. The proposed architecture demonstrates enhanced robustness in the presence of noise, and resilience to a small number of snapshots. Moreover, it is able to resolve angles within the grid resolution. Experimental results demonstrate significant performance gains in the low-SNR regime compared to state-of-the-art methods and without the requirement of any parameter tuning. We relax the assumption that the number of sources is known a priori and present a training method, where the CNN learns to infer the number of sources jointly with the DoAs. Simulation results demonstrate that the proposed CNN can accurately estimate off-grid angles in low SNR, while at the same time the number of sources is successfully inferred for a sufficient number of snapshots. Our robust solution can be applied in several fields, ranging from wireless array sensors to acoustic microphones or sonars.

Risk-based portfolio sensitivity to covariance estimation

ABSTRACT Risk-based portfolio construction methods focus on optimally extracting information from the covariance matrix of asset returns, as opposed to utilising forecasts of expected returns, in determining the portfolio allocation. This improves their robustness to estimation error in means, but this does not mean that they are immune to errors in estimating volatilities and correlations. Using a covariance matrix decomposition that allows separately estimated volatility and correlation models to be recomposed into different models of the covariance matrix, this study examines the empirical performance impact of using an enhanced estimator of the covariance matrix, relative to using the historical sample covariance estimator in the context of six risk-based portfolio optimisations, in a long-only constrained equity market setting. It finds that sensitivity to covariance estimation varies significantly among risk-based portfolio types and that outperformance of the sample historical covariance estimator is possible, but rare. As components of the covariance estimate, among volatility models the EWMA volatilities perform best and GARCH models, poorly. Among correlation models, the Rotationally Invariant Estimator of Bouchaud, Bun, and Potters (2016) shows strong performance, along with the classic Ledoit and Wolf (2003) Single Market Model Estimator.

Enhanced beamspace MUSIC for cost‐effective FMCW automotive radar

As one important sensor for automotive applications, frequency modulated continuous wave (FMCW) radars working at millimetre wave frequency, e.g. 24 or 77 GHz, are drawing much attention in recent years. Compared with range and radial velocity measurement, direction of arrival (DOA) estimation is more challenging in FMCW automotive radars due to the space limitation and computational complexity constraint. The feature of the signal processing procedure further results in the problem of single snapshot and relatively low signal-to-noise ratio (SNR) level. In this study, the authors propose a low-complexity beamspace multiple signal classification (MUSIC) scheme to improve the DOA estimation performance, where (i) the prior information is exploited to improve beamformer design, (ii) a modified MUSIC estimator is formulated to alleviate the performance degradation due to low SNR, (iii) a joint sub-matrix averaging and Toeplitz structure recovery processing is utilised to compensate the sample covariance estimation error resulted from single snapshot, especially for closely located targets. Simulations are given to demonstrate the efficiency of the authors' proposed enhanced beamspace DOA estimation for cost-effective FMCW automotive radar.

Data Fusion With Inverse Covariance Intersection for Prior Covariance Estimation of the Particle Flow Filter

The prior covariance estimation method based on inverse covariance intersection (ICI) is proposed to apply the particle flow filter. The proposed method has better estimate performance and guarantees consistent estimation results compared with previous works. ICI is the recently developed method of ellipsoidal intersection and is used to get the combined estimate of prior covariance. This method integrates the sample covariance estimate, which is unbiased but usually with high variance, together with a more structured but typically a biased target covariance through fusion gains. For verifying the performance of the proposed algorithm, analysis and simulations are performed. Through the simulations, the results are given to illustrate the consistency and accuracy of the proposed algorithm’s estimation and target tracking performance.

Steerable ePCA: Rotationally Invariant Exponential Family PCA.

In photon-limited imaging, the pixel intensities are affected by photon count noise. Many applications require an accurate estimation of the covariance of the underlying 2-D clean images. For example, in X-ray free electron laser (XFEL) single molecule imaging, the covariance matrix of 2-D diffraction images is used to reconstruct the 3-D molecular structure. Accurate estimation of the covariance from low-photon-count images must take into account that pixel intensities are Poisson distributed, hence the classical sample covariance estimator is highly biased. Moreover, in single molecule imaging, including in-plane rotated copies of all images could further improve the accuracy of covariance estimation. In this paper we introduce an efficient and accurate algorithm for covariance matrix estimation of count noise 2-D images, including their uniform planar rotations and possibly reflections. Our procedure, steerable ePCA, combines in a novel way two recently introduced innovations. The first is a methodology for principal component analysis (PCA) for Poisson distributions, and more generally, exponential family distributions, called ePCA. The second is steerable PCA, a fast and accurate procedure for including all planar rotations when performing PCA. The resulting principal components are invariant to the rotation and reflection of the input images. We demonstrate the efficiency and accuracy of steerable ePCA in numerical experiments involving simulated XFEL datasets and rotated face images from Yale Face Database B.

Symmetry-Imposed Rectangular Coprime and Nested Arrays for Direction of Arrival Estimation With Multiple Signal Classification

Linear sparse arrays (coprime and nested arrays) have been studied extensively as a means of performing direction of arrival (DoA) estimation while bypassing Nyquist sampling theorem. However, rectangular sparse arrays have few studies, most of which are based in lattice theory. Although the multiple signal classification (MUSIC) alogrithm can be applied to lattice theory-based sparse arrays, fewer DoAs can be estimated for these arrays than for a full array with the same aperture. One contribution of this paper is the formulation of symmetry-imposed rectangular coprime and nested array designs that have wider contiguous lags than lattice-based arrays. Also, existing algorithms for rectangular arrays employ the direct sample covariance matrix estimate, which has low accuracy. Another contribution of this paper is an alternate method for estimating the covariance matrix that ensures the matrix is block-Toeplitz. Using the proposed covariance matrix estimate, we integrate a MUSIC-based DoA estimation method that applies to both full arrays and sparse arrays. The results show that the covariance estimates produced by the proposed method have higher accuracy than the traditional sample covariance estimates. The results also demonstrate that symmetry-imposed sparse arrays have higher resolution than full rectangular arrays with the same number of sensors.

Parametric spatial covariance models in the ensemble Kalman filter

Several applications rely on data assimilation methods for complex spatio-temporal problems. The focus of this paper is on ensemble-based methods, where some approaches require estimation of covariances between state variables and observations in the assimilation step. Spurious correlations present a challenge in such cases as they can degrade the quality of the ensemble representation of probability distributions. In particular, prediction variability is often underestimated. We propose to replace the sample covariance estimate by a parametric approach using maximum likelihood estimation for a small number of parameters in a spatial covariance model. Parametric covariance and precision estimation are employed in the context of the ensemble Kalman filter, and applied to a Gauss-linear autoregressive model and a geological process model. We learn that parametric approaches reduce the underestimation in prediction variability. Furthermore rich, non-stationary models do not seem to add much over simpler models with fewer parameters.

Multimodel Shrinkage for Knowledge-Aided Space-Time Adaptive Processing

Space-time adaptive processing gets its adaptivity from estimating the second-order statistics of the clutter from secondary data. In practice, the amount of available secondary data may not be sufficient for the sample covariance estimate to be accurate. Existing techniques exploit assumed properties of the clutter covariance, such as rank, to reduce the need for secondary training data. More recently in radar, single-model shrinkage estimators have been shown to reduce the reliance on training data given that a good a priori model for the covariance is available. We extend shrinkage estimation to a regularized multimodel approach that incorporates inexact knowledge of a digital elevation map to reduce the need for large amounts of secondary data. Using both simulation and the KASSPER I dataset, performance of the proposed approach is compared to low-rank estimation and various single-model shrinkage estimation approaches. The proposed approach offers the practical ability to reliably estimate the clutter covariance from a number of training data less than the rank of the clutter covariance matrix.

Precision matrix expansion – efficient use of numerical simulations in estimating errors on cosmological parameters

Computing the inverse covariance matrix (or precision matrix) of large data vectors is crucial in weak lensing (and multi-probe) analyses of the large scale structure of the universe. Analytically computed covariances are noise-free and hence straightforward to invert, however the model approximations might be insufficient for the statistical precision of future cosmological data. Estimating covariances from numerical simulations improves on these approximations, but the sample covariance estimator is inherently noisy, which introduces uncertainties in the error bars on cosmological parameters and also additional scatter in their best fit values. For future surveys, reducing both effects to an acceptable level requires an unfeasibly large number of simulations. In this paper we describe a way to expand the true precision matrix around a covariance model and show how to estimate the leading order terms of this expansion from simulations. This is especially powerful if the covariance matrix is the sum of two contributions, $\smash{\mathbf{C} = \mathbf{A}+\mathbf{B}}$, where $\smash{\mathbf{A}}$ is well understood analytically and can be turned off in simulations (e.g. shape-noise for cosmic shear) to yield a direct estimate of $\smash{\mathbf{B}}$. We test our method in mock experiments resembling tomographic weak lensing data vectors from the Dark Energy Survey (DES) and the Large Synoptic Survey Telecope (LSST). For DES we find that $400$ N-body simulations are sufficient to achive negligible statistical uncertainties on parameter constraints. For LSST this is achieved with $2400$ simulations. The standard covariance estimator would require >$10^5$ simulations to reach a similar precision. We extend our analysis to a DES multi-probe case finding a similar performance.

A perturbation analysis of stochastic matrix Riccati diffusions

Matrix differential Riccati equations are central in filtering and optimal control theory. The purpose of this article is to develop a perturbation theory for a class of stochastic matrix Riccati diffusions. Diffusions of this type arise, for example, in the analysis of ensemble Kalman-Bucy filters since they describe the flow of certain sample covariance estimates. In this context, the random perturbations come from the fluctuations of a mean field particle interpretation of a class of nonlinear diffusions equipped with an interacting sample covariance matrix functional. The main purpose of this article is to derive non-asymptotic Taylor-type expansions of stochastic matrix Riccati flows with respect to some perturbation parameter. These expansions rely on an original combination of stochastic differential analysis and nonlinear semigroup techniques on matrix spaces. The results here quantify the fluctuation of the stochastic flow around the limiting deterministic Riccati equation, at any order. The convergence of the interacting sample covariance matrices to the deterministic Riccati flow is proven as the number of particles tends to infinity. Also presented are refined moment estimates and sharp bias and variance estimates. These expansions are also used to deduce a functional central limit theorem at the level of the diffusion process in matrix spaces.

Using principal component analysis to estimate a high dimensional factor model with high-frequency data

This paper constructs an estimator for the number of common factors in a setting where both the sampling frequency and the number of variables increase. Empirically, we document that the covariance matrix of a large portfolio of US equities is well represented by a low rank common structure with sparse residual matrix. When employed for out-of-sample portfolio allocation, the proposed estimator largely outperforms the sample covariance estimator.

Multi-Waveform Space-Time Adaptive Processing

A new form of space-time adaptive processing (STAP) is presented that leverages additional training data obtained from waveform-diverse pulse compression filters possessing low cross-correlation with the primary waveform that is used for traditional airborne and space-based ground moving target indication. In contrast to traditional training data in which clutter and targets are “focused” in range via pulse compression of the primary waveform, this new set of training data possesses a “smeared” range response that better approximates the identically distributed assumption made during sample covariance estimation. The Multi-Waveform STAP (MuW-STAP or simply $\mu $ -STAP) formulation is shown for both multiple-input multiple-output and single-input multiple-output configurations, with the former retaining the spatially-focused primary emission supplemented by low-power secondary emissions that illuminate sidelobe clutter and the latter a special case of the former. In simulation, signal to interference plus noise ratio analysis reveals enhanced robustness to nonstationary interference compared to standard STAP training data.