Sample Covariance Matrix Estimator Research Articles

Efficient Real-Time Whitening for Blind Eigenvalue-Based Detection in mmWave Full Duplex Cognitive Radio

Although combining Full Duplex (FD) and Cognitive Radio (CR) technologies can provide several benefits, the presence of the residual self-interference (RSI) at the FD transceiver may influence the spectrum sensing capabilities. In particular, focusing on the blind eigenvalue-based detectors (EVDs), this paper shows that operating at mmWave widebands, the RSI is a colored noise strongly affecting the detection performance even when it is characterized by low power. Consequently, we propose here a whitened detector whose novelty consists in the real-time computation of the whitening matrix, differently from previous works which employ an offline computation. The proposed whitening matrix computation uses a sample covariance matrix (SCM) regularization tailored for the FD scenario, instead of the conventional SCM estimator. Particularly, we prove that the proposed whitened detector is robust against RSI power uncertainty, attaining a constant false alarm rate under typical FD operating conditions. Finally, we show that our method provides computational load reduction and faster convergence to the theoretical false alarm probability characteristic of the white noise case. Numerical simulations validate the theoretical analysis confirming the better performance of the proposed whitening approach, particularly when applied to the sphericity test, in comparison to the offline-based approach.

Probabilistic Forecast Reconciliation under the Gaussian Framework

Forecast reconciliation of multivariate time series maps a set of incoherent forecasts into coherent forecasts to satisfy a given set of linear constraints. Available methods in the literature either follow a projection matrix-based approach or an empirical copula-based reordering approach to revise the incoherent future sample paths to obtain reconciled probabilistic forecasts. The projection matrices are estimated either by optimizing a scoring rule such as energy or variogram score or simply using a projection matrix derived for point forecast reconciliation. This article proves that (a) if the incoherent predictive distribution is jointly Gaussian, then MinT (minimum trace) minimizes the logarithmic scoring rule for the hierarchy; and (b) the logarithmic score of MinT for each marginal predictive density is smaller than that of OLS (ordinary least squares). We illustrate these theoretical results using a set of simulation studies and the Australian domestic tourism dataset. The estimation of MinT needs to estimate the covariance matrix of the base forecast errors. We have evaluated the performance using the sample covariance matrix and shrinkage estimator. It was observed that the theoretical properties noted above are greatly impacted by the covariance matrix used and highlighted the importance of estimating it reliably, especially with high dimensional data.

The empirical performance of option implied volatility surface-driven optimal portfolios

We apply a two-step strategy to forecast the dynamics of the volatility surface implicit in option prices to all American-style options written on the stocks that have entered the Dow Jones Industrial Average Index between 2004 and 2016. We explore whether the implied volatilities extracted through the two-step approach help improve the out-of-sample performance of minimum-variance portfolios. We find that, by using option-implied volatilities in estimating the covariance matrix, the ex-post volatility of the minimum-variance portfolio is lower when compared with the equal-weighted portfolio and a minimum-variance portfolio simply derived from the historical, sample covariance matrix estimator. Moreover, over most of our 13-year sample, the realized Sharpe, Sortino and information ratios increase when the sample covariance matrix estimator is replaced with its implied counterpart. However, the benefits of using option-implied information are countered by an increase in portfolio turnover that may imply higher (implicit) transaction costs. We also apply shrinkage methods to both the sample covariance estimator and the implied covariance estimator and note that they often lead to significant improvements in portfolio performance.

Asset Allocation Strategies Using Covariance Matrix Estimators

Abstract The covariance matrix is an important element of many asset allocation strategies. The widely used sample covariance matrix estimator is unstable especially when the number of time observations is small and the number of assets is large or when high-dimensional data is involved in the computation. In this study, we focus on the most important estimators that are applied on a group of Markowitz-type strategies and also on a recently introduced method based on hierarchical tree clustering. The performance tests of the portfolio strategies using different covariance matrix estimators rely on the out-of-sample characteristics of synthetic and real stock data.

K-Covariance: An Approach of Ensemble Covariance Estimation and Undersampling to Stabilize the Covariance Matrix in the Global Minimum Variance Portfolio

A covariance matrix is an important parameter in many computational applications, such as quantitative trading. Recently, a global minimum variance portfolio received great attention due to its performance after the 2007–2008 financial crisis, and this portfolio uses only a covariance matrix to calculate weights for assets. However, the calculation process of that portfolio is sensitive with outliers in the covariance matrix, for example, a sample covariance matrix estimation or linear shrinkage covariance matrix estimations. In this paper, we propose the use of an undersampling technique and ensemble learning to stabilize the covariance matrix by reducing the impacts of outliers on the output of a covariance estimation. Experimenting on an emerging stock market using three performance metrics shows that our approach significantly improves the sample covariance matrix and also a linear shrinkage to the single-index model to a level of two shrinkage estimations, a shrinkage to identity matrix and shrinkage to constant correlation model.

Coupled Regularized Sample Covariance Matrix Estimator for Multiple Classes

The estimation of covariance matrices of multiple classes with limited training data is a difficult problem. The sample covariance matrix (SCM) is known to perform poorly when the number of variables is large compared to the available number of samples. In order to reduce the mean squared error (MSE) of the SCM, regularized (shrinkage) SCM estimators are often used. In this work, we consider regularized SCM (RSCM) estimators for multiclass problems that couple together two different target matrices for regularization: the pooled (average) SCM of the classes and the scaled identity matrix. Regularization toward the pooled SCM is beneficial when the population covariances are similar, whereas regularization toward the identity matrix guarantees that the estimators are positive definite. We derive the MSE optimal tuning parameters for the estimators as well as propose a method for their estimation under the assumption that the class populations follow (unspecified) elliptical distributions with finite fourth-order moments. The MSE performance of the proposed coupled RSCMs are evaluated with simulations and in a regularized discriminant analysis (RDA) classification set-up on real data. The results based on three different real data sets indicate comparable performance to cross-validation but with a significant speed-up in computation time.

Regularized Kalman filter for brain-computer interfaces using local field potential signals

BackgroundBrain-computer interfaces (BCIs) seek to establish a direct connection from brain to computer, to use in applications such as motor prosthesis control, control of a cursor on the monitor, and so on. Hence, the accuracy of movement decoding from brain signals in BCIs is crucial. The Kalman filter (KF) is often used in BCI systems to decode neural activity and estimate kinetic and kinematic parameters. To use the KF, the state transition matrix, the observation matrix and the covariance matrices of the process and measurement noises must be known in advance, however, in many applications these matrices are not known. Typically, to estimate these parameters, the ordinary least squares method and the sample covariance matrix estimator are used. Our purpose is to enhance the decoding performance of the KF in BCI systems by improving the estimation of the mentioned parameters. New MethodHere, we propose the Regularized Kalman Filter (RKF) which implements two fundamental features: 1) Regularizing the regression estimate of the state equation to improve the estimation of the state transition matrix, and 2) Use of shrinkage method to improve the estimation of the unknown measurement noise covariance matrix. We validated the performance of the proposed method using two datasets of local field potentials obtained from motor cortex of a monkey (Estimation of kinematic parameters during hand movement) and three rats (Estimation of the amount of force applied by hand as a kinetic parameter). ResultsThe results demonstrate that the proposed method outperforms the conventional KF, the KF with feature selection, the Partial least squares, and the Ridge regression approaches.

Random matrix improved covariance estimation for a large class of metrics* *This article is an updated version of: Tiomoko M, Couillet R, Bouchard F and Ginolhac G 2019 Random matrix improved covariance estimation for a large class of metrics Proc. Machine Learning Research vol 97 pp 6254–63.

Relying on recent advances in statistical estimation of covariance distances based on random matrix theory, this article proposes an improved covariance and precision matrix estimation method for a wide family of metrics. This method is shown to largely outperform the sample covariance matrix estimate and to compete with state-of-the-art methods, while at the same time being computationally simpler and faster. Applications to linear and quadratic discriminant analyses also show significant gains, therefore suggesting a practical relevance for statistical machine learning.

Covariance Matrix Estimation under Total Positivity for Portfolio Selection

Abstract Selecting the optimal Markowitz portfolio depends on estimating the covariance matrix of the returns of N assets from T periods of historical data. Problematically, N is typically of the same order as T, which makes the sample covariance matrix estimator perform poorly, both empirically and theoretically. While various other general-purpose covariance matrix estimators have been introduced in the financial economics and statistics literature for dealing with the high dimensionality of this problem, we here propose an estimator that exploits the fact that assets are typically positively dependent. This is achieved by imposing that the joint distribution of returns be multivariate totally positive of order 2 (MTP2). This constraint on the covariance matrix not only enforces positive dependence among the assets but also regularizes the covariance matrix, leading to desirable statistical properties such as sparsity. Based on stock market data spanning 30 years, we show that estimating the covariance matrix under MTP2 outperforms previous state-of-the-art methods including shrinkage estimators and factor models.

Linear-shrinkage-based DOA estimation for coherently distributed sources considering mutual coupling in massive MIMO systems

In this paper, an approach of robust direction-of-arrival (DOA) estimation for coherently distributed (CD) sources considering the influence of array mutual coupling is proposed in a large-scale/massive multiple-input–multiple-output (MIMO) system. By exploiting the special structure of coupling matrix, the proposed approach first eliminates its influence through the inherent mechanism. Considering the realistic scenario that the number of available samples is of the same order of magnitude than the number of sensors in massive MIMO systems, the approach then improves the estimation of the sample covariance matrix (SCM) via the linear shrinkage operation, which is known as an efficient strategy to enhance the estimation of SCM in large-scale uniform linear arrays. Finally, according to the linear shrinkage based minimum description length (LS-MDL) criterion and the improved SCM, DOAs are obtained in closed-form expressions. Numerical results are included for illustrating the effectiveness of the proposed approach.

Post pulse compression and partially adaptive multi‐waveform STAP

The recently developed multi-waveform space-time adaptive processing (µ-STAP) formulation incorporates additional training data into the sample covariance matrix estimate by applying multiple different secondary pulse compression filters to the raw received data, where these filters have a relatively low cross-correlation with the transmitted waveform. The inclusion of this additional training data has been shown to improve robustness to non-homogeneous clutter due to a ‘range smearing’ homogenising effect of the secondary filters. Here, the authors introduce post µ-STAP (Pµ-STAP), a new form of µ-STAP that similarly generates additional training data, albeit after pulse compression has already occurred. In addition, we combine Pµ-STAP with well-known partially adaptive STAP techniques to assess whether the enhanced performance is retained for reduced-dimension operation. Specifically, element-space post-Doppler, beam-space pre-Doppler, and beam-space post-Doppler implementations of Pµ-STAP are evaluated via signal-to-interference-plus-noise ratio analysis and minimum detectable Doppler for different simulated clutter environments.

Unit Circle Rectification of the Minimum Variance Distortionless Response Beamformer

The sample matrix inversion (SMI) beamformer implements Capon's minimum variance distortionless (MVDR) beamforming using the sample covariance matrix (SCM). In a snapshot limited environment, the SCM is poorly conditioned resulting in a suboptimal performance from the SMI beamformer. Imposing structural constraints on the SCM estimate to satisfy known theoretical properties of the ensemble MVDR beamformer mitigates the impact of limited snapshots on the SMI beamformer performance. Toeplitz rectification and bounding the norm of weight vector are common approaches for such constrains. This paper proposes the unit circle rectification technique which constraints the SMI beamformer to satisfy a property of the ensemble MVDR beamformer: for narrowband planewave beamforming on a uniform linear array, the zeros of the MVDR weight array polynomial must fall on the unit circle. Numerical simulations show that the resulting unit circle MVDR (UC MVDR) beamformer frequently improves the suppression of both discrete interferers and white background noise compared to the classic SMI beamformer. Moreover, the UC MVDR beamformer is shown to suppress discrete interferers better than the MVDR beamformer diagonally loaded to maximize the SINR.

Symmetry-Imposed Rectangular Coprime and Nested Arrays for Direction of Arrival Estimation With Multiple Signal Classification

Linear sparse arrays (coprime and nested arrays) have been studied extensively as a means of performing direction of arrival (DoA) estimation while bypassing Nyquist sampling theorem. However, rectangular sparse arrays have few studies, most of which are based in lattice theory. Although the multiple signal classification (MUSIC) alogrithm can be applied to lattice theory-based sparse arrays, fewer DoAs can be estimated for these arrays than for a full array with the same aperture. One contribution of this paper is the formulation of symmetry-imposed rectangular coprime and nested array designs that have wider contiguous lags than lattice-based arrays. Also, existing algorithms for rectangular arrays employ the direct sample covariance matrix estimate, which has low accuracy. Another contribution of this paper is an alternate method for estimating the covariance matrix that ensures the matrix is block-Toeplitz. Using the proposed covariance matrix estimate, we integrate a MUSIC-based DoA estimation method that applies to both full arrays and sparse arrays. The results show that the covariance estimates produced by the proposed method have higher accuracy than the traditional sample covariance estimates. The results also demonstrate that symmetry-imposed sparse arrays have higher resolution than full rectangular arrays with the same number of sensors.

Optimal Shrinkage Covariance Matrix Estimation Under Random Sampling From Elliptical Distributions

This paper considers the problem of estimating a high-dimensional (HD) covariance matrix when the sample size is smaller, or not much larger, than the dimensionality of the data, which could potentially be very large. We develop a regularized sample covariance matrix (RSCM) estimator which can be applied in commonly occurring sparse data problems. The proposed RSCM estimator is based on estimators of the unknown optimal (oracle) shrinkage parameters that yield the minimum mean squared error (MMSE) between the RSCM and the true covariance matrix when the data is sampled from an unspecified elliptically symmetric distribution. We propose two variants of the RSCM estimator which differ in the approach in which they estimate the underlying sphericity parameter involved in the theoretical optimal shrinkage parameter. The performance of the proposed RSCM estimators are evaluated with numerical simulation studies. In particular when the sample sizes are low, the proposed RSCM estimators often show a significant improvement over the conventional RSCM estimator by Ledoit and Wolf (2004). We further evaluate the performance of the proposed estimators in classification and portfolio optimization problems with real data wherein the proposed methods are able to outperform the benchmark methods.

Adaptive Coherency Matrix Estimation for Polarimetric SAR Imagery Based on Local Heterogeneity Coefficients

Polarimetric synthetic aperture radar (SAR) images usually contain a mixture of homogeneous and heterogeneous regions, which makes estimation of the coherency matrix a very challenging task. In this paper, we propose an adaptive coherency matrix estimation method that employs local heterogeneity coefficient and leverages the sample covariance matrix estimation to the homogeneous components and the fixed-point estimation to the heterogeneous components. Evaluations were conducted with synthetic polarimetric data and real-world SAR imagery, including UAVSAR, RADARSAT-2, and ESAR. Our experimental results demonstrated that the heterogeneity coefficient effectively characterizes the scattering property of ground objects, which enables adaptive estimation of the coherency matrix in high-resolution polarimetric SAR imagery. Our method was able to handle single- and multilook polarimetric SAR imagery gracefully. Compared with the sample covariance matrix estimator, the fixed-point estimator, and the Lee sigma filtering, our method achieved the best performance for retaining the spatial structure, suppressing speckles, and preserving polarimetric information of SAR imagery with different degrees of heterogeneity.

Robust Capon beamforming with weighted sparse constraint

Adaptive beamforming is widely used in the fields such as radar, sonar, wireless communication to estimate the parameters of the signal of interest (SOI) at the output of a sensor array by data-adaptive spatial filtering and interference suppression. The standard Capon beamformer (SCB) is a typical adaptive beamforming approach which provides a superior performance by minimizing the array output power while simultaneously maintaining the array response under the assumption of distortionless direction of arrival (DOA). However, the advantages in performance of SCB are obtainable only when the number of snapshots available for the sample covariance matrix estimation is large enough and the direction of the SOI is known accurately. When applied to practical situations where the aforementioned two requirements are not satisfied, SCB will suffer high sidelobe levels and performance degradation in the parameter estimates due to lack of measurements and mismatch in the steering vector.A sparsity-constrained Capon beamformer (SCCB) arises to alleviate these problems. Unlike SCB, the constraint in SCCB is composed of two parts: the original array output power constraint part and the sparse constraint part (?1 norm constraint, encouraging sparse distribution in the array responses). However, if the sparse constraint in SCCB is set too large compared with the array output power constraint part, the responses in the directions of interferences will be influenced, and a tradeoff between the ability to reduce the sidelobe levels and the ability to reject the interferences must be made. Thus, based on the SCCB, a new robust Capon beamformer utilizing a weighted sparse constraint is proposed in this paper. In the proposed method, the sparse constraint part is replaced by a weighted sparse constraint, which is applied only to the sidelobe regions of the beampattern. By doing so, the number of the non-zero elements in the sidelobe response is minimized, resulting in an enhanced mainlobe region and suppressed sidelobe ones.In sparse recovery, the sparse constraint (the l1 norm constraint) does not necessarily enforce democratic penalization, which means that larger coefficients are penalized more heavily than smaller coefficients. Based on such a consideration, a weighting matrix can be constructed to put larger weights in the interferences directions to discourage their responses, and put smaller weights to maintain the responses in the remaining parts of the sidelobe regions. In this paper, the weighting matrix is obtained by utilizing the orthogonality between the signal subspace and the noise subspace. Since the steering vectors corresponding to the interferences and the SOI span the same space as the signal subspace, the inner products between the steering vectors in the interference directions and the noise subspace will produce zeroes ideally. By taking the reciprocals of these inner products, large values will yield in the interference directions while small values are obtained in other directions in the sidelobe regions. Using these values as the weights to the sparse constraint, a beampattern with deeper nulls, lower sidelobes, and better robustness to steering vector mismatch is obtainable as compared with SCB and SCCB. Besides, the output SINR is also effectively improved. Numerical simulations and a water-tank experiment are conducted to demonstrate the effectiveness of the proposed method.

Double zero minimum variance distortionless response beamformer

Adaptive beamformers (ABFs) place deep beampattern notches near interferers to suppress the interferers' power in the ABF output. The common sample matrix inversion (SMI) Minimum Variance Distortionless Response (MVDR) ABF computes the beamformer weights by substituting the sample covariance matrix (SCM) for the unknown ensemble covariance matrix in the MVDR expression. Errors in the SCM estimate of interferer direction due to limited sample support or interferer motion degrade the ABFs ability to suppress the interferer. This presentation exploits array polynomial properties to design a robust ABF. The array polynomial for a uniform linear array beamformer is the z-transform of the array weights. The array polynomial zeros on the unit circle correspond to the beampattern nulls. The proposed double zero (DZ) MVDR ABF solves for the MVDR weights using the SCM for a half-aperture subarray, then convolves the half-aperture weights with themselves to obtain the full aperture ABF weights. The resulting array polynomial for the DZ MVDR ABF has second-order zeros, producing broader and deeper notches in the interferer direction. The DZ MVDR ABF outperforms the SMI MVDR and covariance matrix tapered ABFs in simulations with stationary and moving interferers. [Work supported by ONR 321US.]

On the sample covariance matrix estimator of reduced effective rank population matrices, with applications to fPCA

This work provides a unified analysis of the properties of the sample covariance matrix $\Sigma_{n}$ over the class of $p\times p$ population covariance matrices $\Sigma$ of reduced effective rank $r_{e}(\Sigma)$. This class includes scaled factor models and covariance matrices with decaying spectrum. We consider $r_{e}(\Sigma)$ as a measure of matrix complexity, and obtain sharp minimax rates on the operator and Frobenius norm of $\Sigma_{n}-\Sigma$, as a function of $r_{e}(\Sigma)$ and $\|\Sigma\|_{2}$, the operator norm of $\Sigma$. With guidelines offered by the optimal rates, we define classes of matrices of reduced effective rank over which $\Sigma_{n}$ is an accurate estimator. Within the framework of these classes, we perform a detailed finite sample theoretical analysis of the merits and limitations of the empirical scree plot procedure routinely used in PCA. We show that identifying jumps in the empirical spectrum that consistently estimate jumps in the spectrum of $\Sigma$ is not necessarily informative for other goals, for instance for the selection of those sample eigenvalues and eigenvectors that are consistent estimates of their population counterparts. The scree plot method can still be used for selecting consistent eigenvalues, for appropriate threshold levels. We provide a threshold construction and also give a rule for checking the consistency of the corresponding sample eigenvectors. We specialize these results and analysis to population covariance matrices with polynomially decaying spectra, and extend it to covariance operators with polynomially decaying spectra. An application to fPCA illustrates how our results can be used in functional data analysis.

Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional Gaussian distributions

Differential entropy and log determinant of the covariance matrix of a multivariate Gaussian distribution have many applications in coding, communications, signal processing and statistical inference. In this paper we consider in the high-dimensional setting optimal estimation of the differential entropy and the log-determinant of the covariance matrix. We first establish a central limit theorem for the log determinant of the sample covariance matrix in the high-dimensional setting where the dimension p ( n ) can grow with the sample size n . An estimator of the differential entropy and the log determinant is then considered. Optimal rate of convergence is obtained. It is shown that in the case p ( n ) / n → 0 the estimator is asymptotically sharp minimax. The ultra-high-dimensional setting where p ( n ) > n is also discussed.

Robust Estimates of Covariance Matrices in the Large Dimensional Regime

This article studies the limiting behavior of a class of robust population covariance matrix estimators, originally due to Maronna in 1976, in the regime where both the number of available samples and the population size grow large. Using tools from random matrix theory, we prove that, for sample vectors made of independent entries having some moment conditions, the difference between the sample covariance matrix and (a scaled version of) such robust estimator tends to zero in spectral norm, almost surely. This result can be applied to various statistical methods arising from random matrix theory that can be made robust without altering their first order behavior.