Linear Shrinkage Estimators Research Articles

Regularized fingerprinting in detection and attribution of climate change with weight matrix optimizing the efficiency in scaling factor estimation

Detection and attribution analyses play a central role in establishing the causal effect of human activities on global warming. The most commonly used method in such analyses, optimal fingerprinting, is a multiple regression where each covariate has a measurement error whose covariance matrix is the same as that of the regression error up to a known scale. Inferences about the regression coefficients are critical not only for making statements about detection and attribution but also for quantifying the uncertainty in important outcomes derived from detection and attribution analyses. When there are no errors-in-variables (EIV), the optimal weight matrix in estimating the regression coefficients is the precision matrix of the regression error. This matrix, however, is never known and has to be estimated from climate model simulations with regularization. The consequence is that the optimal fingerprinting method is not optimal, as believed in practice. We construct a weight matrix by inverting a nonlinear shrinkage estimate of the error covariance matrix that minimizes loss functions directly targeting the uncertainty of the resulting regression coefficient estimator. The resulting estimator of the regression coefficients is asymptotically optimal as the sample size of the climate model simulations and the matrix dimension go to infinity together with a limiting ratio. When EIVs are present, the estimator of the regression coefficients, based on the proposed weight matrix, is asymptotically more efficient than that based on the inverse of the existing linear shrinkage estimator of the error covariance matrix. The performance of the method is confirmed in finite sample simulation studies mimicking realistic situations in terms of the length of the confidence intervals and empirical coverage rates for the regression coefficients. In an application to detection and attribution analyses of the mean temperature at different spatial scales, the method yielded shorter confidence intervals which are important for such analyses in practice.

Estimation of Large-Dimensional Covariance Matrices via Second-Order Stein-Type Regularization

This paper tackles the problem of estimating the covariance matrix in large-dimension and small-sample-size scenarios. Inspired by the well-known linear shrinkage estimation, we propose a novel second-order Stein-type regularization strategy to generate well-conditioned covariance matrix estimators. We model the second-order Stein-type regularization as a quadratic polynomial concerning the sample covariance matrix and a given target matrix, representing the prior information of the actual covariance structure. To obtain available covariance matrix estimators, we choose the spherical and diagonal target matrices and develop unbiased estimates of the theoretical mean squared errors, which measure the distances between the actual covariance matrix and its estimators. We formulate the second-order Stein-type regularization as a convex optimization problem, resulting in the optimal second-order Stein-type estimators. Numerical simulations reveal that the proposed estimators can significantly lower the Frobenius losses compared with the existing Stein-type estimators. Moreover, a real data analysis in portfolio selection verifies the performance of the proposed estimators.

Improved Large Covariance Matrix Estimation Based on Efficient Convex Combination and Its Application in Portfolio Optimization

The estimation of the covariance matrix is an important topic in the field of multivariate statistical analysis. In this paper, we propose a new estimator, which is a convex combination of the linear shrinkage estimation and the rotation-invariant estimator under the Frobenius norm. We first obtain the optimal parameters by using grid search and cross-validation, and then, we use these optimal parameters to demonstrate the effectiveness and robustness of the proposed estimation in the numerical simulations. Finally, in empirical research, we apply the covariance matrix estimation to the portfolio optimization. Compared to the existing estimators, we show that the proposed estimator has better performance and lower out-of-sample risk in portfolio optimization.

Sparsistent filtering of comovement networks from high-dimensional data

Network filtering is a technique to isolate core subnetworks of large and complex interconnected systems, which has recently found many applications in financial, biological, physical and technological networks among others. We introduce a new technique to filter large dimensional networks arising out of dynamical behavior of the constituent nodes, exploiting their spectral properties. As opposed to the well known network filters that rely on preserving key topological properties of the realized network, our method treats the spectrum as the fundamental object and preserves spectral properties. Applying asymptotic theory of high-dimensional covariance matrix estimation, we show that the proposed filter can be tuned to interpolate between zero filtering to maximal filtering that induces sparsity via thresholding, while having the least spectral distance from a consistent (non-)linear shrinkage estimator. We demonstrate the application of our proposed filter by applying it to covariance networks constructed from financial data, to extract core subnetworks embedded in full networks.

Matrix means and a novel high-dimensional shrinkage phenomenon

Many statistical settings call for estimating a population parameter, most typically the population mean, based on a sample of matrices. The most natural estimate of the population mean is the arithmetic mean, but there are many other matrix means that may behave differently, especially in high dimensions. Here we consider the matrix harmonic mean as an alternative to the arithmetic matrix mean. We show that in certain high-dimensional regimes, the harmonic mean yields an improvement over the arithmetic mean in estimation error as measured by the operator norm. Counter-intuitively, studying the asymptotic behavior of these two matrix means in a spiked covariance estimation problem, we find that this improvement in operator norm error does not imply better recovery of the leading eigenvector. We also show that a Rao-Blackwellized version of the harmonic mean is equivalent to a linear shrinkage estimator studied previously in the high-dimensional covariance estimation literature, while applying a similar Rao-Blackwellization to regularized sample covariance matrices yields a novel nonlinear shrinkage estimator. Simulations complement the theoretical results, illustrating the conditions under which the harmonic matrix mean yields an empirically better estimate.

Improved shrinkage estimators in the beta regression model with application in econometric and educational data

Although beta regression is a very useful tool to model the continuous bounded outcome variable with some explanatory variables, however, in the presence of multicollinearity, the performance of the maximum likelihood estimates for the estimation of the parameters is poor. In this paper, we propose improved shrinkage estimators via Liu estimator to obtain more efficient estimates. Therefore, we defined linear shrinkage, pretest, shrinkage pretest, Stein and positive part Stein estimators to estimate of the parameters in the beta regression model, when some of them have not a significant effect to predict the outcome variable so that a sub-model may be sufficient. We derived the asymptotic distributional biases, variances, and then we conducted extensive Monte Carlo simulation study to obtain the performance of the proposed estimation strategy. Our results showed a great benefit of the new methodologies for practitioners specifically in the applied sciences. We concluded the paper with two real data analysis from economics and education.

A random matrix theory approach to damping in deep learning

We conjecture that the inherent difference in generalisation between adaptive and non-adaptive gradient methods in deep learning stems from the increased estimation noise in the flattest directions of the true loss surface. We demonstrate that typical schedules used for adaptive methods (with low numerical stability or damping constants) serve to bias relative movement towards flat directions relative to sharp directions, effectively amplifying the noise-to-signal ratio and harming generalisation. We further demonstrate that the numerical damping constant used in these methods can be decomposed into a learning rate reduction and linear shrinkage of the estimated curvature matrix. We then demonstrate significant generalisation improvements by increasing the shrinkage coefficient, closing the generalisation gap entirely in both logistic regression and several deep neural network experiments. Extending this line further, we develop a novel random matrix theory based damping learner for second order optimisers inspired by linear shrinkage estimation. We experimentally demonstrate our learner to be very insensitive to the initialised value and to allow for extremely fast convergence in conjunction with continued stable training and competitive generalisation. We also find that our derived method works well with adaptive gradient methods such as Adam.

More accurate estimation of cell composition in bulk expression through robust integration of single-cell information.

The rapid single-cell transcriptomic technology developments have led to an increasing interest in cellular heterogeneity within cell populations. Although cell-type proportions can be obtained directly from single-cell RNA sequencing (scRNA-seq), it is costly and not feasible in every study. Alternatively, with fewer experimental complications, cell-type compositions are characterized from bulk RNA-seq data. Many computational tools have been developed and reported in the literature. However, they fail to appropriately incorporate the covariance structures in both scRNA-seq and bulk RNA-seq datasets in use. We present a covariance-based single-cell decomposition (CSCD) method that estimates cell-type proportions in bulk data through building a reference expression profile based on a single-cell data, and learning gene-specific bulk expression transformations using a constrained linear inverse model. The approach is similar to Bisque, a cell-type decomposition method that was recently developed. Bisque is limited to a univariate model, thus unable to incorporate gene-gene correlations into the analysis. We introduce a more advanced model that successfully incorporates the covariance structures in both scRNA-seq and bulk RNA-seq datasets into the analysis, and fixes the collinearity issue by utilizing a linear shrinkage estimation of the corresponding covariance matrices. We applied CSCD to several publicly available datasets and measured the performance of CSCD, Bisque and six other common methods in the literature. Our results indicate that CSCD is more accurate and comprehensive than most of the existing methods. The R package is available on https://github.com/empiricalbayes/CSCDRNA.

Optimal Shrinkage-Based Portfolio Selection in High Dimensions

ABSTRACT In this article, we estimate the mean-variance portfolio in the high-dimensional case using the recent results from the theory of random matrices. We construct a linear shrinkage estimator which is distribution-free and is optimal in the sense of maximizing with probability 1 the asymptotic out-of-sample expected utility, that is, mean-variance objective function for different values of risk aversion coefficient which in particular leads to the maximization of the out-of-sample expected utility and to the minimization of the out-of-sample variance. One of the main features of our estimator is the inclusion of the estimation risk related to the sample mean vector into the high-dimensional portfolio optimization. The asymptotic properties of the new estimator are investigated when the number of assets p and the sample size n tend simultaneously to infinity such that . The results are obtained under weak assumptions imposed on the distribution of the asset returns, namely the existence of the moments is only required. Thereafter we perform numerical and empirical studies where the small- and large-sample behavior of the derived estimator is investigated. The suggested estimator shows significant improvements over the existent approaches including the nonlinear shrinkage estimator and the three-fund portfolio rule, especially when the portfolio dimension is larger than the sample size. Moreover, it is robust to deviations from normality.

Improved shrinkage estimators in zero-inflated negative binomial regression model

‎Zero-inflated negative binomial model is an appropriate choice to model count response variables with excessive zeros and over-dispersion simultaneously. ‎This paper addressed parameter estimation in the zero-inflated negative binomial model when there are many parameters, ‎so that some of them have not influence on the response variable. ‎We proposed parameter estimation based on the linear shrinkage, ‎pretest, ‎shrinkage pretest, ‎Stein-type, ‎and positive Stein-type estimators. ‎We obtained the asymptotic distributional biases and risks of the suggested estimators theoretically. ‎‎We also conducted a Monte Carlo simulation study ‎to compare the performance of each estimator with the unrestricted estimator using simulated relative efficiency (SRE) criterion.‎‎ ‎The results reveal that the SREs of proposed estimators are higher than the unrestricted estimator. ‎The suggested estimators were applied to the wildlife fish data to appraise their performance.

Linear shrinkage estimation of high-dimensional means

In estimation of a mean vector, consider the case that the mean vector is suspected to be in one or two general linear subspaces. Then it is reasonable to shrink a sample mean vector toward the restricted estimators on the linear subspaces. Motivated from a standard Bayesian argument, we propose single and double shrinkage estimators in which their optimal weights are estimated consistently in high dimension without assuming any specific distributions. Asymptotic relative improvement in risk of shrinkage estimators over the sample mean vector is derived in high dimension, and it is shown that the gain in improvement by shrinkage depends on the linear subspace. Finally, the performance of the linear shrinkage estimators is numerically investigated by simulation.

Source enumeration via RMT estimator with adaptive decision criterion based on linear shrinkage estimation of noise eigenvalues using relatively few samples

Estimating the number of signals embedded in noise is a fundamental problem in array signal processing. The classic random matrix theory (RMT) estimator based on RMT does not consider the bias term of the expectation of the eigenvalue being tested, and thus, its detection performance will be affected by this bias term, and it also suffers from noise uncertainty in its decision threshold. In order to overcome these problems, more accurate expressions for the distribution and the bias term of the expectation of the eigenvalue being tested are firstly derived by utilising the linear shrinkage (LS) technique. Then, a novel LS-RMT estimator is proposed by incorporating the bias term of the expectation of the eigenvalue being tested into the decision criterion of the RMT estimator. Moreover, both the increased under-estimation probability and the increased over-estimation probability of the RMT estimator incurred by this bias term are derived according to the sign of this bias term. Based on these results, a novel LS-RMT estimator with adaptive decision criterion (termed as ‘LS-RMT-ADC estimator’) is proposed by utilising the LS-RMT estimator, and the proposed LS-RMT-ADC estimator can adaptively select its decision threshold and determine the noise variance in the selected decision threshold. Therefore, it can overcome both the higher under-estimation probability and the higher over-estimation probability of the RMT estimator, and can also avoid the noise uncertainty in the decision threshold of the RMT estimator. Finally, simulation results are presented to show that the LS-RMT-ADC estimator outperforms the existing estimators.

Using shrinkage strategies to estimate fixed effects in zero-inflated negative binomial mixed model

In this paper, we address the estimation of fixed effects parameters in the zero-inflated negative binomial mixed model based on shrinkage estimators, namely linear shrinkage, pretest, shrinkage pretest, shrinkage, and positive-shrinkage estimators when the random effects are considered as nuisance parameters. We compare the performance of the shrinkage estimators to unrestricted and restricted estimators when certain prior subspace information is available. The asymptotic distributional biases and risks of the proposed estimators are obtained. We also conduct a Monte Carlo simulation study to compare the performance of each estimator in the sense of simulated relative efficiency. The results of simulation study show that the proposed estimation strategies perform strongly better than the maximum likelihood method. Finally, proposed methodologies are applied to a real dataset to appraise their performances.

Principal regression for high dimensional covariance matrices.

This manuscript presents an approach to perform generalized linear regression with multiple high dimensional covariance matrices as the outcome. In many areas of study, such as resting-state functional magnetic resonance imaging (fMRI) studies, this type of regression can be utilized to characterize variation in the covariance matrices across units. Model parameters are estimated by maximizing a likelihood formulation of a generalized linear model, conditioning on a well-conditioned linear shrinkage estimator for multiple covariance matrices, where the shrinkage coefficients are proposed to be shared across matrices. Theoretical studies demonstrate that the proposed covariance matrix estimator is optimal achieving the uniformly minimum quadratic loss asymptotically among all linear combinations of the identity matrix and the sample covariance matrix. Under certain regularity conditions, the proposed estimator of the model parameters is consistent. The superior performance of the proposed approach over existing methods is illustrated through simulation studies. Implemented to a resting-state fMRI study acquired from the Alzheimer's Disease Neuroimaging Initiative, the proposed approach identified a brain network within which functional connectivity is significantly associated with Apolipoprotein E ε4, a strong genetic marker for Alzheimer's disease.

Oops! I Shrunk the Sample Covariance Matrix Again: Blockbuster Meets Shrinkage

Abstract Existing shrinkage techniques struggle to model the covariance matrix of asset returns in the presence of multiple-asset classes. Therefore, we introduce a Blockbuster shrinkage estimator that clusters the covariance matrix accordingly. Besides the definition and derivation of a new asymptotically optimal linear shrinkage estimator, we propose an adaptive Blockbuster algorithm that clusters the covariance matrix even if the (number of) asset classes are unknown and change over time. It displays superior all-around performance on historical data against a variety of state-of-the-art linear shrinkage competitors. Additionally, we find that for small- and medium-sized investment universes the proposed estimator outperforms even recent nonlinear shrinkage techniques. Hence, this new estimator can be used to deliver more efficient portfolio selection and detection of anomalies in the cross-section of asset returns. Furthermore, due to the general structure of the proposed Blockbuster shrinkage estimator, the application is not restricted to financial problems.

Ridge-type linear shrinkage estimation of the mean matrix of a high-dimensional normal distribution

The estimation of the mean matrix of the multivariate normal distribution is addressed in the high dimensional setting. Efron–Morris-type linear shrinkage estimators with ridge modification for the precision matrix instead of the Moore–Penrose generalized inverse are considered, and the weights in the ridge-type linear shrinkage estimators are estimated in terms of minimizing the Stein unbiased risk estimators under the quadratic loss. It is shown that the ridge-type linear shrinkage estimators with the estimated weights are minimax, and that the estimated weights and the loss function with these estimated weights are asymptotically equal to the optimal counterparts in the Bayesian model with high dimension by using the random matrix theory. The performance of the ridge-type linear shrinkage estimators is numerically compared with the existing estimators including the Efron–Morris and James–Stein estimators.

Linear shrinkage estimation of the variance of a distribution with unknown mean

In estimation of the variance of a distribution with unknown mean, the paper suggests the linear shrinkage estimators motivated from a Bayesian perspective. The so-called Stein’s truncated estimator of the variance can be derived as the linear shrinkage estimator when the distribution is normal. The method of the linear shrinkage estimation is extended to non normal distributions and to linear regression models. The linear shrinkage estimator with the optimal weight estimate, derived without assuming normality, is shown to have a good numerical performance for several distributions.

A new procedure for resampled portfolio with shrinkaged covariance matrix

ABSTRACTDealing with estimation error is an important issue when we implement the mean–variance paradigm for portfolio construction. To tackle the problem, two approaches are proposed in literature, the portfolio resampling technique introduced by Michuad and the well-known shrinkaged covariance matrix method. There are certain evidences on the advantages of shrinkaged covariance over portfolio resampling, however, it is unclear whether a combination of the two approaches could produce a better performance compared with using shrinkaged covariance alone. In this paper, we propose a new algorithm to integrated linear or nonlinear shrinkage estimation with resampled portfolio to achieve a further improvement. Our method are demonstrated via extensive simulation and application in active portfolio management process.

Oops! I Shrunk the Sample Covariance Matrix Again: Blockbuster Meets Shrinkage

Existing shrinkage techniques struggle to model the covariance matrix of asset returns in the presence of multiple-asset classes. Therefore, we introduce a Blockbuster shrinkage estimator that clusters the covariance matrix accordingly. Besides the definition and derivation of a new asymptotically optimal linear shrinkage estimator we propose an adaptive Blockbuster algorithm that clusters the covariance matrix even if the (number of) asset classes are unknown and change over time. It displays superior all-around performance on historical data against a variety of state-of-the-art linear shrinkage competitors. Additionally, we find that for small and medium-sized investment universes the proposed estimator outperforms even recent nonlinear shrinkage techniques. Hence, this new estimator can be used to deliver more efficient portfolio selection and detection of anomalies in the cross-section of asset returns. Furthermore, due to the general structure of the proposed Blockbuster shrinkage estimator the application is not restricted to financial problems.

Weighted covariance matrix estimation

The paper proposes a cross-validated linear shrinkage estimation for population covariance matrices. Moreover we also propose a novel weighted estimator based on the thresholding and shrinkage methods for high dimensional datasets. It is applicable to a wider scope of different structures of covariance matrices. Some theoretical results about the cross-validated shrinkage method and weighted covariance estimation methods are also developed. The finite-sample performance of the proposed methods is illustrated through extensive simulations and real data analysis.