Nonlinear Shrinkage Estimator Research Articles

Nonlinear shrinkage test on a large‐dimensional covariance matrix

This paper is concerned with deriving a new test on a covariance matrix which is based on its nonlinear shrinkage estimator. The distribution of the test statistic is deduced under the null hypothesis in the large‐dimensional setting, that is, when with variables and samples both tending to infinity. The theoretical results are illustrated by means of an extensive simulation study where the new nonlinear shrinkage‐based test is compared with existing approaches, in particular with the commonly used corrected likelihood ratio test, the corrected John test, and the test based on the linear shrinkage approach. It is demonstrated that the new nonlinear shrinkage test possesses better power properties under heteroscedastic alternative.

Cross validation based transfer learning for cross-sectional non-linear shrinkage: A data-driven approach in portfolio optimization

Enhanced covariance estimation approaches, such as (non-)linear shrinkage, are well established in the literature. Non-linear shrinkage estimators generally minimize a certain loss function regarding statistical assumptions about the future covariance matrix. At the same time, the problem of covariance estimation is traditionally considered from a rather restrictive view since the only available data to determine the estimation parameters is given by the return history of the actual portfolio constituents. In this study, we propose a novel and purely data-driven perspective on covariance estimation. We present a non-linear shrinkage estimator that determines the estimation parameters using cross validation to be historically optimal on a disjoint dataset of assets according to the given objective, such as minimum variance or maximum risk-adjusted return. We then transfer the historically optimal estimation parameters learned on the disjoint dataset to the actual covariance estimation problem. Thereby, the sample eigenvalues are corrected in a purely data-driven way, agnostic to theoretically derived parameters. Another benefit of focusing on disjoint data is that we address the problem of limited data availability in high-dimensional estimation problems when the number of assets exceeds the history length. Our empirical evaluation, based on a total of six stock market indices and various problem dimensions, shows that our approach outperforms existing cross-sectional estimators in minimizing variance and maximizing risk-adjusted return. While our study is limited to the cross-section, the method of parameter selection using cross validation and transfer learning can also be combined with other estimators, such as time-series methods.

Matrix means and a novel high-dimensional shrinkage phenomenon

Many statistical settings call for estimating a population parameter, most typically the population mean, based on a sample of matrices. The most natural estimate of the population mean is the arithmetic mean, but there are many other matrix means that may behave differently, especially in high dimensions. Here we consider the matrix harmonic mean as an alternative to the arithmetic matrix mean. We show that in certain high-dimensional regimes, the harmonic mean yields an improvement over the arithmetic mean in estimation error as measured by the operator norm. Counter-intuitively, studying the asymptotic behavior of these two matrix means in a spiked covariance estimation problem, we find that this improvement in operator norm error does not imply better recovery of the leading eigenvector. We also show that a Rao-Blackwellized version of the harmonic mean is equivalent to a linear shrinkage estimator studied previously in the high-dimensional covariance estimation literature, while applying a similar Rao-Blackwellization to regularized sample covariance matrices yields a novel nonlinear shrinkage estimator. Simulations complement the theoretical results, illustrating the conditions under which the harmonic matrix mean yields an empirically better estimate.

A large-dimensional test for cross-sectional anomalies:Efficient sorting revisited

Many researchers seek factors that predict the cross-section of stock returns. In finance, the key is to replicate anomalies by long–short portfolios based on their firm characteristics, with microcap biases alleviated via New York Stock Exchange (NYSE) breakpoints and value-weighted returns. In econometrics, the key is to include a covariance matrix estimator of stock returns for (mimicking) the portfolio construction. This paper marries these two strands of literature in order to test the zoo of cross-sectional anomalies by injecting size controls, basically NYSE breakpoints and value-weighted returns, into efficient sorting. We propose to use a covariance matrix estimator for ultra-high dimensions (up to 5,000) taking into account large, small and microcap stocks. We demonstrate that using a nonlinear shrinkage estimator of the covariance matrix substantially enhances the power of tests for cross-sectional anomalies: On average, t-statistics more than double. Furthermore, the proposed revisited efficient sorting method computes even highly significant factor portfolios net of transaction costs.

Large dynamic covariance matrices: Enhancements based on intraday data

Multivariate GARCH models do not perform well in large dimensions due to the so-called curse of dimensionality. The recent DCC-NL model of Engle et al. (2019) is able to overcome this curse via nonlinear shrinkage estimation of the unconditional correlation matrix. In this paper, we show how performance can be increased further by using open/high/low/close (OHLC) price data instead of simply using daily returns. A key innovation, for the improved modeling of not only dynamic variances but also of dynamic correlations, is the concept of a regularized return, obtained from a volatility proxy in conjunction with a smoothed sign of the observed return.

Optimal Shrinkage-Based Portfolio Selection in High Dimensions

ABSTRACT In this article, we estimate the mean-variance portfolio in the high-dimensional case using the recent results from the theory of random matrices. We construct a linear shrinkage estimator which is distribution-free and is optimal in the sense of maximizing with probability 1 the asymptotic out-of-sample expected utility, that is, mean-variance objective function for different values of risk aversion coefficient which in particular leads to the maximization of the out-of-sample expected utility and to the minimization of the out-of-sample variance. One of the main features of our estimator is the inclusion of the estimation risk related to the sample mean vector into the high-dimensional portfolio optimization. The asymptotic properties of the new estimator are investigated when the number of assets p and the sample size n tend simultaneously to infinity such that . The results are obtained under weak assumptions imposed on the distribution of the asset returns, namely the existence of the moments is only required. Thereafter we perform numerical and empirical studies where the small- and large-sample behavior of the derived estimator is investigated. The suggested estimator shows significant improvements over the existent approaches including the nonlinear shrinkage estimator and the three-fund portfolio rule, especially when the portfolio dimension is larger than the sample size. Moreover, it is robust to deviations from normality.

Design-free estimation of integrated covariance matrices for high-frequency data

The presence of microstructure noise presents a major challenge for estimating the integrated covariance matrices based on high-frequency data. In this paper, we introduce a new method for estimating the integrated covariance matrices that exploits a nonlinear-shrinkage estimation strategy in the high-dimensional setting and a de-noising method in the univariate case. Our estimator is design-free: no structure assumptions are made on the volatility matrix process. We also show that our proposed estimator not only is asymptotically positive definite, but also enjoys a certain desirable estimation efficiency. At last, simulations and financial applications show that the our estimator performs well in comparison with other existing methods.

Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator

Note. The best result in each experiment is highlighted in bold.The optimal solutions of many decision problems such as the Markowitz portfolio allocation and the linear discriminant analysis depend on the inverse covariance matrix of a Gaussian random vector. In “Distributionally Robust Inverse Covariance Estimation: The Wasserstein Shrinkage Estimator,” Nguyen, Kuhn, and Mohajerin Esfahani propose a distributionally robust inverse covariance estimator, obtained by robustifying the Gaussian maximum likelihood problem with a Wasserstein ambiguity set. In the absence of any prior structural information, the estimation problem has an analytical solution that is naturally interpreted as a nonlinear shrinkage estimator. Besides being invertible and well conditioned, the new shrinkage estimator is rotation equivariant and preserves the order of the eigenvalues of the sample covariance matrix. If there are sparsity constraints, which are typically encountered in Gaussian graphical models, the estimation problem can be solved using a sequential quadratic approximation algorithm.

Analytical nonlinear shrinkage of large-dimensional covariance matrices

This paper establishes the first analytical formula for nonlinear shrinkage estimation of large-dimensional covariance matrices. We achieve this by identifying and mathematically exploiting a deep connection between nonlinear shrinkage and nonparametric estimation of the Hilbert transform of the sample spectral density. Previous nonlinear shrinkage methods were of numerical nature: QuEST requires numerical inversion of a complex equation from random matrix theory whereas NERCOME is based on a sample-splitting scheme. The new analytical method is more elegant and also has more potential to accommodate future variations or extensions. Immediate benefits are (i) that it is typically 1000 times faster with basically the same accuracy as QuEST and (ii) that it accommodates covariance matrices of dimension up to 10,000 and more. The difficult case where the matrix dimension exceeds the sample size is also covered.

Large Dynamic Covariance Matrices: Enhancements Based on Intraday Data

Multivariate GARCH models do not perform well in large dimensions due to the so-called curse of dimensionality. The recent DCC-NL model of Engle et al. (2019) is able to overcome this curse via nonlinear shrinkage estimation of the unconditional correlation matrix. In this paper, we show how performance can be increased further by using open/high/low/close (OHLC) price data instead of simply using daily returns. A key innovation, for the improved modeling of not only dynamic variances but also of dynamic correlations, is the concept of a regularized return, obtained from a volatility proxy in conjunction with a smoothed sign of the observed return.

A Large-Dimensional Test for Cross-Sectional Anomalies: Efficient Sorting Revisited

Many researchers seek factors that predict the cross-section of stock returns. In finance, the key is to replicate anomalies by long-short portfolios based on their factor scores, with microcaps alleviated via New York Stock Exchange (NYSE) breakpoints and value-weighted returns. In econometrics, the key is to include a covariance matrix estimator of stock returns for the (mimicking) portfolio construction. This paper marries these two strands of literature in order to test the zoo of cross-sectional anomalies by injecting size controls, basically NYSE breakpoints and value-weighted returns, into efficient sorting. Thus, we propose to use a covariance matrix estimator for ultra-high dimensions (up to 5,000) taking into account large, small and microcap stocks. We demonstrate that using a nonlinear shrinkage estimator of the covariance matrix substantially enhances the power of tests for cross-sectional anomalies: On average, ‘Student’ t-statistics more than double.

A new procedure for resampled portfolio with shrinkaged covariance matrix

ABSTRACTDealing with estimation error is an important issue when we implement the mean–variance paradigm for portfolio construction. To tackle the problem, two approaches are proposed in literature, the portfolio resampling technique introduced by Michuad and the well-known shrinkaged covariance matrix method. There are certain evidences on the advantages of shrinkaged covariance over portfolio resampling, however, it is unclear whether a combination of the two approaches could produce a better performance compared with using shrinkaged covariance alone. In this paper, we propose a new algorithm to integrated linear or nonlinear shrinkage estimation with resampled portfolio to achieve a further improvement. Our method are demonstrated via extensive simulation and application in active portfolio management process.

Direct Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices

This paper establishes the first analytical formula for optimal nonlinear shrinkage of large-dimensional covariance matrices. We achieve this by identifying and mathematically exploiting a deep connection between nonlinear shrinkage and nonparametric estimation of the Hilbert transform of the sample spectral density. Previous nonlinear shrinkage methods were numerical: QuEST requires numerical inversion of a complex equation from random matrix theory whereas NERCOME is based on a sample-splitting scheme. The new analytical approach is more elegant and also has more potential to accommodate future variations or extensions. Immediate benefits are that it is typically 1,000 times faster with the same accuracy, and accommodates covariance matrices of dimension up to 10, 000. The difficult case where the matrix dimension exceeds the sample size is also covered.

Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks

Markowitz (1952) portfolio selection requires an estimator of the covariance matrix of returns. To address this problem, we promote a nonlinear shrinkage estimator that is more flexible than previous linear shrinkage estimators and has just the right number of free parameters (that is, the Goldilocks principle). This number is the same as the number of assets. Our nonlinear shrinkage estimator is asymptotically optimal for portfolio selection when the number of assets is of the same magnitude as the sample size. In backtests with historical stock return data, it performs better than previous proposals and, in particular, it dominates linear shrinkage.

Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks

Markowitz (1952) portfolio selection requires estimates of (i) the vector of expected returns and (ii) the covariance matrix of returns. Many proposals to address the first question exist already. This paper addresses the second question. We promote a new nonlinear shrinkage estimator of the covariance matrix that is more flexible than previous linear shrinkage estimators and has ‘just the right number’ of free parameters (that is, the Goldilocks principle). In a stylized setting, the nonlinear shrinkage estimator is asymptotically optimal for portfolio selection. In addition to theoretical analysis, we establish superior real-life performance of our new estimator using backtest exercises.

Nonlinear shrinkage estimation of large integrated covariance matrices

Integrated covariance matrices arise in intraday models of asset returns, which allow volatility to change over the trading day. When the number of assets is large, the natural estimator of such a matrix suffers from bias due to extreme eigenvalues. We introduce a novel nonlinear shrinkage estimator for the integrated covariance matrix which shrinks the extreme eigenvalues of a realized covariance matrix back to an acceptable level, and enjoys a certain asymptotic efficiency when the number of assets is of the same order as the number of data points. Novel maximum exposure and actual risk bounds are derived when our estimator is used in constructing the minimum variance portfolio. In simulations and a real-data analysis, our estimator performs favourably in comparison with other methods.

Non-linear shrinkage estimation of large-scale structure covariance

Abstract In many astrophysical settings, covariance matrices of large data sets have to be determined empirically from a finite number of mock realizations. The resulting noise degrades inference and precludes it completely if there are fewer realizations than data points. This work applies a recently proposed non-linear shrinkage estimator of covariance to a realistic example from large-scale structure cosmology. After optimizing its performance for the usage in likelihood expressions, the shrinkage estimator yields subdominant bias and variance comparable to that of the standard estimator with a factor of ∼50 less realizations. This is achieved without any prior information on the properties of the data or the structure of the covariance matrix, at a negligible computational cost.

Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks

Markowitz (1952) portfolio selection requires estimates of (i) the vector of expected returns and (ii) the covariance matrix of returns. Many successful proposals to address the first estimation problem exist by now. This paper addresses the second estimation problem. We promote a nonlinear shrinkage estimator of the covariance matrix that is more flexible than previous linear shrinkage estimators and has 'just the right number' of free parameters to estimate (that is, the Goldilocks principle). It turns out that this number is the same as the number of assets in the investment universe. Under certain high-level assumptions, we show that our nonlinear shrinkage estimator is asymptotically optimal for portfolio selection in the setting where the number of assets is of the same magnitude as the sample size. For example, this is the relevant setting for mutual fund managers who invest in a large universe of stocks. In addition to theoretical analysis, we study the real-life performance of our new estimator using backtest exercises on historical stock return data. We find that it performs better than previous proposals for portfolio selection from the literature and, in particular, that it dominates linear shrinkage.

Nonlinear shrinkage estimation of large-dimensional covariance matrices

Many statistical applications require an estimate of a covariance matrix and/or its inverse. When the matrix dimension is large compared to the sample size, which happens frequently, the sample covariance matrix is known to perform poorly and may suffer from ill-conditioning. There already exists an extensive literature concerning improved estimators in such situations. In the absence of further knowledge about the structure of the true covariance matrix, the most successful approach so far, arguably, has been shrinkage estimation. Shrinking the sample covariance matrix to a multiple of the identity, by taking a weighted average of the two, turns out to be equivalent to linearly shrinking the sample eigenvalues to their grand mean, while retaining the sample eigenvectors. Our paper extends this approach by considering nonlinear transformations of the sample eigenvalues. We show how to construct an estimator that is asymptotically equivalent to an oracle estimator suggested in previous work. As demonstrated in extensive Monte Carlo simulations, the resulting bona fide estimator can result in sizeable improvements over the sample covariance matrix and also over linear shrinkage.

Entropy-based correlated shrinkage of spatial random processes

This paper proposes a two-stage correlated non-linear shrinkage estimation methodology for spatial random processes. A block hard thresholding design, based on Shannon’s entropy, is formulated in the first stage. The thresholding design is adaptive to each resolution level, because it depends on the empirical distribution function of the mutual information ratios between empirical wavelet blocks and the random variables of interest, at each scale. In the second stage, a global correlated (inter- and intra-scale) shrinkage is applied to approximate the values of interest of the underlying spatial process. Additionally, a simulation study is developed, in the Gaussian context, to analyze the sensitivity, measured by empirical stochastic ordering, of the entropy-based block hard thresholding stage in relation to the parameters characterizing local variability (fractality) and dependence range of the spatial process of interest, the noise level, and the design of the region of interest.