Empirical Correlation Matrices Research Articles

A Perspective on Correlation-Based Financial Networks and Entropy Measures

In this mini-review, we critically examine the recent work done on correlation-based networks in financial systems. The structure of empirical correlation matrices constructed from the financial market data changes as the individual stock prices fluctuate with time, showing interesting evolutionary patterns, especially during critical events such as market crashes, bubbles, etc. We show that the study of correlation-based networks and their evolution with time is useful for extracting important information of the underlying market dynamics. Also, we present our perspective on the use of recently-developed entropy measures, such as structural entropy and eigen-entropy, for continuous monitoring of correlation-based networks.

Recovery of original individual person data (IPD) inferences from empirical IPD summaries only: Applications to distributed computing under disclosure constraints.

There are many settings where individual person data (IPD) are not available, due to privacy or technical reasons, and one must work with IPD proxies, such as summary statistics, to approximate original IPD inferences, that is, the results of statistical analyses that would ideally have been performed on individual-level data. For instance, in a distributed computing setting, as implemented in the DataSHIELD software framework, different centers can only share IPD proxies to obtain pooled IPD inferences. Such privacy requirements limit the scope of statistical investigation. For example, it can be challenging to perform between-center random-effect regression models. To increase modeling freedom we propose a method that only uses simple nondisclosive summaries of the original IPD as input, such as empirical marginal moments and correlation matrices, and generates artificial data compatible with those summary features. Specifically, data are generated from a Gaussian copula with marginal and joint components specified by the above summaries. The goal is to reproduce original IPD features in the artificial data, such that original IPD inferences are recovered from the artificial data. In an application example, and through simulations, we show that we can recover estimates of a multivariable IPD random-effect logistic regression, from artificial data generated via the Gaussian copula using the above IPD summaries, suggesting the proposed approach provides a generally applicable strategy for distributed computing settings with data protection constraints.

Estimation of Theory-Implied Correlation Matrices

Correlation matrices are ubiquitous in finance. Some key applications include portfolio construction, risk management, and factor/style analysis. Correlation matrices are usually estimated from historical empirical observations or derived from historically estimated factors. It is widely acknowledged that empirical correlation matrices: (a) have poor numerical properties that lead to unreliable estimators; and (b) have poor predictive power. Additionally, factor-based correlation matrices have their own caveats. In particular, estimated factors are typically non-hierarchical and do not allow for interactions at different levels. This contravenes the fact that financial instruments typically exhibit a nested cluster structure (e.g., MSCI’s GICS levels 1-4). This paper introduces a machine learning (ML) algorithm to estimate forward-looking correlation matrices implied by economic theory. Given a particular theoretical representation of the hierarchical structure that governs a universe of securities, the method fits the correlation matrix that complies with that theoretical representation of the future. This particular use case demonstrates how, contrary to popular perception, ML solutions are not black-boxes, and can be applied effectively to develop and test economic theories.

Overlaps between eigenvectors of correlated random matrices

We obtain general, exact formulas for the overlaps between the eigenvectors of large correlated random matrices, with additive or multiplicative noise. These results have potential applications in many different contexts, from quantum thermalisation to high dimensional statistics. We find that the overlaps only depend on measurable quantities, and do not require the knowledge of the underlying "true" (noiseless) matrices. We apply our results to the case of empirical correlation matrices, that allow us to estimate reliably the width of the spectrum of the true correlation matrix, even when the latter is very close to the identity. We illustrate our results on the example of stock returns correlations, that clearly reveal a non trivial structure for the bulk eigenvalues. We also apply our results to the problem of matrix denoising in high dimension.

Scaling in the eigenvalue fluctuations of correlation matrices

The spectra of empirical correlation matrices, constructed from multivariate data, are widely used in many areas of sciences, engineering and social sciences as a tool to understand the information contained in typically large datasets. In the last two decades, random matrix theory-based tools such as the nearest neighbour eigenvalue spacing and eigenvector distributions have been employed to extract the significant modes of variability present in such empirical correlations. In this work, we present an alternative analysis in terms of the recently introduced spacing ratios, which does not require the cumbersome unfolding process. It is shown that the higher order spacing ratio distributions for the Wishart ensemble of random matrices, characterized by the Dyson index $\beta$, is related to the first order spacing ratio distribution with a modified value of co-dimension $\beta'$. This scaling is demonstrated for Wishart ensemble and also for the spectra of empirical correlation matrices drawn from the observed stock market and atmospheric pressure data. Using a combination of analytical and numerics, such scalings in spacing distributions are also discussed.

RMSEA, CFI, and TLI in structural equation modeling with ordered categorical data: The story they tell depends on the estimation methods.

In structural equation modeling, application of the root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker-Lewis index (TLI) highly relies on the conventional cutoff values developed under normal-theory maximum likelihood (ML) with continuous data. For ordered categorical data, unweighted least squares (ULS) and diagonally weighted least squares (DWLS) based on polychoric correlation matrices have been recommended in previous studies. Although no clear suggestions exist regarding the application of these fit indices when analyzing ordered categorical variables, practitioners are still tempted to adopt the conventional cutoff rules. The purpose of our research was to answer the question: Given a population polychoric correlation matrix and a hypothesized model, if ML results in a specific RMSEA value (e.g., .08), what is the RMSEA value when ULS or DWLS is applied? CFI and TLI were investigated in the same fashion. Both simulated and empirical polychoric correlation matrices with various degrees of model misspecification were employed to address the above question. The results showed that DWLS and ULS lead to smaller RMSEA and larger CFI and TLI values than does ML for all manipulated conditions, regardless of whether or not the indices are scaled. Applying the conventional cutoffs to DWLS and ULS, therefore, has a pronounced tendency not to discover model-data misfit. Discussions regarding the use of RMSEA, CFI, and TLI for ordered categorical data are given.

Distribution of the smallest eigenvalue in complex and real correlated Wishart ensembles

For the correlated Gaussian–Wishart ensemble we compute the distribution of the smallest eigenvalue and a related gap probability. We obtain exact results for the complex (β = 2) and for the real case (β = 1). For a particular set of empirical correlation matrices we find universality in the spectral density, for both real and complex ensembles and all kinds of rectangularity. We calculate the asymptotic and universal results for the gap probability and the distribution of the smallest eigenvalue. We use the supersymmetry method, in particular the generalized Hubbard–Stratonovich transformation and superbosonization.

Robustness of a Noise Filtering Procedure for Correlation Matrices: A. Filtering of Deliberately Introduced Noise from Empirical Correlation Matrices

Robustness of a Noise Filtering Procedure for Correlation Matrices: A. Filtering of Deliberately Introduced Noise from Empirical Correlation Matrices

Fine structure of spectral properties for random correlation matrices: An application to financial markets

We study some properties of eigenvalue spectra of financial correlation matrices. In particular, we investigate the nature of the large eigenvalue bulks which are observed empirically, and which have often been regarded as a consequence of the supposedly large amount of noise contained in financial data. We challenge this common knowledge by acting on the empirical correlation matrices of two data sets with a filtering procedure which highlights some of the cluster structure they contain, and we analyze the consequences of such filtering on eigenvalue spectra. We show that empirically observed eigenvalue bulks emerge as superpositions of smaller structures, which in turn emerge as a consequence of cross correlations between stocks. We interpret and corroborate these findings in terms of factor models, and we compare empirical spectra to those predicted by random matrix theory for such models.

Empirical properties of large covariance matrices

The salient properties of large empirical covariance and correlation matrices are studied for three datasets of size 54, 55 and 330. The covariance is defined as a simple cross product of the returns, with weights that decay logarithmically slowly. The key general properties of the covariance matrices are the following. The spectrum of the covariance is very static, except for the top three to 10 eigenvalues, and decay exponentially fast toward zero. The mean spectrum and spectral density show no particular feature that would separate ‘meaningful’ from ‘noisy’ eigenvalues. The spectrum of the correlation is more static, with three to five eigenvalues that have distinct dynamics. The mean projector of rank k on the leading subspace shows that a large part of the dynamics occurs in the eigenvectors. Together, this implies that the reduction of the covariance to a few leading static eigenmodes misses most of the dynamics. Finally, all the analysed properties of the dynamics of the covariance and correlation are similar. This indicates that a covariance estimator correctly evaluates both volatilities and correlations, and separate estimators are not required.

Extracting Correlations from the Market: New Correlation Parameterizations and the Calibration of a Stochastic Volatility LMM to CMS Spread Options

We present a new generic method for constructing correlation parameterizations that are always positive definite, and derive new flexible parametric forms.Furthermore, we use the CMS spread option pricing formula from Kiesel & Lutz to calibrate a stochastic volatility LMM to caplets, swaptions and CMS spread options, and in this way extract the implied correlation information available from the market.We investigate the performance of several correlation parameterizations and compare the implied correlation matrices with the corresponding empirical correlation matrices.

Feature evaluation in fMRI data using random matrix theory

Quantitative descriptors of intrinsic properties of fMRI data can be obtained from the theory of random matrices. We study data reduction based on the comparison of empirical correlation matrices with a suitably chosen ensemble of random positive matrices. Accordingly, data dimensions can be discarded if the quality of fit of the data spectrum deviates locally from the theoretical result, which is derived here analytically. Further, more complex quantities such as the number variance are discussed and shown to be potentially useful in an analogous manner.

FINANCIAL AND OTHER SPATIO-TEMPORAL TIME SERIES: LONG-RANGE CORRELATIONS AND SPECTRAL PROPERTIES

In this paper, we review some of the properties of financial and other spatio-temporal time series generated from coupled map lattices, GARCH(1,1) processes and random processes (for which analytical results are known). We use the Hurst exponent (R/S analysis) and detrended fluctuation analysis as the tools to study the long-time correlations in the time series. We also compare the eigenvalue properties of the empirical correlation matrices, especially in relation to random matrices.

Random matrix approach to multivariate correlations: some limiting cases.

Recent advances have shown that the empirical correlation matrices of dynamical systems can be modeled as random matrices, for most part, chosen from an appropriate ensemble of the random matrix theory (RMT). In this work, we study certain limiting cases where this approach could potentially break down. Using a combination of analytical and numerical tools, we especially study the eigenvalue density and its spacing distribution. We show that the correlation matrices obtained from multivariate spatiotemporal timeseries, in a regime of spatiotemporal chaos, lead to strong deviations from RMT. We illustrate the results with time-series data drawn from coupled map lattices. We also explore the transition to the RMT regime from the limiting cases.

Collective origin of the coexistence of apparent random matrix theory noise and of factors in large sample correlation matrices

Through simple analytical calculations and numerical simulations, we demonstrate the generic existence of a self-organized macroscopic state in any large multivariate system possessing non-vanishing average correlations between a finite fraction of all pairs of elements. The coexistence of an eigenvalue spectrum predicted by random matrix theory (RMT) and a few very large eigenvalues in large empirical correlation matrices is shown to result from a bottom–up collective effect of the underlying time series rather than a top–down impact of factors. Our results, in excellent agreement with previous results obtained on large financial correlation matrices, show that there is relevant information also in the bulk of the eigenvalue spectrum and rationalize the presence of market factors previously introduced in an ad hoc manner.

Statistics of atmospheric correlations.

For a large class of quantum systems, the statistical properties of their spectrum show remarkable agreement with random matrix predictions. Recent advances show that the scope of random matrix theory is much wider. In this work, we show that the random matrix approach can be beneficially applied to a completely different classical domain, namely, to the empirical correlation matrices obtained from the analysis of the basic atmospheric parameters that characterize the state of atmosphere. We show that the spectrum of atmospheric correlation matrices satisfy the random matrix prescription. In particular, the eigenmodes of the atmospheric empirical correlation matrices that have physical significance are marked by deviations from the eigenvector distribution.

RANDOM MATRIX THEORY AND FINANCIAL CORRELATIONS

We show that results from the theory of random matrices are potentially of great interest when trying to understand the statistical structure of the empirical correlation matrices appearing in the study of multivariate financial time series. We find a remarkable agreement between the theoretical prediction (based on the assumption that the correlation matrix is random) and empirical data concerning the density of eigenvalues associated to the time series of the different stocks of the S&P500 (or other major markets). Finally, we give a specific example to show how this idea can be sucessfully implemented for improving risk management.

Model for correlations in stock markets

We propose a group model for correlations in stock markets. In the group model the markets are composed of several groups, within which the stock price fluctuations are correlated. The spectral properties of empirical correlation matrices reported recently are well understood from the model. It provides the connection between the spectral properties of the empirical correlation matrix and the structure of correlations in stock markets.

Noise Dressing of Financial Correlation Matrices

We show that results from the theory of random matrices are potentially of great interest to understand the statistical structure of the empirical correlation matrices appearing in the study of price fluctuations. The central result of the present study is the remarkable agreement between the theoretical prediction (based on the assumption that the correlation matrix is random) and empirical data concerning the density of eigenvalues associated to the time series of the different stocks of the S&P500 (or other major markets). In particular the present study raises serious doubts on the blind use of empirical correlation matrices for risk management.

On the eigenvalue distribution of genetic and phenotypic dispersion matrices: Evidence for a nonrandom organization of quantitative character variation

A quantitative genetic model of “random pleiotropy” is introduced as reference model for detecting the kind and degree of organization in quantitative genetic variation. In this model the genetic dispersion matrix takes the form of G = BBT, where B is a general, real, Gaussian random matrix. The eigenvalue density of the corresponding ensemble of random matrices (ℰG) is considered. The first two moments are derived for variance-covariance matrices G as well as for correlation matrices R, and an approximate expression of the density function is given. The eigenvalue distribution of all empirical correlation matrices deviates from that of a random pleiotropy model by a very large leading eigenvalue associated with a “size factor”. However the frequency-distribution of the remaining eigenvalues shows only minor deviations in mammalian skeletal data. A prevalence of intermediate eigenvalues in insect data may be caused by the inclusion of many functionally unrelated characters. Hence two kinds of deviations from random organization have been found: a “mammal like” and an “insect like” organization. It is concluded that functionally related characters are on the average more tightly correlated than by chance (= “mammal like” organization), while functionally unrelated characters appear to be less correlated than by random pleiotropy (“insect like” organization).