Sample Correlation Matrix Research Articles

Robust Estimation for Number of Factors in High Dimensional Factor Modeling via Spearman Correlation Matrix

Determining the number of factors in high-dimensional factor modeling is essential but challenging, especially when the data are heavy-tailed. In this article, we introduce a new estimator based on the spectral properties of Spearman sample correlation matrix under the high-dimensional setting, where both dimension and sample size tend to infinity proportionally. Our estimator is robust against heavy tails in either the common factors or idiosyncratic errors. The consistency of our estimator is established under mild conditions. Numerical experiments demonstrate the superiority of our estimator compared to existing methods. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Moderate Deviation Principle for the Determinant of Sample Correlation Matrix

ABSTRACTLet be a sequence of independence random vectors from with population correlation matrix , and its sample correlation matrix is denoted as , where the matrix element represents the Pearson correlation coefficient between the and columns of the data matrix . is an interesting part in multivariate analysis, and the likelihood ratio test (LRT) statistic used to test the complete independence of all components of can be expressed as a function of . On this basis, under the condition that the sample size and the dimension satisfy as , we derive the moderate deviation principle (MDP) of the LRT statistic for the hypothesis testing problem vs .

Large sample correlation matrices with unbounded spectrum

In this paper, we demonstrate that the diagonal of a high-dimensional sample covariance matrix stemming from n independent observations of a p-dimensional time series with finite fourth moments can be approximated in spectral norm by the diagonal of the population covariance matrix regardless of the spectral norm of the population covariance matrix. Our assumptions involve p and n tending to infinity, with p/n tending to a constant which might be positive or zero. Consequently, we investigate the asymptotic properties of the sample correlation matrix with a divergent spectrum, and we explore its applications by deriving the limiting spectral distribution for its eigenvalues and analyzing the convergence of divergent and non-divergent spiked eigenvalues under a generalized spiked correlation framework.

Improved Dual Correlation Reduction Network With Affinity Recovery.

Deep graph clustering, which aims to reveal the underlying graph structure and divide the nodes into different clusters without human annotations, is a fundamental yet challenging task. However, we observe that the existing methods suffer from the representation collapse problem and tend to encode samples with different classes into the same latent embedding. Consequently, the discriminative capability of nodes is limited, resulting in suboptimal clustering performance. To address this problem, we propose a novel deep graph clustering algorithm termed improved dual correlation reduction network (IDCRN) through improving the discriminative capability of samples. Specifically, by approximating the cross-view feature correlation matrix to an identity matrix, we reduce the redundancy between different dimensions of features, thus improving the discriminative capability of the latent space explicitly. Meanwhile, the cross-view sample correlation matrix is forced to approximate the designed clustering-refined adjacency matrix to guide the learned latent representation to recover the affinity matrix even across views, thus enhancing the discriminative capability of features implicitly. Moreover, we avoid the collapsed representation caused by the oversmoothing issue in graph convolutional networks (GCNs) through an introduced propagation regularization term, enabling IDCRN to capture the long-range information with the shallow network structure. Extensive experimental results on six benchmarks have demonstrated the effectiveness and efficiency of IDCRN compared with the existing state-of-the-art deep graph clustering algorithms. The code of IDCRN is released at https://github.com/yueliu1999/IDCRN. Besides, we share a collection of deep graph clustering, including papers, codes, and datasets at https://github.com/yueliu1999/Awesome-Deep-Graph-Clustering.

Determining Sample Size Requirements in EFA Solutions: A Simple Empirical Proposal

In unrestricted or exploratory factor analysis (EFA), there is a wide range of recommendations about the size samples should be to attain correct and stable solutions. In general, however, these recommendations are either rules of thumb or based on simulation results. As it is hard to establish the extent to which a particular data set suits the conditions used in a simulation study, the advice produced by simulation studies is not direct enough to be of practical use. Instead of trying to provide general and complex recommendations, in this article, we propose to estimate the sample size that is needed to analyze a data set at hand. The estimation takes into account the specified EFA model. The proposal is based on an intensive simulation process in which the sample correlation matrix is used as a basis for generating data sets from a pseudo-population in which the parent correlation holds exactly, and the criterion for determining the size required is a threshold that quantifies the closeness between the pseudo-population and the sample reproduced correlation matrices. The simulation results suggest that the proposal works well and that the determinants identified agree with those in the literature.

Dimensionality assessment in the presence of wording effects: A network psychometric and factorial approach.

This study proposes a procedure for substantive dimensionality estimation in the presence of wording effects, the inconsistent response to regular and reversed self-report items. The procedure developed consists of subtracting an approximate estimate of the wording effects variance from the sample correlation matrix and then estimating the substantive dimensionality on the residual correlation matrix. This is achieved by estimating a random intercept factor with unit loadings for all the regular and unrecoded reversed items. The accuracy of the procedure was evaluated through an extensive simulation study that manipulated nine relevant variables and employed the exploratory graph analysis (EGA) and parallel analysis (PA) retention methods. The results indicated that combining the proposed procedure with EGA or PA achieved high accuracy in estimating the substantive latent dimensionality, but that EGA was superior. Additionally, the present findings shed light on the complex ways that wording effects impact the dimensionality estimates when the response bias in the data is ignored. A tutorial on substantive dimensionality estimation with the R package EGAnet is offered, as well as practical guidelines for applied researchers.

Bayesian Structural Equation Models of Correlation Matrices

We present a method for Bayesian structural equation modeling of sample correlation matrices as correlation structures. The method transforms the sample correlation matrix to an unbounded vector using the matrix logarithm function. Bayesian inference about the unbounded vector is performed assuming a multivariate-normal likelihood, with a mean based on the transformed model-implied correlation matrix, and a covariance assumed to be of known form. Using Monte Carlo simulation, we examine the performance of the method with normal and ordinal indicators, as well as the capacity of the method to estimate models that account for misspecification. The performance of the approach is often adequate suggesting that the proposed method can be used for Bayesian analysis of correlation structures. We conclude with a discussion of potential applications of the approach, as well as future directions needed to further develop the method.

Exact sampling distribution of the general case sample correlation matrix

The distribution of the sample correlation matrix when the sample comes from a real multivariate normal population and when the population covariance matrix is diagonal or when the population variables are independently distributed is available in the literature. In the general case, that is, when the covariance matrix in the normal population is a general real positive definite matrix, the distribution of the sample correlation matrix is not available in a compact computable form in the literature. Some scattered results on some aspects of the general case are available, but the representations are not easily tractable. This article deals with a covariance structure coming from a real matrix-variate gamma when the scale parameter matrix is a general real positive definite matrix. General procedures for deriving asymptotic chi-square and asymptotic normal for some general moment structures are also presented in this article. The density in some particular cases are given explicitly. Graphs are also provided in the particular cases.

Central limit theorem of linear spectral statistics of high-dimensional sample correlation matrices

A high-dimensional sample correlation matrix is an important random matrix in multivariate statistical analysis. Its central limit theory is one of the main theoretical bases for making statistical inferences on high-dimensional correlation matrices. Under the high-dimensional framework in which the data dimension tends to infinity proportionally with the sample size, we establish the central limit theorems (CLT) for the linear spectral statistics (LSS) of sample correlation matrices in two settings: (1) the population follows an independent component structure; (2) the population follows an elliptical structure, including some heavy-tailed distributions. The results show that the CLTs of the LSS of the sample correlation matrices are very different in the two settings. In particular, even if the population correlation matrix is an identity matrix, the CLTs are different in the two settings. An application of our two established CLTs is provided.

Comparative serum lipid and immunohematological values among adult pulmonary tuberculosis and tuberculosis lymphadenitis cases and their association with sputum bacilli load and time to culture positivity in Northwestern Ethiopia

BackgroundThe serum lipid and immunohematological values of tuberculosis lymphadenitis (TBLN) patients is poorly documented relative to pulmonary tuberculosis (PTB) cases. Therefore, the aim of this study was to investigate the serum lipid and immunohematological values of patients with TBLN in comparison with PTB (PTB) patients.MethodsAn institution-based comparative cross-sectional study was conducted in Northwest Ethiopia from March to December 2021. The study participants were bacteriologically confirmed PTB (n = 82) and TBLN (n = 94) cases with no known comorbidity and whose ages was greater than 18 years and with no current pregnancy. Independent sample t-test, one-way ANOVA, box plot, and correlation matrix were used to analyze the data.ResultsThe body mass index (BMI), CD4 + T cell count, and high-density lipoprotein-Cholesterol (HDL-C) values were significantly higher among TBLN cases compared with PTB cases. Additionally, the total white blood cell (WBC) count, hemoglobin (Hb), total Cholesterol (CHO) and creatinine (Cr) values were relatively higher among TBLN than PTB (P > 0.05). On the reverse, the platelet count and triacylglycerol (TAG) values were relatively higher among PTB than in TBLN cases. While the mean days of culture positivity were 11.6 days for TBLN, the mean days of culture positivity were 14.0 days for PTB. Anemia and serum lipid values showed no correlation with sputum bacilli load and time to culture positivity.ConclusionTuberculous lymphadenitis patients were well-endowed with serum lipid, immunological and nutritional status compared with PTB cases. Hence, the high incidence rate of TBLN in Ethiopia could not be explained by low peripheral immunohematological values, malnutrition, Anemia, and dyslipidemia. Further study for identifying the predictors for TBLN in Ethiopia is highly desirable.

CORRELATION MATRIX OF EQUI-CORRELATED NORMAL POPULATION: FLUCTUATION OF THE LARGEST EIGENVALUE, SCALING OF THE BULK EIGENVALUES, AND STOCK MARKET

Given an N-dimensional sample of size [Formula: see text] form a sample correlation matrix [Formula: see text]. Suppose that N and T tend to infinity with [Formula: see text] converging to a fixed finite constant [Formula: see text]. If the population is a factor model, then the eigenvalue distribution of [Formula: see text] almost surely converges weakly to Marčenko–Pastur distribution such that the index is Q and the scale parameter is the limiting ratio of the specific variance to the ith variable [Formula: see text]. For an N-dimensional normal population with equi-correlation coefficient [Formula: see text], which is a one-factor model, for the largest eigenvalue [Formula: see text] of [Formula: see text], we prove that [Formula: see text] converges to the equi-correlation coefficient [Formula: see text] almost surely. These results suggest an important role of an equi-correlated normal population and a factor model in Laloux et al. [(2000) Random matrix theory and financial correlations, International Journal of Theoretical and Applied Finance3 (3), 391–397]: the histogram of the eigenvalue of sample correlation matrix of the returns of stock prices fits the density of Marčenko–Pastur distribution of index [Formula: see text] and scale parameter [Formula: see text]. Moreover, we provide the limiting distribution of the largest eigenvalue of a sample covariance matrix of an equi-correlated normal population. We discuss the phase transition as to the decay rate of the equi-correlation coefficient in N.

Using multiple outcomes in intervention studies: improving power while controlling type I errors.

Background The CONSORT guidelines for clinical trials recommend using a single primary outcome, to guard against excess false positive findings when multiple measures are considered. However, statistical power can be increased while controlling the familywise error rate if multiple outcomes are included. The MEff statistic is well-suited to this purpose, but is not well-known outside genetics. Methods Data were simulated for an intervention study, with a given sample size (N), effect size (E) and correlation matrix for a suite of outcomes ( R). Using the variance of eigenvalues from the correlation matrix, we compute MEff, the effective number of variables that the alpha level should be divided by to control the familywise error rate. Various scenarios are simulated to consider how MEff is affected by the pattern of pairwise correlations within a set of outcomes. The power of the MEff approach is compared to Bonferroni correction, and a principal component analysis (PCA). Results In many situations, power can be increased by inclusion of multiple outcomes. Differences in power between MEff and Bonferroni correction are small if intercorrelations between outcomes are low, but the advantage of MEff is more evident as intercorrelations increase. PCA is superior in cases where the impact on outcomes is fairly uniform, but MEff is applicable when intervention effects are inconsistent across measures. Conclusions The optimal method for correcting for multiple testing depends on the underlying data structure, with PCA being superior if outcomes are all indicators of a common underlying factor. Both Bonferroni correction and MEff can be applied post hoc to evaluate published intervention studies, with MEff being superior when outcomes are moderately or highly correlated. A lookup table is provided to give alpha levels for use with Meff for cases where the correlation between outcome measures can be estimated.

Spectral Properties of Rescaled Sample Correlation Matrix

Spectral Properties of Rescaled Sample Correlation Matrix

High-Accuracy DOA Estimation Based on an Improved Sample Correlation Matrix

In a direction-finding process, high-resolution subspace-based algorithms are the most popular ones. It is well-known that their performance of direction of arrival estimation mainly depends on the accuracy of the signal subspace. However, the traditional methods of capturing the signal subspace do not mine the information hidden in the array output in depth, which may restrict their application to some extent. In this study, we elaborate on a novel scheme to extract the signal subspace through refinement of the correlation matrix of the array output. In the developed scheme, a collection of spatial temporal correlation matrices is firstly established. Then, we define a weighting vector for the correlation matrices, and take the weighted average of the correlation matrices as the covariance matrix of the array output. It is clear that this covariance matrix is more general than the traditional covariance matrix, and the signal subspace can be optimized through adjustment of the weighting vector. In this study, we present an optimal weighting vector by adopting the particle swarm optimization. Simulation results demonstrate that the proposed approach has better performance in root mean square error compared to the existing schemes.

The density of the sample correlations under elliptical symmetry with or without the truncated variance-ratio

The probability density function (pdf) of the sample correlation coefficient, when the associated sample variance-ratio is variously truncated, is given for the bivariate elliptical distribution (elliptical symmetry) with some moments. It is derived that the joint pdf of the sample variance-ratios and correlation matrix with possible truncation under multivariate elliptical symmetry is unchanged irrespective of distinct elliptical distributions. A general condition for transformed sample variances and covariances to have an unchanged pdf under elliptical symmetry is given. Examples satisfying this condition are shown for the intraclass correlation, coefficient alpha and principal component analysis.

Properties of eigenvalues and eigenvectors of large-dimensional sample correlation matrices

This paper is to study the properties of eigenvalues and eigenvectors of high-dimensional sample correlation matrices. We first improve the result of Jiang (Sankhyā 66 (2004) 35–48), Xiao and Zhou (J. Theoret. Probab. 23 (2010) 1–20) and the Theorem 1 of El Karoui (Ann. Appl. Probab. 19 (2009) 2362–2405), both concerning the limiting spectral distribution and the extreme eigenvalues of sample correlation matrices, by allowing a more general fourth moment condition. Then, we establish a central limit theorem (CLT) for the linear statistics of the eigenvectors of large sample correlation matrices. We discover that the difference between the functional CLT of the sample covariance matrix and the sample correlation matrix is fundamentally influenced by the direction of a nonrandom projection vector. In the special case where the square root of the correlation matrix is identity, the difference will be determined by the sum of the fourth powers of the entries of the projection vector. These results also indicate that the eigenmatrix of sample correlation matrices is not asymptotically Haar if the underlying distribution is Gaussian. In other words, the normalization based on the sample variances affects the asymptotic properties of the eigenmatrix of the Wishart matrix. Furthermore, we establish a theorem concerning CLT for the linear statistics of the eigenvectors of large sample covariance matrices. This theorem improves the main results in Bai, Miao and Pan (Ann. Probab. 35 (2007) 1532–1572), which requires the assumption that the fourth moment of the underlying variable matches the one of Gaussian distribution, as well as Theorem 1.3 in Pan and Zhou (Ann. Appl. Probab. 18 (2008) 1232–1270), which relaxed the Gaussian like fourth moment requirement but assumes the maximum entries of the projection vector converge to 0 uniformly. We illustrate the usefulness of the theoretical results through an application in communications.

Limiting distributions for eigenvalues of sample correlation matrices from heavy-tailed populations

Consider a p-dimensional population x∈Rp with i.i.d. coordinates that are regularly varying with index α∈(0,2). Since the variance of x is infinite, the diagonal elements of the sample covariance matrix Sn=n−1 ∑ i=1nxixi′ based on a sample x1,…,xn from the population tend to infinity as n increases and it is of interest to use instead the sample correlation matrix Rn={diag(Sn)}−1/2Sn{diag(Sn)}−1/2. This paper finds the limiting distributions of the eigenvalues of Rn when both the dimension p and the sample size n grow to infinity such that p/n→γ∈(0,∞). The family of limiting distributions {Hα,γ} is new and depends on the two parameters α and γ. The moments of Hα,γ are fully identified as sum of two contributions: the first from the classical Marčenko–Pastur law and a second due to heavy tails. Moreover, the family {Hα,γ} has continuous extensions at the boundaries α=2 and α=0 leading to the Marčenko–Pastur law and a modified Poisson distribution, respectively. Our proofs use the method of moments, the path-shortening algorithm developed in [18] (Stochastic Process. Appl. 128 (2018) 2779–2815) and some novel graph counting combinatorics. As a consequence, the moments of Hα,γ are expressed in terms of combinatorial objects such as Stirling numbers of the second kind. A simulation study on these limiting distributions Hα,γ is also provided for comparison with the Marčenko–Pastur law.

A Dichotomous Behavior of Guttman-Kaiser Criterion from Equi-Correlated Normal Population

We consider a p-dimensional, centered normal population such that all variables have a positive variance σ2 and any correlation coefficient between different variables is a given nonnegative constant ρ < 1. Suppose that both the sample size n and population dimension p tend to infinity with p/n → c > 0. We prove that the limiting spectral distribution of a sample correlation matrix is the Marcenko-Pastur distribution of index c and scale parameter 1 − ρ. By the limiting spectral distributions, we rigorously show the limiting behavior of widespread stopping rules Guttman-Kaiser criterion and cumulative-percentage-of-variation rule in PCA andEFA. As a result, we establish the following dichotomous behavior of Guttman-Kaiser criterion when both n and p are large, but p/n is small: (1) the criterion retains a small number of variables for ρ > 0, as suggested by Kaiser, Humphreys, and Tucker [Kaiser, H. F. (1992). On Cliff’s formula, the Kaiser-Guttman rule and the number of factors. Percept. Mot. Ski. 74]; and (2) the criterion retains p/2 variables for ρ = 0, as in a simulation study [Yeomans, K. A. and Golder, P. A. (1982). The Guttman-Kaiser criterion as a predictor of the number of common factors. J. Royal Stat. Soc. Series D. 31(3)].

Deep Graph Clustering via Dual Correlation Reduction

Deep graph clustering, which aims to reveal the underlying graph structure and divide the nodes into different groups, has attracted intensive attention in recent years. However, we observe that, in the process of node encoding, existing methods suffer from representation collapse which tends to map all data into the same representation. Consequently, the discriminative capability of the node representation is limited, leading to unsatisfied clustering performance. To address this issue, we propose a novel self-supervised deep graph clustering method termed Dual Correlation Reduction Network (DCRN) by reducing information correlation in a dual manner. Specifically, in our method, we first design a siamese network to encode samples. Then by forcing the cross-view sample correlation matrix and cross-view feature correlation matrix to approximate two identity matrices, respectively, we reduce the information correlation in the dual-level, thus improving the discriminative capability of the resulting features. Moreover, in order to alleviate representation collapse caused by over-smoothing in GCN, we introduce a propagation regularization term to enable the network to gain long-distance information with the shallow network structure. Extensive experimental results on six benchmark datasets demonstrate the effectiveness of the proposed DCRN against the existing state-of-the-art methods. The code of DCRN is available at https://github.com/yueliu1999/DCRN and a collection (papers, codes and, datasets) of deep graph clustering is shared at https://github.com/yueliu1999/Awesome-Deep-Graph-Clustering on Github.

High-dimensional correlation matrix estimation for general continuous data with Bagging technique

High-dimensional covariance matrix estimation plays a central role in multivariate statistical analysis. It is well-known that the sample covariance matrix is singular when the sample size is smaller than the dimension of the variable, but the covariance estimate must be positive-definite. This motivates some modifications of the sample covariance matrix to preserve its efficient estimation of pairwise covariance. In this paper, we modify the sample correlation matrix using the Bagging technique. The proposed Bagging estimator is flexible for general continuous data. Under some mild conditions, we show theoretically that the Bagging estimator can ensure positive-definiteness with probability one in finite samples. We also prove the consistency of the bootstrap estimator of Pearson correlation and the consistency of our Bagging estimator when the dimension p is fixed. Simulation results and a real application are provided to demonstrate that our method strikes a better balance between RMSE and likelihood, and is more robust, than other existing estimators.