Pastur Law Research Articles

Noncommutative Probability and Multiplicative Cascades

Various aspects of standard model particle physics might be explained by a suitably rich algebra acting on itself, as suggested by Furey (2015). The present paper develops the asymptotics of large causal tree diagrams that combine freely independent elements in such an algebra. The Marčenko–Pastur law and Wigner’s semicircle law are shown to emerge as limits of normalized sum-over-paths of nonnegative elements assigned to the edges of causal trees. These results are established in the setting of noncommutative probability. Trees with classically independent positive edge weights (random multiplicative cascades) were originally proposed by Mandelbrot as a model displaying the fractal features of turbulence. The novelty of the present work is the use of noncommutative (free) probability to allow the edge weights to take values in an algebra. An application to theoretical neuroscience is also discussed.

Analysis of grid operation state based on improved maximum eigenvalue sample covariance matrix algorithm

Traditional static threshold–based state analysis methods can be applied to specific signal-to-noise ratio situations but may present poor performance in the presence of large sizes and complexity of power system. In this article, an improved maximum eigenvalue sample covariance matrix algorithm is proposed, where a Marchenko–Pastur law–based dynamic threshold is introduced by taking all the eigenvalues exceeding the supremum into account for different signal-to-noise ratio situations, to improve the calculation efficiency and widen the application fields of existing methods. The comparison analysis based on IEEE 39-Bus system shows that the proposed algorithm outperforms the existing solutions in terms of calculation speed, anti-interference ability, and universality to different signal-to-noise ratio situations.

Marchenko–Pastur law with relaxed independence conditions

We prove the Marchenko–Pastur law for the eigenvalues of [Formula: see text] sample covariance matrices in two new situations where the data does not have independent coordinates. In the first scenario — the block-independent model — the [Formula: see text] coordinates of the data are partitioned into blocks in such a way that the entries in different blocks are independent, but the entries from the same block may be dependent. In the second scenario — the random tensor model — the data is the homogeneous random tensor of order [Formula: see text], i.e. the coordinates of the data are all [Formula: see text] different products of [Formula: see text] variables chosen from a set of [Formula: see text] independent random variables. We show that Marchenko–Pastur law holds for the block-independent model as long as the size of the largest block is [Formula: see text], and for the random tensor model as long as [Formula: see text]. Our main technical tools are new concentration inequalities for quadratic forms in random variables with block-independent coordinates, and for random tensors.

Application of random matrix model in multiple abnormal sources detection and location based on PMU monitoring data in distribution network

With the conversion of the global power economy and energy structure, access to a large amount of renewable energy has led to a decrease in power system inertia. The slight abnormal disturbance in the distribution network may have a significant impact on social and economic development. Aim at enhancing power stability and system resiliency; this study focuses on the detection and location of multiple abnormal sources in the distribution network. Most traditional methods use models relying on precise line parameters, subject to poor adaptability to the distribution network with a large number of nodes, and rapidly changing topology. Therefore, this study proposes a novel random matrix model, driven by monitoring data from phasor measurement units distributed on the overhead transmission lines. In this model, linear shrinkage (LS) theory, and Marchenko–Pastur law are combined for noise reduction to ensure the dynamic character and anti-noise ability. Moreover, data dimensions and sample points may be at the same level in an extensive scale network. The LS and standard condition number rule (SCN) are used for estimating the number of abnormal sources. Finally, the effectiveness of this paper's model is verified in PSCAD. The results indicate that the method has specific dynamic performance and anti-noise ability.

Early anomaly detection and localisation in distribution network: a data‐driven approach

The measurement data collected from the supervisory control and data acquisition (SCADA) system installed in distribution network can reflect the operational state of the network effectively. In this study, a random matrix theory-based approach is developed for early anomaly detection and localisation by using the data. For every feeder in the distribution network, a corresponding data matrix is formed. Based on the Marchenko–Pastur law for the empirical spectral analysis of covariance ‘signal+noise’ matrix, the linear eigenvalue statistics are introduced to indicate the anomaly, and the outliers and their corresponding eigenvectors are analysed for locating the anomaly. As for the low observability feeders in the distribution network, an increasing data dimension algorithm is designed for the formulated low-dimensional matrices being more accurately analysed. The developed approach can detect and localise the anomaly at an early stage, and it is robust to random disturbance and measurement error. Cases on Matpower simulation data and real SCADA data corroborate the feasibility of the approach.

Random Matrices From Linear Codes and Wigner’s Semicircle Law

In this paper, we consider a new normalization of matrices obtained by choosing distinct codewords at random from linear codes over finite fields and find that under some natural algebraic conditions of the codes their empirical spectral distribution converges to Wigner’s semicircle law as the length of the codes goes to infinity. One such condition is that the dual distance of the codes is at least five. This is analogous to previous work on the empirical spectral distribution of similar matrices obtained in this fashion that converges to the Marchenko–Pastur law.

On the Marchenko–Pastur law in analog bipartite spin-glasses**The authors are pleased to dedicate this paper to Giorgio Parisi on the occasion of his seventieth birthday.

In recent decades, statistical mechanics of disordered systems (mainly spin-glasses) has become one of the main tools to investigate complex systems, probably due to the celebrated Replica Symmetry Breaking scheme of Parisi Theory and its deep implications.In this work we consider the analog bipartite spin-glass (or real-valued restricted Boltzmann machine in a neural-network jargon), whose variables (those quenched as well as those dynamical) share standard Gaussian distributions. First, via Guerra’s interpolation technique, we express its quenched free-energy in terms of the natural order parameters of the theory (namely the self- and two-replica overlaps), then, we re-obtain the same result by using the replica-trick: a mandatory tribute, given the special occasion. Next, we show that the quenched free-energy of this model is the functional generator of the moments of the matrix whose entries are the correlation coefficients between the weights connecting the two layers of the spin-glass (i.e. the Wishart matrix in random matrix theory or the Hebbian coupling in neural networks): as weights are quenched stochastic variables, this acts as a powerful tool to inspect random matrices. In particular, we find that the Stieltjes transform of the spectral density of the correlation matrix is determined by the (replica-symmetric) quenched free-energy of the bipartite spin-glass model. In this setup, we re-obtain the Marchenko–Pastur law in a very simple way.

Active Eavesdropping Detection Based on Large-Dimensional Random Matrix Theory for Massive MIMO-Enabled IoT

The increasing Internet-of-Things (IoT) applications will take a significant share of the services of the fifth generation mobile network (5G). However, IoT devices are vulnerable to security threats due to the limitation of their simple hardware and communication protocol. Massive multiple-input multiple-output (massive MIMO) is recognized as a promising technique to support massive connections of IoT devices, but it faces potential physical layer breaches. An active eavesdropper can compromises the communication security of massive MIMO systems by purposely contaminating the uplink pilots. According to the random matrix theory (RMT), the eigenvalue distribution of a large dimensional matrix composed of data samples converges to the limit spectrum distribution that can be characterized by matrix dimensions. With the assistance of RMT, we propose an active eavesdropping detection method in this paper. The theoretical limit spectrum distribution is exploited to determine the distribution range of the eigenvalues of a legitimate user signal. In addition the noise components are removed using the Marčenko–Pastur law of RMT. Hypothesis testing is then carried out to determine whether the spread range of eigenvalues is “normal” or not. Simulation results show that, compared with the classical Minimum Description Length (MDL)-based detection algorithm, the proposed method significantly improves active eavesdropping detection performance.

$e$PCA: High dimensional exponential family PCA

Many applications involve large datasets with entries from exponential family distributions. Our main motivating application is photon-limited imaging, where we observe images with Poisson distributed pixels. We focus on X-ray Free Electron Lasers (XFEL), a quickly developing technology whose goal is to reconstruct molecular structure. In XFEL, estimating the principal components of the noiseless distribution is needed for denoising and for structure determination. However, the standard method, Principal Component Analysis (PCA), can be inefficient in non-Gaussian noise. Motivated by this application, we develop $e$PCA (exponential family PCA), a new methodology for PCA on exponential families. $e$PCA is a fast method that can be used very generally for dimension reduction and denoising of large data matrices with exponential family entries. We conduct a substantive XFEL data analysis using $e$PCA. We show that $e$PCA estimates the PCs of the distribution of images more accurately than PCA and alternatives. Importantly, it also leads to better denoising. We also provide theoretical justification for our estimator, including the convergence rate and the Marchenko–Pastur law in high dimensions. An open-source implementation is available.

The PPT square conjecture holds generically for some classes of independent states

Let be a random pure state on , where ψ is a random unit vector uniformly distributed on the sphere in . Let be a random induced state whose distribution is , and let be a random induced state following the same distribution independent of . Let ρ be a random state induced by the entanglement swapping of and . We show that the empirical spectrum of converges almost surely to the Marcenko–Pastur law with parameter c2 as and . As an application, we prove that the state ρ is separable generically if are PPT entangled.

The spectral norm of random inner-product kernel matrices

We study an “inner-product kernel” random matrix model, whose empirical spectral distribution was shown by Xiuyuan Cheng and Amit Singer to converge to a deterministic measure in the large n and p limit. We provide an interpretation of this limit measure as the additive free convolution of a semicircle law and a Marchenko–Pastur law. By comparing the tracial moments of this matrix to an additive deformation of a Wigner matrix, we establish that for odd kernel functions, the spectral norm of this matrix converges almost surely to the edge of the limiting spectrum. Our study is motivated by the analysis of a covariance thresholding procedure for the statistical detection and estimation of sparse principal components, and our results characterize the limit of the largest eigenvalue of the thresholded sample covariance matrix in the null setting.

Detail-Preserving Image Denoising via Adaptive Clustering and Progressive PCA Thresholding

This paper proposes a detail-preserving image denoising method via cluster-wise progressive principal component analysis (PCA) thresholding based on the Marchenko–Pastur (MP) law in random matrix theory. According to random matrix theory, an efficient and stable noise-level estimation method is also presented. Specifically, a global Gaussian noise level is estimated by interpreting the relationship between noise and eigenvalues of PCA for noisy patch matrices via the MP law in conjunction with the observation that vectors extracted from a noise-free image often lie in a low-dimensional subspace. Before noise removal, an adaptive clustering method is developed to automatically determine a suitable number of clusters segregating patches with different features (edges and textures). To denoise each cluster matrix, progressive PCA thresholding is performed. First, a hard thresholding of singular values in the singular value decomposition domain based on the MP law is applied to find a low-rank approximation to the cluster matrix. Second, the remaining noises of the low-rank matrix are further removed in the PCA transform domain using a special soft thresholding, i.e., the linear minimum mean-square-error technique with locally estimated parameters. The experiments show that the proposed method not only achieves state-of-the-art denoising performance in terms of quantitative indices, but also preserves visually important image details best.

Spectral analysis of high-dimensional sample covariance matrices with missing observations

We study high-dimensional sample covariance matrices based on independent random vectors with missing coordinates. The presence of missing observations is common in modern applications such as climate studies or gene expression micro-arrays. A weak approximation on the spectral distribution in the “large dimension $d$ and large sample size $n$” asymptotics is derived for possibly different observation probabilities in the coordinates. The spectral distribution turns out to be strongly influenced by the missingness mechanism. In the null case under the missing at random scenario where each component is observed with the same probability $p$, the limiting spectral distribution is a Marcenko–Pastur law shifted by $(1-p)/p$ to the left. As $d/n\rightarrow y\in(0,1)$, the almost sure convergence of the extremal eigenvalues to the respective boundary points of the support of the limiting spectral distribution is proved, which are explicitly given in terms of $y$ and $p$. Eventually, the sample covariance matrix is positive definite if $p$ is larger than \[1-(1-\sqrt{y})^{2},\] whereas this is not true any longer if $p$ is smaller than this quantity.

Almost sure convergence of the largest and smallest eigenvalues of high-dimensional sample correlation matrices

In this paper, we show that the largest and smallest eigenvalues of a sample correlation matrix stemming from n independent observations of a p-dimensional time series with iid components converge almost surely to (1+γ)2 and (1−γ)2, respectively, as n→∞, if p∕n→γ∈(0,1] and the truncated variance of the entry distribution is “almost slowly varying”, a condition we describe via moment properties of self-normalized sums. Moreover, the empirical spectral distributions of these sample correlation matrices converge weakly, with probability 1, to the Marčenko–Pastur law, which extends a result in Bai and Zhou (2008). We compare the behavior of the eigenvalues of the sample covariance and sample correlation matrices and argue that the latter seems more robust, in particular in the case of infinite fourth moment. We briefly address some practical issues for the estimation of extreme eigenvalues in a simulation study.In our proofs we use the method of moments combined with a Path-Shortening Algorithm, which efficiently uses the structure of sample correlation matrices, to calculate precise bounds for matrix norms. We believe that this new approach could be of further use in random matrix theory.

On Structure Testing for Component Covariance Matrices of a High Dimensional Mixture

Summary By studying the family of p-dimensional scale mixtures, the paper shows for the first time a non-trivial example where the eigenvalue distribution of the corresponding sample covariance matrix does not converge to the celebrated Marčenko–Pastur law. A different and new limit is found and characterized. The reasons for failure of the Marčenko–Pastur limit in this situation are found to be a strong dependence between the p-co-ordinates of the mixture. Next, we address the problem of testing whether the mixture has a spherical covariance matrix. To analyse the traditional John's-type test we establish a novel and general central limit theorem for linear statistics of eigenvalues of the sample covariance matrix. It is shown that John's test and its recent high dimensional extensions both fail for high dimensional mixtures, precisely because of the different spectral limit above. As a remedy, a new test procedure is constructed afterwards for the sphericity hypothesis. This test is then applied to identify the covariance structure in model-based clustering. It is shown that the test has much higher power than the widely used integrated classification likelihood and Bayesian information criteria in detecting non-spherical component covariance matrices of a high dimensional mixture.

Kernel estimators for Mar c ˘ enko–Pastur law of quaternion sample covariance matrices

In this paper, we introduce kernel estimators for the Marc˘enko–Pastur law of quaternion sample covariance matrix, which is an extension from real and complex ones. Especially, we indicate that the classical empirical spectral distributions of the quaternion sample covariance matrices still tend to Marc˘enko–Pastur law when their elements have infinite variances, but are in the domain of attraction of normal law.

Marčenko-Pastur law for Kendall’s tau

We prove that Kendall’s Rank correlation matrix converges to the Marčenko Pastur law, under the assumption that observations are i.i.d random vectors $X_1, \ldots , X_n$ with components that are independent and absolutely continuous with respect to the Lebesgue measure. This is the first result on the empirical spectral distribution of a multivariate $U$-statistic.

Spectral analysis of the Moore–Penrose inverse of a large dimensional sample covariance matrix

For a sample of n independent identically distributed p-dimensional centered random vectors with covariance matrix Σn let S̃n denote the usual sample covariance (centered by the mean) and Sn the non-centered sample covariance matrix (i.e. the matrix of second moment estimates), where p>n. In this paper, we provide the limiting spectral distribution and central limit theorem for linear spectral statistics of the Moore–Penrose inverse of Sn and S̃n. We consider the large dimensional asymptotics when the number of variables p→∞ and the sample size n→∞ such that p/n→c∈(1,+∞). We present a Marchenko–Pastur law for both types of matrices, which shows that the limiting spectral distributions for both sample covariance matrices are the same. On the other hand, we demonstrate that the asymptotic distribution of linear spectral statistics of the Moore–Penrose inverse of S̃n differs in the mean from that of Sn.

On the Marčenko–Pastur law for linear time series

This paper is concerned with extensions of the classical Mar\v{c}enko-Pastur law to time series. Specifically, $p$-dimensional linear processes are considered which are built from innovation vectors with independent, identically distributed (real- or complex-valued) entries possessing zero mean, unit variance and finite fourth moments. The coefficient matrices of the linear process are assumed to be simultaneously diagonalizable. In this setting, the limiting behavior of the empirical spectral distribution of both sample covariance and symmetrized sample autocovariance matrices is determined in the high-dimensional setting $p/n\to c\in (0,\infty)$ for which dimension $p$ and sample size $n$ diverge to infinity at the same rate. The results extend existing contributions available in the literature for the covariance case and are one of the first of their kind for the autocovariance case.

On the principal components of sample covariance matrices

We introduce a class of $$M \times M$$ sample covariance matrices $${\mathcal {Q}}$$ which subsumes and generalizes several previous models. The associated population covariance matrix $$\Sigma = \mathbb {E}{\mathcal {Q}}$$ is assumed to differ from the identity by a matrix of bounded rank. All quantities except the rank of $$\Sigma - I_M$$ may depend on $$M$$ in an arbitrary fashion. We investigate the principal components, i.e. the top eigenvalues and eigenvectors, of $${\mathcal {Q}}$$ . We derive precise large deviation estimates on the generalized components $$\langle {\mathbf{{w}}} , {\varvec{\xi }_i}\rangle $$ of the outlier and non-outlier eigenvectors $$\varvec{\xi }_i$$ . Our results also hold near the so-called BBP transition, where outliers are created or annihilated, and for degenerate or near-degenerate outliers. We believe the obtained rates of convergence to be optimal. In addition, we derive the asymptotic distribution of the generalized components of the non-outlier eigenvectors. A novel observation arising from our results is that, unlike the eigenvalues, the eigenvectors of the principal components contain information about the subcritical spikes of $$\Sigma $$ . The proofs use several results on the eigenvalues and eigenvectors of the uncorrelated matrix $${\mathcal {Q}}$$ , satisfying $$\mathbb {E}{\mathcal {Q}} = I_M$$ , as input: the isotropic local Marchenko–Pastur law established in Bloemendal et al. (Electron J Probab 19:1–53, 2014), level repulsion, and quantum unique ergodicity of the eigenvectors. The latter is a special case of a new universality result for the joint eigenvalue–eigenvector distribution.