Marchenko Pastur Research Articles

A functional renormalization group for signal detection and stochastic ergodicity breaking

Signal detection is one of the main challenges in data science. As often happens in data analysis, the signal in the data may be corrupted by noise. There is a wide range of techniques that aim to extract the relevant degrees of freedom from data. However, some problems remain difficult. This is notably the case for signal detection in almost continuous spectra when the signal-to-noise ratio is small enough. This paper follows a recent bibliographic line, which tackles this issue with field-theoretical methods. Previous analysis focused on equilibrium Boltzmann distributions for an effective field representing the degrees of freedom of data. It was possible to establish a relation between signal detection and Z2 -symmetry breaking. In this paper, we consider a stochastic field framework inspired by the so-called ‘model A’, and show that the ability to reach, or not reach, an equilibrium state is correlated with the shape of the dataset. In particular, by studying the renormalization group of the model, we show that the weak ergodicity prescription is always broken for signals that are small enough, when the data distribution is close to the Marchenko–Pastur law. This, in particular, enables the definition of a detection threshold in the regime where the signal-to-noise ratio is small enough.

Studies on the Marchenko–Pastur Law

In free probability, the theory of Cauchy–Stieltjes Kernel (CSK) families has recently been introduced. This theory is about a set of probability measures defined using the Cauchy kernel similarly to natural exponential families in classical probability that are defined by means of the exponential kernel. Within the context of CSK families, this article presents certain features of the Marchenko–Pastur law based on the Fermi convolution and the t-deformed free convolution. The Marchenko–Pastur law holds significant theoretical and practical implications in various fields, particularly in the analysis of random matrices and their applications in statistics, signal processing, and machine learning. In the specific context of CSK families, our study of the Marchenko–Pastur law is summarized as follows: Let K+(μ)={Qmμ(dx);m∈(m0μ,m+μ)} be the CSK family generated by a non-degenerate probability measure μ with support bounded from above. Denote by Qmμ•s the Fermi convolution power of order s>0 of the measure Qmμ. We prove that if Qmμ•s∈K+(μ), then μ is of the Marchenko–Pastur type law. The same result is obtained if we replace the Fermi convolution • with the t-deformed free convolution t.

Monotonicity of the logarithmic energy for random matrices

It is well known that the semi-circle law, which is the limiting distribution in the Wigner theorem, is the minimizer of the logarithmic energy penalized by the second moment. A very similar fact holds for the Girko and Marchenko–Pastur theorems. In this work, we shed the light on an intriguing phenomenon suggesting that this functional is monotonic along the mean empirical spectral distribution in terms of the matrix dimension. This is reminiscent of the monotonicity of the Boltzmann entropy along the Boltzmann equation, the monotonicity of the free energy along ergodic Markov processes, and the Shannon monotonicity of entropy or free entropy along the classical or free central limit theorem. While we only verify this monotonicity phenomenon for the Gaussian unitary ensemble, the complex Ginibre ensemble, and the square Laguerre unitary ensemble, numerical simulations suggest that it is actually more universal. We obtain along the way explicit formulas of the logarithmic energy of the models which can be of independent interest.

Wigner- and Marchenko–Pastur-Type Limit Theorems for Jacobi Processes

AbstractWe study Jacobi processes $$(X_{t})_{t\ge 0}$$ ( X t ) t ≥ 0 on $$[-1,1]^N$$ [ - 1 , 1 ] N and $$[1,\infty [^N$$ [ 1 , ∞ [ N which are motivated by the Heckman–Opdam theory and associated integrable particle systems. These processes depend on three positive parameters and degenerate in the freezing limit to solutions of deterministic dynamical systems. In the compact case, these models tend for $$t\rightarrow \infty $$ t → ∞ to the distributions of the $$\beta $$ β -Jacobi ensembles and, in the freezing case, to vectors consisting of ordered zeros of one-dimensional Jacobi polynomials. We derive almost sure analogues of Wigner’s semicircle and Marchenko–Pastur limit laws for $$N\rightarrow \infty $$ N → ∞ for the empirical distributions of the N particles on some local scale. We there allow for arbitrary initial conditions, which enter the limiting distributions via free convolutions. These results generalize corresponding stationary limit results in the compact case for $$\beta $$ β -Jacobi ensembles and, in the deterministic case, for the empirical distributions of the ordered zeros of Jacobi polynomials. The results are also related to free limit theorems for multivariate Bessel processes, $$\beta $$ β -Hermite and $$\beta $$ β -Laguerre ensembles, and the asymptotic empirical distributions of the zeros of Hermite and Laguerre polynomials for $$N\rightarrow \infty $$ N → ∞ .

Cauchy-Stieltjes Kernel families: Some properties

In the setting of the theory of Cauchy-Stieltjes Kernel (CSK) families, we present in this article some properties related to the Marchenko–Pastur and Semicircle laws. Let K+(μ)={Qmμ(dx);m∈(m1μ,m+μ)} be the CSK family generated by a non degenerate probability measure μ with support bounded from above. Firstly, for 0<t≠1 such that Qmμ⊞t is well defined, we prove that if Qmμ⊞t∈K+(μ) then the measure μ is a scale transformation of the Marchenko–Pastur law. Secondly, consider Tβ:x⟼x+β, for β∈R∖{0}. We prove that if Tβ(Qmμ)∈K+(μ), then the measure μ is an affine transformation of the Semicircle law.

An asymptotically exact estimate of the median noise eigenvalue of sample covariance matrices

The Dominant Mode Rejection beamformer [Abraham &amp; Owsley (1990)] averages the noise subspace eigenvalues of the sample covariance matrix (SCM) to estimate the background noise power. This noise power estimate from snapshot deficient SCMs can be unreliable for large arrays in time-varying environments with limited snapshots. Median filtering offers robustness to outliers and subspace mismatch for these challenging problems. Recent numerical experiments [Campos Anchieta &amp; Buck (2022)] identified a simple regression relating the median sample eigenvalue to the true background power for snapshot deficient SCMs. However, the complicated expression of the Marchenko-Pastur (MP) distribution for the noise eigenvalues of the SCM thwarted prior attempts to find a closed form estimator for the MP distribution median. This talk exploits a coordinate transform to obtain a closed-form median of the MP distribution that is exact for the leading term in the power series expansion of the distribution. The new median estimator is more accurate than the previous numerical regression when the ratio of sensors to snapshots is 2 or more. Given the central role of SCM eigenvalues in principal component analysis and constant false alarm rate detectors, the new median expression should find application in other data science algorithms for underwater acoustics. [Work supported by ONR Code 321US.]

Local Marchenko–Pastur law at the hard edge of the sample covariance ensemble

Consider an N by N matrix X of complex entries with iid real and imaginary parts. We show that the local density of eigenvalues of X*X converges to the Marchenko–Pastur law on the optimal scale with probability 1. We also obtain rigidity of the eigenvalues in the bulk and near both hard and soft edges. Here we avoid logarithmic and polynomial corrections by working directly with high powers of expectation of the Stieltjes transforms. We work under the assumption that the entries have a finite fourth moment and are truncated at N1/4, or alternatively with exploding moments. In this work we simplify and adapt the methods from prior papers of Götze–Tikhomirov [Probab. Relat. Fields 165(1–2), 163–233 (2016)] and Cacciapuoti–Maltsev–Schlein [Probab. Theory Relat. Fields 163(1–2), 1–59 (2015)] to covariance matrices.

Random matrices associated to Young diagrams

We consider the singular values of certain Young diagram shaped random matrices. For block-shaped random matrices, the empirical distribution of the squares of the singular eigenvalues converges almost surely to a distribution whose moments are a generalization of the Catalan numbers. The limiting distribution is the density of a product of rescaled independent Beta random variables and its Stieltjes–Cauchy transform has a hypergeometric representation. In special cases we recover the Marchenko–Pastur and Dykema–Haagerup measures of square and triangular random matrices, respectively. We find a further factorization of the moments in terms of two complex-valued random variables that generalizes the factorization of the Marchenko–Pastur law as product of independent uniform and arcsine random variables.

Random Tensor Networks with Non-trivial Links

Random tensor networks are a powerful toy model for understanding the entanglement structure of holographic quantum gravity. However, unlike holographic quantum gravity, their entanglement spectra are flat. It has therefore been argued that a better model consists of random tensor networks with link states that are not maximally entangled, i.e., have non-trivial spectra. In this work, we initiate a systematic study of the entanglement properties of these networks. We employ tools from free probability, random matrix theory, and one-shot quantum information theory to study random tensor networks with bounded and unbounded variation in link spectra, and in cases where a subsystem has one or multiple minimal cuts. If the link states have bounded spectral variation, the limiting entanglement spectrum of a subsystem with two minimal cuts can be expressed as a free product of the entanglement spectra of each cut, along with a Marchenko–Pastur distribution. For a class of states with unbounded spectral variation, analogous to semiclassical states in quantum gravity, we relate the limiting entanglement spectrum of a subsystem with two minimal cuts to the distribution of the minimal entanglement across the two cuts. In doing so, we draw connections to previous work on split transfer protocols, entanglement negativity in random tensor networks, and Euclidean path integrals in quantum gravity.

Decomposing cryptocurrency high-frequency price dynamics into recurring and noisy components

This paper investigates the temporal patterns of activity in the cryptocurrency market with a focus on Bitcoin, Ethereum, Dogecoin, and WINkLink from January 2020 to December 2022. Market activity measures—logarithmic returns, volume, and transaction number, sampled every 10 s, were divided into intraday and intraweek periods and then further decomposed into recurring and noise components via correlation matrix formalism. The key findings include the distinctive market behavior from traditional stock markets due to the nonexistence of trade opening and closing. This was manifested in three enhanced-activity phases aligning with Asian, European, and U.S. trading sessions. An intriguing pattern of activity surge in 15-min intervals, particularly at full hours, was also noticed, implying the potential role of algorithmic trading. Most notably, recurring bursts of activity in bitcoin and ether were identified to coincide with the release times of significant U.S. macroeconomic reports, such as Nonfarm payrolls, Consumer Price Index data, and Federal Reserve statements. The most correlated daily patterns of activity occurred in 2022, possibly reflecting the documented correlations with U.S. stock indices in the same period. Factors that are external to the inner market dynamics are found to be responsible for the repeatable components of the market dynamics, while the internal factors appear to be substantially random, which manifests itself in a good agreement between the empirical eigenvalue distributions in their bulk and the random-matrix theory predictions expressed by the Marchenko–Pastur distribution. The findings reported support the growing integration of cryptocurrencies into the global financial markets.

Almost-Hermitian random matrices and bandlimited point processes

We study the distribution of eigenvalues of almost-Hermitian random matrices associated with the classical Gaussian and Laguerre unitary ensembles. In the almost-Hermitian setting, which was pioneered by Fyodorov, Khoruzhenko and Sommers in the case of GUE, the eigenvalues are not confined to the real axis, but instead have imaginary parts which vary within a narrow “band” about the real line, of height proportional to tfrac{1}{N}, where N denotes the size of the matrices. We study vertical cross-sections of the 1-point density as well as microscopic scaling limits, and we compare with other results which have appeared in the literature in recent years. Our approach uses Ward’s equation and a property which we call “cross-section convergence”, which relates the large-N limit of the cross-sections of the density of eigenvalues with the equilibrium density for the corresponding Hermitian ensemble: the semi-circle law for GUE and the Marchenko–Pastur law for LUE. As an application of our approach, we prove the bulk universality of the almost-circular ensembles.

Large sample covariance matrices of Gaussian observations with uniform correlation decay

We derive the Marchenko–Pastur (MP) law for sample covariance matrices of the form Vn=1nXXT, where X is a p×n data matrix and p/n→y∈(0,∞) as n,p→∞. We assume the data in X stems from a correlated joint normal distribution. In particular, the correlation acts both across rows and across columns of X, and we do not assume a specific correlation structure, such as separable dependencies. Instead, we assume that correlations converge uniformly to zero at a speed of an/n, where an may grow mildly to infinity. We employ the method of moments tightly: We identify the exact condition on the growth of an which will guarantee that the moments of the empirical spectral distributions (ESDs) converge to the MP moments. If the condition is not met, we can construct an ensemble for which all but finitely many moments of the ESDs diverge. We also investigate the operator norm of Vn under a uniform correlation bound of C/nδ, where C,δ>0 are fixed, and observe a phase transition at δ=1. In particular, convergence of the operator norm to the maximum of the support of the MP distribution can only be guaranteed if δ>1. The analysis leads to an example for which the MP law holds almost surely, but the operator norm remains stochastic in the limit, and we provide its exact limiting distribution.

Biwhitening Reveals the Rank of a Count Matrix.

Estimating the rank of a corrupted data matrix is an important task in data analysis, most notably for choosing the number of components in PCA. Significant progress on this task was achieved using random matrix theory by characterizing the spectral properties of large noise matrices. However, utilizing such tools is not straightforward when the data matrix consists of count random variables, e.g., Poisson, in which case the noise can be heteroskedastic with an unknown variance in each entry. In this work, we focus on a Poisson random matrix with independent entries and propose a simple procedure, termed biwhitening, for estimating the rank of the underlying signal matrix (i.e., the Poisson parameter matrix) without any prior knowledge. Our approach is based on the key observation that one can scale the rows and columns of the data matrix simultaneously so that the spectrum of the corresponding noise agrees with the standard Marchenko-Pastur (MP) law, justifying the use of the MP upper edge as a threshold for rank selection. Importantly, the required scaling factors can be estimated directly from the observations by solving a matrix scaling problem via the Sinkhorn-Knopp algorithm. Aside from the Poisson, our approach is extended to families of distributions that satisfy a quadratic relation between the mean and the variance, such as the generalized Poisson, binomial, negative binomial, gamma, and many others. This quadratic relation can also account for missing entries in the data. We conduct numerical experiments that corroborate our theoretical findings, and showcase the advantage of our approach for rank estimation in challenging regimes. Furthermore, we demonstrate the favorable performance of our approach on several real datasets of single-cell RNA sequencing (scRNA-seq), High-Throughput Chromosome Conformation Capture (Hi-C), and document topic modeling.

Using Marchenko–Pastur SVD and Linear MMSE Estimation for Reducing Image Noise

The degradation in visual quality of images is often seen due to a variety of noise added inevitably at the time of image acquisition. Its restoration has thus become a fundamental and significant problem in image processing. Many attempts are made in recent past to efficiently denoise images. But, the best possible solution to this problem is still an open research problem. This paper validates the effectiveness of one such popular image denoising approach, where an adaptive image patch clustering is followed by the two step denoising algorithm in Principal Component Analysis (PCA) domain. First step uses Marchenko–Pastur law based hard thresholding of singular values in the singular value decomposition (SVD) domain and the second step removes remaining noise in PCA domain using Linear Minimum Mean-Squared-Error (LMMSE), a soft thresholding. The experimentation is conducted on gray-scale images corrupted by four different noise types namely speckle, salt &amp; pepper, Gaussian, and Poisson. The efficiency of image denoising is quantified in terms of popular image quality metrics peak signal-to-noise ratio (PSNR), structural similarity (SSIM), feature similarity (FSIM), and the denoising time. The comprehensive performance analysis of the denoising approach against the four noise models underlies its suitability to various applications. This certainly gives the new researchers a direction for selection of image denoising method.

Random matrices theory elucidates the nonequilibrium critical phenomena

The earlier times of the evolution of a magnetic system contain more information than we can imagine. Capturing correlation matrices built from different time evolutions of a simple testbed spin system, as the spin-1/2 and spin-1 Ising models, we analyzed the density of eigenvalues for different temperatures of the so called Wishart matrices. We observe a transition in the shape of the distribution that presents a gap of eigenvalues for temperatures lower than the critical temperature, or in its roundness, with a continuous migration to the Marchenko–Pastur law in the paramagnetic phase. We consider the analysis a promising method to be applied in other spin systems, with or without defined Hamiltonian, to characterize phase transitions. Our approach differs from the alternatives in literature since it uses the concept of magnetization matrix, not the spatial matrix of single spins.

Superposition of random plane waves in high spatial dimensions: Random matrix approach to landscape complexity

Motivated by current interest in understanding statistical properties of random landscapes in high-dimensional spaces, we consider a model of the landscape in RN obtained by superimposing M &amp;gt; N plane waves of random wavevectors and amplitudes and further restricted by a uniform parabolic confinement in all directions. For this landscape, we show how to compute the “annealed complexity,” controlling the asymptotic growth rate of the mean number of stationary points as N → ∞ at fixed ratio α = M/N &amp;gt; 1. The framework of this computation requires us to study spectral properties of N × N matrices W = KTKT, where T is a diagonal matrix with M mean zero independent and identically distributed (i.i.d.) real normally distributed entries, and all MN entries of K are also i.i.d. real normal random variables. We suggest to call the latter Gaussian Marchenko–Pastur ensemble as such matrices appeared in the seminal 1967 paper by those authors. We compute the associated mean spectral density and evaluate some moments and correlation functions involving products of characteristic polynomials for such matrices.

A characterization of the Marchenko–Pastur probability measure

Consider K+(ν)={Q(m,ν)(dx);m∈(m0(ν),m+(ν))} the Cauchy-Stieltjes Kernel (CSK) family generated by a non degenerate probability measure ν with support bounded from above. Let X1,X2,…,Xn be boolean independent random variables with common distribution an element from K+(ν). i.e X1∼Q(m,ν) for some m∈(m0(ν),m+(ν)). We prove that the distribution of ∑i=1nXi belongs to the CSK family K+(ν) if and only if the probability measure ν is a scale transformation of the Marchenko–Pastur probability measure.

Local Laws for Sparse Sample Covariance Matrices

We proved the local Marchenko–Pastur law for sparse sample covariance matrices that corresponded to rectangular observation matrices of order n×m with n/m→y (where y&gt;0) and sparse probability npn&gt;logβn (where β&gt;0). The bounds of the distance between the empirical spectral distribution function of the sparse sample covariance matrices and the Marchenko–Pastur law distribution function that was obtained in the complex domain z∈D with Imz&gt;v0&gt;0 (where v0) were of order log4n/n and the domain bounds did not depend on pn while npn&gt;logβn.

Local Law for Singular Values of Oscillatory Matrices

AbstractWe continue the study of spectra of oscillatory random matrices with fully dependent rows. Motivated by the $d$-dimensional skew-shift dynamics from ergodic theory, we introduce the $N\times N$ random matrices $$ \begin{align*} &amp;X_{j,k}=\exp\left(2\pi \textrm{i} \sum_{q=1}^d\ \omega_{j,q} k^q\right), \end{align*}$$where $\{\omega _{j,q}\}_{1\leq j\leq N, 1\leq q\leq d}$ is a collection of i.i.d. random variables and $d$ is a fixed integer. We prove that as $N\to \infty $ the distribution of singular values converges to the local Marchenko–Pastur law up to scales $N^{-\theta _d}$ for an explicit $\theta _d&amp;gt;0$, as long as $d\geq 18$. Our approach provides a novel mechanism—deterministic oscillatory cancellations—for universal spectral laws. The proof blends techniques from random matrix theory, harmonic analysis, and analytic number theory such as strong estimates on the number of solutions to Diophantine equations in the form of Vinogradov’s main conjecture, proved by Bourgain–Demeter–Guth.

The conjugate gradient algorithm on a general class of spiked covariance matrices

We consider the conjugate gradient algorithm applied to a general class of spiked sample covariance matrices. The main result of the paper is that the norms of the error and residual vectors at any finite step concentrate on deterministic values determined by orthogonal polynomials with respect to a deformed Marchenko–Pastur law. The first-order limits and fluctuations are shown to be universal. Additionally, for the case where the bulk eigenvalues lie in a single interval we show a stronger universality result in that the asymptotic rate of convergence of the conjugate gradient algorithm only depends on the support of the bulk, provided the spikes are well-separated from the bulk. In particular, this shows that the classical condition number bound for the conjugate gradient algorithm is pessimistic for spiked matrices.