Empirical Spectral Distribution Research Articles

Strong convergence of the empirical spectral distribution of a unified matrix model

Let [Formula: see text] is a positive definite matrix and [Formula: see text] with [Formula: see text] being independent and identically distributed (i.i.d.) centered real random variables. Consider the matrix [Formula: see text] where [Formula: see text] and [Formula: see text] are two projection matrices (deterministic or random) satisfying [Formula: see text] and [Formula: see text]. Additionally, if [Formula: see text] and [Formula: see text] are random matrices, they are independent of [Formula: see text]. In this paper, we demonstrate that the empirical spectral distribution of [Formula: see text] converges almost surely to a non-random distribution when [Formula: see text] and [Formula: see text], assuming [Formula: see text] has finite second moment.

Spectral Pseudorandomness and the Road to Improved Clique Number Bounds for Paley Graphs

We study subgraphs of Paley graphs of prime order p induced on the sets of vertices extending a given independent set of size a to a larger independent set. Using a sufficient condition due to Kunisky (2023), we show that estimates on a new family of necklace character sums would imply that, as p → ∞ , the empirical spectral distributions of the adjacency matrices of any sequence of such subgraphs have the same weak limit (after rescaling) as those of subgraphs induced on a random set including each vertex independently with probability 2 − a , namely, a Kesten-McKay law with parameter 2 a . We prove the necessary estimates for a = 1, obtaining in the process an alternate proof of a character sum equidistribution result of Xi (2022), and provide numerical evidence for this weak convergence for a ≥ 2 . We also conjecture that the minimum eigenvalue of any such sequence converges (after rescaling) to the left edge of the corresponding Kesten-McKay law, and provide numerical evidence for this convergence. Finally, we show that, once a ≥ 3 , this (conjectural) convergence of the minimum eigenvalue would imply bounds on the clique number of the Paley graph improving on the current state of the art due to Hanson and Petridis (2021), and that this convergence for all a ≥ 1 would imply that the clique number is o ( p ) .

Central limit theorem for linear eigenvalue statistics of the adjacency matrices of random simplicial complexes

AbstractWe study the adjacency matrix of the Linial–Meshulam complex model, which is a higher‐dimensional generalization of the Erdős–Rényi graph model. Recently, Knowles and Rosenthal proved that the empirical spectral distribution of the adjacency matrix is asymptotically given by Wigner's semicircle law in a diluted regime. In this article, we prove a central limit theorem for the linear eigenvalue statistics for test functions of polynomial growth that is of class on a closed interval. The proof is based on higher‐dimensional combinatorial enumerations and concentration properties of random symmetric matrices. Furthermore, when the test function is a polynomial function, we obtain the explicit formula for the variance of the limiting Gaussian distribution.

Monotonicity of the logarithmic energy for random matrices

It is well known that the semi-circle law, which is the limiting distribution in the Wigner theorem, is the minimizer of the logarithmic energy penalized by the second moment. A very similar fact holds for the Girko and Marchenko–Pastur theorems. In this work, we shed the light on an intriguing phenomenon suggesting that this functional is monotonic along the mean empirical spectral distribution in terms of the matrix dimension. This is reminiscent of the monotonicity of the Boltzmann entropy along the Boltzmann equation, the monotonicity of the free energy along ergodic Markov processes, and the Shannon monotonicity of entropy or free entropy along the classical or free central limit theorem. While we only verify this monotonicity phenomenon for the Gaussian unitary ensemble, the complex Ginibre ensemble, and the square Laguerre unitary ensemble, numerical simulations suggest that it is actually more universal. We obtain along the way explicit formulas of the logarithmic energy of the models which can be of independent interest.

Eigenvalue distributions of high-dimensional matrix processes driven by fractional Brownian motion

In this paper, we study high-dimensional behavior of empirical spectral distributions [Formula: see text] for a class of [Formula: see text] symmetric/Hermitian random matrices, whose entries are generated from the solution of stochastic differential equation driven by fractional Brownian motion with Hurst parameter [Formula: see text]. For Wigner-type matrices, we obtain almost sure relative compactness of [Formula: see text] in [Formula: see text] following the approach in [1]; for Wishart-type matrices, we obtain tightness of [Formula: see text] on [Formula: see text] by tightness criterions provided in Appendix B. The limit of [Formula: see text] as [Formula: see text] is also characterized.

Separable sample covariance matrices under elliptical populations with applications

This paper is to investigate the spectral properties of separable covariance matrices under elliptical populations. The separable covariance matrix model can handle both cross-row and cross-column correlations thus gain more popularity recently. Under the high-dimensional setting where the dimension p p and the sample size n n tend to infinity proportionally, we find the limit of the empirical spectral distribution and establish the central limit theorems (CLT) for linear spectral statistics of such kinds of sample covariance matrices. Some applications of our established CLT are also given.

A study on the spectral properties of covariance type matrices with isotropic log-concave column vectors.

The limiting spectral distribution of matrix [Formula: see text] is considered in this paper. Existing results always focus on the condition of modifying Tn, but for Xn, it is usually assumed to be a matrix composed of n × N independent identically distributed elements. Here we specify the joint distribution of column vectors of Xn. In particular, entries on the same column of Xn are correlated, in contrast with more common independence assumptions. Assuming that the columns of Xn are random vectors following the isotropic log-concave distribution, and under some additional regularity conditions, we prove that the empirical spectral distribution [Formula: see text] of matrix Bn converges to a deterministic probability distribution F almost surely. Moreover, the Stieltjes transformation m = m(z) of F satisfies a deterministic form of equation, and for any [Formula: see text], it is the unique solution of the equation.

Stability of the Lanczos algorithm on matrices with regular spectral distributions

We study the stability of the Lanczos algorithm run on problems whose eigenvector empirical spectral distribution is near to a reference measure with well-behaved orthogonal polynomials. We give a backwards stability result which can be upgraded to a forward stability result when the reference measure has a density supported on a single interval with square root behavior at the endpoints. Our analysis implies the Lanczos algorithm run on many large random matrix models is in fact forward stable, and hence nearly deterministic, even when computations are carried out in finite precision arithmetic. Since the Lanczos algorithm is not forward stable in general, this provides yet another example of the fact that random matrices are far from “any old matrix”, and care must be taken when using them to test numerical algorithms.

Empirically Driven multiwavelength K-corrections at low redshift

ABSTRACT K-corrections – a necessary ingredient for converting between flux in observed bands to flux in rest-frame bands – are critical for comparing galaxies at differing redshifts. These corrections often rely on fits to empirical or theoretical spectral energy distribution (SED) templates of galaxies. However, templates can only produce reliable K-corrections in regimes where SED models are robust. For instance, the templates utilized in some popular software packages are not well-constrained in some bands (e.g. WISE W4 in Kcorrect), which results in ill-behaved K-corrections. We address this shortcoming by developing an empirically driven approach to K-corrections that limits the dependence on SED templates. We perform a polynomial fit for the K-correction as a function of a galaxy’s rest-frame colour determined in a pair of well-constrained bands (e.g. 0(g − r)) and redshift, exploiting the fact that galaxy SEDs can be approximated as a one-parameter family at low redshift. For bands well-constrained by SED templates, our empirically driven K-corrections yield results comparable to the SED fitting methods used by Kcorrect and the GSWLC-M2 catalogue (the updated medium-deep GALEX–SDSS–WISE Legacy Catalogue). However, our method dramatically outperforms Kcorrect derived K-corrections for WISE W4. Our method is also robust to incorrect template assumptions outside of the optical bands and enforces that the K-correction must be zero at z = 0. Our K-corrected photometry and code are publicly available.

Phase transition for the generalized two-community stochastic block model

AbstractWe study the problem of detecting the community structure from the generalized stochastic block model with two communities (G2-SBM). Based on analysis of the Stieljtes transform of the empirical spectral distribution, we prove a Baik–Ben Arous–Péché (BBP)-type transition for the largest eigenvalue of the G2-SBM. For specific models, such as a hidden community model and an unbalanced stochastic block model, we provide precise formulas for the two largest eigenvalues, establishing the gap in the BBP-type transition.

Limiting empirical spectral distribution for the non-backtracking matrix of an Erdős-Rényi random graph

AbstractIn this note, we give a precise description of the limiting empirical spectral distribution for the non-backtracking matrices for an Erdős-Rényi graph $G(n,p)$ assuming $np/\log n$ tends to infinity. We show that derandomizing part of the non-backtracking random matrix simplifies the spectrum considerably, and then, we use Tao and Vu’s replacement principle and the Bauer-Fike theorem to show that the partly derandomized spectrum is, in fact, very close to the original spectrum.

On the concavity of the TAP free energy in the SK model

We analyze the Hessian of the Thouless–Anderson–Palmer (TAP) free energy for the Sherrington–Kirkpatrick model, below the de Almeida–Thouless line, evaluated in Bolthausen’s approximate solutions of the TAP equations. We show that the empirical spectral distribution weakly converges to a measure with negative support below the AT line, and that the support includes zero on the AT line. In this “macroscopic” sense, we show that TAP free energy is concave in the order parameter of the theory, i.e. the random spin-magnetizations. This proves a spectral interpretation of the AT line. We also find different magnetizations than Bolthausen’s approximate solutions at which the Hessian of the TAP free energy has positive outlier eigenvalues. In particular, when the magnetizations are assumed to be independent of the disorder, we prove that Plefka’s second condition is equivalent to all eigenvalues being negative. On this occasion, we extend the convergence result of Capitaine et al. (Electron. J. Probab. 16, no. 64, 2011) for the largest eigenvalue of perturbed complex Wigner matrices to the GOE.

A Riemann–Hilbert Approach to the Perturbation Theory for Orthogonal Polynomials: Applications to Numerical Linear Algebra and Random Matrix Theory

Abstract We establish a new perturbation theory for orthogonal polynomials using a Riemann–Hilbert approach and consider applications in numerical linear algebra and random matrix theory. This new approach shows that the orthogonal polynomials with respect to two measures can be effectively compared using the difference of their Stieltjes transforms on a suitably chosen contour. Moreover, when two measures are close and satisfy some regularity conditions, we use the theta functions of a hyperelliptic Riemann surface to derive explicit and accurate expansion formulae for the perturbed orthogonal polynomials. In contrast to other approaches, a key strength of the methodology is that estimates can remain valid as the degree of the polynomial grows. The results are applied to analyze several numerical algorithms from linear algebra, including the Lanczos tridiagonalization procedure, the Cholesky factorization, and the conjugate gradient algorithm. As a case study, we investigate these algorithms applied to a general spiked sample covariance matrix model by considering the eigenvector empirical spectral distribution and its limits. For the first time, we give precise estimates on the output of the algorithms, applied to this wide class of random matrices, as the number of iterations diverges. In this setting, beyond the first order expansion, we also derive a new mesoscopic central limit theorem for the associated orthogonal polynomials and other quantities relevant to numerical algorithms.

The eigenvector LSD of information plus noise matrices and its application to linear regression model

In this paper, we investigate the limit of the eigenvector empirical spectral distribution (VESD) of large dimensional information plus noise matrices Cn=1nTn1/2(Xn+Rn)(Xn+Rn)′Tn1/2, where Xn are p×n random matrices with independent and identically distributed entries, Tn and Rn are non-random matrices. It is shown that as p/n→c∈(0,∞), the VESD of Cn has a deterministic limit under some mild conditions. The theoretical result is applied to the model selection problems in high dimensional linear regression model.

Large sample covariance matrices of Gaussian observations with uniform correlation decay

We derive the Marchenko–Pastur (MP) law for sample covariance matrices of the form Vn=1nXXT, where X is a p×n data matrix and p/n→y∈(0,∞) as n,p→∞. We assume the data in X stems from a correlated joint normal distribution. In particular, the correlation acts both across rows and across columns of X, and we do not assume a specific correlation structure, such as separable dependencies. Instead, we assume that correlations converge uniformly to zero at a speed of an/n, where an may grow mildly to infinity. We employ the method of moments tightly: We identify the exact condition on the growth of an which will guarantee that the moments of the empirical spectral distributions (ESDs) converge to the MP moments. If the condition is not met, we can construct an ensemble for which all but finitely many moments of the ESDs diverge. We also investigate the operator norm of Vn under a uniform correlation bound of C/nδ, where C,δ>0 are fixed, and observe a phase transition at δ=1. In particular, convergence of the operator norm to the maximum of the support of the MP distribution can only be guaranteed if δ>1. The analysis leads to an example for which the MP law holds almost surely, but the operator norm remains stochastic in the limit, and we provide its exact limiting distribution.

Limit Theorem for Spectra of Laplace Matrix of Random Graphs

We consider the limit of the empirical spectral distribution of Laplace matrices of generalized random graphs. Applying the Stieltjes transform method, we prove under general conditions that the limit spectral distribution of Laplace matrices converges to the free convolution of the semicircular law and the normal law.

Partial Linear Eigenvalue Statistics for Non-Hermitian Random Matrices

For an $n \times n$ independent-entry random matrix $X_n$ with eigenvalues $\lambda_1, \dots, \lambda_n$, the seminal work of Rider and Silverstein [Ann. Probab., 34 (2006), pp. 2118--2143] asserts that the fluctuations of the linear eigenvalue statistics $\sum_{i=1}^n f(\lambda_i)$ converge to a Gaussian distribution for sufficiently nice test functions $f$. We study the fluctuations of $\sum_{i=1}^{n-K} f(\lambda_i)$, where $K$ randomly chosen eigenvalues have been removed from the sum. In this case, we identify the limiting distribution and show that it need not be Gaussian. Our results hold for the case when $K$ is fixed as well as for the case when $K$ tends to infinity with $n$. The proof utilizes the predicted locations of the eigenvalues introduced by E. Meckes and M. Meckes, [Ann. Fac. Sci. Toulouse Math. (6), 24 (2015), pp. 93--117]. As a consequence of our methods, we obtain a rate of convergence for the empirical spectral distribution of $X_n$ to the circular law in Wasserstein distance, which may be of independent interest.

Sparse matrices: convergence of the characteristic polynomial seen from infinity

We prove that the reverse characteristic polynomial det(In−zAn) of a random n×n matrix An with iid Bernoulli(d∕n) entries converges in distribution towards the random infinite product ∏ℓ=1∞(1−zℓ)Yℓ where Yℓ are independent Poisson(dℓ∕ℓ) random variables. We show that this random function is a Poisson analog of more classical Gaussian objects such as the Gaussian holomorphic chaos. As a byproduct, we obtain new simple proofs of previous results on the asymptotic behaviour of extremal eigenvalues of sparse Erdős-Rényi digraphs: for every d>1, the greatest eigenvalue of An is close to d and the second greatest is smaller than d, a Ramanujan-like property for irregular digraphs. For d<1, the only non-zero eigenvalues of An converge to a Poisson multipoint process on the unit circle. Our results also extend to the semi-sparse regime where d is allowed to grow to ∞ with n, slower than no(1). We show that the reverse characteristic polynomial converges towards a more classical object written in terms of the exponential of a log-correlated real Gaussian field. In the semi-sparse regime, the empirical spectral distribution of An∕ dn converges to the circle distribution; as a consequence of our results, the second eigenvalue sticks to the edge of the circle.

On the eigenvectors of large-dimensional sample spatial sign covariance matrices

To investigate the limiting behavior of eigenvectors of the sample spatial sign covariance matrix (SSCM), the eigenvector empirical spectral distribution (VESD) is defined with weights depending on the eigenvectors. In this paper, we first show that the VESD of a large-dimensional sample SSCM converges to a generalized Marčenko–Pastur distribution when both the dimension p of observations and the sample size n tend to infinity proportionally. Further, the central limit theorem of linear spectral statistics of VESD is established, which suggests that the eigenmatrix of sample SSCM and the classical sample covariance matrix are asymptotically the same.

Paving the way forEuclid and JWST via probabilistic selection of high-redshift quasars

ABSTRACT We introduce a probabilistic approach to select 6 ≤ $z$ ≤ 8 quasar candidates for spectroscopic follow-up, which is based on density estimation in the high-dimensional space inhabited by the optical and near-infrared photometry. Densities are modelled as Gaussian mixtures with principled accounting of errors using the extreme deconvolution (XD) technique, generalizing an approach successfully used to select lower redshift ($z$ ≤ 3) quasars. We train the probability density of contaminants on 1902 071 7-d flux measurements from the 1076 deg2 overlapping area from the Dark Energy Camera Legacy Survey (DECaLS) ($z$), VIKING (YJHKs), and unWISE (W1W2) imaging surveys, after requiring they dropout of DECaLS g and r, whereas the distribution of high-$z$ quasars are trained on synthetic model photometry. Extensive simulations based on these density distributions and current estimates of the quasar luminosity function indicate that this method achieves a completeness of $\ge 56{{\ \rm per\ cent}}$ and an efficiency of $\ge 5{{\ \rm per\ cent}}$ for selecting quasars at 6 &amp;lt; $z$ &amp;lt; 8 with JAB &amp;lt; 21.5. Among the classified sources are 8 known 6 &amp;lt; $z$ &amp;lt; 7 quasars, of which 2/8 are selected suggesting a completeness $\simeq 25{{\ \rm per\ cent}}$, whereas classifying the 6 known (JAB &amp;lt; 21.5) quasars at $z$ &amp;gt; 7 from the entire sky, we select 5/6 or a completeness of $\simeq 80{{\ \rm per\ cent}}$. The failure to select the majority of 6 &amp;lt; $z$ &amp;lt; 7 quasars arises because our quasar density model is based on an empirical quasar spectral energy distribution model that underestimates the scatter in the distribution of fluxes. This new approach to quasar selection paves the way for efficient spectroscopic follow-up of Euclid quasar candidates with ground-based telescopes and James Webb Space Telescope.