Independent Entries Research Articles

The all-or-nothing phenomenon in sparse linear regression

We study the problem of recovering a hidden binary k -sparse p -dimensional vector \beta from n noisy linear observations Y=X\beta+W , where X_{ij} are i.i.d. \mathcal{N}(0,1) and W_i are i.i.d. \mathcal{N}(0,\sigma^2) . A closely related hypothesis testing problem is to distinguish the pair (X,Y) generated from this structured model from a corresponding null model where (X,Y) consist of purely independent Gaussian entries. In the low sparsity k=o(\sqrt{p}) and high signal-to-noise ratio k/\sigma^2 \to \infty regime, we establish an “all-or-nothing” information-theoretic phase transition at a critical sample size n^{\ast}=2 k\log (p/k) /\log (1+k/\sigma^2) , resolving a conjecture of Gamarnik and Zadik (2017). Specifically, we show that if \liminf_{p\to \infty} n/n^{\ast}>1 , then the maximum likelihood estimator almost perfectly recovers the hidden vector with high probability and moreover the true hypothesis can be detected with a vanishing error probability. Conversely, if \limsup_{p\to \infty} n/n^{\ast}<1 , then it becomes information-theoretically impossible even to recover an arbitrarily small but fixed fraction of the hidden vector support, or to test hypotheses strictly better than random guess. Our proof of the impossibility result builds upon two key techniques, which could be of independent interest. First, we use a conditional second moment method to upper bound the Kullback–Leibler (KL) divergence between the structured and the null model. Second, inspired by the celebrated area theorem, we establish a lower bound to the minimum mean squared estimation error of the hidden vector in terms of the KL divergence between the two models. An extended abstract version of this work appeared at the Proceedings of the Conference on Learning Theory (COLT), 2019.

The smallest singular value of heavy-tailed not necessarily i.i.d. random matrices via random rounding

We show the existence of a net near the sphere, such that the values of any matrix on the sphere and on the net are compared via a regularized Hilbert-Schmidt norm, which we introduce. This allows to construct an efficient net which controls the length of Ax for any random matrix A with independent columns (no other assumptions are required). As a consequence we show that the smallest singular value σn (A) of an N × n random matrix A with i.i.d. mean zero, variance one entries enjoys the following small ball estimate, for any ϵ > 0 $$P({\sigma _n}(A) < \epsilon (\sqrt {N + 1} - \sqrt n )) \le {(C\epsilon \,\log \,1/\epsilon )^{N - n + 1}} + {e^{ - cN}}.$$ The proof of this result requires working with matrices whose rows are not independent, and, therefore, the fact that the theorem about discretization works for matrices with dependent rows, is crucial. Furthermore, in the case of the square n×n matrix A with independent entries having concentration function separated from 1, i.i.d. rows, and such that $$\mathbb{E}\left\| A \right\|_{HS}^2 \le c{n^2}$$ , one has $$P({\sigma _n}(A) < {\epsilon \over {\sqrt n }} \le C\epsilon + {e^{ - cn}},$$ for any ϵ > 0. In addition, for $$\epsilon > {c \over {\sqrt n }}$$ the assumption of i.i.d. rows is not required. Our estimates generalize the previous results of Rudelson and Vershynin [29], [30], which required the sub-gaussian mean zero variance one assumptions, as well as the work of Rebrova and Tikhomirov [25], where mean zero variance 1 and i.i.d. entries were required.

Tackling Small Eigen-Gaps: Fine-Grained Eigenvector Estimation and Inference Under Heteroscedastic Noise

This paper aims to address two fundamental challenges arising in eigenvector estimation and inference for a low-rank matrix from noisy observations: 1) how to estimate an unknown eigenvector when the eigen-gap (i.e. the spacing between the associated eigenvalue and the rest of the spectrum) is particularly small; 2) how to perform estimation and inference on linear functionals of an eigenvector—a sort of “fine-grained” statistical reasoning that goes far beyond the usual <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\ell _{2}$ </tex-math></inline-formula> analysis. We investigate how to address these challenges in a setting where the unknown <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n\times n$ </tex-math></inline-formula> matrix is symmetric and the additive noise matrix contains independent (and non-symmetric) entries. Based on eigen-decomposition of the asymmetric data matrix, we propose estimation and uncertainty quantification procedures for an unknown eigenvector, which further allow us to reason about linear functionals of an unknown eigenvector. The proposed procedures and the accompanying theory enjoy several important features: 1) distribution-free (i.e. prior knowledge about the noise distributions is not needed); 2) adaptive to heteroscedastic noise; 3) minimax optimal under Gaussian noise. Along the way, we establish valid procedures to construct confidence intervals for the unknown eigenvalues. All this is guaranteed even in the presence of a small eigen-gap (up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$O(\sqrt {n/\mathrm {poly}\log (n)}\,)$ </tex-math></inline-formula> times smaller than the requirement in prior theory), which goes significantly beyond what generic matrix perturbation theory has to offer.

Extremal eigenvalues of sample covariance matrices with general population

We consider the eigenvalues of sample covariance matrices of the form Q=(Σ1/2X)(Σ1/2X)∗. The sample X is an M×N rectangular random matrix with real independent entries and the population covariance matrix Σ is a positive definite diagonal matrix independent of X. Assuming that the limiting spectral density of Σ exhibits convex decay at the right edge of the spectrum, in the limit M,N→∞ with N/M→d∈(0,∞), we find a certain threshold d+ such that for d>d+ the limiting spectral distribution of Q also exhibits convex decay at the right edge of the spectrum. In this case, the largest eigenvalues of Q are determined by the order statistics of the eigenvalues of Σ, and in particular, the limiting distribution of the largest eigenvalue of Q is given by a Weibull distribution. In case d<d+, we also prove that the limiting distribution of the largest eigenvalue of Q is Gaussian if the entries of Σ are i.i.d. random variables. While Σ is considered to be random mostly, the results also hold for deterministic Σ with some additional assumptions.

Asymptotic behaviour of linear eigenvalue statistics of Hankel matrices

We study linear eigenvalue statistics of band Hankel matrices with Brownian motion entries. We prove that the centred, normalized linear eigenvalue statistics of band Hankel matrices obey a central limit theorem (CLT) type result. We also discuss the convergence of linear eigenvalue statistics of band Hankel matrices with independent entries for odd power monomials. Our method is based on trace formula, moment method and some results of process convergence.

Characteristic Polynomials of Random Banded Hessenberg Matrices and Hermite–Padé Approximation

We consider a class of random banded Hessenberg matrices with independent entries having identical distributions along diagonals. The distributions may be different for entries belonging to different diagonals. For a sequence of \(n\times n\) matrices in the class considered, we investigate the asymptotic behavior of their empirical spectral distribution as n tends to infinity.

Spectral radius of random matrices with independent entries

We consider random $n\times n$ matrices $X$ with independent and centered entries and a general variance profile. We show that the spectral radius of $X$ converges with very high probability to the square root of the spectral radius of the variance matrix of $X$ when $n$ tends to infinity. We also establish the optimal rate of convergence, that is a new result even for general i.i.d. matrices beyond the explicitly solvable Gaussian cases. The main ingredient is the proof of the local inhomogeneous circular law [arXiv:1612.07776] at the spectral edge.

Fluctuations of linear eigenvalue statistics of reverse circulant and symmetric circulant matrices with independent entries

In this article, we study the fluctuations of linear eigenvalue statistics of reverse circulant (RCn) and symmetric circulant (SCn) matrices with independent entries that satisfy some moment conditions. We show that 1nTrϕ(RCn) and 1nTrϕ(SCn) obey the central limit theorem type result, where ϕ is a polynomial test function.

Optimal fast Johnson\u2013Lindenstrauss embeddings for large data sets

Johnson–Lindenstrauss embeddings are widely used to reduce the dimension and thus the processing time of data. To reduce the total complexity, also fast algorithms for applying these embeddings are necessary. To date, such fast algorithms are only available either for a non-optimal embedding dimension or up to a certain threshold on the number of data points. We address a variant of this problem where one aims to simultaneously embed larger subsets of the data set. Our method follows an approach by Nelson et al. (New constructions of RIP matrices with fast multiplication and fewer rows. In: Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1515-1528, 2014): a subsampled Hadamard transform maps points into a space of lower, but not optimal dimension. Subsequently, a random matrix with independent entries projects to an optimal embedding dimension. For subsets whose size scales at least polynomially in the ambient dimension, the complexity of this method comes close to the number of operations just to read the data under mild assumptions on the size of the data set that are considerably less restrictive than in previous works. We also prove a lower bound showing that subsampled Hadamard matrices alone cannot reach an optimal embedding dimension. Hence, the second embedding cannot be omitted.

Deep Boltzmann Machines: Rigorous Results at Arbitrary Depth

A class of deep Boltzmann machines is considered in the simplified framework of a quenched system with Gaussian noise and independent entries. The quenched pressure of a K-layers spin glass model is studied allowing interactions only among consecutive layers. A lower bound for the pressure is found in terms of a convex combination of K Sherrington–Kirkpatrick models and used to study the annealed and replica symmetric regimes of the system. A map with a one-dimensional monomer–dimer system is identified and used to rigorously control the annealed region at arbitrary depth K with the methods introduced by Heilmann and Lieb. The compression of this high-noise region displays a remarkable phenomenon of localisation of the processing layers. Furthermore, a replica symmetric lower bound for the limiting quenched pressure of the model is obtained in a suitable region of the parameters and the replica symmetric pressure is proved to have a unique stationary point.

Genomic Epidemiology of SARS-CoV-2 in Madrid, Spain, during the First Wave of the Pandemic: Fast Spread and Early Dominance by D614G Variants.

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) was first detected in Madrid, Spain, on 25 February 2020. It increased in frequency very fast and by the end of May more than 70,000 cases had been confirmed by reverse transcription-polymerase chain reaction (RT-PCR). To study the lineages and the diversity of the viral population during this first epidemic wave in Madrid we sequenced 224 SARS-CoV-2 viral genomes collected from three hospitals from February to May 2020. All the known major lineages were found in this set of samples, though B.1 and B.1.5 were the most frequent ones, accounting for more than 60% of the sequences. In parallel with the B lineages and sublineages, the D614G mutation in the Spike protein sequence was detected soon after the detection of the first coronavirus disease 19 (COVID-19) case in Madrid and in two weeks became dominant, being found in 80% of the samples and remaining at this level during all the study periods. The lineage composition of the viral population found in Madrid was more similar to the European population than to the publicly available Spanish data, underlining the role of Madrid as a national and international transport hub. In agreement with this, phylodynamic analysis suggested multiple independent entries before the national lockdown and air transportation restrictions.

ASYMMETRY HELPS: EIGENVALUE AND EIGENVECTOR ANALYSES OF ASYMMETRICALLY PERTURBED LOW-RANK MATRICES.

This paper is concerned with the interplay between statistical asymmetry and spectral methods. Suppose we are interested in estimating a rank-1 and symmetric matrix , yet only a randomly perturbed version M is observed. The noise matrix M - M ⋆ is composed of independent (but not necessarily homoscedastic) entries and is, therefore, not symmetric in general. This might arise if, for example, we have two independent samples for each entry of M ⋆ and arrange them in an asymmetric fashion. The aim is to estimate the leading eigenvalue and the leading eigenvector of M ⋆. We demonstrate that the leading eigenvalue of the data matrix M can be times more accurate (up to some log factor) than its (unadjusted) leading singular value of M in eigenvalue estimation. Moreover, the eigen-decomposition approach is fully adaptive to heteroscedasticity of noise, without the need of any prior knowledge about the noise distributions. In a nutshell, this curious phenomenon arises since the statistical asymmetry automatically mitigates the bias of the eigenvalue approach, thus eliminating the need of careful bias correction. Additionally, we develop appealing non-asymptotic eigenvector perturbation bounds; in particular, we are able to bound the perterbation of any linear function of the leading eigenvector of M (e.g. entrywise eigenvector perturbation). We also provide partial theory for the more general rank-r case. The takeaway message is this: arranging the data samples in an asymmetric manner and performing eigen-decomposition could sometimes be quite beneficial.

Positive solutions for large random linear systems

Consider a large linear system where $A_n$ is a $n\times n$ matrix with independent real standard Gaussian entries, $\boldsymbol{1}_n$ is a $n\times 1$ vector of ones and with unknown the $n\times 1$ vector $\boldsymbol{x}_n$ satisfying $$ \boldsymbol{x}_n = \boldsymbol{1}_n +\frac 1{\alpha_n\sqrt{n}} A_n \boldsymbol{x}_n\, . $$ We investigate the (componentwise) positivity of the solution $\boldsymbol{x}_n$ depending on the scaling factor $\alpha_n$ as the dimension $n$ goes to $\infty$. We prove that there is a sharp phase transition at the threshold $\alpha^*_n =\sqrt{2\log n}$: below the threshold ($\alpha_n\ll \sqrt{2\log n}$), $\boldsymbol{x}_n$ has negative components with probability tending to 1 while above ($\alpha_n\gg \sqrt{2\log n}$), all the vector's components are eventually positive with probability tending to 1. At the critical scaling $\alpha^*_n$, we provide a heuristics to evaluate the probability that $\boldsymbol{x}_n$ is positive. Such linear systems arise as solutions at equilibrium of large Lotka-Volterra systems of differential equations, widely used to describe large biological communities with interactions such as foodwebs for instance. In the domaine of positivity of the solution $\boldsymbol{x}_n$, that is when $\alpha_n\gg \sqrt{2\log n}$, we establish that the Lotka-Volterra system of differential equations whose solution at equilibrium is precisely $\x_n$ is stable in the sense that its jacobian $$ {\mathcal J}(\boldsymbol{x}_n) = \mathrm{diag}(\boldsymbol{x}_n)\left(-I_n + \frac {A_n}{\alpha_n\sqrt{n}}\right) $$ has all its eigenvalues with negative real part with probability tending to one. Our results shed a new light and complement the understanding of feasibility and stability issues for large biological communities with interaction.

The spectral norm of random lifts of matrices

We study the spectral norm of random lifts of matrices. Given an n×n symmetric matrix A, and a centered distribution π on k×k(k≥2) symmetric matrices with spectral norm at most 1, let the matrix random lift A(k,π) be the random symmetric kn×kn matrix (AijXij)1≤i<j≤n, where Xij are independent samples from π. We prove that E‖A(k,π)‖≲max i ∑j Aij2+maxij|Aij| log(kn). This result can be viewed as an extension of existing spectral bounds on random matrices with independent entries, providing further instances where the multiplicative logn factor in the Non-Commutative Khintchine inequality can be removed. As a direct application of our result, we prove an upper bound of 2(1+ϵ) Δ+O( log(kn)) on the new eigenvalues for random k-lifts of a fixed G=(V,E) with |V|=n and maximum degree Δ, compared to the previous result of O( Δlog(kn)) by Oliveira [Oli10a] and the recent breakthrough by Bordenave and Collins [BC19] which gives 2 Δ−1+o(1) as k→∞ for Δ-regular graph G.

Rate of convergence for products of independent non-Hermitian random matrices

We study the rate of convergence of the empirical spectral distribution of products of independent non-Hermitian random matrices to the power of the Circular Law. The distance to the deterministic limit distribution will be measured in terms of a uniform Kolmogorov-like distance. First, we prove that for products of Ginibre matrices, the optimal rate is given by O(1∕n), which is attained with overwhelming probability up to a logarithmic correction. Avoiding the edge, the rate of convergence of the mean empirical spectral distribution is even faster. Second, we show that also products of matrices with independent entries attain this optimal rate in the bulk up to a logarithmic factor. In the case of Ginibre matrices, we apply a saddlepoint approximation to a double contour integral representation of the density and in the case of matrices with independent entries we make use of techniques from local laws.

Recursive max-linear models with propagating noise

Recursive max-linear vectors model causal dependence between node variables by a structural equation model, expressing each node variable as a max-linear function of its parental nodes in a directed acyclic graph (DAG) and some exogenous innovation. For such a model, there exists a unique minimum DAG, represented by the Kleene star matrix of its edge weight matrix, which identifies the model and can be estimated. For a more realistic statistical modeling we introduce some random observational noise. A probabilistic analysis of this new noisy model reveals that the unique minimum DAG representing the distribution of the non-noisy model remains unchanged and identifiable. Moreover, the distribution of the minimum ratio estimators of the model parameters at their left limits are completely determined by the distribution of the noise variables up to a positive constant. Under a regular variation condition on the noise variables we prove that the estimated Kleene star matrix converges to a matrix of independent Weibull entries after proper centering and scaling.

Multi-Layer Bilinear Generalized Approximate Message Passing

In this paper, we extend the bilinear generalized approximate message passing (BiG-AMP) approach, originally proposed for high-dimensional generalized bilinear regression, to the multi-layer case for the handling of cascaded problem such as matrix-factorization problem arising in relay communication among others. Assuming statistically independent matrix entries with known priors, the new algorithm called ML-BiGAMP could approximate the general sum-product loopy belief propagation (LBP) in the high-dimensional limit enjoying a substantial reduction in computational complexity. We demonstrate that, in large system limit, the asymptotic MSE performance of ML-BiGAMP could be fully characterized via a set of simple one-dimensional equations termed state evolution (SE). We establish that the asymptotic MSE predicted by ML-BiGAMP' SE matches perfectly the exact MMSE predicted by the replica method, which is well-known to be Bayes-optimal but infeasible in practice. This consistency indicates that the ML-BiGAMP may still retain the same Bayes-optimal performance as the MMSE estimator in high-dimensional applications, although ML-BiGAMP's computational burden is far lower. As an illustrative example of the general ML-BiGAMP, we provide a detector design that could estimate the channel fading and the data symbols jointly with high precision for the two-hop amplify-and-forward relay communication systems.

Some patterned matrices with independent entries

Patterned random matrices such as the reverse circulant, the symmetric circulant, the Toeplitz and the Hankel matrices and their almost sure limiting spectral distribution (LSD), have attracted much attention. Under the assumption that the entries are taken from an i.i.d. sequence with finite variance, the LSD are tied together by a common thread — the [Formula: see text]th moment of the limit equals a weighted sum over different types of pair-partitions of the set [Formula: see text] and are universal. Some results are also known for the sparse case. In this paper, we generalize these results by relaxing significantly the i.i.d. assumption. For our models, the limits are defined via a larger class of partitions and are also not universal. Several existing and new results for patterned matrices, their band and sparse versions, as well as for matrices with continuous and discrete variance profile follow as special cases.

Process convergence of fluctuations of linear eigenvalue statistics of random circulant matrices

We discuss the process convergence of the time dependent fluctuations of linear eigenvalue statistics of random circulant matrices with independent Brownian motion entries, as the dimension of the matrix tends to [Formula: see text]. Our derivation is based on the trace formula of circulant matrix, method of moments and some combinatorial techniques.

Nominal correlation of inhomogeneous random sequences

In this paper, we consider random sequences with independent entries that are not necessarily identically distributed and obtain bounds on the maximal and minimal correlation of such sequences. We also determine sufficient conditions under which the scaled maximal and minimal correlations both converge to a common nominal value.