High-dimensional Regime Research Articles

Penalized ensemble Kalman filters for high dimensional non-linear systems.

The ensemble Kalman filter (EnKF) is a data assimilation technique that uses an ensemble of models, updated with data, to track the time evolution of a usually non-linear system. It does so by using an empirical approximation to the well-known Kalman filter. However, its performance can suffer when the ensemble size is smaller than the state space, as is often necessary for computationally burdensome models. This scenario means that the empirical estimate of the state covariance is not full rank and possibly quite noisy. To solve this problem in this high dimensional regime, we propose a computationally fast and easy to implement algorithm called the penalized ensemble Kalman filter (PEnKF). Under certain conditions, it can be theoretically proven that the PEnKF will be accurate (the estimation error will converge to zero) despite having fewer ensemble members than state dimensions. Further, as contrasted to localization methods, the proposed approach learns the covariance structure associated with the dynamical system. These theoretical results are supported with simulations of several non-linear and high dimensional systems.

Limiting laws for extreme eigenvalues of large-dimensional spiked Fisher matrices with a divergent number of spikes

Consider the p×p matrix that is the product of a population covariance matrix and the inverse of another population covariance matrix. Suppose that their difference has a divergent rank with respect to p, when two samples of sizes n and T from the two populations are available, we construct its corresponding sample version. In the high-dimensional regime where both n and T are proportional to p, we investigate the limiting laws for extreme (spiked) eigenvalues of the sample (spiked) Fisher matrix when the number of spikes is divergent and these spikes are unbounded. We derive the convergence in probability of these spiked eigenvalues after scaling, and the central limit theorem for normalized spiked eigenvalues.

De-Biased Sparse PCA: Inference for Eigenstructure of Large Covariance Matrices

Sparse principal component analysis has become one of the most widely used techniques for dimensionality reduction in high-dimensional datasets. While many methods are available for point estimation of eigenstructure in high-dimensional settings, in this paper we propose methodology for uncertainty quantification, such as construction of confidence intervals and tests for the principal eigenvector and the corresponding largest eigenvalue. We base our methodology on an M-estimator with Lasso penalty which achieves minimax optimal rates and is used to construct a de-biased sparse PCA estimator. The novel estimator has a Gaussian limiting distribution and can be used for hypothesis testing or support recovery of the first eigenvector. The empirical performance of the new estimator is demonstrated on synthetic data and we also show that the estimator compares favourably with the classical PCA in moderately high-dimensional regimes.

Non-parametric sign prediction of high-dimensional correlation matrix coefficients

We introduce a method to predict which correlation matrix coefficients are likely to change their signs in the future in the high-dimensional regime, i.e., when the number of features is larger than the number of samples per feature. The stability of correlation signs, two-by-two relationships, is found to depend on three-by-three relationships inspired by Heider social cohesion theory in this regime. We apply our method to US and Hong Kong equities historical data to illustrate how the structure of correlation matrices influences the stability of the sign of its coefficients.

A CLT in Stein’s Distance for Generalized Wishart Matrices and Higher-Order Tensors

AbstractWe study the a central limit theorem for sums of independent tensor powers, $\frac{1}{\sqrt{d}}\sum \limits _{i=1}^d X_i^{\otimes p}$. We focus on the high-dimensional regime where $X_i \in{\mathbb{R}}^n$ and $n$ may scale with $d$. Our main result is a proposed threshold for convergence. Specifically, we show that, under some regularity assumption, if $n^{2p-1}\ll d$, then the normalized sum converges to a Gaussian. The results apply, among others, to symmetric uniform log-concave measures and to product measures. This generalizes several results found in the literature. Our main technique is a novel application of optimal transport to Stein’s method, which accounts for the low-dimensional structure, which is inherent in $X_i^{\otimes p}$.

Large-sample properties of unsupervised estimation of the linear discriminant using projection pursuit

We study the estimation of the linear discriminant with projection pursuit, a method that is unsupervised in the sense that it does not use the class labels in the estimation. Our viewpoint is asymptotic and, as our main contribution, we derive central limit theorems for estimators based on three different projection indices, skewness, kurtosis, and their convex combination. The results show that in each case the limiting covariance matrix is proportional to that of linear discriminant analysis (LDA), a supervised estimator of the discriminant. An extensive comparative study between the asymptotic variances reveals that projection pursuit gets arbitrarily close in efficiency to LDA when the distance between the groups is large enough and their proportions are reasonably balanced. Additionally, we show that consistent unsupervised estimation of the linear discriminant can be achieved also in high-dimensional regimes where the dimension grows at a suitable rate to the sample size, for example, pn=o(n1∕3) is sufficient under skewness-based projection pursuit. We conclude with a real data example and a simulation study investigating the validity of the obtained asymptotic formulas for finite samples.

Multi-Reference Alignment in High Dimensions: Sample Complexity and Phase Transition

Multi-reference alignment entails estimating a signal in $\mathbb{R}^L$ from its circularly-shifted and noisy copies. This problem has been studied thoroughly in recent years, focusing on the finite-dimensional setting (fixed $L$). Motivated by single-particle cryo-electron microscopy, we analyze the sample complexity of the problem in the high-dimensional regime $L\to\infty$. Our analysis uncovers a phase transition phenomenon governed by the parameter $\alpha = L/(\sigma^2\log L)$, where $\sigma^2$ is the variance of the noise. When $\alpha>2$, the impact of the unknown circular shifts on the sample complexity is minor. Namely, the number of measurements required to achieve a desired accuracy $\varepsilon$ approaches $\sigma^2/\varepsilon$ for small $\varepsilon$; this is the sample complexity of estimating a signal in additive white Gaussian noise, which does not involve shifts. In sharp contrast, when $\alpha\leq 2$, the problem is significantly harder and the sample complexity grows substantially quicker with $\sigma^2$.

Information Theoretic Limits for Phase Retrieval With Subsampled Haar Sensing Matrices

We study information theoretic limits of recovering an unknown ${n}$ dimensional, complex signal vector ${x}_{\star} $ with unit norm from ${m}$ magnitude-only measurements of the form ${y}_{i} = |({A}{x}_{\star} )_{i}|^{2}, \; {i} = 1,2 {\dots }, {m}$ , where ${A}$ is the sensing matrix. This is known as the Phase Retrieval problem and models practical imaging systems where measuring the phase of the observations is difficult. Since in a number of applications, the sensing matrix has orthogonal columns, we model the sensing matrix as a subsampled Haar matrix formed by picking ${n}$ columns of a uniformly random ${m} \times {m}$ unitary matrix. We study this problem in the high dimensional asymptotic regime, where ${m},{n} \rightarrow \infty $ , while ${m}/{n} \rightarrow \delta $ with $\delta $ being a fixed number, and show that if ${m} , then any estimator is asymptotically orthogonal to the true signal vector ${x}_{\star} $ . This lower bound is sharp since when ${m} > (2+{o}_{n}(1)) \cdot {n} $ , estimators that achieve a non trivial asymptotic correlation with the signal vector are known from previous works.

Information-Theoretic Limits for the Matrix Tensor Product

This article studies a high-dimensional inference problem involving the matrix tensor product of random matrices. This problem generalizes a number of contemporary data science problems including the spiked matrix models used in sparse principal component analysis and covariance estimation and the stochastic block model used in network analysis. The main results are single-letter formulas (i.e., analytical expressions that can be approximated numerically) for the mutual information and the minimum mean-squared error (MMSE) in the Bayes optimal setting where the distributions of all random quantities are known. We provide non-asymptotic bounds and show that our formulas describe exactly the leading order terms in the mutual information and MMSE in the high-dimensional regime where the number of rows n and number of columns d scale with d = O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">α</sup> ) for some α <; 1/20. On the technical side, this article introduces some new techniques for the analysis of high-dimensional matrix-valued signals. Specific contributions include a novel extension of the adaptive interpolation method that uses order-preserving positive semidefinite interpolation paths, and a variance inequality between the overlap and the free energy that is based on continuous-time I-MMSE relations.

Generalized Autoregressive Linear Models for Discrete High-Dimensional Data

Fitting multivariate autoregressive (AR) models is fundamental for time-series data analysis in a wide range of applications. This article considers the problem of learning a p-lag multivariate AR model where each time step involves a linear combination of the past p states followed by a probabilistic, possibly nonlinear, mapping to the next state. The problem is to learn the linear connectivity tensor from observations of the states. We focus on the sparse setting, which arises in applications with a limited number of direct connections between variables. For such problems, 11-regularized maximum likelihood estimation (or M-estimation more generally) is often straightforward to apply and works well in practice. However, the analysis of such methods is difficult due to the feedback in the state dynamic and the presence of nonlinearities, especially when the underlying process is non-Gaussian. Our main result provides a bound on the mean-squared error of the estimated connectivity tensor as a function of the sparsity and the number of samples, for a class of discrete multivariate AR models, in the high-dimensional regime. Importantly, the bound indicates that, with sufficient sparsity, consistent estimation is possible in cases where the number of samples is significantly less than the total number of parameters.

The Spiked Matrix Model With Generative Priors

We investigate the statistical and algorithmic properties of random neural-network generative priors in a simple inference problem: spiked-matrix estimation. We establish a rigorous expression for the performance of the Bayes-optimal estimator in the high-dimensional regime, and identify the statistical threshold for weak-recovery of the spike. Next, we derive a message-passing algorithm taking into account the latent structure of the spike, and show that its performance is asymptotically optimal for natural choices of the generative network architecture. The absence of an algorithmic gap in this case is in stark contrast to known results for sparse spikes, another popular prior for modelling low-dimensional signals, and for which no algorithm is known to achieve the optimal statistical threshold. Finally, we show that linearising our message passing algorithm yields a simple spectral method also achieving the optimal threshold for reconstruction. We conclude with an experiment on a real data set showing that our bespoke spectral method outperforms vanilla PCA.

Asymptotic normality for random simplices and convex bodies in high dimensions

Central limit theorems for the log-volume of a class of random convex bodies in R n \mathbb {R}^n are obtained in the high-dimensional regime, that is, as n → ∞ n\to \infty . In particular, the case of random simplices pinned at the origin and simplices where all vertices are generated at random is investigated. The coordinates of the generating vectors are assumed to be independent and identically distributed with subexponential tails. In addition, asymptotic normality is also established for random convex bodies (including random simplices pinned at the origin) when the spanning vectors are distributed according to a radially symmetric probability measure on the n n -dimensional ℓ p \ell _p -ball. In particular, this includes the cone and the uniform probability measure.

High-dimensional dynamics of generalization error in neural networks

We perform an analysis of the average generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant “high-dimensional” regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the generalization error and training error dynamics of learning and analyze how they depend on the dimensionality of data and signal to noise ratio of the learning problem. We find that the dynamics of gradient descent learning naturally protect against overtraining and overfitting in large networks. Overtraining is worst at intermediate network sizes, when the effective number of free parameters equals the number of samples, and thus can be reduced by making a network smaller or larger. Additionally, in the high-dimensional regime, low generalization error requires starting with small initial weights. We then turn to non-linear neural networks, and show that making networks very large does not harm their generalization performance. On the contrary, it can in fact reduce overtraining, even without early stopping or regularization of any sort. We identify two novel phenomena underlying this behavior in overcomplete models: first, there is a frozen subspace of the weights in which no learning occurs under gradient descent; and second, the statistical properties of the high-dimensional regime yield better-conditioned input correlations which protect against overtraining. We demonstrate that standard application of theories such as Rademacher complexity are inaccurate in predicting the generalization performance of deep neural networks, and derive an alternative bound which incorporates the frozen subspace and conditioning effects and qualitatively matches the behavior observed in simulation.

Beta polytopes and Poisson polyhedra: f-vectors and angles

We study random polytopes of the form [X1,…,Xn] defined as convex hulls of independent and identically distributed random points X1,…,Xn in Rd with one of the following densities:fd,β(x)=cd,β(1−‖x‖2)β,‖x‖<1,(beta distribution, β>−1) orf˜d,β(x)=c˜d,β(1+‖x‖2)−β,x∈Rd,(beta' distribution, β>d/2). Here, cd,β and c˜d,β are normalizing constants. This setting also includes the uniform distribution on the unit sphere and the standard normal distribution as limiting cases. We derive exact and asymptotic formulae for the expected number of k-faces of [X1,…,Xn] for arbitrary k∈{0,1,…,d−1}. We prove that for any such k this expected number is strictly monotonically increasing with n. Also, we compute the expected internal and external angles of these polytopes at faces of every dimension and, more generally, the expected conic intrinsic volumes of their tangent cones. By passing to the large n limit in the beta' case, we compute the expected f-vector of the convex hull of Poisson point processes with power-law intensity function. Using convex duality, we derive exact formulae for the expected number of k-faces of the zero cell for a class of isotropic Poisson hyperplane tessellations in Rd. This family includes the zero cell of a classical stationary and isotropic Poisson hyperplane tessellation and the typical cell of a stationary Poisson–Voronoi tessellation as special cases. In addition, we prove precise limit theorems for this f-vector in the high-dimensional regime, as d→∞. Finally, we relate the d-dimensional beta and beta' distributions to the generalized Pareto distributions known in extreme-value theory.

Simulation smoothing for nowcasting with large mixed-frequency VARs

Mixed-frequency VAR models deal with data sampled at different frequencies while remaining within the realms of VARs. Estimation of mixed-frequency VARs makes use of simulation smoothing, but as the size of the model grows, these models quickly become prohibitive in nowcasting situations using the standard procedure. Two algorithms that alleviate the computational efficiency of the simulation smoothing algorithm are therefore proposed. The preferred choice is an adaptive algorithm, which augments the state vector as necessary to sample the monthly variables that are missing at the end of the sample. For large VARs, considerable improvements in speed can be shown by using the proposed adaptive algorithm. The algorithm therefore provides a crucial building block for bringing the mixed-frequency VAR model to the high-dimensional regime.

An adaptable generalization of Hotelling’s $T^{2}$ test in high dimension

We propose a two-sample test for detecting the difference between mean vectors in a high-dimensional regime based on a ridge-regularized Hotelling’s $T^{2}$. To choose the regularization parameter, a method is derived that aims at maximizing power within a class of local alternatives. We also propose a composite test that combines the optimal tests corresponding to a specific collection of local alternatives. Weak convergence of the stochastic process corresponding to the ridge-regularized Hotelling’s $T^{2}$ is established and used to derive the cut-off values of the proposed test. Large sample properties are verified for a class of sub-Gaussian distributions. Through an extensive simulation study, the composite test is shown to compare favorably against a host of existing two-sample test procedures in a wide range of settings. The performance of the proposed test procedures is illustrated through an application to a breast cancer data set where the goal is to detect the pathways with different DNA copy number alterations across breast cancer subtypes.

A Flexible Framework for Hypothesis Testing in High Dimensions

SummaryHypothesis testing in the linear regression model is a fundamental statistical problem. We consider linear regression in the high dimensional regime where the number of parameters exceeds the number of samples (p &amp;gt; n). To make informative inference, we assume that the model is approximately sparse, i.e. the effect of covariates on the response can be well approximated by conditioning on a relatively small number of covariates whose identities are unknown. We develop a framework for testing very general hypotheses regarding the model parameters. Our framework encompasses testing whether the parameter lies in a convex cone, testing the signal strength, and testing arbitrary functionals of the parameter. We show that the procedure proposed controls the type I error, and we also analyse the power of the procedure. Our numerical experiments confirm our theoretical findings and demonstrate that we control the false positive rate (type I error) near the nominal level and have high power. By duality between hypotheses testing and confidence intervals, the framework proposed can be used to obtain valid confidence intervals for various functionals of the model parameters. For linear functionals, the length of confidence intervals is shown to be minimax rate optimal.

Generalization bounds for regularized portfolio selection with market side information

Drawing on statistical learning theory, we derive out-of-sample and optimality guarantees about the investment strategy obtained from a regularized portfolio optimization model which attempts to exploit side information about the financial market in order to reach an optimal risk-return tradeoff. This side information might include for instance recent stock returns, volatility indexes, financial news indicators, etc. In particular, we demonstrate that a regularized investment policy that linearly combines this side information in a way that is optimal from the perspective of a random sample set is guaranteed to perform also relatively well (i.e., within a perturbing factor of ) with respect to the unknown distribution that generated this sample set. We also demonstrate that these performance guarantees are lost in a high-dimensional regime where the size of the side information vector is of an order that is comparable to the sample size. We further extend these results to the case where non-linear investment policies are considered using a kernel operator and show that with radial basis function kernels the performance guarantees become insensitive to how much side information is used. Finally, we illustrate our findings with a set of numerical experiments involving financial data for the NASDAQ composite index.

Multiple Change Points Detection in Low Rank and Sparse High Dimensional Vector Autoregressive Models

Identifying change/break points in multivariate time series represents a canonical problem in signal processing, due to numerous applications related to anomaly detection problems. The underlying detection methodology heavily depends on the nature of the mechanism determining the temporal dynamics of the data. Vector auto-regressive models (VAR) constitute a widely used model in diverse areas, including surveillance applications, economics/finance and neuroscience. In this work, we consider piece-wise stationary VAR models exhibiting break points between the corresponding stationary segments, wherein the transition matrices that govern the model's temporal evolution are decomposed into a common low-rank component and time evolving sparse ones. Further, we assume that the number of available time points are smaller than the number of model parameters and hence we are operating in a high-dimensional regime . We develop a three-step strategy that accurately detects the number of change points together with their location and subsequently estimates the model parameters in each stationary segment. The effectiveness of the proposed procedure is illustrated on both synthetic and real data sets.

Debiased Inference of Average Partial Effects in Single-Index Models: Comment on Wooldridge and Zhu

There has recently been a considerable amount of interest in developing methods for statistical inference in high-dimensional regimes with more covariates than data points (Javanmard and Montanari ...