High-dimensional Regime Research Articles

Enhanced Response Envelope via Envelope Regularization

The response envelope model provides substantial efficiency gains over the standard multivariate linear regression by identifying the material part of the response to the model and by excluding the immaterial part. In this article, we propose the enhanced response envelope by incorporating a novel envelope regularization term based on a nonconvex manifold formulation. It is shown that the enhanced response envelope can yield better prediction risk than the original envelope estimator. The enhanced response envelope naturally handles high-dimensional data for which the original response envelope is not serviceable without necessary remedies. In an asymptotic high-dimensional regime where the ratio of the number of predictors over the number of samples converges to a nonzero constant, we characterize the risk function and reveal an interesting double descent phenomenon for the envelope model. A simulation study confirms our main theoretical findings. Simulations and real data applications demonstrate that the enhanced response envelope does have significantly improved prediction performance over the original envelope method, especially when the number of predictors is close to or moderately larger than the number of samples. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Limit behavior in high-dimensional regime for Wishart tensors with Rosenblatt entries

In this paper, we consider an initial [Formula: see text] random matrix with non-Gaussian correlated entries on each row and independent entries from one row to another. The correlation on the rows is given by the correlation of the increments of the Rosenblatt process, which is a non-Gaussian self-similar process with stationary increments, living in the second Wiener chaos. To this initial matrix, we associate a Wishart tensor of length [Formula: see text]. We study the limit behavior in distribution of this Wishart tensor in the high-dimensional regime, i.e. when [Formula: see text] are large enough. We prove that the vector corresponding to the [Formula: see text]-Wishart tensor converges to an [Formula: see text]-dimensional non-Gaussian vector, with Rosenblatt random variables on its hyper-diagonals and zeros outside the hyper-diagonals.

The twin peaks of learning neural networks

Recent works demonstrated the existence of a double-descent phenomenon for the generalization error of neural networks, where highly overparameterized models escape overfitting and achieve good test performance, at odds with the standard bias-variance trade-off described by statistical learning theory. In the present work, we explore a link between this phenomenon and the increase of complexity and sensitivity of the function represented by neural networks. In particular, we study the Boolean mean dimension (BMD), a metric developed in the context of Boolean function analysis. Focusing on a simple teacher-student setting for the random feature model, we derive a theoretical analysis based on the replica method that yields an interpretable expression for the BMD, in the high dimensional regime where the number of data points, the number of features, and the input size grow to infinity. We find that, as the degree of overparameterization of the network is increased, the BMD reaches an evident peak at the interpolation threshold, in correspondence with the generalization error peak, and then slowly approaches a low asymptotic value. The same phenomenology is then traced in numerical experiments with different model classes and training setups. Moreover, we find empirically that adversarially initialized models tend to show higher BMD values, and that models that are more robust to adversarial attacks exhibit a lower BMD.

Estimating high-dimensional Hawkes process with time-dependent covariates

The Hawkes process models have been recently become a popular tool for modeling and analysis of neural spike train data. Despite this popularity, existing methodological and theoretical work has mainly focused on models without assumption of time-varying covariates, such as stimulus drive. In this article, motivated by neuronal spike trains study in the presence of stimuli, we propose a new Hawkes process model, where covariates are included in the intensity function. We consider the problem of simultaneous variable selection and estimation for the Hawkes process in the high-dimensional regime. Estimation of the intensity function of the high-dimensional point process is considered within a nonparametric framework, applying B-splines and the SCAD penalty for matters of sparsity. We apply the Doob-Kolmogorov inequality to establish the consistency of the resulting estimators. Finally, we illustrate the performance of our proposal through simulation and demonstrate its utility by applying it to the neuron spike train data set.

Limit Behavior in High-Dimensional Regime for the Wishart Tensors in Wiener Chaos

Limit Behavior in High-Dimensional Regime for the Wishart Tensors in Wiener Chaos

The universal law of generalization holds for naturalistic stimuli.

Shepard's universal law of generalization is a remarkable hypothesis about how intelligent organisms should perceive similarity. In its broadest form, the universal law states that the level of perceived similarity between a pair of stimuli should decay as a concave function of their distance when embedded in an appropriate psychological space. While extensively studied, evidence in support of the universal law has relied on low-dimensional stimuli and small stimulus sets that are very different from their real-world counterparts. This is largely because pairwise comparisons-as required for similarity judgments-scale quadratically in the number of stimuli. We provide strong evidence for the universal law in a naturalistic high-dimensional regime by analyzing an existing data set of 214,200 human similarity judgments and a newly collected data set of 390,819 human generalization judgments (N = 2,406 U.S. participants) across three sets of natural images. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Asymptotic Behavior of Adversarial Training in Binary Linear Classification.

Adversarial training using empirical risk minimization (ERM) is the state-of-the-art method for defense against adversarial attacks, that is, against small additive adversarial perturbations applied to test data leading to misclassification. Despite being successful in practice, understanding the generalization properties of adversarial training in classification remains widely open. In this article, we take the first step in this direction by precisely characterizing the robustness of adversarial training in binary linear classification. Specifically, we consider the high-dimensional regime where the model dimension grows with the size of the training set at a constant ratio. Our results provide exact asymptotics for both standard and adversarial test errors under general lq -norm bounded perturbations ( q ≥ 1 ) in both discriminative binary models and generative Gaussian-mixture models with correlated features. We use our sharp error formulae to explain how the adversarial and standard errors depend upon the over-parameterization ratio, the data model, and the attack budget. Finally, by comparing with the robust Bayes estimator, our sharp asymptotics allow us to study the fundamental limits of adversarial training.

Extrapolated Cross-Validation for Randomized Ensembles

Ensemble methods such as bagging and random forests are ubiquitous in various fields, from finance to genomics. Despite their prevalence, the question of the efficient tuning of ensemble parameters has received relatively little attention. This article introduces a cross-validation method, Extrapolated Cross-Validation (ECV), for tuning the ensemble and subsample sizes in randomized ensembles. Our method builds on two primary ingredients: initial estimators for small ensemble sizes using out-of-bag errors and a novel risk extrapolation technique that leverages the structure of prediction risk decomposition. By establishing uniform consistency of our risk extrapolation technique over ensemble and subsample sizes, we show that ECV yields δ-optimal (with respect to the oracle-tuned risk) ensembles for squared prediction risk. Our theory accommodates general predictors, only requires mild moment assumptions, and allows for high-dimensional regimes where the feature dimension grows with the sample size. As a practical case study, we employ ECV to predict surface protein abundances from gene expressions in single-cell multiomics using random forests under a computational constraint on the maximum ensemble size. Compared to sample-split and K-fold cross-validation, ECV achieves higher accuracy by avoiding sample splitting. Meanwhile, its computational cost is considerably lower owing to the use of the risk extrapolation technique. Supplementary materials for this article are available online.

A note on the prediction error of principal component regression in high dimensions

We analyze the prediction error of principal component regression (PCR) and prove high probability bounds for the corresponding squared risk conditional on the design. Our first main result shows that PCR performs comparably to the oracle method obtained by replacing empirical principal components by their population counterparts, provided that an effective rank condition holds. On the other hand, if the latter condition is violated, then empirical eigenvalues start to have a significant upward bias, resulting in a self-induced regularization of PCR. Our approach relies on the behavior of empirical eigenvalues, empirical eigenvectors and the excess risk of principal component analysis in high-dimensional regimes.

The volume of random simplices from elliptical distributions in high dimension

Random simplices and more general random convex bodies of dimension p in Rn with p≤n are considered, which are generated by random vectors having an elliptical distribution. In the high-dimensional regime, that is, if p→∞ and n→∞ in such a way that p/n→γ∈(0,1), a central and a stable limit theorem for the logarithmic volume of random simplices and random convex bodies is shown. The result follows from a related central limit theorem for the log-determinant of p×n random matrices whose rows are copies of a random vector with an elliptical distribution, which is established as well.

Retire: Robust expectile regression in high dimensions

High-dimensional data can often display heterogeneity due to heteroscedastic variance or inhomogeneous covariate effects. Penalized quantile and expectile regression methods offer useful tools to detect heteroscedasticity in high-dimensional data. The former is computationally challenging due to the non-smooth nature of the check loss, and the latter is sensitive to heavy-tailed error distributions. In this paper, we propose and study (penalized) robust expectile regression (retire), with a focus on iteratively reweighted ℓ1-penalization which reduces the estimation bias from ℓ1-penalization and leads to oracle properties. Theoretically, we establish the statistical properties of the retire estimator under two regimes: (i) low-dimensional regime in which d≪n; (ii) high-dimensional regime in which s≪n≪d with s denoting the number of significant predictors. In the high-dimensional setting, we thoroughly analyze the statistical properties of the solution path of iteratively reweighted ℓ1-penalized retire estimation, adapted from the local linear approximation algorithm for folded-concave regularization. Under a mild minimum signal strength condition, we demonstrate that with as few as log(logd) iterations, the final iterate of our proposed approach achieves the oracle convergence rate. At each iteration, we solve the weighted ℓ1-penalized convex program using a semismooth Newton coordinate descent algorithm. Numerical studies demonstrate the promising performance of the proposed procedure in comparison to both non-robust and quantile regression based alternatives.

Phase transitions for support recovery under local differential privacy

We address the problem of variable selection in a high-dimensional but sparse mean model, under the additional constraint that only privatized data are available for inference. The original data are vectors with independent entries having a symmetric, strongly log-concave distribution on \mathbb{R} . For this purpose, we adopt a recent generalization of classical minimax theory to the framework of local \alpha -differential privacy. We provide lower and upper bounds on the rate of convergence for the expected Hamming loss over classes of at most s -sparse vectors whose non-zero coordinates are separated from 0 by a constant a>0 . As corollaries, we derive necessary and sufficient conditions (up to log factors) for exact recovery and for almost full recovery. When we restrict our attention to non-interactive mechanisms that act independently on each coordinate our lower bound shows that, contrary to the non-private setting, both exact and almost full recovery are impossible whatever the value of a in the high-dimensional regime such that n \alpha^2/ d^2\lesssim 1 . However, in the regime n\alpha^2/d^2\gg \log(d) we can exhibit a critical value a^* (up to a logarithmic factor) such that exact and almost full recovery are possible for all a\gg a^* and impossible for a\leq a^* . We show that these results can be improved when allowing for all non-interactive (that act globally on all coordinates) locally \alpha -differentially private mechanisms in the sense that phase transitions occur at lower levels.

Optimal non-asymptotic concentration of centered empirical relative entropy in the high-dimensional regime

This document establishes the optimal non-asymptotic concentration of the Kullback–Leibler divergence between the empirical distribution and the true distribution around its mean in the regime of K≫n, where K and n are the alphabet size and sample size, respectively.

High-dimensional regime for Wishart matrices based on the increments of the solution to the stochastic heat equation

High-dimensional regime for Wishart matrices based on the increments of the solution to the stochastic heat equation

Minimax optimal clustering of bipartite graphs with a generalized power method

Abstract Clustering bipartite graphs is a fundamental task in network analysis. In the high-dimensional regime where the number of rows $n_{1}$ and the number of columns $n_{2}$ of the associated adjacency matrix are of different order, the existing methods derived from the ones used for symmetric graphs can come with sub-optimal guarantees. Due to increasing number of applications for bipartite graphs in the high-dimensional regime, it is of fundamental importance to design optimal algorithms for this setting. The recent work of Ndaoud et al. (2022, IEEE Trans. Inf. Theory, 68, 1960–1975) improves the existing upper-bound for the misclustering rate in the special case where the columns (resp. rows) can be partitioned into $L = 2$ (resp. $K = 2$) communities. Unfortunately, their algorithm cannot be extended to the more general setting where $K \neq L \geq 2$. We overcome this limitation by introducing a new algorithm based on the power method. We derive conditions for exact recovery in the general setting where $K \neq L \geq 2$, and show that it recovers the result in Ndaoud et al. (2022, IEEE Trans. Inf. Theory, 68, 1960–1975). We also derive a minimax lower bound on the misclustering error when $K=L$ under a symmetric version of our model, which matches the corresponding upper bound up to a factor depending on $K$.

Kernel Approximation on Algebraic Varieties

Low-rank approximation of kernels is a fundamental mathematical problem with widespread algorithmic applications. Often the kernel is restricted to an algebraic variety, e.g., in problems involving sparse or low-rank data. We show that significantly better approximations are obtainable in this setting: the rank required to achieve a given error depends on the variety’s dimension rather than the ambient dimension, which is typically much larger. This is true in both high-precision and high-dimensional regimes. Our results are presented for smooth isotropic kernels, the predominant class of kernels used in applications. Our main technical insight is to approximate smooth kernels by polynomial kernels and to leverage two key properties of polynomial kernels that hold when they are restricted to a variety. First, their ranks decrease exponentially in the variety’s co-dimension. Second, their maximum values are governed by their values over a small set of points. Together, our results provide a general approach for exploiting (approximate) “algebraic structure” in datasets in order to efficiently solve large-scale data science problems.

Bayesian causal inference: a critical review.

This paper provides a critical review of the Bayesian perspective of causal inference based on the potential outcomes framework. We review the causal estimands, assignment mechanism, the general structure of Bayesian inference of causal effects and sensitivity analysis. We highlight issues that are unique to Bayesian causal inference, including the role of the propensity score, the definition of identifiability, the choice of priors in both low- and high-dimensional regimes. We point out the central role of covariate overlap and more generally the design stage in Bayesian causal inference. We extend the discussion to two complex assignment mechanisms: instrumental variable and time-varying treatments. We identify the strengths and weaknesses of the Bayesian approach to causal inference. Throughout, we illustrate the key concepts via examples. This article is part of the theme issue 'Bayesian inference: challenges, perspectives, and prospects'.

Learning curves for the multi-class teacher–student perceptron

One of the most classical results in high-dimensional learning theory provides a closed-form expression for the generalisation error of binary classification with a single-layer teacher–student perceptron on i.i.d. Gaussian inputs. Both Bayes-optimal (BO) estimation and empirical risk minimisation (ERM) were extensively analysed in this setting. At the same time, a considerable part of modern machine learning practice concerns multi-class classification. Yet, an analogous analysis for the multi-class teacher–student perceptron was missing. In this manuscript we fill this gap by deriving and evaluating asymptotic expressions for the BO and ERM generalisation errors in the high-dimensional regime. For Gaussian teacher, we investigate the performance of ERM with both cross-entropy and square losses, and explore the role of ridge regularisation in approaching Bayes-optimality. In particular, we observe that regularised cross-entropy minimisation yields close-to-optimal accuracy. Instead, for Rademacher teacher we show that a first-order phase transition arises in the BO performance.

Random or Nonrandom Signal in High-Dimensional Regimes

This paper proposes a new hypothesis test to check the randomness and nonrandomness of the contaminated high dimensional signal. Specifically, for a signal plus noise model, we propose a statistic to distinguish whether the corresponding signal is random or not. In order to analyze the performance of the proposed method, we also prove two important results for signal plus noise models: 1) No eigenvalues outside the support of the limiting spectral distribution of the noncentered and centered sample covariance matrix; and 2) Exact separation of eigenvalues of the noncentered and centered sample covariance matrix. Simulation studies demonstrate that our proposed method works well under a variety of settings.

Penalization-induced shrinking without rotation in high dimensional GLM regression: a cavity analysis

In high dimensional regression, where the number of covariates is of the order of the number of observations, ridge penalization is often used as a remedy against overfitting. Unfortunately, for correlated covariates such regularisation typically induces in generalized linear models not only shrinking of the estimated parameter vector, but also an unwanted rotation relative to the true vector. We show analytically how this problem can be removed by using a generalization of ridge penalization, and we analyse the asymptotic properties of the corresponding estimators in the high dimensional regime, using the cavity method. Our results also provide a quantitative rationale for tuning the parameter controlling the amount of shrinking. We compare our theoretical predictions with simulated data and find excellent agreement.