Ambient Dimension Research Articles

Optimized Tail Bounds for Random Matrix Series.

Random matrix series are a significant component of random matrix theory, offering rich theoretical content and broad application prospects. In this paper, we propose modified versions of tail bounds for random matrix series, including matrix Gaussian (or Rademacher) and sub-Gaussian and infinitely divisible (i.d.) series. Unlike present studies, our results depend on the intrinsic dimension instead of ambient dimension. In some cases, the intrinsic dimension is much smaller than ambient dimension, which makes the modified versions suitable for high-dimensional or infinite-dimensional setting possible. In addition, we obtain the expectation bounds for random matrix series based on the intrinsic dimension.

The Dimension of the Set of ψ-Badly Approximable Points in All Ambient Dimensions: On a Question of Beresnevich and Velani

Abstract Let $\psi :{\mathbb{N}} \to [0,\infty )$, $\psi (q)=q^{-(1+\tau )}$ and let $\psi $-badly approximable points be those vectors in ${\mathbb{R}}^{d}$ that are $\psi $-well approximable, but not $c\psi $-well approximable for arbitrarily small constants $c>0$. We establish that the $\psi $-badly approximable points have the Hausdorff dimension of the $\psi $-well approximable points, the dimension taking the value $(d+1)/(\tau +1)$ familiar from theorems of Besicovitch and Jarník. The method of proof is an entirely new take on the Mass Transference Principle (MTP) by Beresnevich and Velani (Annals, 2006); namely, we use the colloquially named “delayed pruning” to construct a sufficiently large $\liminf $ set and combine this with ideas inspired by the proof of the MTP to find a large $\limsup $ subset of the $\liminf $ set. Our results are a generalisation of some $1$-dimensional results due to Bugeaud and Moreira (Acta Arith, 2011), but our method of proof is nothing alike.

Convergence rates for sums-of-squares hierarchies with correlative sparsity

AbstractThis work derives upper bounds on the convergence rate of the moment-sum-of-squares hierarchy with correlative sparsity for global minimization of polynomials on compact basic semialgebraic sets. The main conclusion is that both sparse hierarchies based on the Schmüdgen and Putinar Positivstellensätze enjoy a polynomial rate of convergence that depends on the size of the largest clique in the sparsity graph but not on the ambient dimension. Interestingly, the sparse bounds outperform the best currently available bounds for the dense hierarchy when the maximum clique size is sufficiently small compared to the ambient dimension and the performance is measured by the running time of an interior point method required to obtain a bound on the global minimum of a given accuracy.

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

The expressive power of deep neural networks is manifested by their remarkable ability to approximate multivariate functions in a way that appears to overcome the curse of dimensionality. This ability is exemplified by their success in solving high-dimensional problems where traditional numerical solvers fail due to their limitations in accurately representing high-dimensional structures. To provide a theoretical framework for explaining this phenomenon, we analyze the approximation of Hölder functions defined on a d-dimensional smooth manifold M embedded in RD, with d≪D, using deep neural networks. We prove that the uniform convergence estimates of the approximation and generalization errors by deep neural networks with ReLU activation functions do not depend on the ambient dimension D of the function but only on its lower manifold dimension d, in a precise sense. Our result improves existing results from the literature where approximation and generalization errors were shown to depend weakly on D.

Quantitative convergence of the vectorial Allen–Cahn equation towards multiphase mean curvature flow

Phase-field models such as the Allen–Cahn equation may give rise to the formation and evolution of geometric shapes, a phenomenon that may be analyzed rigorously in suitable scaling regimes. In its sharp-interface limit, the vectorial Allen–Cahn equation with a potential with N\geq 3 distinct minima has been conjectured to describe the evolution of branched interfaces by multiphase mean curvature flow. In the present work, we give a rigorous proof for this statement in two and three ambient dimensions and for a suitable class of potentials: as long as a strong solution to multiphase mean curvature flow exists, solutions to the vectorial Allen–Cahn equation with well-prepared initial data converge towards multiphase mean curvature flow in the limit of vanishing interface width parameter \varepsilon\searrow 0 . We even establish the rate of convergence O(\varepsilon^{1/2}) . Our approach is based on the gradient-flow structure of the Allen–Cahn equation and its limiting motion: building on the recent concept of “gradient-flow calibrations” for multiphase mean curvature flow, we introduce a notion of relative entropy for the vectorial Allen–Cahn equation with multi-well potential. This enables us to overcome the limitations of other approaches, e.g. avoiding the need for a stability analysis of the Allen–Cahn operator or additional convergence hypotheses for the energy at positive times.

Approximate Policy Iteration With Deep Minimax Average Bellman Error Minimization.

In this work, we investigate the utilization of deep approximate policy iteration (DAPI) in estimating the optimal action-value function Q* within the context of reinforcement learning, employing rectified linear unit (ReLU) ResNet as the underlying framework. The iterative process of DAPI incorporates the minimax average Bellman error minimization principle. It employs ReLU ResNet to estimate the fixed point of the Bellman equation, which is aligned with the estimated greedy policy. Through error propagation, we derive nonasymptotic error bounds between Q* and the estimated Q function induced by the output greedy policy in DAPI. To effectively control the Bellman residual error, we address both the statistical and approximation errors associated with the α -mixing dependent data derived from Markov decision processes, using the techniques of empirical process and deep approximation theory, respectively. Furthermore, we present a novel generalization bound for ReLU ResNet in the presence of dependent data, as well as an approximation bound for ReLU ResNet within the Hölder class. Notably, this approximation bound contributes to a significant improvement in the dependence on the ambient dimension, transitioning from an exponential relationship to a polynomial one. The derived nonasymptotic error bounds explicitly depend on factors such as the sample size, the ambient dimension (in polynomial terms), and the width and depth of the neural networks. Consequently, these bounds serve as valuable theoretical guidelines for appropriately setting the hyperparameters, thereby enabling the achievement of the desired convergence rate during the training process of DAPI.

A unified recovery of structured signals using atomic norm

Abstract In many applications, we seek to recover signals from linear measurements far fewer than the ambient dimension, given the signals have exploitable structures such as sparse vectors or low rank matrices. In this paper, we work in a general setting where signals are approximately sparse in a so-called atomic set. We provide general recovery results stating that a convex programming can stably and robustly recover signals if the null space of the sensing map satisfies certain properties. Moreover, we argue that such null space property can be satisfied with high probability if each measurement is sub-Gaussian even when the number of measurements are very few. Some new results for recovering signals sparse in a frame, and recovering low rank matrices are also derived as a result.

Deep nonparametric estimation of intrinsic data structures by chart autoencoders: Generalization error and robustness

Autoencoders have demonstrated remarkable success in learning low-dimensional latent features of high-dimensional data across various applications. Assuming that data are sampled near a low-dimensional manifold, we employ chart autoencoders, which encode data into low-dimensional latent features on a collection of charts, preserving the topology and geometry of the data manifold. Our paper establishes statistical guarantees on the generalization error of chart autoencoders, and we demonstrate their denoising capabilities by considering n noisy training samples, along with their noise-free counterparts, on a d-dimensional manifold. By training autoencoders, we show that chart autoencoders can effectively denoise the input data with normal noise. We prove that, under proper network architectures, chart autoencoders achieve a squared generalization error in the order of n−2d+2log4⁡n, which depends on the intrinsic dimension of the manifold and only weakly depends on the ambient dimension and noise level. We further extend our theory on data with noise containing both normal and tangential components, where chart autoencoders still exhibit a denoising effect for the normal component. As a special case, our theory also applies to classical autoencoders, as long as the data manifold has a global parametrization. Our results provide a solid theoretical foundation for the effectiveness of autoencoders, which is further validated through several numerical experiments.

Deep estimation for Q⁎ with minimax Bellman error minimization

In this paper we consider the estimation of optimal state-action value function Q⁎ with ReLU ResNet based on minimax Bellman error minimization. We construct the non-asymptotic error bounds for the minimax estimator and the estimated Q function induced by the estimated greedy policy. To bound the Bellman residual error, we guarantee the approximation errors based on deep approximation theory and the statistical ones by utilizing empirical processes taking into account the Markov decision process dependency. We provide a novel generalization bound with dependent data and an approximation bound in the Hölder class which are of independent interest. This bound depends on the sample size, the ambient dimension, the width and depth of the neural network, which can bring prior insights into tuning these hyper-parameters to achieve a desired convergence rate in practice. Furthermore, the bound circumvents the curse of dimensionality if the distribution of state-action pairs is assumed to be supported on a set of low intrinsic dimension.

Sliding stability and uniqueness for the set Y × Y

This article is dedicated to discuss the sliding stability and the uniqueness property for the 2-dimensional minimal cone Y×Y in R4. This problem is motivated by the classification of singularities for Almgren minimal sets, a model for Plateau's problem in the setting of sets. Minimal cones are blow up limits of Almgren minimal sets, thus the list of all minimal cones gives all possible types of singularities that can occur for minimal sets.As proved in [16], when several 2-dimensional Almgren (resp. topological) minimal cones are Almgren (resp. topological) sliding stable, and Almgren (resp. topological) unique, the almost orthogonal union of them stays minimal. Hence if several minimal cones admit sliding stability and uniqueness properties, then we can use their almost orthogonal unions to generate new families of minimal cones. One then naturally ask which minimal cones admit these two properties.This list of known 2-dimensional minimal cones in arbitrary ambient dimension is not long, and the stability and uniqueness properties for all the known 2-dimensional minimal cones, except for Y×Y, have already been established in the previous works [18,17]. Among all the known 2-dimensional minimal cones, Y×Y is the only one whose stability and uniqueness properties were left unsolved. This is due to two main reasons: 1) Y×Y is the only known minimal cone which is essentially of codimension larger than 1—that is, we cannot decompose it into transversal unions of minimal cones of codimension 1. 2) Y×Y lives in R4, where we know very little about which types of singularities can occur in a minimal set. This makes it difficult to control and estimate the measures of all possible competitors. Due to the above two issues, new ideas are required here for solving the problem.We give affirmative answers to this problem for the stability and uniqueness properties for Y×Y in this paper: we prove that the set Y×Y is both Almgren sliding stable, and Almgren unique; for the topological case, we prove its topological sliding stability and topological uniqueness for the coefficient group Z2. This result, along with the results in [16,18,17], allows us to use all the known 2-dimensional minimal cones to generate new 2-dimensional minimal cones by taking almost orthogonal unions.

Distributed (ATC) Gradient Descent for High Dimension Sparse Regression

We study linear regression from data distributed over a network of agents (with no master node) by means of LASSO estimation, in <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">high-dimension</i> , which allows the ambient dimension to grow faster than the sample size. While there is a vast literature of distributed algorithms applicable to the problem, statistical and computational guarantees of most of them remain unclear in high dimension. This paper provides a first statistical study of the Distributed Gradient Descent (DGD) in the Adapt-Then-Combine (ATC) form. Our theory shows that, under standard notions of restricted strong convexity and smoothness of the loss functions–which hold with high probability for standard data generation models–suitable conditions on the network connectivity and algorithm tuning, DGD-ATC converges globally at a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">linear</i> rate to an estimate that is within the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">centralized</i> statistical precision of the model. In the worst-case scenario, the total number of communications to statistical optimality grows logarithmically with the ambient dimension, which improves on the communication complexity of DGD in the Combine-Then-Adapt (CTA) form, scaling linearly with the dimension. This reveals that mixing gradient information among agents, as DGD-ATC does, is critical in high-dimensions to obtain favorable rate scalings.

Lower bounds on the low-distortion embedding dimension of submanifolds of Rn

Let M be a smooth submanifold of Rn equipped with the Euclidean (chordal) metric. This note considers the smallest dimension m for which there exists a bi-Lipschitz function f:M↦Rm with bi-Lipschitz constants close to one. The main result bounds the best achievable embedding dimension m below in terms of the Lipschitz constants of f as well as the reach, volume, diameter, and dimension of M. This new lower bound is then applied to show that prior upper bounds for m via random matrices by Eftekhari and Wakin [6] are near-optimal in a wide range of settings. This supports random linear maps as being nearly as efficient as nonlinear maps at reducing the ambient dimension for manifold data in many situations. Along the way, we also prove results concerning the impossibility of achieving better nonlinear measurement maps with the Restricted Isometry Property (RIP) in compressive sensing applications.

The effect of intrinsic dimension on the Bayes-error of projected quadratic discriminant classification

High-dimensionality is a common hurdle in machine learning and pattern classification; mitigating its effects has attracted extensive research efforts. It has been found in a recent NeurIPS paper that, when the data possesses a low effective dimension, the predictive performance of a discriminative quadratic classifier with nuclear norm regularisation enjoys a reduced (logarithmic) dependence on the ambient dimension and depends on the effective dimension instead, while other regularisers are insensitive to the effective dimension. In this paper, we show that dependence on the effective dimension is also exhibited by the Bayes error of the generative Quadratic Discriminant Analysis (QDA) classifier, without any explicit regularisation, under three linear dimensionality reduction schemes. Specifically, we derive upper bounds on the Bayes error of QDA, which adapt to the effective dimension, and entirely bypass any dependence on the ambient dimension. Our findings complement previous results on compressive QDA that were obtained under compressive sensing type assumptions on the covariance structure. In contrast, our bounds make no a-priori assumptions on the covariance structure, in turn they tighten in the presence of benign traits of the covariance. We corroborate our findings with numerical experiments.

Learning theory for inferring interaction kernels in second-order interacting agent systems

Modeling the complex interactions of systems of particles or agents is a fundamental problem across the sciences, from physics and biology, to economics and social sciences. In this work, we consider second-order, heterogeneous, multivariable models of interacting agents or particles, within simple environments. We describe a nonparametric inference framework to efficiently estimate the latent interaction kernels which drive these dynamical systems. We develop a learning theory which establishes strong consistency and optimal nonparametric min–max rates of convergence for the estimators, as well as provably accurate predicted trajectories. The optimal rates only depends on intrinsic dimension of interactions, which is typically much smaller than the ambient dimension. Our arguments are based on a coercivity condition which ensures that the interaction kernels can be estimated in stable fashion. The numerical algorithm presented to build the estimators is parallelizable, performs well on high-dimensional problems, and its performance is tested on a variety of complex dynamical systems.

Compressed sensing of low-rank plus sparse matrices

Expressing a matrix as the sum of a low-rank matrix plus a sparse matrix is a flexible model capturing global and local features in data. This model is the foundation of robust principle component analysis (Candes et al., 2011) (Chandrasekaran et al., 2009), and popularized by dynamic-foreground/static-background separation (Bouwmans et al., 2016) amongst other applications. Compressed sensing, matrix completion, and their variants (Eldar and Kutyniok, 2012) (Foucart and Rauhut, 2013) have established that data satisfying low complexity models can be efficiently measured and recovered from a number of measurements proportional to the model complexity rather than the ambient dimension. This manuscript develops similar guarantees showing that $m\times n$ matrices that can be expressed as the sum of a rank-$r$ matrix and a $s$-sparse matrix can be recovered by computationally tractable methods from $\mathcal{O}(r(m+n-r)+s)\log(mn/s)$ linear measurements. More specifically, we establish that the restricted isometry constants for the aforementioned matrices remain bounded independent of problem size provided $p/mn$, $s/p$, and $r(m+n-r)/p$ reman fixed. Additionally, we show that semidefinite programming and two hard threshold gradient descent algorithms, NIHT and NAHT, converge to the measured matrix provided the measurement operator's RIC's are sufficiently small. Numerical experiments illustrating these results are shown for synthetic problems, dynamic-foreground/static-background separation, and multispectral imaging.

Star DGT: a robust Gabor transform for speech denoising

In this paper, we address the speech denoising problem, where Gaussian and coloured additive noises are to be removed from a given speech signal. Our approach is based on a redundant, analysis-sparse representation of the original speech signal. We pick an eigenvector of the Zauner unitary matrix and—under certain assumptions on the ambient dimension—we use it as window vector to generate a spark deficient Gabor frame. The analysis operator associated with such a frame, is a (highly) redundant Gabor transform, which we use as a sparsifying transform in the denoising procedure. We conduct computational experiments on real-world speech data, using as baseline three Gabor transforms generated by state-of-the-art window vectors in time-frequency analysis and compare their performance to the proposed Gabor transform. The results show that the proposed redundant Gabor transform outperforms previous ones consistently for all types of examined signals of noise.

Kernel Approximation on Algebraic Varieties

Low-rank approximation of kernels is a fundamental mathematical problem with widespread algorithmic applications. Often the kernel is restricted to an algebraic variety, e.g., in problems involving sparse or low-rank data. We show that significantly better approximations are obtainable in this setting: the rank required to achieve a given error depends on the variety’s dimension rather than the ambient dimension, which is typically much larger. This is true in both high-precision and high-dimensional regimes. Our results are presented for smooth isotropic kernels, the predominant class of kernels used in applications. Our main technical insight is to approximate smooth kernels by polynomial kernels and to leverage two key properties of polynomial kernels that hold when they are restricted to a variety. First, their ranks decrease exponentially in the variety’s co-dimension. Second, their maximum values are governed by their values over a small set of points. Together, our results provide a general approach for exploiting (approximate) “algebraic structure” in datasets in order to efficiently solve large-scale data science problems.

Provably Efficient Reinforcement Learning with Linear Function Approximation

Modern reinforcement learning (RL) is commonly applied to practical problems with an enormous number of states, where function approximation must be deployed to approximate either the value function or the policy. The introduction of function approximation raises a fundamental set of challenges involving computational and statistical efficiency, especially given the need to manage the exploration/exploitation trade-off. As a result, a core RL question remains open: how can we design provably efficient RL algorithms that incorporate function approximation? This question persists even in a basic setting with linear dynamics and linear rewards, for which only linear function approximation is needed. This paper presents the first provable RL algorithm with both polynomial run time and polynomial sample complexity in this linear setting, without requiring a “simulator” or additional assumptions. Concretely, we prove that an optimistic modification of least-squares value iteration—a classical algorithm frequently studied in the linear setting—achieves [Formula: see text] regret, where d is the ambient dimension of feature space, H is the length of each episode, and T is the total number of steps. Importantly, such regret is independent of the number of states and actions. Funding: This work was supported by the Defense Advanced Research Projects Agency program on Lifelong Learning Machines.

Bi-Lipschitz embeddings of quasiconformal trees

A quasiconformal tree is a doubling metric tree in which the diameter of each arc is bounded above by a fixed multiple of the distance between its endpoints. In this paper we show that every quasiconformal tree bi-Lipschitz embeds in some Euclidean space, with the ambient dimension and the bi-Lipschitz constant depending only on the doubling and bounded turning constants of the tree. This answers Question 1.6 of David and Vellis [Illinois J. Math. 66 (2022), pp. 189–244].

Bootstrapping the operator norm in high dimensions: Error estimation for covariance matrices and sketching

Although the operator (spectral) norm is one of the most widely used metrics for covariance estimation, comparatively little is known about the fluctuations of error in this norm. To be specific, let Σˆ denote the sample covariance matrix of n i.i.d. observations in Rp that arise from a population matrix Σ, and let Tn=n‖Σˆ−Σ‖ op. In the setting where the eigenvalues of Σ have a decay profile of the form λj(Σ)≍j−2β, we analyze how well the bootstrap can approximate the distribution of Tn. Our main result shows that up to factors of log(n), the bootstrap can approximate the distribution of Tn with respect to the Kolmogorov metric at the rate of n−β−1∕2 6β+4, which does not depend on the ambient dimension p. In addition, we offer a supporting result of independent interest that establishes a high-probability upper bound for Tn based on flexible moment assumptions. More generally, we discuss the consequences of our work beyond covariance matrices, and show how the bootstrap can be used to estimate the errors of sketching algorithms in randomized numerical linear algebra (RandNLA). An illustration of these ideas is also provided with a climate data example.