Ambient Dimension Research Articles

A dimensionality reduction technique for unconstrained global optimization of functions with low effective dimensionality

Abstract We investigate the unconstrained global optimization of functions with low effective dimensionality, which are constant along certain (unknown) linear subspaces. Extending the technique of random subspace embeddings in Wang et al. (2016, J. Artificial Intelligence Res., 55, 361–387), we study a generic Random Embeddings for Global Optimization (REGO) framework that is compatible with any global minimization algorithm. Instead of the original, potentially large-scale optimization problem, within REGO, a Gaussian random, low-dimensional problem with bound constraints is formulated and solved in a reduced space. We provide novel probabilistic bounds for the success of REGO in solving the original, low effective-dimensionality problem, which show its independence of the (potentially large) ambient dimension and its precise dependence on the dimensions of the effective and embedding subspaces. These results significantly improve existing theoretical analyses by providing the exact distribution of a reduced minimizer and its Euclidean norm and by the general assumptions required on the problem. We validate our theoretical findings by extensive numerical testing of REGO with three types of global optimization solvers, illustrating the improved scalability of REGO compared with the full-dimensional application of the respective solvers.

The constitution of the city and the critique of critical urban theory

A theoretical account of the genesis and internal spatial structure of cities is given. The essence of the urbanisation process is described in terms of the following main developmental phases: (a) the emergence of relationships based on specialisation and interdependence in society; (b) the pre-eminent role of the division of labour within these relationships and its recomposition in dense spatial nodes of human activity; and (c) the concomitant formation of the networked intra-urban spaces of the city. These phases are then contextualised within three intertwined dimensions of urban materiality, namely, an internal dimension (the internal organisation and spatial dynamics of the city), a socially ambient dimension (the relational structure of society at large) and an exogenous dimension (the geographic outside of the city). In light of this account, an evaluative review of what I designate ‘the new critical urban theory’ is carried out, with special reference to planetary urbanisation, postcolonial urban theory and comparativist methodologies. I argue that while every individual city represents a uniquely complex combination of social conjunctures, there are nonetheless definite senses in which urban phenomena are susceptible to investigation at the highest levels of theoretical generality.

Stability of Talagrand’s Gaussian transport-entropy inequality via the Föllmer process

We establish a dimension-free improvement of Talagrand’s Gaussian transport-entropy inequality, under the assumption that the measures satisfy a Poincare inequality. We also study stability of the inequality, in terms of relative entropy, when restricted to measures whose covariance matrix trace is smaller than the ambient dimension. In case the covariance matrix is strictly smaller than the identity, we give dimension-free estimates which depend on its eigenvalues. To complement our results, we show that our conditions cannot be relaxed, and that there exist measures with covariance larger than the identity, for which the inequality is not stable, in relative entropy. To deal with these examples, we show that, without any assumptions, one can always get quantitative stability estimates in terms of relative entropy to Gaussian mixtures. Our approach gives rise to a new point of view which sheds light on the hierarchy between Fisher information, entropy and transportation distance, and may be of independent interest. In particular, it implies that the described results apply verbatim to the log-Sobolev inequality and improve upon some known estimates in the literature.

Optimal fast Johnson\u2013Lindenstrauss embeddings for large data sets

Johnson–Lindenstrauss embeddings are widely used to reduce the dimension and thus the processing time of data. To reduce the total complexity, also fast algorithms for applying these embeddings are necessary. To date, such fast algorithms are only available either for a non-optimal embedding dimension or up to a certain threshold on the number of data points. We address a variant of this problem where one aims to simultaneously embed larger subsets of the data set. Our method follows an approach by Nelson et al. (New constructions of RIP matrices with fast multiplication and fewer rows. In: Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1515-1528, 2014): a subsampled Hadamard transform maps points into a space of lower, but not optimal dimension. Subsequently, a random matrix with independent entries projects to an optimal embedding dimension. For subsets whose size scales at least polynomially in the ambient dimension, the complexity of this method comes close to the number of operations just to read the data under mild assumptions on the size of the data set that are considerably less restrictive than in previous works. We also prove a lower bound showing that subsampled Hadamard matrices alone cannot reach an optimal embedding dimension. Hence, the second embedding cannot be omitted.

FANOK: Knockoffs in Linear Time

We describe a series of algorithms that efficiently implement Gaussian model-X knockoffs to control the false discovery rate on large-scale feature selection problems. Identifying the knockoff distribution requires solving a large-scale semidefinite program for which we derive several efficient methods. One handles generic covariance matrices and has a complexity scaling as $\mathcal{O}(p^3)$, where $p$ is the ambient dimension, while another assumes a rank-$k$ factor model on the covariance matrix to reduce this complexity bound to $\mathcal{O}(pk^2)$. We review an efficient procedure to estimate factor models and show that under a factor model assumption, we can sample knockoff covariates with complexity linear in the dimension. We test our methods on problems with $p$ as large as 500 000.

Density estimation on an unknown submanifold

We investigate the estimation of a density f from a n-sample on an Euclidean space RD, when the data are supported by an unknown submanifold M of possibly unknown dimension d<D, under a reach condition. We investigate several nonparametric kernel methods, with data-driven bandwidths that incorporate some learning of the geometry via a local dimension estimator. When f has Holder smoothness β and M has regularity α, our estimator achieves the rate n−α∧β∕(2α∧β+d) for a pointwise loss. The rate does not depend on the ambient dimension D and we establish that our procedure is asymptotically minimax for α≥β. Following Lepski’s principle, a bandwidth selection rule is shown to achieve smoothness adaptation. We also investigate the case α≤β: by estimating in some sense the underlying geometry of M, we establish in dimension d=1 that the minimax rate is n−β∕(2β+1) proving in particular that it does not depend on the regularity of M. Finally, a numerical implementation is conducted on some case studies in order to confirm the practical feasibility of our estimators.

Hölder Parameterization of Iterated Function Systems and a Self-Affine Phenomenon

Abstract We investigate the Hölder geometry of curves generated by iterated function systems (IFS) in a complete metric space. A theorem of Hata from 1985 asserts that every connected attractor of an IFS is locally connected and path-connected. We give a quantitative strengthening of Hata’s theorem. First we prove that every connected attractor of an IFS is (1/s)-Hölder path-connected, where s is the similarity dimension of the IFS. Then we show that every connected attractor of an IFS is parameterized by a (1/ α)-Hölder curve for all α &gt; s. At the endpoint, α = s, a theorem of Remes from 1998 already established that connected self-similar sets in Euclidean space that satisfy the open set condition are parameterized by (1/s)-Hölder curves. In a secondary result, we show how to promote Remes’ theorem to self-similar sets in complete metric spaces, but in this setting require the attractor to have positive s-dimensional Hausdorff measure in lieu of the open set condition. To close the paper, we determine sharp Hölder exponents of parameterizations in the class of connected self-affine Bedford-McMullen carpets and build parameterizations of self-affine sponges. An interesting phenomenon emerges in the self-affine setting. While the optimal parameter s for a self-similar curve in ℝ n is always at most the ambient dimension n, the optimal parameter s for a self-affine curve in ℝ n may be strictly greater than n.

Maximum likelihood estimation for totally positive log‐concave densities

AbstractWe study nonparametric maximum likelihood estimation for two classes of multivariate distributions that imply strong forms of positive dependence; namely log‐supermodular (MTP2) distributions and log‐L♮‐concave (LLC) distributions. In both cases we also assume log‐concavity in order to ensure boundedness of the likelihood function. Given n independent and identically distributed random vectors from one of our distributions, the maximum likelihood estimator (MLE) exists a.s. and is unique a.e. with probability one when n≥3. This holds independently of the ambient dimension d. We conjecture that the MLE is always the exponential of a tent function. We prove this result for samples in {0,1}d or in under MTP2, and for samples in under LLC. Finally, we provide a conditional gradient algorithm for computing the maximum likelihood estimate.

Invariant hypersurfaces and nodal components for codimension one singular foliations

It is known that there is at least an invariant analytic curve passing through each of the components in the complement of nodal singularities, after reduction of singularities of a germ of singular foliation in $$({\mathbb {C}}^2,0)$$ . Here, we state and prove a generalization of this property to any ambient dimension.

Doubling coverings via resolution of singularities and preparation

In this paper, we provide asymptotic upper bounds on the complexity in two (closely related) situations. We confirm for the total doubling coverings and not only for the chains the expected bounds of the form [Formula: see text] This is done in a rather general setting, i.e. for the [Formula: see text]-complement of a polynomial zero-level hypersurface [Formula: see text] and for the regular level hypersurfaces [Formula: see text] themselves with no assumptions on the singularities of [Formula: see text]. The coefficient [Formula: see text] is the ambient dimension [Formula: see text] in the first case and [Formula: see text] in the second case. However, the question of a uniform behavior of the coefficient [Formula: see text] remains open. As a second theme, we confirm in arbitrary dimension the upper bound for the number of a-charts covering a real semi-algebraic set [Formula: see text] of dimension [Formula: see text] away from the [Formula: see text]-neighborhood of a lower dimensional set [Formula: see text], with bound of the form [Formula: see text] holding uniformly in the complexity of [Formula: see text]. We also show an analogue for level sets with parameter away from the [Formula: see text]-neighborhood of a low dimensional set. More generally, the bounds are obtained also for real subanalytic and real power-subanalytic sets.

Polynomial Matrix Completion for Missing Data Imputation and Transductive Learning

This paper develops new methods to recover the missing entries of a high-rank or even full-rank matrix when the intrinsic dimension of the data is low compared to the ambient dimension. Specifically, we assume that the columns of a matrix are generated by polynomials acting on a low-dimensional intrinsic variable, and wish to recover the missing entries under this assumption. We show that we can identify the complete matrix of minimum intrinsic dimension by minimizing the rank of the matrix in a high dimensional feature space. We develop a new formulation of the resulting problem using the kernel trick together with a new relaxation of the rank objective, and propose an efficient optimization method. We also show how to use our methods to complete data drawn from multiple nonlinear manifolds. Comparative studies on synthetic data, subspace clustering with missing data, motion capture data recovery, and transductive learning verify the superiority of our methods over the state-of-the-art.

New explicit solutions to the p-Laplace equation based on isoparametric foliations

In contrast to an infinite family of explicit examples of two-dimensional p-harmonic functions obtained by G. Aronsson in the late 80s, there is very little known about the higher-dimensional case. In this paper, we show how to use isoparametric polynomials to produce diverse examples of p-harmonic and biharmonic functions. Remarkably, for some distinguished values of p and the ambient dimension n this yields first examples of rational and algebraic p-harmonic functions. Moreover, we show that there are no p-harmonic polynomials of the isoparametric type. This supports a negative answer to a question proposed in 1980 by J. Lewis.

Learning rates for partially linear functional models with high dimensional scalar covariates

This paper is concerned with learning rates for partial linear functional models (PLFM) within reproducing kernel Hilbert spaces (RKHS), where all the covariates consist of two parts: functional-type covariates and scalar ones. As opposed to frequently used functional principal component analysis for functional models, the finite number of basis functions in the proposed approach can be generated automatically by taking advantage of reproducing property of RKHS. This avoids additional computational costs on PCA decomposition and the choice of the number of principal components. Moreover, the coefficient estimators with bounded covariates converge to the true coefficients with linear rates, as if the functional term in PLFM has no effect on the linear part. In contrast, the prediction error for the functional estimator is significantly affected by the ambient dimension of the scalar covariates. Finally, we develop the proposed numerical algorithm for the proposed penalized approach, and some simulated experiments are implemented to support our theoretical results.

An Online Algorithm for Smoothed Online Convex Optimization

We consider Online Convex Optimization (OCO) in the setting where the costs are m-strongly convex and the online learner pays a switching cost for changing decisions between rounds. We show that the recently proposed Online Balanced Descent (OBD) algorithm is constant competitive in this setting, with competitive ratio 3+O(1/m), irrespective of the ambient dimension. We demonstrate the generality of our approach by showing that the OBD framework can be used to construct competitive a algorithm for LQR control.

The scaling limit of high-dimensional online independent component analysis**This article is an updated version of: Wang C and Lu Y M 2017 The scaling limit of high-dimensional online independent component analysis Advances in Neural Information Processing Systems 30 ed I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan and R Garnett (Red Hook, NY: Curran Associates,

We analyze the dynamics of an online algorithm for independent component analysis in the high-dimensional scaling limit. As the ambient dimension tends to infinity, and with proper time scaling, we show that the time-varying joint empirical measure of the target feature vector and the estimates provided by the algorithm will converge weakly to a deterministic measured-valued process that can be characterized as the unique solution of a nonlinear PDE. Numerical solutions of this PDE, which involves two spatial variables and one time variable, can be efficiently obtained. These solutions provide detailed information about the performance of the ICA algorithm, as many practical performance metrics are functionals of the joint empirical measures. Numerical simulations show that our asymptotic analysis is accurate even for moderate dimensions. In addition to providing a tool for understanding the performance of the algorithm, our PDE analysis also provides useful insight. In particular, in the high-dimensional limit, the original coupled dynamics associated with the algorithm will be asymptotically ‘decoupled’, with each coordinate independently solving a 1D effective minimization problem via stochastic gradient descent. Exploiting this insight to design new algorithms for achieving optimal trade-offs between computational and statistical efficiency may prove an interesting line of future research.

On the volume of non-central sections of a cube

Let Qn be the cube of side length one centered at the origin in Rn, and let F be an affine (n−d)-dimensional subspace of Rn having distance to the origin less than or equal to 12, where 0<d<n. We show that the (n−d)-dimensional volume of the section Qn∩F is bounded below by a value c(d) depending only on the codimension d but not on the ambient dimension n or a particular subspace F. In the case of hyperplanes, d=1, we show that c(1)=117 is a possible choice. We also consider a complex analogue of this problem for a hyperplane section of the polydisc.

Rigorous restricted isometry property of low-dimensional subspaces

Dimensionality reduction is in demand to reduce the complexity of solving large-scale problems with data lying in latent low-dimensional structures in machine learning and computer version. Motivated by such need, in this work we study the Restricted Isometry Property (RIP) of Gaussian random projections for low-dimensional subspaces in $\mathbb{R}^N$, and rigorously prove that the projection Frobenius norm distance between any two subspaces spanned by the projected data in $\mathbb{R}^n$ ($n<N$) remain almost the same as the distance between the original subspaces with probability no less than $1 - {\rm e}^{-\mathcal{O}(n)}$. Previously the well-known Johnson-Lindenstrauss (JL) Lemma and RIP for sparse vectors have been the foundation of sparse signal processing including Compressed Sensing. As an analogy to JL Lemma and RIP for sparse vectors, this work allows the use of random projections to reduce the ambient dimension with the theoretical guarantee that the distance between subspaces after compression is well preserved.

Information based complexity for high dimensional sparse functions

We investigate optimal algorithms for optimizing and approximating a general high dimensional smooth and sparse function from the perspective of information based complexity. Our algorithms and analyses reveal several interesting characteristics for these tasks. In particular, somewhat surprisingly, we show that the optimal sample complexity for optimization or high precision approximation is independent of the ambient dimension. In addition, we show that the benefit of randomization could be substantial for these problems.

Persistent homology for low-complexity models

We show that recent results on randomized dimension reduction schemes that exploit structural properties of data can be applied in the context of persistent homology. In the spirit of compressed sensing, the dimension reduction is determined by the Gaussian width of a structure associated with the dataset, rather than its size, and such a reduction can be computed efficiently. We further relate the Gaussian width to the doubling dimension of a finite metric space, which appears in the study of the complexity of other methods for approximating persistent homology. We can, therefore, literally replace the ambient dimension by an intrinsic notion of dimension related to the structure of the data.

Bubbling analysis and geometric convergence results for free boundary minimal surfaces

We investigate the limit behaviour of sequences of free boundary minimal hypersurfaces with bounded index and volume, by presenting a detailed blow-up analysis near the points where curvature concentration occurs. Thereby, we derive a general quantization identity for the total curvature functional, valid in ambient dimension less than eight and applicable to possibly improper limit hypersurfaces. In dimension three, this identity can be combined with the Gauss-Bonnet theorem to provide a constraint relating the topology of the free boundary minimal surfaces in a converging sequence, of their limit, and of the bubbles or half-bubbles that occur as blow-up models. We present various geometric applications of these tools, including a description of the behaviour of index one free boundary minimal surfaces inside a 3-manifold of non-negative scalar curvature and strictly mean convex boundary. In particular, in the case of compact, simply connected, strictly mean convex domains in ℝ 3 unconditional convergence occurs for all topological types except the disk and the annulus, and in those cases the possible degenerations are classified.