Low Effective Dimension Research Articles

Variable transformations in combination with wavelets and ANOVA for high-dimensional approximation

We use hyperbolic wavelet regression for the fast reconstruction of high-dimensional functions having only low-dimensional variable interactions. Compactly supported periodic Chui-Wang wavelets are used for the tensorized hyperbolic wavelet basis on the torus. With a variable transformation, we are able to transform the approximation rates and fast algorithms from the torus to other domains. We perform and analyze scattered data approximation for smooth but arbitrary density functions by using a least squares method. The corresponding system matrix is sparse due to the compact support of the wavelets, which leads to a significant acceleration of the matrix vector multiplication. For non-periodic functions, we propose a new extension method. A proper choice of the extension parameter together with the piecewise polynomial Chui-Wang wavelets extends the functions appropriately. In every case, we are able to bound the approximation error with high probability. Additionally, if the function has a low effective dimension (i.e., only interactions of a few variables), we qualitatively determine the variable interactions and omit ANOVA terms with low variance in a second step in order to decrease the approximation error. This allows us to suggest an adapted model for the approximation. Numerical results show the efficiency of the proposed method.

Dimension of Activity in Random Neural Networks.

Neural networks are high-dimensional nonlinear dynamical systems that process information through the coordinated activity of many connected units. Understanding how biological and machine-learning networks function and learn requires knowledge of the structure of this coordinated activity, information contained, for example, in cross covariances between units. Self-consistent dynamical mean field theory (DMFT) has elucidated several features of random neural networks-in particular, that they can generate chaotic activity-however, a calculation of cross covariances using this approach has not been provided. Here, we calculate cross covariances self-consistently via a two-site cavity DMFT. We use this theory to probe spatiotemporal features of activity coordination in a classic random-network model with independent and identically distributed (i.i.d.) couplings, showing an extensive but fractionally low effective dimension of activity and a long population-level timescale. Our formulas apply to a wide range of single-unit dynamics and generalize to non-i.i.d. couplings. As an example of the latter, we analyze the case of partially symmetric couplings.

The effect of intrinsic dimension on the Bayes-error of projected quadratic discriminant classification

High-dimensionality is a common hurdle in machine learning and pattern classification; mitigating its effects has attracted extensive research efforts. It has been found in a recent NeurIPS paper that, when the data possesses a low effective dimension, the predictive performance of a discriminative quadratic classifier with nuclear norm regularisation enjoys a reduced (logarithmic) dependence on the ambient dimension and depends on the effective dimension instead, while other regularisers are insensitive to the effective dimension. In this paper, we show that dependence on the effective dimension is also exhibited by the Bayes error of the generative Quadratic Discriminant Analysis (QDA) classifier, without any explicit regularisation, under three linear dimensionality reduction schemes. Specifically, we derive upper bounds on the Bayes error of QDA, which adapt to the effective dimension, and entirely bypass any dependence on the ambient dimension. Our findings complement previous results on compressive QDA that were obtained under compressive sensing type assumptions on the covariance structure. In contrast, our bounds make no a-priori assumptions on the covariance structure, in turn they tighten in the presence of benign traits of the covariance. We corroborate our findings with numerical experiments.

Fast hyperbolic wavelet regression meets ANOVA

We use hyperbolic wavelet regression for the fast reconstruction of high-dimensional functions having only low dimensional variable interactions. Compactly supported periodic Chui–Wang wavelets are used for the tensorized hyperbolic wavelet basis. In a first step we give a self-contained characterization of tensor product Sobolev–Besov spaces on the d-torus with arbitrary smoothness in terms of the decay of such wavelet coefficients. In the second part we perform and analyze scattered-data approximation using a hyperbolic cross type truncation of the basis expansion for the associated least squares method. The corresponding system matrix is sparse due to the compact support of the wavelets, which leads to a significant acceleration of the matrix vector multiplication. In case of i.i.d. samples we can even bound the approximation error with high probability by loosing only log -terms that do not depend on d compared to the best approximation. In addition, if the function has low effective dimension (i.e. only interactions of few variables), we qualitatively determine the variable interactions and omit ANOVA terms with low variance in a second step in order to increase the accuracy. This allows us to suggest an adapted model for the approximation. Numerical results show the efficiency of the proposed method.

ADT: AI-Driven network Telemetry processing on routers

Network monitoring is a pivotal part of network management and operations. It is responsible for monitoring the behavior of the network to assure its functionality within expectation and to guarantee a smooth-running environment for enabling of various services. Therefore, operators are interested in gaining a comprehensive assessment of their network elements and tracking operational changes to facilitate timely correction of any deviation. Commonly, this assessment is achieved by performing regular manual checks of different operational counters and defining expert rules from known root causes. The common approach requires the maintenance of a regularly updated set of rules and only goes as far as the operator’s pre-gained knowledge of the system. With the growing complexity of the networks as well as the availability of more data, a more efficient monitoring approach is necessary to address the emerging network monitoring requirements.In this paper, a novel unsupervised approach is proposed that is capable of exploring a broader set of counters (not limited to the handpicked Key Performance Indicators (KPIs)). The goal is to leverage the dependencies between the counters in order to discover complex state changes that might have otherwise slipped the operator’s view. This paper proposes ADT, an AI-driven telemetry processing solution that facilitates monitoring of a larger set of counters. The Detector block of ADT is known as DESTIN, a multivariate unsupervised change detection for high dimensional time-series data of originally low effective dimension, which provides near real-time state assessment of network devices. The efficiency of the proposed approach is demonstrated and compared with well-known methodologies on an experimental test-bed. The method’s performance is also explored extensively considering different criteria such as traffic type, device and the type of events to identify its potentials and limitations. The datasets used for the evaluation are made publicly available.

A low-rank ensemble Kalman filter for elliptic observations

We propose a regularization method for ensemble Kalman filtering (EnKF) with elliptic observation operators. Commonly used EnKF regularization methods suppress state correlations at long distances. For observations described by elliptic partial differential equations, such as the pressure Poisson equation (PPE) in incompressible fluid flows, distance localization should be used cautiously, as we cannot disentangle slowly decaying physical interactions from spurious long-range correlations. This is particularly true for the PPE, in which distant vortex elements couple nonlinearly to induce pressure. Instead, these inverse problems have a low effective dimension: low-dimensional projections of the observations strongly inform a low-dimensional subspace of the state space. We derive a low-rank factorization of the Kalman gain based on the spectrum of the Jacobian of the observation operator. The identified eigenvectors generalize the source and target modes of the multipole expansion, independently of the underlying spatial distribution of the problem. Given rapid spectral decay, inference can be performed in the low-dimensional subspace spanned by the dominant eigenvectors. This low-rank EnKF is assessed on dynamical systems with Poisson observation operators, where we seek to estimate the positions and strengths of point singularities over time from potential or pressure observations. We also comment on the broader applicability of this approach to elliptic inverse problems outside the context of filtering.

Observation of Non-Markovian Spin Dynamics in a Jaynes-Cummings-Hubbard Model Using a Trapped-Ion Quantum Simulator.

The Jaynes-Cummings-Hubbard (JCH) model is a fundamental many-body model for light-matter interaction. As a leading platform for quantum simulation, the trapped ion system has realized the JCH model for two to three ions. Here, we report the quantum simulation of the JCH model using up to 32 ions. We verify the simulation results even for large ion numbers by engineering low excitations and thus low effective dimensions; then we extend to 32 excitations for an effective dimension of 2^{77}, which is difficult for classical computers. By regarding the phonon modes as baths, we explore Markovian or non-Markovian spin dynamics in different parameter regimes of the JCH model, similar to quantum emitters in a structured photonic environment. We further examine the dependence of the non-Markovian dynamics on the effective Hilbert space dimension. Our Letter demonstrates the trapped ion system as a powerful quantum simulator for many-body physics and open quantum systems.

Global optimization using random embeddings

We propose a random-subspace algorithmic framework for global optimization of Lipschitz-continuous objectives, and analyse its convergence using novel tools from conic integral geometry. X-REGO randomly projects, in a sequential or simultaneous manner, the high-dimensional original problem into low-dimensional subproblems that can then be solved with any global, or even local, optimization solver. We estimate the probability that the randomly-embedded subproblem shares (approximately) the same global optimum as the original problem. This success probability is then used to show almost sure convergence of X-REGO to an approximate global solution of the original problem, under weak assumptions on the problem (having a strictly feasible global solution) and on the solver (guaranteed to find an approximate global solution of the reduced problem with sufficiently high probability). In the particular case of unconstrained objectives with low effective dimension, we propose an X-REGO variant that explores random subspaces of increasing dimension until finding the effective dimension of the problem, leading to X-REGO globally converging after a finite number of embeddings, proportional to the effective dimension. We show numerically that this variant efficiently finds both the effective dimension and an approximate global minimizer of the original problem.

Dimension independent excess risk by stochastic gradient descent

One classical canon of statistics is that large models are prone to overfitting, and model selection procedures are necessary for high dimensional data. However, many overparameterized models, such as neural networks, perform very well in practice, although they are often trained with simple online methods and regularization. The empirical success of overparameterized models, which is often known as benign overfitting, motivates us to have a new look at the statistical generalization theory for online optimization. In particular, we present a general theory on the excess risk of stochastic gradient descent (SGD) solutions for both convex and locally non-convex loss functions. We further discuss data and model conditions that lead to a “low effective dimension”. Under these conditions, we show that the excess risk either does not depend on the ambient dimension p or depends on p via a poly-logarithmic factor. We also demonstrate that in several widely used statistical models, the “low effective dimension” arises naturally in overparameterized settings. The studied statistical applications include both convex models such as linear regression and logistic regression and non-convex models such as M-estimator and two-layer neural networks.

Experimental realization of a quantum image classifier via tensor-network-based machine learning

Quantum machine learning aspires to overcome intractability that currently limits its applicability to practical applications. However, quantum machine learning itself is limited by low effective dimensions achievable in state-of-the-art experiments. Here, we demonstrate highly successful classifications of real-life images using photonic qubits, combining a quantum tensor-network representation of hand-written digits and entanglement-based optimization. Specifically, we focus on binary classification for hand-written zeroes and ones, whose features are cast into the tensor-network representation, further reduced by optimization based on entanglement entropy and encoded into two-qubit photonic states. We then demonstrate image classification with a high success rate exceeding 98%, through successive gate operations and projective measurements. Although we work with photons, our approach is amenable to other physical realizations such as nitrogen-vacancy centers, nuclear spins, and trapped ions, and our scheme can be scaled to efficient multi-qubit encodings of features in the tensor-product representation, thereby setting the stage for quantum-enhanced multi-class classification.

Baseline Comparisons of Complementary Sampling Methods for Assembly Driven by Short-Ranged Pair Potentials toward Fast and Flexible Hybridization.

This work measures baseline sampling characteristics that highlight fundamental differences between sampling methods for assembly driven by short-ranged pair potentials. Such granular comparison is essential for fast, flexible, and accurate hybridization of complementary methods. Besides sampling speed, efficiency, and accuracy of uniform grid coverage, other sampling characteristics measured are (i) accuracy of covering narrow low energy regions that have low effective dimension (ii) ability to localize sampling to specific basins, and (iii) flexibility in sampling distributions. As a proof of concept, we compare a recently developed geometric methodology EASAL (Efficient Atlasing and Search of Assembly Landscapes) and the traditional Monte Carlo (MC) method for sampling the energy landscape of two assembling trans-membrane helices, driven by short-range pair potentials. By measuring the above-mentioned sampling characteristics, we demonstrate that EASAL provides localized and accurate coverage of crucial regions of the energy landscape of low effective dimension, under flexible sampling distributions, with much fewer samples and computational resources than MC sampling. EASAL's empirically validated theoretical guarantees permit credible extrapolation of these measurements and comparisons to arbitrary number and size of assembling units. Promising avenues for hybridizing the complementary advantages of the two methods are discussed.

Automatic control variates for option pricing using neural networks

Abstract Many pricing problems boil down to the computation of a high-dimensional integral, which is usually estimated using Monte Carlo. In fact, the accuracy of a Monte Carlo estimator with M simulations is given by σ M {\frac{\sigma}{\sqrt{M}}} . Meaning that its convergence is immune to the dimension of the problem. However, this convergence can be relatively slow depending on the variance σ of the function to be integrated. To resolve such a problem, one would perform some variance reduction techniques such as importance sampling, stratification, or control variates. In this paper, we will study two approaches for improving the convergence of Monte Carlo using Neural Networks. The first approach relies on the fact that many high-dimensional financial problems are of low effective dimensions. We expose a method to reduce the dimension of such problems in order to keep only the necessary variables. The integration can then be done using fast numerical integration techniques such as Gaussian quadrature. The second approach consists in building an automatic control variate using neural networks. We learn the function to be integrated (which incorporates the diffusion model plus the payoff function) in order to build a network that is highly correlated to it. As the network that we use can be integrated exactly, we can use it as a control variate.

A modeling substorm dynamics of the magnetosphere using self-organized criticality approach

Responses of Earth magnetic field during substorms exhibits a number of characteristics features such as the power-law spectra of fluctuations on different scales and signatures of low effective dimensions. Due the magnetosphere are constantly out-equilibrium their behavior is similar to real sandpiles during substorms, features of self-organized criticality (SOC) systems. Thus, in this work we presented a simple mathematical model to AE-index based on self-organizing sandpile mentioned by Uritsky and Pudovkin (1998), but we input the energy dissipation process inside the model. The statistical and multifractal tools to characterization of dynamical processes were used. The results also were compared with results from a classical geomagnetic event of the July, 2000, including the Bastille Day intense geomagnetic storm on 15 July.

Decentralized High-Dimensional Bayesian Optimization With Factor Graphs

This paper presents a novel decentralized high-dimensional Bayesian optimization (DEC-HBO) algorithm that, in contrast to existing HBO algorithms, can exploit the interdependent effects of various input components on the output of the unknown objective function f for boosting the BO performance and still preserve scalability in the number of input dimensions without requiring prior knowledge or the existence of a low (effective) dimension of the input space. To realize this, we propose a sparse yet rich factor graph representation of f to be exploited for designing an acquisition function that can be similarly represented by a sparse factor graph and hence be efficiently optimized in a decentralized manner using distributed message passing. Despite richly characterizing the interdependent effects of the input components on the output of f with a factor graph, DEC-HBO can still guarantee no-regret performance asymptotically. Empirical evaluation on synthetic and real-world experiments (e.g., sparse Gaussian process model with 1811 hyperparameters) shows that DEC-HBO outperforms the state-of-the-art HBO algorithms.

Solving High-Dimensional Multi-Objective Optimization Problems with Low Effective Dimensions

Multi-objective (MO) optimization problems require simultaneously optimizing two or more objective functions. An MO algorithm needs to find solutions that reach different optimal balances of the objective functions, i.e., optimal Pareto front, therefore, high dimensionality of the solution space can hurt MO optimization much severer than single-objective optimization, which was little addressed in previous studies. This paper proposes a general, theoretically-grounded yet simple approach ReMO, which can scale current derivative-free MO algorithms to the high-dimensional non-convex MO functions with low effective dimensions, using random embedding. We prove the conditions under which an MO function has a low effective dimension, and for such functions, we prove that ReMO possesses the desirable properties of optimal Pareto front preservation, time complexity reduction, and rotation perturbation invariance. Experimental results indicate that ReMO is effective for optimizing the high-dimensional MO functions with low effective dimensions, and is even effective for the high-dimensional MO functions where all dimensions are effective but most only have a small and bounded effect on the function value.

Embedded Bandits for Large-Scale Black-Box Optimization

Random embedding has been applied with empirical success to large-scale black-box optimization problems with low effective dimensions. This paper proposes the EmbeddedHunter algorithm, which incorporates the technique in a hierarchical stochastic bandit setting, following the optimism in the face of uncertainty principle and breaking away from the multiple-run framework in which random embedding has been conventionally applied similar to stochastic black-box optimization solvers. Our proposition is motivated by the bounded mean variation in the objective value for a low-dimensional point projected randomly into the decision space of Lipschitz-continuous problems. In essence, the EmbeddedHunter algorithm expands optimistically a partitioning tree over a low-dimensional — equal to the effective dimension of the problem —search space based on a bounded number of random embeddings of sampled points from the low-dimensional space. In contrast to the probabilistic theoretical guarantees of multiple-run random-embedding algorithms, the finite-time analysis of the proposed algorithm presents a theoretical upper bound on the regret as a function of the algorithm's number of iterations. Furthermore, numerical experiments were conducted to validate its performance. The results show a clear performance gain over recently proposed random embedding methods for large-scale problems, provided the intrinsic dimensionality is low.

A new method to explore the structure of effective dimensions for functions

Effective dimension, an indicator for the difficulty of high-dimensional integration, describes whether a function can be well approximated by low-dimensional terms or sums of low-order terms. Some problems in option pricing are believed to have low effective dimensions, which help explain the success of quasi-Monte Carlo (QMC) methods recently observed in financial engineering. This paper provides a way of studying the structure of effective dimensions by finding a proper space the function of interest belongs to and then determining the effective dimension of that space. To this end, we extend the definitions of effective dimensions to weighted function spaces with product-order-dependent weights and give bounds on norms and variances. Furthermore, we show that the proposed method is applicable to functions arising in option pricing and consequently offers some hints on the performance of QMC methods.

Scaling Simultaneous Optimistic Optimization for High-Dimensional Non-Convex Functions with Low Effective Dimensions

Simultaneous optimistic optimization (SOO) is a recently proposed global optimization method with a strong theoretical foundation. Previous studies have shown that SOO has a good performance in low-dimensional optimization problems, however, its performance is unsatisfactory when the dimensionality is high. This paper adapts random embedding to scaling SOO, resulting in the RESOO algorithm. We prove that the simple regret of RESOO depends only on the effective dimension of the problem, while that of SOO depends on the dimension of the solution space. Empirically, on some high-dimensional non-convex testing functions as well as hyper-parameter tuning tasks for multi-class support vector machines, RESOO shows significantly improved performance from SOO.

Pricing Derivative Securities Using Integrated Quasi--Monte Carlo Methods with Dimension Reduction and Discontinuity Realignment

The dimension and the smoothness of the integrands are the two key factors that affect the efficiency of the quasi--Monte Carlo (QMC) method. The first factor implies that the QMC method can have high performance on a problem with low effective dimension even though its nominal dimension can be very large. The second factor suggests that the QMC method can be very effective on a problem that is smooth or even on a problem that is discontinuous as long as its discontinuities are parallel to the axes. Motivated by these findings, methods such as the linear transformation (LT) method of [J. Imai and K. S. Tan, J. Comput. Finance, 10 (2006), pp. 129--155] and the orthogonal transformation (OT) method of [X. Wang and K. S. Tan, Management Sci., 59 (2013), pp. 376--389] have been proposed to increase the efficiency of QMC. However, both of these methods are unsatisfactory in that they only achieve optimality single sidedly in the sense that the LT method enhances QMC by effective dimension reduction while the O...

An Improved Algorithm for Pricing Derivatives Using Sobol Quasirandom Sequences

It is often argued that Quasi-Monte Carlo Methods (QMC ) only work for problems of low effective dimension that encompass most of financial problems. We will show here some evidence that, with the Sobol construction, they can be suited for problems with high effective dimension in the truncation sense as well as (moderately) high effective dimension in the superposition sense. We propose a methodology for obtaining the initial direction integers that are a central piece for generation of good Sobol sequences and show several examples both financial and nonfinancial where they can be used. Our sequences can deal with up to 35,000-dimensional problems.