Decay Of Eigenvalues Research Articles

Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime* *This article is an updated version of: Cui H, Loureiro B, Krzakala F and Zdeborová L 2021 Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin,

In this manuscript we consider kernel ridge regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the features co-variance. These decays were, however, provided for sizeably different setups, namely in the noiseless case with constant regularization and in the noisy optimally regularized case. Intermediary settings have been left substantially uncharted. In this work, we unify and extend this line of work, providing characterization of all regimes and excess error decay rates that can be observed in terms of the interplay of noise and regularization. In particular, we show the existence of a transition in the noisy setting between the noiseless exponents to its noisy values as the sample complexity is increased. Finally, we illustrate how this crossover can also be observed on real data sets.

CANONICAL THRESHOLDING FOR NON-SPARSE HIGH-DIMENSIONAL LINEAR REGRESSION.

We consider a high-dimensional linear regression problem. Unlike many papers on the topic, we do not require sparsity of the regression coefficients; instead, our main structural assumption is a decay of eigenvalues of the covariance matrix of the data. We propose a new family of estimators, called the canonical thresholding estimators, which pick largest regression coefficients in the canonical form. The estimators admit an explicit form and can be linked to LASSO and Principal Component Regression (PCR). A theoretical analysis for both fixed design and random design settings is provided. Obtained bounds on the mean squared error and the prediction error of a specific estimator from the family allow to clearly state sufficient conditions on the decay of eigenvalues to ensure convergence. In addition, we promote the use of the relative errors, strongly linked with the out-of-sample R 2. The study of these relative errors leads to a new concept of joint effective dimension, which incorporates the covariance of the data and the regression coefficients simultaneously, and describes the complexity of a linear regression problem. Some minimax lower bounds are established to showcase the optimality of our procedure. Numerical simulations confirm good performance of the proposed estimators compared to the previously developed methods.

Unified Analysis of Periodization-Based Sampling Methods for Matérn Covariances

The periodization of a stationary Gaussian random field on a sufficiently large torus comprising the spatial domain of interest is the basis of various efficient computational methods, such as the classical circulant embedding technique using the fast Fourier transform for generating samples on uniform grids. For the family of Matérn covariances with smoothness index $\nu$ and correlation length $\lambda$, we analyze the nonsmooth periodization (corresponding to classical circulant embedding) and an alternative procedure using a smooth truncation of the covariance function. We solve two open problems: the first concerning the $\nu$-dependent asymptotic decay of eigenvalues of the resulting circulant in the nonsmooth case, the second concerning the required size in terms of $\nu$, $\lambda$ of the torus when using a smooth periodization. In doing this we arrive at a complete characterization of the performance of these two approaches. Both our theoretical estimates and the numerical tests provided here show substantial advantages of smooth truncation.

Proper orthogonal decomposition of atomistic flow simulations

An adaptive proper orthogonal decomposition based on time windows (WPOD) for analysis of velocity fields from atomistic simulations is presented. The method effectively separates the field into ensemble average and fluctuation components, and can be applied to both stationary and non-stationary flows in simple and complex geometries. The criteria to distinguish POD modes representing ensemble average and fluctuation components are based on adaptive examination of eigenvalue decomposition, specifically on the rate of decay of POD eigenvalues and analysis of POD eigenvectors. The WPOD method is efficient and its superior accuracy leads to smooth field gradients that can be used effectively in multiscale formulations. The method has been applied in molecular dynamics and dissipative particle dynamics simulations. We demonstrate the method for several cases including steady and unsteady flow and red blood cell flow, but the same approach can be used in atomistic simulations of materials.

TEMPORARY DIMENSIONS OF MULTIVARIATE DATA FROM PALEOCLIMATE RECORDS — A NOVEL MEASURE FOR DYNAMIC CHARACTERIZATION OF LONG-TERM CLIMATE CHANGE

Standard techniques of nonlinear data analysis are difficult to apply to short or insufficiently resolved multivariate time series. We demonstrate that dimension estimation based on the decay of eigenvalues of the covariance matrix yields qualitatively robust characteristics even in the case of very short measurement series. The uncertainty of these characteristics is discussed for both: synthetic data sets and measurement data. We apply this approach to measurements of trace element abundances in a marine sediment core obtained at the East Antarctic coast and discuss the results from a paleoclimate point of view.

Eigenvalue distribution of Mercer‐like kernels

AbstractWe study eigenvalues of positive definite kernels of L2 integral operators on arbitrary intervals. Assuming integrability and uniform continuity of the kernel on the diagonal, we show that the eigenvalue distribution is totally determined by the smoothness of the kernel together with its decay rate at infinity along the diagonal. Moreover, the rate of decay of eigenvalues depends on both these quantities in a symmetrical way. Our main result treats all possible orders of differentiability and all possible rates of decay of the kernel; the known optimal results for eigenvalue distribution of positive definite kernels in compact intervals are particular cases. (© 2007 WILEY‐VCH Verlag GmbH & Co. KGaA, Weinheim)

Lower Bounds and Nonuniform Time Discretization for Approximation of Stochastic Heat Equations

We study algorithms for approximation of the mild solution of stochastic heat equations on the spatial domain ]0, 1[d. The error of an algorithm is defined in L2-sense. We derive lower bounds for the error of every algorithm that uses a total of N evaluations of one-dimensional components of the driving Wiener process W. For equations with additive noise we derive matching upper bounds and we construct asymptotically optimal algorithms. The error bounds depend on N and d, and on the decay of eigenvalues of the covariance of W in the case of nuclear noise. In the latter case the use of nonuniform time discretizations is crucial.

On Hybrid Boundary Control of Flexible Systems

Simple proofs are presented which show that hybrid boundary control systems are asymptotically, but not exponentially, stable. The asymptotic rates of decay of eigenvalues are determined analytically for both second and fourth-order systems.

Eigenvalues of Weakly Singular Integral Operators

We determine the asymptotic order of decay of eigenvalues of weakly singular integral operators. The singularities are of quite general form, containing power and logarithmic terms. We give a unified elementary proof of all known results in this area. Our approach applies also in the case where the power order of the singularity is equal to the dimension of the domain and the logarithmic order is less than — 1. This case has not been considered previously. Furthermore, we show the optimality of the upper estimates in a rather strong sense. In particular, we give a partial positive answer to the conjecture of [3].