Reproducing Kernel Hilbert Space Norm Research Articles

Sample Complexity and Minimax Properties of Exponentially Stable Regularized Estimators

Recent studies have shown how regularization may play an important role in linear system identification. An effective approach consists of searching for the impulse response in a high-dimensional space, e.g., a reproducing kernel Hilbert space (RKHS). Complexity is then controlled using a regularizer, e.g., the RKHS norm, able to encode smoothness and stability information. Examples are RKHSs induced by the so-called stable spline or tuned-correlated kernels, which contain a parameter that regulates impulse response exponential decay. In this article, we derive nonasymptotic upper bounds on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _2$</tex-math></inline-formula> error of these regularized schemes and study their optimality in order (in the minimax sense). Under white noise inputs and Gaussian measurement noises, we obtain conditions which ensure the optimal convergence rate for all the class of stable spline estimators and several generalizations. Theoretical findings are then illustrated via a numerical experiment.

Koopman spectra in reproducing kernel Hilbert spaces

Every invertible, measure-preserving dynamical system induces a Koopman operator, which is a linear, unitary evolution operator acting on the $L^2$ space of observables associated with the invariant measure. Koopman eigenfunctions represent the quasiperiodic, or non-mixing, component of the dynamics. The extraction of these eigenfunctions and their associated eigenfrequencies from a given time series is a non-trivial problem when the underlying system has a dense point spectrum, or a continuous spectrum behaving similarly to noise. This paper describes methods for identifying Koopman eigenfrequencies and eigenfunctions from a discretely sampled time series generated by such a system with unknown dynamics. Our main result gives necessary and sufficient conditions for a Fourier function, defined on $N$ states sampled along an orbit of the dynamics, to be extensible to a Koopman eigenfunction on the whole state space, lying in a reproducing kernel Hilbert space (RKHS). In particular, we show that such an extension exists if and only if the RKHS norm of the Fourier function does not diverge as $ N \to \infty $, in which case the corresponding Fourier frequency is also a Koopman eigenfrequency. If such an RKHS extension does not exist, we can still construct an $L^2$ approximation of the eigenfunction. Numerical experiments on mixed-spectrum systems with weak periodic components demonstrate that this approach has significantly higher skill in identifying Koopman eigenfrequencies compared to conventional spectral estimation techniques based on the discrete Fourier transform.

Some Notes on Error Analysis for Kernel Based Regularized Interpolation

Kernel based regularized interpolation is one of the most important methods for approximating functions. The theory behind the kernel based regularized interpolation is the well-known Representer Theorem, which shows the form of approximation function in the reproducing kernel Hilbert spaces. Because of the advantages of the kernel based regularized interpolation, it is widely used in many mathematical and engineering applications, for example, dimension reduction and dimension estimation. However, the performance of the approximation is not fully understood from the theoretical perspective. In other word, the error analysis for the kernel based regularized interpolation is lacking. In this paper, some error bounds in terms of the reproducing kernel Hilbert space norm and Sobolev space norm are given to understand the behavior of the approximation function.

Fast and strong convergence of online learning algorithms

In this paper, we study the online learning algorithm without explicit regularization terms. This algorithm is essentially a stochastic gradient descent scheme in a reproducing kernel Hilbert space (RKHS). The polynomially decaying step size in each iteration can play a role of regularization to ensure the generalization ability of online learning algorithm. We develop a novel capacity dependent analysis on the performance of the last iterate of online learning algorithm. The contribution of this paper is two-fold. First, our nice analysis can lead to the convergence rate in the standard mean square distance which is the best so far. Second, we establish, for the first time, the strong convergence of the last iterate with polynomially decaying step sizes in the RKHS norm. We demonstrate that the theoretical analysis established in this paper fully exploits the fine structure of the underlying RKHS, and thus can lead to sharp error estimates of online learning algorithm.

Efficient nonparametric Bayesian inference for $X$-ray transforms

We consider the statistical inverse problem of recovering a function $f:M\to \mathbb{R}$, where $M$ is a smooth compact Riemannian manifold with boundary, from measurements of general $X$-ray transforms $I_{a}(f)$ of $f$, corrupted by additive Gaussian noise. For $M$ equal to the unit disk with “flat” geometry and $a=0$ this reduces to the standard Radon transform, but our general setting allows for anisotropic media $M$ and can further model local “attenuation” effects—both highly relevant in practical imaging problems such as SPECT tomography. We study a nonparametric Bayesian inference method based on standard Gaussian process priors for $f$. The posterior reconstruction of $f$ corresponds to a Tikhonov regulariser with a reproducing kernel Hilbert space norm penalty that does not require the calculation of the singular value decomposition of the forward operator $I_{a}$. We prove Bernstein–von Mises theorems for a large family of one-dimensional linear functionals of $f$, and they entail that posterior-based inferences such as credible sets are valid and optimal from a frequentist point of view. In particular we derive the asymptotic distribution of smooth linear functionals of the Tikhonov regulariser, which attains the semiparametric information lower bound. The proofs rely on an invertibility result for the “Fisher information” operator $I_{a}^{*}I_{a}$ between suitable function spaces, a result of independent interest that relies on techniques from microlocal analysis. We illustrate the performance of the proposed method via simulations in various settings.

Gradient descent for robust kernel-based regression

In this paper, we study the gradient descent algorithm generated by a robust loss function over a reproducing kernel Hilbert space (RKHS). The loss function is defined by a windowing function G and a scale parameter σ, which can include a wide range of commonly used robust losses for regression. There is still a gap between theoretical analysis and optimization process of empirical risk minimization based on loss: the estimator needs to be global optimal in the theoretical analysis while the optimization method can not ensure the global optimality of its solutions. In this paper, we aim to fill this gap by developing a novel theoretical analysis on the performance of estimators generated by the gradient descent algorithm. We demonstrate that with an appropriately chosen scale parameter σ, the gradient update with early stopping rules can approximate the regression function. Our elegant error analysis can lead to convergence in the standard L2 norm and the strong RKHS norm, both of which are optimal in the mini-max sense. We show that the scale parameter σ plays an important role in providing robustness as well as fast convergence. The numerical experiments implemented on synthetic examples and real data set also support our theoretical results.

Refined bounds for online pairwise learning algorithms

Abstract Motivated by the recent growing interest in pairwise learning problems, we study the generalization performance of Online Pairwise lEaRning Algorithm (OPERA) in a reproducing kernel Hilbert space (RKHS) without an explicit regularization. The convergence rates established in this paper can be arbitrarily closed to O ( T − 1 2 ) within T iterations and largely improve the existing convergence rates for OPERA. Our novel analysis is conducted by showing an almost boundedness of the iterates encountered in the learning process with high probability after establishing an induction lemma on refining the RKHS norm estimate of the iterates.

The Connection Between Bayesian Estimation of a Gaussian Random Field and RKHS.

Reconstruction of a function from noisy data is key in machine learning and is often formulated as a regularized optimization problem over an infinite-dimensional reproducing kernel Hilbert space (RKHS). The solution suitably balances adherence to the observed data and the corresponding RKHS norm. When the data fit is measured using a quadratic loss, this estimator has a known statistical interpretation. Given the noisy measurements, the RKHS estimate represents the posterior mean (minimum variance estimate) of a Gaussian random field with covariance proportional to the kernel associated with the RKHS. In this brief, we provide a statistical interpretation when more general losses are used, such as absolute value, Vapnik or Huber. Specifically, for any finite set of sampling locations (that includes where the data were collected), the maximum a posteriori estimate for the signal samples is given by the RKHS estimate evaluated at the sampling locations. This connection establishes a firm statistical foundation for several stochastic approaches used to estimate unknown regularization parameters. To illustrate this, we develop a numerical scheme that implements a Bayesian estimator with an absolute value loss. This estimator is used to learn a function from measurements contaminated by outliers.

Learning using state space kernel machines

AbstractReconstruction of a function from noisy data is often formulated as a regularized optimization problem whose solution closely matches an observed data set and also has a small reproducing kernel Hilbert space norm. The loss functions that measure agreement between the data and the function are often smooth (e.g. the least squares penalty), but non-smooth loss functions are of interest in many applications. Using the least squares penalty, large machine learning problems with kernels amenable to a stochastic state space representation (which we call state space kernel machines) have been solved using a Kalman smoothing approach. In this paper we extend this approach for state space kernel machines to the Vapnik penalty, a particularly important non-smooth penalty that is robust to outlier noise in the measurements and induces a sparse representation of the reconstructed function. We exploit the structure of such models using interior point methods that efficiently solve the functional recovery problem with the Vapnik penalty, and demonstrate the effectiveness of the method on a numerical experiment. The computational effort of our approach scales linearly with the number of data points.

Regularization in kernel learning

Under mild assumptions on the kernel, we obtain the best known error rates in a regularized learning scenario taking place in the corresponding reproducing kernel Hilbert space (RKHS). The main novelty in the analysis is a proof that one can use a regularization term that grows significantly slower than the standard quadratic growth in the RKHS norm.

Learning rates for regularized classifiers using multivariate polynomial kernels

Regularized classifiers (a leading example is support vector machine) are known to be a kind of kernel-based classification methods generated from Tikhonov regularization schemes, and the polynomial kernels are the original and also probably the most important kernels used in them. In this paper, we provide an error analysis for the regularized classifiers using multivariate polynomial kernels. We introduce Bernstein–Durrmeyer polynomials, whose reproducing kernel Hilbert space norms and approximation properties in L 1 space play a key role in the analysis of regularization error. We also introduce the standard estimation of sample error, and derive explicit learning rates for these algorithms.

Information Consistency of Nonparametric Gaussian Process Methods

Bayesian nonparametric models are widely and successfully used for statistical prediction. While posterior consistency properties are well studied in quite general settings, results have been proved using abstract concepts such as metric entropy, and they come with subtle conditions which are hard to validate and not intuitive when applied to concrete models. Furthermore, convergence rates are difficult to obtain. By focussing on the concept of information consistency for Bayesian Gaussian process (GP)models, consistency results and convergence rates are obtained via a regret bound on cumulative log loss. These results depend strongly on the covariance function of the prior process, thereby giving a novel interpretation to penalization with reproducing kernel Hilbert space norms and to commonly used covariance function classes and their parameters. The proof of the main result employs elementary convexity arguments only. A theorem of Widom is used in order to obtain precise convergence rates for several covariance functions widely used in practice.

Model selection in nonparametric hazard regression

We propose a novel model selection method for a nonparametric extension of the Cox proportional hazard model, in the framework of smoothing splines ANOVA models. The method automates the model building and model selection processes simultaneously by penalizing the reproducing kernel Hilbert space norms. On the basis of a reformulation of the penalized partial likelihood, we propose an efficient algorithm to compute the estimate. The solution demonstrates great flexibility and easy interpretability in modeling relative risk functions for censored data. Adaptive choice of the smoothing parameter is discussed. Both simulations and a real example suggest that our proposal is a useful tool for multivariate function estimation and model selection in survival analysis.

Approximation with polynomial kernels and SVM classifiers

This paper presents an error analysis for classification algorithms generated by regularization schemes with polynomial kernels. Explicit convergence rates are provided for support vector machine (SVM) soft margin classifiers. The misclassification error can be estimated by the sum of sample error and regularization error. The main difficulty for studying algorithms with polynomial kernels is the regularization error which involves deeply the degrees of the kernel polynomials. Here we overcome this difficulty by bounding the reproducing kernel Hilbert space norm of Durrmeyer operators, and estimating the rate of approximation by Durrmeyer operators in a weighted L1 space (the weight is a probability distribution). Our study shows that the regularization parameter should decrease exponentially fast with the sample size, which is a special feature of polynomial kernels.

Wavelet kernel penalized estimation for non-equispaced design regression

The paper considers regression problems with univariate design points. The design points are irregular and no assumptions on their distribution are imposed. The regression function is retrieved by a wavelet based reproducing kernel Hilbert space (RKHS) technique with the penalty equal to the sum of blockwise RKHS norms. In order to simplify numerical optimization, the problem is replaced by an equivalent quadratic minimization problem with an additional penalty term. The computational algorithm is described in detail and is implemented with both the sets of simulated and real data. Comparison with existing methods showed that the technique suggested in the paper does not oversmooth the function and is superior in terms of the mean squared error. It is also demonstrated that under additional assumptions on design points the method achieves asymptotic optimality in a wide range of Besov spaces.

Wavelet-based functional reconstruction and extrapolation of fractional random fields

Least-squares linear functional reconstruction and extrapolation of a random field defining the random input of a linear system represented by an integral equation is considered. This problem is solved for a class of random fields with reproducing kernel Hilbert space norm equivalent to the norm of a Sobolev space of an appropriate fractional order. More specifically, functional reconstruction and extrapolation formulae are derived from generalized wavelet-based orthogonal expansions of the input and output random fields in the class considered (see Angulo and Ruiz-Medina, 1999, for the ordinary case). In the Gaussian and ordinary case, the results derived also provide sample-path functional reconstruction and extrapolation formulae. Simulation studies are carried out for systems defined in terms of fractional integration of fractional Brownian motion.

Functional analytic framework for model selection

Model selection is one of the most important tasks in the identification of black-box systems. In this paper, we give a novel model selection method from the viewpoint of functional analysis. We formulate the system identification problem as a function approximation problem in a reproducing kernel Hilbert space (RKHS), where the approximation error is measured by the RKHS norm. Within this framework, we derive an estimator of the approximation error called the subspace information criterion (SIC) and show its properties.

Some relations among RKHS norms, Fredholm equations, and innovations representations

We first show how reproducing kernel Hilbert space (RKHS) norms can be determined for a large class of covariance functions by methods based on the solution of a Riccati differential equation or a Wiener-Hopf integral equation. Efficient numerical algorithms for such equations have been extensively studied, especially in the control literature. The innovations representations enter in that it is they that suggest the form of the RKHS norms. From the RKHS norms, we show how recursive solutions can be obtained for certain Fredholm equations of the first kind that are widely used in certain approaches to detection theory. Our approach specifies a unique solution: moreover, the algorithms used are well suited to the treatment of increasing observation intervals.