Vector-valued Reproducing Kernel Hilbert Space Research Articles

KLMS algorithm in Vector-Valued RKHS for online nonlinear MIMO systems identification

This paper proposes an extension of the KLMS algorithm to the Vector-Valued Reproducing Kernel Hilbert Space (VV-RKHS), enabling the use of the operator-valued kernel (OV-kernel) in the online identification of multi-input multi-output (MIMO) nonlinear systems. The yielded multivariate kernel model offers more design flexibility and involves fewer parameters than other vector-valued KLMS algorithms present in the literature. Conditions ensuring the convergence of the proposed OV-KLMS algorithm are given. Experiments on a multivariate chaotic attractor as well as numerical simulations are carried out and show the effectiveness of the proposed algorithm

Stein Variational Gradient Descent on Infinite-Dimensional Space and Applications to Statistical Inverse Problems

In this paper, we propose an infinite-dimensional version of the Stein variational gradient descent (iSVGD) method for solving Bayesian inverse problems. The method can generate approximate samples from posteriors efficiently. Based on the concepts of operator-valued kernels and vector-valued reproducing kernel Hilbert spaces, a rigorous definition is given for the infinite-dimensional objects, e.g., the Stein operator, which are proved to be the limit of finite-dimensional ones. Moreover, a more efficient iSVGD with preconditioning operators is constructed by generalizing the change of variables formula and introducing a regularity parameter. The proposed algorithms are applied to an inverse problem of the steady state Darcy flow equation. Numerical results confirm our theoretical findings and demonstrate the potential applications of the proposed approach in the posterior sampling of large-scale nonlinear statistical inverse problems.

Vector-Valued Reproducing Kernel Hilbert $$C^*$$-Modules

The aim of this paper is to present a unified framework in the setting of Hilbert \(C^*\)-modules for the scalar- and vector-valued reproducing kernel Hilbert spaces and \(C^*\)-valued reproducing kernel spaces. We investigate conditionally negative definite kernels with values in the \(C^*\)-algebra of adjointable operators acting on a Hilbert \(C^*\)-module. In addition, we show that there exists a two-sided connection between positive definite kernels and reproducing kernel Hilbert \(C^*\)-modules. Furthermore, we explore some conditions under which a function is in the reproducing kernel module and present an interpolation theorem. Moreover, we study some basic properties of the so-called relative reproducing kernel Hilbert \(C^*\)-modules and give a characterization of dual modules. Among other things, we prove that every conditionally negative definite kernel gives us a reproducing kernel Hilbert \(C^*\)-module and a certain map. Several examples illustrate our investigation.

Learning rates for multi-task regularization networks

Multi-task learning is an important trend of machine learning in facing the era of artificial intelligence and big data. Despite a large amount of researches on learning rate estimates of various single-task machine learning algorithms, there is little parallel work for multi-task learning. We present mathematical analysis on the learning rate estimate of multi-task learning based on the theory of vector-valued reproducing kernel Hilbert spaces and matrix-valued reproducing kernels. For the typical multi-task regularization networks, an explicit learning rate dependent both on the number of sample data and the number of tasks is obtained. It reveals that the generalization ability of multi-task learning algorithms is indeed affected as the number of tasks increases.

Commutant Lifting and Nevanlinna–Pick Interpolation in Several Variables

This paper concerns a commutant lifting theorem and a Nevanlinna–Pick type interpolation result in the setting of multipliers from vector-valued Drury–Arveson space to a large class of vector-valued reproducing kernel Hilbert spaces over the unit ball in \({\mathbb {C}}^n\). The special case of reproducing kernel Hilbert spaces includes all natural examples of Hilbert spaces like Hardy space, Bergman space and weighted Bergman spaces over the unit ball.

Mismatch Removal Based on Gaussian Mixture Model for Aircraft Surface Texture Mapping

Aiming at the fact of lower efficiency and higher time cost for feature matching in aircraft surface texture mapping process, a novel mismatch removal method based on Gaussian mixture model is proposed to increase correct corresponding feature matching point pairs. The detection and initial point sets for corresponding pairs are carried out, and a vector field is interpolated between the two matching of ORB feature points. The Gaussian mixture model(GMM) is introduced and a prior is taken to force the smoothness of the field, which is based on the Tikhonov regularization in vector-valued reproducing kernel Hilbert space(RKHS). In order to obtain the optimal estimation, the MAP solution of a Bayesian model with latent variables, which could be performed by Expectation Maximization (EM) algorithm, is utilized to determine the correct correspondence. The experimental results show that the algorithm could remove mismatches effectively and the classification for feature points is excellent. The calculation time is greatly reduced, which enhanced real-time performance of aircraft surface texture mapping process.

A novel wind power probabilistic forecasting approach based on joint quantile regression and multi-objective optimization

Probabilistic forecasts of wind power generation play a significant role in operation management and decision-making for wind power producers and power system operators. This study introduces a novel joint quantile model based on the use of Mercer’s kernels in the vector-valued RKHSs (Reproducing Kernel Hilbert Spaces) for the nonparametric probabilistic forecasts of wind power. The proposed approach takes the autocorrelation of the (nonlinearly) transformed input and output into consideration and improves the model flexibility by constructing composite kernels. Besides, the model is optimized by a new meta-heuristics algorithm named Multi-Objective Salp Swarm Optimization Algorithm (MSSA), which mathematically models and mimics the behavior of salp swarms for solving multiple objective optimization problems. Under this model framework, several conditional quantiles of wind power can be estimated and predicted at the same time. Moreover, due to the joint estimation process, the problem of quantile curve crossing can be effectively weakened during the process of model training. The qualitative and quantitative performances of the proposed method are tested and verified on a set of real case studies. The results show that the proposed algorithm provides superior outputs compared to the well-known and recent algorithms in the literature.

Generalized linear models for geometrical current predictors: An application to predict garment fit

The aim of this article is to model an ordinal response variable in terms of vector-valued functional data included on a vector-valued reproducing kernel Hilbert space (RKHS). In particular, we focus on the vector-valued RKHS obtained when a geometrical object (body) is characterized by a current and on the ordinal regression model. A common way to solve this problem in functional data analysis is to express the data in the orthonormal basis given by decomposition of the covariance operator. But our data present very important differences with respect to the usual functional data setting. On the one hand, they are vector-valued functions, and on the other, they are functions in an RKHS with a previously defined norm. We propose to use three different bases: the orthonormal basis given by the kernel that defines the RKHS, a basis obtained from decomposition of the integral operator defined using the covariance function and a third basis that combines the previous two. The three approaches are compared and applied to an interesting problem: building a model to predict the fit of children's garment sizes, based on a 3D database of the Spanish child population. Our proposal has been compared with alternative methods that explore the performance of other classifiers (Support Vector Machine and [Formula: see text]-NN), and with the result of applying the classification method proposed in this work, from different characterizations of the objects (landmarks and multivariate anthropometric measurements instead of currents), obtaining in all these cases worst results.

Supervised classification of geometrical objects by integrating currents and functional data analysis

This paper focuses on the application of supervised classification techniques to a set of geometrical objects (bodies) characterized by currents, in particular, discriminant analysis and some nonparametric methods. A current is a relevant mathematical object to model geometrical data, like hypersurfaces, through integration of vector fields over them. As a consequence of the choice of a vector-valued reproducing kernel Hilbert space (RKHS) as a test space to integrate over hypersurfaces, it is possible to consider that hypersurfaces are embedded in this Hilbert space. This embedding enables us to consider classification algorithms of geometrical objects. We present a method to apply supervised classification techniques in the obtained vector-valued RKHS. This method is based on the eigenfunction decomposition of the kernel. The novelty of this paper is therefore the reformulation of a size and shape supervised classification problem in functional data analysis terms using the theory of currents and vector-valued RKHSs. This approach is applied to a 3D database obtained from an anthropometric survey of the Spanish child population with a potential application to online sales of children’s wear.

Dynamic mode decomposition in vector-valued reproducing kernel Hilbert spaces for extracting dynamical structure among observables

Understanding nonlinear dynamical systems (NLDSs) is challenging in a variety of engineering and scientific fields. Dynamic mode decomposition (DMD), which is a numerical algorithm for the spectral analysis of Koopman operators, has been attracting attention as a way of obtaining global modal descriptions of NLDSs without requiring explicit prior knowledge. However, since existing DMD algorithms are in principle formulated based on the concatenation of scalar observables, it is not directly applicable to data with dependent structures among observables, which take, for example, the form of a sequence of graphs. In this paper, we formulate Koopman spectral analysis for NLDSs with structures among observables and propose an estimation algorithm for this problem. This method can extract and visualize the underlying low-dimensional global dynamics of NLDSs with structures among observables from data, which can be useful in understanding the underlying dynamics of such NLDSs. To this end, we first formulate the problem of estimating spectra of the Koopman operator defined in vector-valued reproducing kernel Hilbert spaces, and then develop an estimation procedure for this problem by reformulating tensor-based DMD. As a special case of our method, we propose the method named as Graph DMD, which is a numerical algorithm for Koopman spectral analysis of graph dynamical systems, using a sequence of adjacency matrices. We investigate the empirical performance of our method by using synthetic and real-world data.

Semi-supervised cross-modal common representation learning with vector-valued manifold regularization

While cross-media data, like text, image, audio, video and 3D model, has been the main form of big data, there is a current dearth of research on cross-media retrieval. In this paper, we focus on how to learn the common representation of heterogeneous data which is a key challenge for cross-media retrieval. Most existing approaches linearly project original low-level feature into a joint feature space for isomorphic data representation. However, linear projection cannot capture most complex cross-modal correlation with high nonlinearity. In this paper, we propose a novel feature learning algorithm, which is semi-supervised cross-modal vector-valued manifold regularization (SCVM), to explore common representation of heterogeneous data. SCVM jointly explores low-level feature correlation and semantic information in a unified framework. Based on manifold regularization, we learn cross-media features from vector-valued reproducing kernel Hilbert spaces (RKHS) by kernel transformation on both labeled and unlabeled samples; moreover, we impose smoothness constraints of possible solutions to improve retrieval accuracy. Comparing with the current state-of-the-art approaches on two public datasets, comprehensive experimental results show superior performance of our SCVM. The method is more robust and stable when extended from two media types to five media types, which is very attractive in practical application.

Some Properties of Reproducing Kernel Banach and Hilbert Spaces

This paper is devoted to the study of reproducing kernel Hilbert spaces. We focus on multipliers of reproducing kernel Banach and Hilbert spaces. In particular, we try to extend this concept and prove some related theorems. Moreover, we focus on reproducing kernels in vector-valued reproducing kernel Hilbert spaces. In particular, we extend reproducing kernels to relative reproducing kernels and prove some theorems in this subject.

Unsupervised classification of children’s bodies using currents

Object classification according to their shape and size is of key importance in many scientific fields. This work focuses on the case where the size and shape of an object is characterized by a current. A current is a mathematical object which has been proved relevant to the modeling of geometrical data, like submanifolds, through integration of vector fields along them. As a consequence of the choice of a vector-valued reproducing kernel Hilbert space (RKHS) as a test space for integrating manifolds, it is possible to consider that shapes are embedded in this Hilbert Space. A vector-valued RKHS is a Hilbert space of vector fields; therefore, it is possible to compute a mean of shapes, or to calculate a distance between two manifolds. This embedding enables us to consider size-and-shape clustering algorithms. These algorithms are applied to a 3D database obtained from an anthropometric survey of the Spanish child population with a potential application to online sales of children’s wear.

A unifying framework in vector-valued reproducing kernel Hilbert spaces for manifold regularization and co-regularized multi-view learning

This paper presents a general vector-valued reproducing kernel Hilbert spaces (RKHS) framework for the problem of learning an unknown functional dependency between a structured input space and a structured output space. Our formulation encompasses both Vector-valued Manifold Regularization and Co-regularized Multi-view Learning, providing in particular a unifying framework linking these two important learning approaches. In the case of the least square loss function, we provide a closed form solution, which is obtained by solving a system of linear equations. In the case of Support Vector Machine (SVM) classification, our formulation generalizes in particular both the binary Laplacian SVM to the multi-class, multi-view settings and the multi-class Simplex Cone SVM to the semi-supervised, multi-view settings. The solution is obtained by solving a single quadratic optimization problem, as in standard SVM, via the Sequential Minimal Optimization (SMO) approach. Empirical results obtained on the task of object recognition, using several challenging data sets, demonstrate the competitiveness of our algorithms compared with other state-of-the-art methods.

Learn the Lagrangian: A Vector-Valued RKHS Approach to Identifying Lagrangian Systems.

We study the modeling of Lagrangian systems with multiple degrees of freedom. Based on system dynamics, canonical parametric models require ad hoc derivations and sometimes simplification for a computable solution; on the other hand, due to the lack of prior knowledge in the system's structure, modern nonparametric models in machine learning face the curse of dimensionality, especially in learning large systems. In this paper, we bridge this gap by unifying the theories of Lagrangian systems and vector-valued reproducing kernel Hilbert space. We reformulate Lagrangian systems with kernels that embed the governing Euler-Lagrange equation-the Lagrangian kernels-and show that these kernels span a subspace capturing the Lagrangian's projection as inverse dynamics. By such property, our model uses only inputs and outputs as in machine learning and inherits the structured form as in system dynamics, thereby removing the need for the mundane derivations for new systems as well as the generalization problem in learning from scratches. In effect, it learns the system's Lagrangian, a simpler task than directly learning the dynamics. To demonstrate, we applied the proposed kernel to identify the robot inverse dynamics in simulations and experiments. Our results present a competitive novel approach to identifying Lagrangian systems, despite using only inputs and outputs.

Multikernel least mean square algorithm.

The multikernel least-mean-square algorithm is introduced for adaptive estimation of vector-valued nonlinear and nonstationary signals. This is achieved by mapping the multivariate input data to a Hilbert space of time-varying vector-valued functions, whose inner products (kernels) are combined in an online fashion. The proposed algorithm is equipped with novel adaptive sparsification criteria ensuring a finite dictionary, and is computationally efficient and suitable for nonstationary environments. We also show the ability of the proposed vector-valued reproducing kernel Hilbert space to serve as a feature space for the class of multikernel least-squares algorithms. The benefits of adaptive multikernel (MK) estimation algorithms are illuminated in the nonlinear multivariate adaptive prediction setting. Simulations on nonlinear inertial body sensor signals and nonstationary real-world wind signals of low, medium, and high dynamic regimes support the approach.

Refinement of operator-valued reproducing kernels

This paper studies the construction of a refinement kernel for a given operator-valued reproducing kernel such that the vector-valued reproducing kernel Hilbert space of the refinement kernel conta...

Operator algebras of functions

We present some general theorems about operator algebras that are algebras of functions on sets, including theories of local algebras, residually finite-dimensional operator algebras and algebras that can be represented as the scalar multipliers of a vector-valued reproducing kernel Hilbert space. We use these to further develop a quantized function theory for various domains that extends and unifies Agler's theory of commuting contractions and the Arveson–Drury–Popescu theory of commuting row contractions. We obtain analogous factorization theorems, prove that the algebras that we obtain are dual operator algebras and show that for many domains, supremums over all commuting tuples of operators satisfying certain inequalities are obtained over all commuting tuples of matrices.