Infinite Dimensional Feature Space Research Articles

Adaptive data-driven fault-tolerant control for nonlinear systems: Koopman-based virtual actuator approach

This paper proposes an adaptive data-driven fault-tolerant control scheme using the Koopman operator for unknown dynamics subjected to nonlinearities, time-varying loss of effectiveness, and additive actuator faults. The main objective of this method is to design a virtual actuator to hide actuator faults from the view of the system’s nominal controller without having any prior knowledge about the system’s underlying dynamics. The designed virtual actuator is placed between the faulty plant and the nominal controller of the system to keep the dynamical system’s performance consistent before and after the occurrence of actuator faults. Based on the Koopman operator theory, an equivalent Koopman predictor is first obtained using the process data only, without knowing the governing equations of the underlying dynamics. Koopman operator is an infinite-dimensional, linear operator which takes the nonlinear process data into an infinite-dimensional feature space where the dynamic data correlations have linear behavior. Next, based on the approximated system’s Koopman operator, a virtual actuator is designed and implemented without knowing the system’s nominal controller. Needless to use a separate fault detection, isolation, and identification module to perform fault-tolerant control, the current method leverages the adaptive framework to keep the system’s desired performance in facing time-varying additive and loss of effectiveness actuator faults. Finally, the approach’s efficacy is demonstrated using simulation on a two-link manipulator benchmark, and a comparison study is presented.

A Kernel-Expanded Stochastic Neural Network

Abstract The deep neural network suffers from many fundamental issues in machine learning. For example, it often gets trapped into a local minimum in training, and its prediction uncertainty is hard to be assessed. To address these issues, we propose the so-called kernel-expanded stochastic neural network (K-StoNet) model, which incorporates support vector regression as the first hidden layer and reformulates the neural network as a latent variable model. The former maps the input vector into an infinite dimensional feature space via a radial basis function kernel, ensuring the absence of local minima on its training loss surface. The latter breaks the high-dimensional non-convex neural network training problem into a series of low-dimensional convex optimization problems, and enables its prediction uncertainty easily assessed. The K-StoNet can be easily trained using the imputation-regularized optimization algorithm. Compared to traditional deep neural networks, K-StoNet possesses a theoretical guarantee to asymptotically converge to the global optimum and enables the prediction uncertainty easily assessed. The performances of the new model in training, prediction and uncertainty quantification are illustrated by simulated and real data examples.

Kernel-based Partial Permutation Test for Detecting Heterogeneous Functional Relationship

We propose a kernel-based partial permutation test for checking the equality of functional relationship between response and covariates among different groups. The main idea, which is intuitive and easy to implement, is to keep the projections of the response vector Y on leading principle components of a kernel matrix fixed and permute Y’s projections on the remaining principle components. The proposed test allows for different choices of kernels, corresponding to different classes of functions under the null hypothesis. First, using linear or polynomial kernels, our partial permutation tests are exactly valid in finite samples for linear or polynomial regression models with Gaussian noise; similar results straightforwardly extend to kernels with finite feature spaces. Second, by allowing the kernel feature space to diverge with the sample size, the test can be large-sample valid for a wider class of functions. Third, for general kernels with possibly infinite-dimensional feature space, the partial permutation test is exactly valid when the covariates are exactly balanced across all groups, or asymptotically valid when the underlying function follows certain regularized Gaussian processes. We further suggest test statistics using likelihood ratio between two (nested) GPR models, and propose computationally efficient algorithms utilizing the EM algorithm and Newton’s method, where the latter also involves Fisher scoring and quadratic programming and is particularly useful when EM suffers from slow convergence. Extensions to correlated and non-Gaussian noises have also been investigated theoretically or numerically. Furthermore, the test can be extended to use multiple kernels together and can thus enjoy properties from each kernel. Both simulation study and application illustrate the properties of the proposed test.

Optimal Transport in Reproducing Kernel Hilbert Spaces: Theory and Applications.

In this paper, we present a mathematical and computational framework for comparing and matching distributions in reproducing kernel Hilbert spaces (RKHS). This framework, called optimal transport in RKHS, is a generalization of the optimal transport problem in input spaces to (potentially) infinite-dimensional feature spaces. We provide a computable formulation of Kantorovich's optimal transport in RKHS. In particular, we explore the case in which data distributions in RKHS are Gaussian, obtaining closed-form expressions of both the estimated Wasserstein distance and optimal transport map via kernel matrices. Based on these expressions, we generalize the Bures metric on covariance matrices to infinite-dimensional settings, providing a new metric between covariance operators. Moreover, we extend the correlation alignment problem to Hilbert spaces, giving a new strategy for matching distributions in RKHS. Empirically, we apply the derived formulas under the Gaussianity assumption to image classification and domain adaptation. In both tasks, our algorithms yield state-of-the-art performances, demonstrating the effectiveness and potential of our framework.

Kernel methods for the approximation of some key quantities of nonlinear systems

We introduce a data-based approach to estimating key quantities which arise in the study of nonlinear control systems and random nonlinear dynamical systems. Our approach hinges on the observation that much of the existing linear theory may be readily extended to nonlinear systems -with a reasonable expectation of success -once the nonlinear system has been mapped into a high or infinite dimensional feature space. In particular, we embed a nonlinear system in a reproducing kernel Hilbert space where linear theory can be used to develop computable, non-parametric estimators approximating controllability and observability energy functions for nonlinear systems. In all cases the relevant quantities are estimated from simulated or observed data. It is then shown that the controllability energy estimator provides a key means for approximating the invariant measure of an ergodic, stochastically forced nonlinear system.

Fast hand posture classification using depth features extracted from random line segments

In this paper we propose a set of fast-computable depth features for static hand posture classification from a single depth image. The proposed features, which are extracted from pixels on randomly positioned line segments, are specially designed as low-level cues to be used in random forest classifier which combines the cues to discover high-level unseen informative structure in an infinite dimensional feature space. The proposed features, while being simple, can effectively capture both hand geometry shape and depth texture information. The accuracy and speed performance of the recognition algorithm based on the proposed features is evaluated with American Sign Language (ASL) finger spelling dataset and with two new hand posture datasets. The proposed algorithm has a recognition accuracy rate that is comparable to the state-of-the-art methods, while being much faster in both training and testing phases. Our implementation of the proposed algorithm runs at about 600fps using only one thread of an i7 CPU. A pre-trained demo program is available to public.

Bayesian Active Remote Sensing Image Classification

In recent years, kernel methods, in particular support vector machines (SVMs), have been successfully introduced to remote sensing image classification. Their properties make them appropriate for dealing with a high number of image features and a low number of available labeled spectra. The introduction of alternative approaches based on (parametric) Bayesian inference has been quite scarce in the more recent years. Assuming a particular prior data distribution may lead to poor results in remote sensing problems because of the specificities and complexity of the data. In this context, the emerging field of nonparametric Bayesian methods constitutes a proper theoretical framework to tackle the remote sensing image classification problem. This paper exploits the Bayesian modeling and inference paradigm to tackle the problem of kernel-based remote sensing image classification. This Bayesian methodology is appropriate for both finite- and infinite-dimensional feature spaces. The particular problem of active learning is addressed by proposing an incremental/active learning approach based on three different approaches: 1) the maximum differential of entropies; 2) the minimum distance to decision boundary; and 3) the minimum normalized distance. Parameters are estimated by using the evidence Bayesian approach, the kernel trick, and the marginal distribution of the observations instead of the posterior distribution of the adaptive parameters. This approach allows us to deal with infinite-dimensional feature spaces. The proposed approach is tested on the challenging problem of urban monitoring from multispectral and synthetic aperture radar data and in multiclass land cover classification of hyperspectral images, in both purely supervised and active learning settings. Similar results are obtained when compared to SVMs in the supervised mode, with the advantage of providing posterior estimates for classification and automatic parameter learning. Comparison with random sampling as well as standard active learning methods such as margin sampling and entropy-query-by-bagging reveals a systematic overall accuracy gain and faster convergence with the number of queries.

Robust PCA as Bilinear Decomposition With Outlier-Sparsity Regularization

Principal component analysis (PCA) is widely used for dimensionality reduction, with well-documented merits in various applications involving high-dimensional data, including computer vision, preference measurement, and bioinformatics. In this context, the fresh look advocated here permeates benefits from variable selection and compressive sampling, to robustify PCA against outliers. A least-trimmed squares estimator of a low-rank bilinear factor analysis model is shown closely related to that obtained from an $\ell_0$-(pseudo)norm-regularized criterion encouraging sparsity in a matrix explicitly modeling the outliers. This connection suggests robust PCA schemes based on convex relaxation, which lead naturally to a family of robust estimators encompassing Huber's optimal M-class as a special case. Outliers are identified by tuning a regularization parameter, which amounts to controlling sparsity of the outlier matrix along the whole robustification path of (group) least-absolute shrinkage and selection operator (Lasso) solutions. Beyond its neat ties to robust statistics, the developed outlier-aware PCA framework is versatile to accommodate novel and scalable algorithms to: i) track the low-rank signal subspace robustly, as new data are acquired in real time; and ii) determine principal components robustly in (possibly) infinite-dimensional feature spaces. Synthetic and real data tests corroborate the effectiveness of the proposed robust PCA schemes, when used to identify aberrant responses in personality assessment surveys, as well as unveil communities in social networks, and intruders from video surveillance data.

Infrared point target detection with improved template matching

Detecting point targets in infrared images is a difficult task. Template matching is simple and easy to implement for completing this task. However, it has some shortcomings. We propose an improved template matching method for detecting targets. Different from the classic template matching, the projection coefficients obtained from principal component analysis are used as templates and the nonlinear correlation is proposed to measure the similarity, the matching degree. The correlation in original space can not capture the higher-order statistical property of images. So its detection performance is not satisfying. We introduce the nonlinear correlation, which computes the correlation coefficients in a higher-dimensional feature space or even in an infinite-dimensional feature space, to capture the higher-order statistics. The detection performance is improved greatly. Results of experiments show that the improved method is competent to detect infrared point targets.

Manifold Mapping Machine

Nonlinear classification has been a non-trivial task in machine learning for the past decades. In recent years, kernel machines have successfully generalized the inner-product based linear classifiers to nonlinear ones by transforming data into some high or infinite dimensional feature space. However, due to their implicit space transformation and unobservable latent feature space, it is hard to have an intuitive understanding of their working mechanism. In this paper, we propose a comprehensible framework for nonlinear classifier design, called Manifold Mapping Machine (M 3). M 3 can generalize any linear classifier to nonlinear by transforming data into some low-dimensional feature space explicitly. To demonstrate the effectiveness of M 3 framework, we further present an algorithmic implementation of M 3 named Supervised Spectral Space Classifier (S 3C). Compared with the kernel classifiers, S 3C can achieve similar or even better data separation by mapping data into the low-dimensional spectral space, allowing both of its mapped data and new feature space to be examined directly. Moreover, with the discriminative information integrated into the spectral space transformation, the classification performance of S 3C is more robust than that of the kernel classifiers. Experimental results show that S 3C is superior to other state-of-the-art nonlinear classifiers on both synthetic and real-world data sets.

On linear separability of data sets in feature space

In this paper we focus our topic on linear separability of two data sets in feature space, including finite and infinite data sets. We first develop a method to construct a mapping that maps original data set into a high dimensional feature space, on which inner product is defined by a dot product kernel. Our method can also be applied to the Gaussian kernels. Via this mapping, structure of features in the feature space is easily observed, and the linear separability of data sets in feature space could be studied. We obtain that any two finite sets of data with empty overlap in original input space will become linearly separable in an infinite dimensional feature space. For two infinite data sets, we present several sufficient and necessary conditions for their linear separability in feature space. We also obtain a meaningful formula to judge linear separability of two infinite data sets in feature space by information in original input space.

Kernel PCA for novelty detection

Kernel principal component analysis (kernel PCA) is a non-linear extension of PCA. This study introduces and investigates the use of kernel PCA for novelty detection. Training data are mapped into an infinite-dimensional feature space. In this space, kernel PCA extracts the principal components of the data distribution. The squared distance to the corresponding principal subspace is the measure for novelty. This new method demonstrated a competitive performance on two-dimensional synthetic distributions and on two real-world data sets: handwritten digits and breast-cancer cytology.

Learning Bounds for Kernel Regression Using Effective Data Dimensionality

Kernel methods can embed finite-dimensional data into infinite-dimensional feature spaces. In spite of the large underlying feature dimensionality, kernel methods can achieve good generalization ability. This observation is often wrongly interpreted, and it has been used to argue that kernel learning can magically avoid the "curse-of-dimensionality" phenomenon encountered in statistical estimation problems. This letter shows that although using kernel representation, one can embed data into an infinite-dimensional feature space; the effective dimensionality of this embedding, which determines the learning complexity of the underlying kernel machine, is usually small. In particular, we introduce an algebraic definition of a scale-sensitive effective dimension associated with a kernel representation. Based on this quantity, we derive upper bounds on the generalization performance of some kernel regression methods. Moreover, we show that the resulting convergent rates are optimal under various circumstances.

An Expectation-Maximization Approach to Nonlinear Component Analysis

The proposal of considering nonlinear principal component analysis as a kernel eigenvalue problem has provided an extremely powerful method of extracting nonlinear features for a number of classification and regression applications. Whereas the utilization of Mercer kernels makes the problem of computing principal components in, possibly, infinite-dimensional feature spaces tractable, there are still the attendant numerical problems of diagonalizing large matrices. In this contribution, we propose an expectation-maximization approach for performing kernel principal component analysis and show this to be a computationally efficient method, especially when the number of data points is large.