Positive Semi-definite Kernel Research Articles

Positive semidefinite kernels that are axially symmetric on the sphere and stationary in time: spectral and semi-spectral theory, and constructive approaches

Positive semidefinite kernels that are axially symmetric on the sphere and stationary in time: spectral and semi-spectral theory, and constructive approaches

Sparse Gaussian processes for solving nonlinear PDEs

This article proposes an efficient numerical method for solving nonlinear partial differential equations (PDEs) based on sparse Gaussian processes (SGPs). Gaussian processes (GPs) have been extensively studied for solving PDEs by formulating the problem of finding a reproducing kernel Hilbert space (RKHS) to approximate a PDE solution. The approximated solution lies in the span of base functions generated by evaluating derivatives of different orders of kernels at sample points. However, the RKHS specified by GPs can result in an expensive computational burden due to the cubic computation order of the matrix inverse. We conjecture that a solution exists on a “condensed” subspace that can achieve similar approximation performance, and we propose a SGP-based method to reformulate the optimization problem in the “condensed” subspace. This significantly reduces the computation burden while retaining desirable accuracy. The paper rigorously formulates this problem and provides error analysis and numerical experiments to demonstrate the effectiveness of this method. The numerical experiments show that the SGP method uses fewer than half the uniform samples as inducing points and achieves comparable accuracy to the GP method using the same number of uniform samples, resulting in a significant reduction in computational cost.Our contributions include formulating the nonlinear PDE problem as an optimization problem on a “condensed” subspace of RKHS using SGP, as well as providing an existence proof and rigorous error analysis. Furthermore, our method can be viewed as an extension of the GP method to account for general positive semi-definite kernels.

Reproducing Kernel Hilbert Spaces of Smooth Fractal Interpolation Functions

The theory of reproducing kernel Hilbert spaces (RKHSs) has been developed into a powerful tool in mathematics and has lots of applications in many fields, especially in kernel machine learning. Fractal theory provides new technologies for making complicated curves and fitting experimental data. Recently, combinations of fractal interpolation functions (FIFs) and methods of curve estimations have attracted the attention of researchers. We are interested in the study of connections between FIFs and RKHSs. The aim is to develop the concept of smooth fractal-type reproducing kernels and RKHSs of smooth FIFs. In this paper, a linear space of smooth FIFs is considered. A condition for a given finite set of smooth FIFs to be linearly independent is established. For such a given set, we build a fractal-type positive semi-definite kernel and show that the span of these linearly independent smooth FIFs is the corresponding RKHS. The nth derivatives of these FIFs are investigated, and properties of related positive semi-definite kernels and the corresponding RKHS are studied. We also introduce subspaces of these RKHS which are important in curve-fitting applications.

Deep Time Series Forecasting With Shape and Temporal Criteria.

This paper addresses the problem of multi-step time series forecasting for non-stationary signals that can present sudden changes. Current state-of-the-art deep learning forecasting methods, often trained with variants of the MSE, lack the ability to provide sharp predictions in deterministic and probabilistic contexts. To handle these challenges, we propose to incorporate shape and temporal criteria in the training objective of deep models. We define shape and temporal similarities and dissimilarities, based on a smooth relaxation of Dynamic Time Warping (DTW) and Temporal Distortion Index (TDI), that enable to build differentiable loss functions and positive semi-definite (PSD) kernels. With these tools, we introduce DILATE (DIstortion Loss including shApe and TimE), a new objective for deterministic forecasting, that explicitly incorporates two terms supporting precise shape and temporal change detection. For probabilistic forecasting, we introduce STRIPE++ (Shape and Time diverRsIty in Probabilistic for Ecasting), a framework for providing a set of sharp and diverse forecasts, where the structured shape and time diversity is enforced with a determinantal point process (DPP) diversity loss. Extensive experiments and ablations studies on synthetic and real-world datasets confirm the benefits of leveraging shape and time features in time series forecasting.

A catalogue of nonseparable positive semidefinite kernels on the product of two spheres

A catalogue of nonseparable positive semidefinite kernels on the product of two spheres

Fast Deterministic Approximation of Symmetric Indefinite Kernel Matrices with High Dimensional Datasets

Kernel methods are used frequently in various applications of machine learning. For large-scale high dimensional applications, the success of kernel methods hinges on the ability to operate certain large dense kernel matrix $K$. An enormous amount of literature has been devoted to the study of symmetric positive semidefinite (SPSD) kernels, where Nyström methods compute a low-rank approximation to the kernel matrix via choosing landmark points. In this paper, we study the Nyström method for approximating both symmetric indefinite kernel matrices as well SPSD ones. We first develop a theoretical framework for general symmetric kernel matrices, which provides a theoretical guidance for the selection of landmark points. We then leverage discrepancy theory to propose the anchor net method for computing accurate Nyström approximations with optimal complexity. The anchor net method operates entirely on the dataset without requiring the access to $K$ or its matrix-vector product. Results on various types of kernels (both indefinite and SPSD ones) and machine learning datasets demonstrate that the new method achieves better accuracy and stability with lower computational cost compared to the state-of-the-art Nyström methods.

The Gauss hypergeometric covariance kernel for modeling second-order stationary random fields in Euclidean spaces: its compact support, properties and spectral representation

This paper presents a parametric family of compactly-supported positive semidefinite kernels aimed to model the covariance structure of second-order stationary isotropic random fields defined in the d-dimensional Euclidean space. Both the covariance and its spectral density have an analytic expression involving the hypergeometric functions $${}_2F_1$$ and $${}_1F_2$$ , respectively, and four real-valued parameters related to the correlation range, smoothness and shape of the covariance. The presented hypergeometric kernel family contains, as special cases, the spherical, cubic, penta, Askey, generalized Wendland and truncated power covariances and, as asymptotic cases, the Matérn, Laguerre, Tricomi, incomplete gamma and Gaussian covariances, among others. The parameter space of the univariate hypergeometric kernel is identified and its functional properties—continuity, smoothness, transitive upscaling (montée) and downscaling (descente)—are examined. Several sets of sufficient conditions are also derived to obtain valid stationary bivariate and multivariate covariance kernels, characterized by four matrix-valued parameters. Such kernels turn out to be versatile, insofar as the direct and cross-covariances do not necessarily have the same shapes, correlation ranges or behaviors at short scale, thus associated with vector random fields whose components are cross-correlated but have different spatial structures.

Indefinite twin support vector machine with DC functions programming

Twin support vector machine (TWSVM) is an efficient algorithm for binary classification. However, the lack of the structural risk minimization principle restrains the generalization of TWSVM and the guarantee of convex optimization constraints TWSVM to only use positive semi-definite kernels (PSD). In this paper, we propose a novel TWSVM for indefinite kernel called indefinite twin support vector machine with difference of convex functions programming (ITWSVM-DC). The indefinite TWSVM (ITWSVM) leverages a maximum margin regularization term to improve the generalization of TWSVM and a smooth quadratic hinge loss function to make the model continuously differentiable. The representer theorem is applied to the ITWSVM and the convexity of the ITWSVM is analyzed. In order to address the non-convex optimization problem when the kernel is indefinite, a difference of convex functions (DC) is used to decompose the non-convex objective function into the subtraction of two convex functions and a line search method is applied in the DC algorithm to accelerate the convergence rate. A theoretical analysis illustrates that ITWSVM-DC can converge to a local optimum and extensive experiments on indefinite and positive semi-definite kernels show the superiority of ITWSVM-DC.

Covariance and correlation measures on a graph in a generalized bag-of-paths formalism

Abstract This work derives closed-form expressions computing the expectation of co-presence and of number of co-occurrences of nodes on paths sampled from a network according to general path weights (a bag of paths). The underlying idea is that two nodes are considered as similar when they often appear together on (preferably short) paths of the network. The different expressions are obtained for both regular and hitting paths and serve as a basis for computing new covariance and correlation measures between nodes, which are valid positive semi-definite kernels on a graph. Experiments on semi-supervised classification problems show that the introduced similarity measures provide competitive results compared to other state-of-the-art distance and similarity measures between nodes.

A Simulation-induced Regularization Method for System Identification

Abstract In the past decade, regularization methods for system identification have attracted a great deal of attention in the system identification community. For regularization method with regularization in quadratic form, there are different ways to design the regularization, e.g., through designing a positive semidefinite kernel or a filter. In this paper, we propose a new regularization method, where the regularization is in essence induced by simulating a carefully designed linear system driven by a white Gaussian noise and this regularization method is thus called the simulation-induced regularization method (SIRM). In contrast with the kernel or filter based regularization methods, SIRM has the advantages that it is free of the explicit expression of the regularization and moreover, has a linear computational complexity.

METASET: Exploring Shape and Property Spaces for Data-Driven Metamaterials Design

Abstract Data-driven design of mechanical metamaterials is an increasingly popular method to combat costly physical simulations and immense, often intractable, geometrical design spaces. Using a precomputed dataset of unit cells, a multiscale structure can be quickly filled via combinatorial search algorithms, and machine learning models can be trained to accelerate the process. However, the dependence on data induces a unique challenge: an imbalanced dataset containing more of certain shapes or physical properties can be detrimental to the efficacy of data-driven approaches. In answer, we posit that a smaller yet diverse set of unit cells leads to scalable search and unbiased learning. To select such subsets, we propose METASET, a methodology that (1) uses similarity metrics and positive semi-definite kernels to jointly measure the closeness of unit cells in both shape and property spaces and (2) incorporates Determinantal Point Processes for efficient subset selection. Moreover, METASET allows the trade-off between shape and property diversity so that subsets can be tuned for various applications. Through the design of 2D metamaterials with target displacement profiles, we demonstrate that smaller, diverse subsets can indeed improve the search process as well as structural performance. By eliminating inherent overlaps in a dataset of 3D unit cells created with symmetry rules, we also illustrate that our flexible method can distill unique subsets regardless of the metric employed. Our diverse subsets are provided publicly for use by any designer.

The Application of the Positive Semi-Definite Kernel Space for SVM in Quality Prediction

Background:A transformation toward 4th Generation Industrial Revolution (Industry 4.0) is being led by Germany based on Cyber-Physical System-enabled manufacturing and service innovation. Smart manufacturing is an important feature of Industry 4.0 which uses the networked manufacturing systems for smart production. Current manufacturing systems (5M1E systems) require deeper mining of the data which is generated from manufacturing process.Objective:To map low-dimensional embedding into the input space would meet the requirement of “kernel trick” to solve a problem in feature space. On the other hand, the distance can be calculated more precisely.Methods:In this research, we proposed a positive semi-definite kernel space by using a constant additive method based on a kernel view of ISOMAP. There were 6 steps in the algorithm.Results:The classification precision of KMLSVM was better than SVM in the enterprise data set, in which SVM selected the RBF kernel and optimized its parameters.Conclusion:We adopted the additive constant method in kernel space construction and the positive semi-definite kernel was built. The typical mixed data set of an enterprise was used in simulation. We compared the SVM and KMLSVM in this data set and optimized the SVM kernel function parameters. The simulation results demonstrated the KMLSVM was a better algorithm in mix type data set than SVM.

A simple proof of necessity in the McCullough-Quiggin theorem

A short and simple proof of necessity in the McCullough-Quiggin characterization of positive semi-definite kernels with the complete Pick property is presented.

Kernel Absolute Summability Is Sufficient but Not Necessary for RKHS Stability

Regularized approaches have been successfully applied to linear system identification in recent years; many of them model unknown impulse responses exploiting the so-called Reproducing Kernel Hilbert Spaces (RKHSs) that enjoy the notable property of being in one-to-one correspondence with the class of positive semidefinite kernels. The necessary and sufficient condition for a RKHS to be stable, i.e., to contain only BIBO stable linear dynamic systems, has been known in the literature since at least 2006. However, an open question still persists and concerns the equivalence of such a condition with the absolute summability of the kernel. This paper provides a definite answer to this matter by proving that such correspondence does not hold. A counterexample is introduced that illustrates the existence of stable RKHSs that are induced by nonabsolutely summable kernels.

Sparsification of core set models in non-metric supervised learning

Supervised learning employing positive semi definite kernels has gained wide attraction and lead to a variety of successful machine learning approaches. The restriction to positive semi definite kernels and a hilbert space is common to simplify the mathematical derivations of the respective learning methods, but is also limiting because more recent research indicates that non-metric, and therefore non positive semi definite, data representations are often more effective. This challenge is addressed by multiple approaches and recently dedicated algorithms for so called indefinite learning have been proposed. Along this line, the Krĕin space Support Vector Machine (KSVM) and variants are very efficient classifiers for indefinite learning problems, but with a non-sparse decision function. This very dense decision function prevents practical applications due to a costly out of sample extension. We focus on this problem and provide two post processing techniques to sparsify models as obtained by a Krĕin space SVM approach. In particular we consider the indefinite Core Vector Machine and indefinite Core Vector Regression Machine which are both efficient for psd kernels, but suffer from the same dense decision function, if the Krĕin space approach is used. We evaluate the influence of different levels of sparsity and employ a Nyström approach to address large scale problems. Experiments show that our algorithm is similar efficient as the non-sparse Krĕin space Support Vector Machine but with substantially lower costs, such that also problems of larger scale can be processed.

Extreme learning machines with expectation kernels

Extreme learning machines (ELMs), especially kernel ELMs (KELMs), have achieved great success in providing efficient and effective solutions to classification applications. This paper proposes a simple but effective expectation kernel ELM (EKELM) to improve the classification abilities of ELMs. EKELM is based on a new family of positive semidefinite (PSD) kernel functions, namely expectation kernels (EKs), to learn similarities between data samples by combining the advances of random feature mapping and conventional kernel functions. EK provides a new insight to model ELMs from the perspective of kernel approximation using random sampling techniques. Particularly, we show that the distribution of random sampling weights, i.e. the input weight of ELM, has deep influences on the classification performance, and we use Gaussian distributions to generate random weights in EKELM. Besides, the number of sampling samples, i.e. the number of hidden neurons in ELM, can be reduced by using a proper nonlinear kernel function. We test the proposed EKELM on 20 benchmark classification datasets, the results prove that using a Gaussian distribution to generate random sampling weights has better performance than uniform distribution. Also, the results show that nonlinear RBF EKELM achieves comparable classification performance compared with the conventional RBF kernel, and requires much less random samples compared with linear EKELM.

Kernel Treelets

A new method for hierarchical clustering of data points is presented. It combines treelets, a particular multiresolution decomposition of data, with a mapping on a reproducing kernel Hilbert space. The proposed approach, called kernel treelets (KT), uses this mapping to go from a hierarchical clustering over attributes (the natural output of treelets) to a hierarchical clustering over data. KT effectively substitutes the correlation coefficient matrix used in treelets with a symmetric and positive semi-definite matrix efficiently constructed from a symmetric and positive semi-definite kernel function. Unlike most clustering methods, which require data sets to be numeric, KT can be applied to more general data and yields a multiresolution sequence of orthonormal bases on the data directly in feature space. The effectiveness and potential of KT in clustering analysis are illustrated with some examples.

Adaptive Superpixel Generation for SAR Images With Linear Feature Clustering and Edge Constraint

Due to the speckle noise and complex geometric distortions within SAR images, it is still a challenge to develop a stable method that can produce superpixels with both high boundary adherence and visual compactness with low computational costs at the same time. In this paper, we propose an adaptive superpixel generation approach with linear feature clustering and edge constraint for synthetic aperture radar (SAR) images, which consists of three stages. First, the local gradient ratio pattern of each pixel in SAR imagery is extracted as features, which was previously proposed by us for SAR target recognition and has been proven to be insensitive to speckle noise. Second, we propose to use the feature-ratio-based edge detector with Gauss-shaped window instead of the traditional rectangle-shaped window to obtain the edge strength map and final edges for SAR images. Finally, a modified normalized cut (Ncut)-based superpixel generation strategy is adopted using a distance metric that simultaneously measures both the feature similarity and space proximity. In this strategy, we approximate the similarity measure through a positive semidefinite kernel function rather than directly using the traditional eigen-based algorithm. Therefore, the objective functions of weighted local K- means and Ncuts can achieve the same optimum point by appropriately weighting each point in this feature space, which greatly reduces the computation cost. During the linear feature clustering, the coefficient of variation is used to automatically determine the tradeoff factor between the feature similarity and space proximity, which helps change the superpixel shape and size adaptively according to the image homogeneity. Furthermore, the edge information is also introduced to constrain the clustering for the sake of high boundary adherence. By bridging the local K-means clustering and Ncuts, as well as the benefits of edge constraint, our method not only produces superpixels with good boundary adherence but also captures the global image structure information. Experimental results with simulated and real SAR images demonstrate the effectiveness of our proposed method, which performs better than other state-of-the-art algorithms.

Plenary: Michelle Craske (UCLA) – Neuroscience informed treatments for anxiety and depression: Time: Friday, 07/Sep/2018: 11:00am–12:15am

Extreme learning machines (ELMs), especially kernel ELMs (KELMs), have achieved great success in providing efficient and effective solutions to classification applications. This paper proposes a simple but effective expectation kernel ELM (EKELM) to improve the classification abilities of ELMs. EKELM is based on a new family of positive semidefinite (PSD) kernel functions, namely expectation kernels (EKs), to learn similarities between data samples by combining the advances of random feature mapping and conventional kernel functions. EK provides a new insight to model ELMs from the perspective of kernel approximation using random sampling techniques. Particularly, we show that the distribution of random sampling weights, i.e. the input weight of ELM, has deep influences on the classification performance, and we use Gaussian distributions to generate random weights in EKELM. Besides, the number of sampling samples, i.e. the number of hidden neurons in ELM, can be reduced by using a proper nonlinear kernel function. We test the proposed EKELM on 20 benchmark classification datasets, the results prove that using a Gaussian distribution to generate random sampling weights has better performance than uniform distribution. Also, the results show that nonlinear RBF EKELM achieves comparable classification performance compared with the conventional RBF kernel, and requires much less random samples compared with linear EKELM.

Invariant weakly positive semidefinite kernels with values in topologically ordered $*$-spaces

We consider weakly positive semidefinite kernels valued in ordered $*$-spaces with or without certain topological properties, and investigate their linearisations (Kolmogorov decompositions) as well as their reproducing kernel spaces. The spaces of realisations are of VE (Vector Euclidean) or VH (Vector Hilbert) type, more precisely, vector spaces that possess gramians (vector valued inner products). The main results refer to the case when the kernels are invariant under certain actions of $*$-semigroups and show under which conditions $*$-representations on VE-spaces, or VH-spaces in the topological case, can be obtained. Finally we show that these results unify most of dilation type results for invariant positive semidefinite kernels with operator values as well as recent results on positive semidefinite maps on $*$-semigroups with values operators from a locally bounded topological vector space to its conjugate $Z$-dual space, for $Z$ an ordered $*$-space.