Dot Product Kernel Research Articles

A Local Gaussian Process Regression Approach to Frequency Response Function Estimation

Frequency response function (FRF) estimation is a classical subject in system Identification. In the past two decades, there have been remarkable advances in developing local methods for this subject, e.g., the local polynomial method, local rational method, and iterative local rational method. The recent concentrations for local methods are two issues: the model order selection and the Identification of lightly damped systems. To address these two issues, we propose a new local method called local Gaussian process regression (LGPR). We show that the frequency response function locally is either analytic or resonant, and this prior knowledge can be embedded into a kernel-based regularized estimate through a dot-product kernel plus a resonance kernel induced by a second-order resonant system. The LGPR provides a new route to tackle the aforementioned issues. In the numerical simulations, the LGPR shows the best FRF estimation accuracy compared with the existing local methods, and moreover, the LGPR is more robust with respect to sample size and noise level.

Precise learning curves and higher-order scaling limits for dot-product kernel regression *

As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improvements under different model and data scaling regimes. Currently, theoretical understanding of the learning curves (LCs) that characterize how the prediction error depends on the number of samples is restricted to either large-sample asymptotics () or, for certain simple data distributions, to the high-dimensional asymptotics in which the number of samples scales linearly with the dimension (). There is a wide gulf between these two regimes, including all higher-order scaling relations , which are the subject of the present paper. We focus on the problem of kernel ridge regression for dot-product kernels and present precise formulas for the mean of the test error, bias and variance, for data drawn uniformly from the sphere with isotropic random labels in the rth-order asymptotic scaling regime with held constant. We observe a peak in the LC whenever for any integer r, leading to multiple sample-wise descent and non-trivial behavior at multiple scales. We include a colab (available at: https://tinyurl.com/2nzym7ym) notebook that reproduces the essential results of the paper.

A BLIS-like matrix multiplication for machine learning in the RISC-V ISA-based GAP8 processor

We address the efficient realization of matrix multiplication (gemm), with application in the convolution operator for machine learning, for the RISC-V core present in the GreenWaves GAP8 processor. Our approach leverages BLIS (Basic Linear Algebra Instantiation Software) to develop an implementation that (1) re-organizes the gemm algorithm adapting its micro-kernel to exploit the hardware-supported dot product kernel in the GAP8; (2) explicitly orchestrates the data transfers across the hierarchy of scratchpad memories via DMA (direct memory access); and (3) operates with integer arithmetic.

Dynamic Mode Decomposition for Continuous Time Systems with the Liouville Operator

Dynamic mode decomposition (DMD) has become synonymous with the Koopman operator, where continuous time dynamics are discretized and examined using Koopman (i.e. composition) operators. Using the newly introduced “occupation kernels,” the present manuscript develops an approach to DMD that treats continuous time dynamics directly through the Liouville operator. This manuscript outlines the technical and theoretical differences between Koopman-based DMD for discrete time systems and Liouville-based DMD for continuous time systems, which includes an examination of Koopman and Liouville operators over several reproducing kernel Hilbert spaces. While Liouville operators are modally unbounded, this manuscript introduces the concept of a scaled Liouville operator, which, for many dynamical systems, is a compact operator over the native space of the exponential dot product kernel. Compactness of scaled Liouville operators allows for norm convergence of Liouville-based DMD, which is a decided advantage over Koopman-based DMD.

Sparse random feature maps for the item-multiset kernel

Random feature maps are a promising tool for large-scale kernel methods. Since most random feature maps generate dense random features causing memory explosion, it is hard to apply them to very-large-scale sparse datasets. The factorization machines and related models, which use feature combinations efficiently, scale well for large-scale sparse datasets and have been used in many applications. However, their optimization problems are typically non-convex. Therefore, although they are optimized by using gradient-based iterative methods, such methods cannot find global optimum solutions in general and require a large number of iterations for convergence. In this paper, we define the item-multiset kernel, which is a generalization of the itemset kernel and dot product kernels. Unfortunately, random feature maps for the itemset kernel and dot product kernels cannot approximate the item-multiset kernel. We thus develop a method that converts an item-multiset kernel into an itemset kernel, enabling the item-multiset kernel to be approximated by using a random feature map for the itemset kernel. We propose two random feature maps for the itemset kernel, which run faster and are more memory efficient than the existing feature map for the itemset kernel. They also generate sparse random features when the original (input) feature vector is sparse and thus linear models using proposed methods . Experiments using real-world datasets demonstrated the effectiveness of the proposed methodology: linear models using the proposed random feature maps ran from 10 to 100 times faster than ones based on existing methods.

Learning deep kernels in the space of monotone conjunctive polynomials

• The monotone Conjunctive Polynomial feature space is here presented. • Homogeneous polynomial kernels can be decomposed as a combination of monotone conjunctive kernels. • The optimal monotone Conjunctive Polynomial kernel can be learned via Multiple Kernel Learning . • The proposed approach proved to be robust to overfitting. • The monotone Conjunctive Polynomial kernel has a deep structure. Dot-product kernels is a large family of kernel functions based on dot-product between examples. A recent result states that any dot-product kernel can be decomposed as a non-negative linear combination of homogeneous polynomial kernels of different degrees, and it is possible to learn the coefficients of the combination by exploiting the Multiple Kernel Learning (MKL) paradigm. In this paper it is proved that, under mild conditions, any homogeneous polynomial kernel defined on binary valued data can be decomposed in a parametrized finite linear non-negative combination of monotone conjunctive kernels. MKL has been employed to learn the parameters of the combination. Furthermore, we show that our solution produces a deep kernel whose feature space consists of hierarchically organized features of increasing complexity. We also emphasize the connection between our solution and existing deep kernel learning frameworks. A wide empirical assessment is presented to evaluate the proposed framework, and to compare it against the baselines on several categorical and binary datasets.

A Double-Variational Bayesian Framework in Random Fourier Features for Indefinite Kernels.

Random Fourier features (RFFs) have been successfully employed to kernel approximation in large-scale situations. The rationale behind RFF relies on Bochner's theorem, but the condition is too strict and excludes many widely used kernels, e.g., dot-product kernels (violates the shift-invariant condition) and indefinite kernels [violates the positive definite (PD) condition]. In this article, we present a unified RFF framework for indefinite kernel approximation in the reproducing kernel Kreĭn spaces (RKKSs). Besides, our model is also suited to approximate a dot-product kernel on the unit sphere, as it can be transformed into a shift-invariant but indefinite kernel. By the Kolmogorov decomposition scheme, an indefinite kernel in RKKS can be decomposed into the difference of two unknown PD kernels. The spectral distribution of each underlying PD kernel can be formulated as a nonparametric Bayesian Gaussian mixtures model. Based on this, we propose a double-infinite Gaussian mixture model in RFF by placing the Dirichlet process prior. It takes full advantage of high flexibility on the number of components and has the capability of approximating indefinite kernels on a wide scale. In model inference, we develop a non-conjugate variational algorithm with a sub-sampling scheme for the posterior inference. It allows for the non-conjugate case in our model and is quite efficient due to the sub-sampling strategy. Experimental results on several large classification data sets demonstrate the effectiveness of our nonparametric Bayesian model for indefinite kernel approximation when compared to other representative random feature-based methods.

Evaluating attainable memory bandwidth of parallel programming models via BabelStream

Many scientific codes consist of memory bandwidth bound kernels. One major advantage of many-core devices such as general purpose graphics processing units (GPGPUs) and the Intel Xeon Phi is their focus on providing increased memory bandwidth over traditional CPU architectures. Peak memory bandwidth is usually unachievable in practice and so benchmarks are required to measure a practical upper bound on expected performance. We augment the standard STREAM kernels with a dot product kernel to investigate the performance of simple reduction operations on large arrays. The choice of programming model should ideally not limit the achievable performance on a device. BabelStream (formally GPU-STREAM) has been updated to incorporate a wide variety of the latest parallel programming models, all implementing the same parallel scheme. As such this tool can be used as a kind of Rosetta Stone which provides both a cross-platform and cross-programming model array of results of achievable memory bandwidth.

Optimized BiCGStab Based GPU Accelerated Computation of Incompressible Viscous Flows by the $$\psi $$ ψ –v Formulation

In this work, we present an optimization strategy for implementing the BiCGStab iterative solver on graphic processing units (GPU) for computing incompressible viscous flows governed by the unsteady Navier–Stokes (N–S) equations on a CUDA platform. A recently developed $$\psi $$ –v formulation is used to discretize the biharmonic form of the N–S equation. Special emphasis was given on optimizing the matrix-vector multiplication and the dot product kernels in GPU as the bulk of the computational efforts was seen to be occupied by these two kernels. The parallel code was implemented on the benchmark problem of lid-driven square cavity flow and remarkable speed-up up to 20 times was achieved on finer grids. The GPU implementation enabled us to compute the flow on extremely fine grids and very small scales were resolved with remarkable accuracy.

Evaluating attainable memory bandwidth of parallel programming models via BabelStream

Many scientific codes consist of memory bandwidth bound kernels. One major advantage of many-core devices such as general purpose graphics processing units (GPGPUs) and the Intel Xeon Phi is their focus on providing increased memory bandwidth over traditional CPU architectures. Peak memory bandwidth is usually unachievable in practice and so benchmarks are required to measure a practical upper bound on expected performance. We augment the standard STREAM kernels with a dot product kernel to investigate the performance of simple reduction operations on large arrays. The choice of programming model should ideally not limit the achievable performance on a device. BabelStream (formally GPU-STREAM) has been updated to incorporate a wide variety of the latest parallel programming models, all implementing the same parallel scheme. As such this tool can be used as a kind of Rosetta Stone which provides both a cross-platform and cross-programming model array of results of achievable memory bandwidth.

Learning deep kernels in the space of dot product polynomials

Recent literature has shown the merits of having deep representations in the context of neural networks. An emerging challenge in kernel learning is the definition of similar deep representations. In this paper, we propose a general methodology to define a hierarchy of base kernels with increasing expressiveness and combine them via multiple kernel learning (MKL) with the aim to generate overall deeper kernels. As a leading example, this methodology is applied to learning the kernel in the space of Dot-Product Polynomials (DPPs), that is a positive combination of homogeneous polynomial kernels (HPKs). We show theoretical properties about the expressiveness of HPKs that make their combination empirically very effective. This can also be seen as learning the coefficients of the Maclaurin expansion of any definite positive dot product kernel thus making our proposed method generally applicable. We empirically show the merits of our approach comparing the effectiveness of the kernel generated by our method against baseline kernels (including homogeneous and non homogeneous polynomials, RBF, etc...) and against another hierarchical approach on several benchmark datasets.

Learning With Jensen-Tsallis Kernels.

Jensen-type [Jensen-Shannon (JS) and Jensen-Tsallis] kernels were first proposed by Martins et al. (2009). These kernels are based on JS divergences that originated in the information theory. In this paper, we extend the Jensen-type kernels on probability measures to define positive-definite kernels on Euclidean space. We show that the special cases of these kernels include dot-product kernels. Since Jensen-type divergences are multidistribution divergences, we propose their multipoint variants, and study spectral clustering and kernel methods based on these. We also provide experimental studies on benchmark image database and gene expression database that show the benefits of the proposed kernels compared with the existing kernels. The experiments on clustering also demonstrate the use of constructing multipoint similarities.

Sharp estimates for eigenvalues of integral operators generated by dot product kernels on the sphere

We obtain explicit formulas for the eigenvalues of integral operators generated by continuous dot product kernels defined on the sphere via the usual gamma function. Using them, we present both, a procedure to describe sharp bounds for the eigenvalues and their asymptotic behavior near 0. We illustrate our results with examples, among them the integral operator generated by a Gaussian kernel. Finally, we sketch complex versions of our results to cover the cases when the sphere sits in a Hermitian space.

Characterization and sequence prediction of structural variations in α-helix

BackgroundThe structure conservation in various α-helix subclasses reveals the sequence and context dependent factors causing distortions in the α-helix. The sequence-structure relationship in these subclasses can be used to predict structural variations in α-helix purely based on its sequence. We train support vector machine(SVM) with dot product kernel function to discriminate between regular α-helix and non-regular α-helices purely based on the sequences, which are represented with various overall and position specific propensities of amino acids.ResultsWe characterize the structural distortions in five α-helix subclasses. The sequence structure correlation in the subclasses reveals that the increased propensity of proline, histidine, serine, aspartic acid and aromatic amino acids are responsible for the distortions in regular α-helix. The N-terminus of regular α-helix prefers neutral and acidic polar amino acids, while the C-terminus prefers basic polar amino acid. Proline is preferred in the first turn of regular α-helix , while it is preferred to produce kinked and curved subclasses. The SVM discriminates between regular α-helix and the rest with precision of 80.97% and recall of 88.05%.ConclusionsThe correlation between structural variation in helices and their sequences is manifested by the performance of SVM based on sequence features. The results presented here are useful for computational design of helices. The results are also useful for prediction of structural perturbations in helix sequence purely based on its sequence.

Graph kernels for chemical compounds using topological and three-dimensional local atom pair environments

Approaches that can predict the biological activity or properties of a chemical compound are an important application of machine learning. In this paper, we introduce a new kernel function for measuring the similarity between chemical compounds and for learning their related properties and activities. The method is based on local atom pair environments which can be rapidly computed by using the topological all-shortest paths matrix and the geometrical distance matrix of a molecular graph as lookup tables. The local atom pair environments are stored in prefix search trees, so called tries, for an efficient comparison. The kernel can be either computed as an optimal assignment kernel or as a corresponding convolution kernel over all local atom similarities. We implemented the Tanimoto kernel, min kernel, minmax kernel and the dot product kernel as local kernels, which are computed recursively by traversing the tries. We tested the approach on eight structure-activity and structure-property molecule benchmark data sets from the literature. The models were trained with ε - support vector regression and support vector classification. The local atom pair kernels showed to be at least competitive to state-of-the-art kernels in seven out of eight cases in a direct comparison. A comparison against literature results using similar experimental setups as in the original works confirmed these findings. The method is easy to implement and has robust default parameters.

On conditionally positive definite dot product kernels

Let m and n be fixed, positive integers and Open image in new window a space composed of real polynomials in m variables. The authors study functions f: ℝ → ℝ which map Gram matrices, based upon n points of ℝm, into matrices, which are nonnegative definite with respect to Open image in new window . Among other things, the authors discuss continuity, differentiability, convexity, and convexity in the sense of Jensen, of such functions.

A new extension of kernel feature and its application for visual recognition

In this paper, we first conceive a new perception of the kernel feature. The kernel subspace methods can be regarded as two independent steps: an explicit kernel feature extraction step and a linear subspace analysis step on the extracted kernel features. The kernel feature vector of an image is composed of dot products between the image and all the training images using nonlinear dot product kernel. Then, based on this perception, we further extend the kernel feature vector of an image to a kernel feature matrix for visual recognition. This extension takes different representation cues of images into account, respectively, while only global average information is used in the traditional kernel methods. From the view of dot product as similarity, this extension means using multiple similarities to measure two images, which is more accordant to human vision. In order to efficiently deal with the problem of numerical computation, a matrix-based kernel discriminant analysis algorithm is employed to learn discriminating kernel features for visual recognition. Experiments on the FERET face database, the COIL-100 object database, and the Wang's nature image database show the advantage of the proposed method.

Power Series Kernels

We introduce a class of analytic positive definite multivariate kernels which includes infinite dot product kernels as sometimes used in machine learning, certain new nonlinearly factorizable kernels, and a kernel which is closely related to the Gaussian. Each such kernel reproduces in a certain “native” Hilbert space of multivariate analytic functions. If functions from this space are interpolated in scattered locations by translates of the kernel, we prove spectral convergence rates of the interpolants and all derivatives. By truncation of the power series of the kernel-based interpolants, we constructively generalize the classical Bernstein theorem concerning polynomial approximation of analytic functions to the multivariate case. An application to machine learning algorithms is presented.

On linear separability of data sets in feature space

In this paper we focus our topic on linear separability of two data sets in feature space, including finite and infinite data sets. We first develop a method to construct a mapping that maps original data set into a high dimensional feature space, on which inner product is defined by a dot product kernel. Our method can also be applied to the Gaussian kernels. Via this mapping, structure of features in the feature space is easily observed, and the linear separability of data sets in feature space could be studied. We obtain that any two finite sets of data with empty overlap in original input space will become linearly separable in an infinite dimensional feature space. For two infinite data sets, we present several sufficient and necessary conditions for their linear separability in feature space. We also obtain a meaningful formula to judge linear separability of two infinite data sets in feature space by information in original input space.

Learning the Kernel Function via Regularization

We study the problem of finding an optimal kernel from a prescribed convex set of kernels K for learning a real-valued function by regularization. We establish for a wide variety of regularization functionals that this leads to a convex optimization problem and, for square loss regularization, we characterize the solution of this problem. We show that, although K may be an uncountable set, the optimal kernel is always obtained as a convex combination of at most m+2 basic kernels, where m is the number of data examples. In particular, our results apply to learning the optimal radial kernel or the optimal dot product kernel.