Positive Semi-definite Kernel Research Articles

An empirical approach for probing the definiteness of kernels

Models like support vector machines or Gaussian process regression often require positive semi-definite kernels. These kernels may be based on distance functions. While definiteness is proven for common distances and kernels, a proof for a new kernel may require too much time and effort for users who simply aim at practical usage. Furthermore, designing definite distances or kernels may be equally intricate. Finally, models can be enabled to use indefinite kernels. This may deteriorate the accuracy or computational cost of the model. Hence, an efficient method to determine definiteness is required. We propose an empirical approach. We show that sampling as well as optimization with an evolutionary algorithm may be employed to determine definiteness. We provide a proof of concept with 16 different distance measures for permutations. Our approach allows to disprove definiteness if a respective counterexample is found. It can also provide an estimate of how likely it is to obtain indefinite kernel matrices. This provides a simple, efficient tool to decide whether additional effort should be spent on designing/selecting a more suitable kernel or algorithm.

GoGP: scalable geometric-based Gaussian process for online regression

One of the most challenging problems in Gaussian process regression is to cope with large-scale datasets and to tackle an online learning setting where data instances arrive irregularly and continuously. In this paper, we introduce a novel online Gaussian process model that scales efficiently with large-scale datasets. Our proposed GoGP is constructed based on the geometric and optimization views of the Gaussian process regression, hence termed geometric-based online GP (GoGP). We developed theory to guarantee that with a good convergence rate our proposed algorithm always offers a sparse solution, which can approximate the true optima up to any level of precision specified a priori. Moreover, to further speed up the GoGP accompanied with a positive semi-definite and shift-invariant kernel such as the well-known Gaussian kernel and also address the curse of kernelization problem, wherein the model size linearly rises with data size accumulated over time in the context of online learning, we proposed to approximate the original kernel using the Fourier random feature kernel. The model of GoGP with Fourier random feature (i.e., GoGP-RF) can be stored directly in a finite-dimensional random feature space, hence being able to avoid the curse of kernelization problem and scalable efficiently and effectively with large-scale datasets. We extensively evaluated our proposed methods against the state-of-the-art baselines on several large-scale datasets for online regression task. The experimental results show that our GoGP(s) delivered comparable, or slightly better, predictive performance while achieving a magnitude of computational speedup compared with its rivals under online setting. More importantly, its convergence behavior is guaranteed through our theoretical analysis, which is rapid and stable while achieving lower errors.

The deep kernelized autoencoder

Autoencoders learn data representations (codes) in such a way that the input is reproduced at the output of the network. However, it is not always clear what kind of properties of the input data need to be captured by the codes. Kernel machines have experienced great success by operating via inner-products in a theoretically well-defined reproducing kernel Hilbert space, hence capturing topological properties of input data. In this paper, we enhance the autoencoder's ability to learn effective data representations by aligning inner products between codes with respect to a kernel matrix. By doing so, the proposed kernelized autoencoder allows learning similarity-preserving embeddings of input data, where the notion of similarity is explicitly controlled by the user and encoded in a positive semi-definite kernel matrix. Experiments are performed for evaluating both reconstruction and kernel alignment performance in classification tasks and visualization of high-dimensional data. Additionally, we show that our method is capable to emulate kernel principal component analysis on a denoising task, obtaining competitive results at a much lower computational cost.

Applications of Explicit Non-Linear Feature Maps in Steganalysis

Currently, the most popular detectors of content-adaptive image steganography are built using machine learning with images represented with rich features. Such high-dimensional descriptors, however, prevent utilization of more complex and potentially more accurate machine learning paradigms, such as kernelized support vector machines, due to infeasibly expensive training. In this paper, we demonstrate that explicit non-linear feature maps coupled with simple classifiers improve the accuracy of current steganalysis detectors built as binary classifiers as well as quantitative detectors in the form of payload regressors. The non-linear map is obtained by approximating a symmetric positive semi-definite kernel on selected pairs of cover features. Exponential forms of kernels derived from symmetrized Ali-Silvey distances improve the detection accuracy of binary detectors and lower the error of quantitative detectors across all tested steganographic schemes on grayscale and color images. The learned non-linear map only weakly depends on the cover source and its learning has a low computational complexity. The technique can also be used for unsupervised feature dimensionality reduction. For payload regressors, the dimensionality can be significantly reduced while simultaneously decreasing the estimation error.

Adaptive spectrum transformation by topology preservation on indefinite proximity data

Similarity-based representation generates indefinite matrices, which are inconsistent with classical kernel-based learning frameworks. In this paper, we present an adaptive spectrum transformation method that provides a positive semidefinite (psd) kernel consistent with the intrinsic geometry of proximity data. In the proposed method, an indefinite similarity matrix is rectified by maximizing the Euclidian factor (EF) criterion, which represents the similarity of the resulting feature space to Euclidean space. This maximization is achieved by modifying volume elements through applying a conformal transform over the similarity matrix. We performed several experiments to evaluate the performance of the proposed method in comparison with flip, clip, shift, and square spectrum transformation techniques on similarity matrices. Applying the resulting psd matrices as kernels in dimensionality reduction and clustering problems confirms the success of the proposed approach in adapting to data and preserving its topological information. Our experiments show that in classification applications, the superiority of the proposed method is considerable when the negative eigenfraction of the similarity matrix is significant.

Low-Rank Factorization of Determinantal Point Processes

Determinantal point processes (DPPs) have garnered attention as an elegant probabilistic model of set diversity. They are useful for a number of subset selection tasks, including product recommendation. DPPs are parametrized by a positive semi-definite kernel matrix. In this work we present a new method for learning the DPP kernel from observed data using a low-rank factorization of this kernel. We show that this low-rank factorization enables a learning algorithm that is nearly an order of magnitude faster than previous approaches, while also providing for a method for computing product recommendation predictions that is far faster (up to 20x faster or more for large item catalogs) than previous techniques that involve a full-rank DPP kernel. Furthermore, we show that our method provides equivalent or sometimes better test log-likelihood than prior full-rank DPP approaches.

Representations of $$\varvec{*}$$ ∗ -Semigroups Associated to Invariant Kernels with Values Continuously Adjointable Operators

We consider positive semidefinite kernels valued in the \(*\)-algebra of continuous and continuously adjointable operators on a VH-space (Vector Hilbert space in the sense of Loynes) and that are invariant under actions of \(*\)-semigroups. For such a kernel we obtain two necessary and sufficient boundedness conditions in order for there to exist \(*\)-representations of the underlying \(*\)-semigroup on a VH-space linearisation, equivalently, on a reproducing kernel VH-space. We exhibit several situations when the latter boundedness condition is automatically fulfilled. For example, when specialising to the case of Hilbert modules over locally \(C^*\)-algebras, we show that both boundedness conditions are automatically fulfilled and, consequently, this general approach provides a rather direct proof of the general Stinespring–Kasparov type dilation theorem for completely positive maps on locally \(C^*\)-algebras and with values adjointable operators on Hilbert modules over locally \(C^*\)-algebras.

Operator models for Hilbert locally C^∗-modules

We single out the concept of concrete Hilbert module over a locally $C^*$-algebra by means of locally bounded operators on certain strictly inductive limits of Hilbert spaces. Using this concept, we construct an operator model for all Hilbert locally $C^*$-modules and, as an application, we obtain a direct construction of the exterior tensor product of Hilbert locally $C^*$-modules. These are obtained as consequences of a general dilation theorem for positive semidefinite kernels invariant under an action of a $*$-semigroup with values locally bounded operators. As a by-product, we obtain two Stinespring type theorems for completely positive maps on locally $C^*$-algebras and with values locally bounded operators.

Representations of ⁎-semigroups associated to invariant kernels with values adjointable operators

We consider positive semidefinite kernels valued in the ⁎-algebra of adjointable operators on a VE-space (Vector Euclidean space) and that are invariant under actions of ⁎-semigroups. A rather general dilation theorem is stated and proved: for these kind of kernels, representations of the ⁎-semigroup on either the VE-spaces of linearisation of the kernels or on their reproducing kernel VE-spaces are obtainable. We point out the reproducing kernel fabric of dilation theory and we show that the general theorem unifies many dilation results at the non-topological level.

Metric and non-metric proximity transformations at linear costs

Domain specific (dis-)similarity or proximity measures used e.g. in alignment algorithms of sequence data are popular to analyze complicated data objects and to cover domain specific data properties. Without an underlying vector space these data are given as pairwise (dis-)similarities only. The few available methods for such data focus widely on similarities and do not scale to large datasets. Kernel methods are very effective for metric similarity matrices, also at large scale, but costly transformations are necessary starting with non-metric (dis-) similarities. We propose an integrative combination of Nyström approximation, potential double centering and eigenvalue correction to obtain valid kernel matrices at linear costs in the number of samples. By the proposed approach effective kernel approaches become accessible. Experiments with several larger (dis-)similarity datasets show that the proposed method achieves much better runtime performance than the standard strategy while keeping competitive model accuracy. The main contribution is an efficient and accurate technique, to convert (potentially non-metric) large scale dissimilarity matrices into approximated positive semi-definite kernel matrices at linear costs.

Subgradient-based neural network for nonconvex optimization problems in support vector machines with indefinite kernels

Support vector machines (SVMs) with positive semidefinite kernels yield convex quadratic programming problems. SVMs with indefinite kernels yield nonconvex quadratic programming problems. Most optimization methods for SVMs rely on the convexity of objective functions and are not efficient for solving such nonconvex problems. In this paper, we propose a subgradient-based neural network (SGNN) for the problems cast by SVMs with indefinite kernels. It is shown that the state of the proposed neural network has finite length, and as a consequence it converges toward a singleton. The coincidence between the solution and the slow solution of SGNN is also proved starting from the initial value of SGNN. Moreover, we employ the Łojasiewicz inequality to exploit the convergence rate of trajectory of SGNN. The obtained results show that each trajectory is either exponentially convergent, or convergent in finite time, toward a singleton belonging to the set of constrained critical points through a quantitative evaluation of the Łojasiewicz exponent at the equilibrium points. This method is easy to implement without adding any new parameters. Three benchmark data sets from the University of California, Irvine machine learning repository are used in the numerical tests. Experimental results show the efficiency of the proposed neural network.

Sparse conformal prediction for dissimilarity data

Existing classification algorithms focus on vectorial data given in Euclidean space or representations by means of positive semi-definite kernel matrices. Many real world data, like biological sequences are not vectorial, often non-euclidean and given only in the form of (dis-)similarities between examples, requesting for efficient and interpretable models. Vectorial embeddings or transformations to get a valid kernel are limited and current dissimilarity classifiers often lead to dense complex models which are hard to interpret by domain experts. They also fail to provide additional information about the confidence of the classification. In this paper we propose a prototype-based conformal classifier for dissimilarity data. It is based on a prototype dissimilarity learner and extended by the conformal prediction methodology. It (i) can deal with dissimilarity data characterized by an arbitrary symmetric dissimilarity matrix, (ii) offers intuitive classification in terms of sparse prototypical class representatives, (iii) leads to state-of-the-art classification results supported by a confidence measure and (iv) the model complexity is automatically adjusted. In experiments on dissimilarity data we investigate the effectiveness with respect to accuracy and model complexity in comparison to different state of the art classifiers.

Entropy-Relevant Dimensions in the Kernel Feature Space: Cluster-Capturing Dimensionality Reduction

The nonlinear mapping to the feature space is a very important concept in kernel-based machine learning for signal processing, within the framework of positive semidefinite (psd) kernels. Given labeled data, algorithms such as support vector machines (SVMs) or projection methods such as Fisher discriminant analysis may be executed in the feature space. For unsupervised dimensionality reduction in the feature space, the most common approach is to perform principal component analysis (PCA) in that space, thus maximally capturing the variability of the feature space data, however, doing so without necessarily capturing any cluster structure in the data. In this tutorial, we review the theory behind feature space mapping and review recent advances that broaden the understanding and interpretability of the mapping in terms of a key input space quantity, particularly the quadratic Renyi entropy of the data via the eigenvalues and eigenfunctions of a psd convolution operator. Focusing on the unsupervised case, we describe the identification of entropy-relevant dimensions in the feature space. We review recent results showing that these dimensions capture structure in the data in the form of clusters, and that they are, in general, different from the kernel PCA (KPCA) dimensions. Differences between these approaches to dimensionality reduction for visualization and clustering are illustrated.

Robust kernel density estimation

We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical M-estimation. We interpret the KDE based on a positive semi-definite kernel as a sample mean in the associated reproducing kernel Hilbert space. Since the sample mean is sensitive to outliers, we estimate it robustly via M-estimation, yielding a robust kernel density estimator (RKDE). An RKDE can be computed efficiently via a kernelized iteratively re-weighted least squares (IRWLS) algorithm. Necessary and sufficient conditions are given for kernelized IRWLS to converge to the global minimizer of the M-estimator objective function. The robustness of the RKDE is demonstrated with a representer theorem, the influence function, and experimental results for density estimation and anomaly detection.

Correlation Kernels for Support Vector Machines Classification with Applications in Cancer Data

High dimensional bioinformatics data sets provide an excellent and challenging research problem in machine learning area. In particular, DNA microarrays generated gene expression data are of high dimension with significant level of noise. Supervised kernel learning with an SVM classifier was successfully applied in biomedical diagnosis such as discriminating different kinds of tumor tissues. Correlation Kernel has been recently applied to classification problems with Support Vector Machines (SVMs). In this paper, we develop a novel and parsimonious positive semidefinite kernel. The proposed kernel is shown experimentally to have better performance when compared to the usual correlation kernel. In addition, we propose a new kernel based on the correlation matrix incorporating techniques dealing with indefinite kernel. The resulting kernel is shown to be positive semidefinite and it exhibits superior performance to the two kernels mentioned above. We then apply the proposed method to some cancer data in discriminating different tumor tissues, providing information for diagnosis of diseases. Numerical experiments indicate that our method outperforms the existing methods such as the decision tree method and KNN method.

A generalization of Haussler's convolution kernel

Haussler's convolution kernel provides an effiective framework for engineering positive semidefinite kernels, and has a wide range of applications. On the other hand, the mapping kernel that we int...

Applying the Possibilistic c-Means Algorithm in Kernel-Induced Spaces

In this paper, we study a kernel extension of the classic possibilistic c-means. In the proposed extension, we implicitly map input patterns into a possibly high-dimensional space by means of positive semidefinite kernels. In this new space, we model the mapped data by means of the possibilistic clustering algorithm. We study in more detail the special case where we model the mapped data using a single cluster only, since it turns out to have many interesting properties. The modeled memberships in kernel-induced spaces yield a modeling of generic shapes in the input space. We analyze in detail the connections to one-class support vector machines and kernel density estimation, thus, suggesting that the proposed algorithm can be used in many scenarios of unsupervised learning. In the experimental part, we analyze the stability and the accuracy of the proposed algorithm on some synthetic and real datasets. The results show high stability and good performances in terms of accuracy.

Support vector machine classification with indefinite kernels

In this paper, we propose a method for support vector machine classification using indefinite kernels. Instead of directly minimizing or stabilizing a nonconvex loss function, our method simultaneously finds the support vectors and a proxy kernel matrix used in computing the loss. This can be interpreted as a robust classification problem where the indefinite kernel matrix is treated as a noisy observation of the true positive semidefinite kernel. Our formulation keeps the problem convex and relatively large problems can be solved efficiently using the analytic center cutting plane method. We compare the performance of our technique with other methods on several data sets.

Examining the relative influence of familial, genetic, and environmental covariate information in flexible risk models

We present a method for examining the relative influence of familial, genetic, and environmental covariate information in flexible nonparametric risk models. Our goal is investigating the relative importance of these three sources of information as they are associated with a particular outcome. To that end, we developed a method for incorporating arbitrary pedigree information in a smoothing spline ANOVA (SS-ANOVA) model. By expressing pedigree data as a positive semidefinite kernel matrix, the SS-ANOVA model is able to estimate a log-odds ratio as a multicomponent function of several variables: one or more functional components representing information from environmental covariates and/or genetic marker data and another representing pedigree relationships. We report a case study on models for retinal pigmentary abnormalities in the Beaver Dam Eye Study. Our model verifies known facts about the epidemiology of this eye lesion--found in eyes with early age-related macular degeneration--and shows significantly increased predictive ability in models that include all three of the genetic, environmental, and familial data sources. The case study also shows that models that contain only two of these data sources, that is, pedigree-environmental covariates, or pedigree-genetic markers, or environmental covariates-genetic markers, have comparable predictive ability, but less than the model with all three. This result is consistent with the notions that genetic marker data encode--at least in part--pedigree data, and that familial correlations encode shared environment data as well.

A Compact [0,1]-valued First-order Lukasiewicz Logic with Identity on Hilbert Space

By an MV-set, we understand a pair (E,X) where X is a set of unit vectors in a Hilbert space E such that the linear span of X is dense in E, and 〈v,w〉 ≥ 0 for all v,w ∈ X. The scalar product 〈v,w〉 ∈ [0,1] is the identity degree of v and w. Building on MV-sets and continuous functions and relations defined on them, we construct a compact [0,1]-valued first-order Łukasiewicz logic, whose set of unsatisfiable formulas is recursively enumerable. In the particular case when X is an orthonormal basis of E we recover classical Skolem first-order logic with identity, constants, functions and relations. Our main tools are the Kolmogorov dilation theorem for positive semidefinite kernels, and the Tarski–Seidenberg decision method for elementary algebra and geometry.