Full Kernel Matrix Research Articles

Efficient Parameter Optimisation for Quantum Kernel Alignment: A Sub-sampling Approach in Variational Training

Quantum machine learning with quantum kernels for classification problems is a growing area of research. Recently, quantum kernel alignment techniques that parameterise the kernel have been developed, allowing the kernel to be trained and therefore aligned with a specific dataset. While quantum kernel alignment is a promising technique, it has been hampered by considerable training costs because the full kernel matrix must be constructed at every training iteration. Addressing this challenge, we introduce a novel method that seeks to balance efficiency and performance. We present a sub-sampling training approach that uses a subset of the kernel matrix at each training step, thereby reducing the overall computational cost of the training. In this work, we apply the sub-sampling method to synthetic datasets and a real-world breast cancer dataset and demonstrate considerable reductions in the number of circuits required to train the quantum kernel while maintaining classification accuracy.

Augmented low-rank methods for gaussian process regression

This paper presents techniques to improve the prediction accuracy of approximation methods used in Gaussian process regression models. Conventional methods such as Nystrom and subset of data methods rely on low-rank approximations to the kernel matrix derived from a set of representative data points. Prediction accuracy suffers when the number of representative points is small or when the length scale is small. The techniques proposed here augment the set of representative points with neighbors of each test input to improve accuracy. Our approach leverages the general structure of the problem through the low-rank approximation and improves its accuracy further by exploiting locality at each test input. Computations involving neighbor points are cast as updates to the base approximation which result in significant savings. To ensure numerical stability, prediction is done via orthogonal projection onto the subspace of the kernel approximation derived from the augmented set. Experiments on synthetic and real datasets show that our approach is robust with respect to changes in length scale and matches the prediction accuracy of the full kernel matrix while using fewer points for kernel approximation. This results in faster and more accurate predictions compared to conventional methods.

Scalable Gaussian Process Classification With Additive Noise for Non-Gaussian Likelihoods.

Gaussian process classification (GPC) provides a flexible and powerful statistical framework describing joint distributions over function space. Conventional GPCs, however, suffer from: 1) poor scalability for big data due to the full kernel matrix and 2) intractable inference due to the non-Gaussian likelihoods. Hence, various scalable GPCs have been proposed through: 1) the sparse approximation built upon a small inducing set to reduce the time complexity and 2) the approximate inference to derive analytical evidence lower bound (ELBO). However, these scalable GPCs equipped with analytical ELBO are limited to specific likelihoods or additional assumptions. In this work, we present a unifying framework that accommodates scalable GPCs using various likelihoods. Analogous to GP regression (GPR), we introduce additive noises to augment the probability space for: 1) the GPCs with step, (multinomial) probit, and logit likelihoods via the internal variables and 2) particularly, the GPC using softmax likelihood via the noise variables themselves. This leads to unified scalable GPCs with analytical ELBO by using variational inference. Empirically, our GPCs showcase superiority on extensive binary/multiclass classification tasks with up to two million data points.

Spatial analysis made easy with linear regression and kernels

Kernel methods are a popular technique for extending linear models to handle non-linear spatial problems via a mapping to an implicit, high-dimensional feature space. While kernel methods are computationally cheaper than an explicit feature mapping, they are still subject to cubic cost on the number of points. Given only a few thousand locations, this computational cost rapidly outstrips the currently available computational power. This paper aims to provide an overview of kernel methods from first-principals (with a focus on ridge regression) and progress to a review of random Fourier features (RFF), a method that enables the scaling of kernel methods to big datasets. We show how the RFF method is capable of approximating the full kernel matrix, providing a significant computational speed-up for a negligible cost to accuracy and can be incorporated into many existing spatial methods using only a few lines of code. We give an example of the implementation of RFFs on a simulated spatial data set to illustrate these properties. Lastly, we summarise the main issues with RFFs and highlight some of the advanced techniques aimed at alleviating them. At each stage, the associated R code is provided.

Fully Symmetric Kernel Quadrature

Kernel quadratures and other kernel-based approximation methods typically suffer from prohibitive cubic time and quadratic space complexity in the number of function evaluations. The problem arises because a system of linear equations needs to be solved. In this article we show that the weights of a kernel quadrature rule can be computed efficiently and exactly for up to tens of millions of nodes if the kernel, integration domain, and measure are fully symmetric and the node set is a union of fully symmetric sets. This is based on the observations that in such a setting there are only as many distinct weights as there are fully symmetric sets and that these weights can be solved from a linear system of equations constructed out of row sums of certain submatrices of the full kernel matrix. We present several numerical examples that show feasibility, both for a large number of nodes and in high dimensions, of the developed fully symmetric kernel quadrature rules. Most prominent of the fully symmetric kernel quadrature rules we propose are those that use sparse grids.

CLAss-Specific Subspace Kernel Representations and Adaptive Margin Slack Minimization for Large Scale Classification.

In kernel-based classification models, given limited computational power and storage capacity, operations over the full kernel matrix becomes prohibitive. In this paper, we propose a new supervised learning framework using kernel models for sequential data processing. The framework is based on two components that both aim at enhancing the classification capability with a subset selection scheme. The first part is a subspace projection technique in the reproducing kernel Hilbert space using a CLAss-specific Subspace Kernel representation for kernel approximation. In the second part, we propose a novel structural risk minimization algorithm called the adaptive margin slack minimization to iteratively improve the classification accuracy by an adaptive data selection. We motivate each part separately, and then integrate them into learning frameworks for large scale data. We propose two such frameworks: the memory efficient sequential processing for sequential data processing and the parallelized sequential processing for distributed computing with sequential data acquisition. We test our methods on several benchmark data sets and compared with the state-of-the-art techniques to verify the validity of the proposed techniques.

Speech enhancement method based on low-rank approximation in a reproducing kernel Hilbert space

Abstract Speech signal is corrupted unavoidably by noisy environment in subway, factory, and restaurant or speech from other speakers in speech communication. Speech enhancement methods have been widely studied to minimize noise influence in different linear transform domain, such as discrete Fourier transform domain, Karhunen-Loeve transform domain or discrete cosine transform domain. Kernel method as a nonlinear transform has received a lot of interest recently and is commonly used in many applications including audio signal processing. However this kind of method typically suffers from the computational complexity. In this paper, we propose a speech enhancement algorithm using low-rank approximation in a reproducing kernel Hilbert space to reduce storage space and running time with very little performance loss in the enhanced speech. We also analyze the root mean squared error bound between the enhanced vectors obtained by the approximation kernel matrix and the full kernel matrix. Simulations show that the proposed method can improve the computation speed of the algorithm with the approximate performance compared with that of the full kernel matrix.

Approximate kernel competitive learning

Kernel competitive learning has been successfully used to achieve robust clustering. However, kernel competitive learning (KCL) is not scalable for large scale data processing, because (1) it has to calculate and store the full kernel matrix that is too large to be calculated and kept in the memory and (2) it cannot be computed in parallel. In this paper we develop a framework of approximate kernel competitive learning for processing large scale dataset. The proposed framework consists of two parts. First, it derives an approximate kernel competitive learning (AKCL), which learns kernel competitive learning in a subspace via sampling. We provide solid theoretical analysis on why the proposed approximation modelling would work for kernel competitive learning, and furthermore, we show that the computational complexity of AKCL is largely reduced. Second, we propose a pseudo-parallelled approximate kernel competitive learning (PAKCL) based on a set-based kernel competitive learning strategy, which overcomes the obstacle of using parallel programming in kernel competitive learning and significantly accelerates the approximate kernel competitive learning for large scale clustering. The empirical evaluation on publicly available datasets shows that the proposed AKCL and PAKCL can perform comparably as KCL, with a large reduction on computational cost. Also, the proposed methods achieve more effective clustering performance in terms of clustering precision against related approximate clustering approaches.

On the Sample Complexity of Random Fourier Features for Online Learning

We study the sample complexity of random Fourier features for online kernel learning—that is, the number of random Fourier features required to achieve good generalization performance. We show that when the loss function is strongly convex and smooth, online kernel learning with random Fourier features can achieve an O (log T / T ) bound for the excess risk with only O (1/λ 2 ) random Fourier features, where T is the number of training examples and λ is the modulus of strong convexity. This is a significant improvement compared to the existing result for batch kernel learning that requires O ( T ) random Fourier features to achieve a generalization bound O (1/√T). Our empirical study verifies that online kernel learning with a limited number of random Fourier features can achieve similar generalization performance as online learning using full kernel matrix. We also present an enhanced online learning algorithm with random Fourier features that improves the classification performance by multiple passes of training examples and a partial average.

Interactive image segmentation via kernel propagation

In this paper, we propose a new approach to interactive image segmentation via kernel propagation (KP), called KP Cut. The key to success in interactive image segmentation is to preserve characteristics of the user׳s interactive input and maintain data-coherence effectively. To achieve this, we employ KP which is very effective in propagating the given supervised information into the entire data set. KP first learns a small-size seed-kernel matrix, and then propagates it into a large-size full-kernel matrix. It is based on a learned kernel, and thus can fit the given data better than a predefined kernel. Based on KP, we first generate a small-size seed-kernel matrix from the user׳s interactive input. Then, the seed-kernel matrix is propagated into the full-kernel matrix of the entire image. During the propagation, foreground objects are effectively segmented from background. Experimental results demonstrate that KP Cut effectively extracts foreground objects from background, and outperforms the state-of-the-art methods for interactive image segmentation.

Automatic image segmentation using constraint learning and propagation

In this paper, we propose automatic image segmentation using constraint learning and propagation. Recently, kernel learning is receiving much attention because a learned kernel can fit the given data better than a predefined kernel. To effectively learn the constraints generated by initial seeds for image segmentation, we employ kernel propagation (KP) based on kernel learning. The key idea of KP is first to learn a small-sized seed-kernel matrix and then propagate it into a large-sized full-kernel matrix. By applying KP to automatic image segmentation, we design a novel segmentation method to achieve high performance. First, we generate pairwise constraints, i.e., must-link and cannot-link, from initially selected seeds to make the seed-kernel matrix. To select the optimal initial seeds, we utilize global k-means clustering (GKM) and self-tuning spectral clustering (SSC). Next, we propagate the seed-kernel matrix into the full-kernel matrix of the entire image, and thus image segmentation results are obtained. We test our method on the Berkeley segmentation database, and the experimental results demonstrate that the proposed method is very effective in automatic image segmentation.

Error Bounds on the SCISSORS Approximation Method

The SCISSORS method for approximating chemical similarities has shown excellent empirical performance on a number of real-world chemical data sets but lacks theoretically proven bounds on its worst-case error performance. This paper first proves reductions showing SCISSORS to be equivalent to two previous kernel methods: kernel principal components analysis and the rank-k Nyström approximation of a Gram matrix. These reductions allow the use of generalization bounds on these techniques to show that the expected error in SCISSORS approximations of molecular similarity kernels is bounded in expected pairwise inner product error, in matrix 2-norm and Frobenius norm for full kernel matrix approximations and in root-mean-square deviation for approximated matrices. Finally, we show that the actual performance of SCISSORS is significantly better than these worst-case bounds, indicating that chemical space is well-structured for chemical sampling algorithms.

Two-stage nonparametric kernel leaning: From label propagation to kernel propagation

We introduce a kernel learning algorithm, called kernel propagation (KP), to learn a nonparametric kernel from a mixture of a few pairwise constraints and plentiful unlabeled samples. Specifically, KP consists of two stages: the first is to learn a small-sized sub-kernel matrix just restricted to the samples with constrains, and the second is to propagate this learned sub-kernel matrix into a large-sized full-kernel matrix over all samples. As an interesting fact, our approach exposes a natural connection between KP and label propagation (LP), that is, one LP can naturally induce its KP counterpart. Thus, we develop three KPs from the three typical LPs correspondingly. Following the idea in KP, we also naturally develop an out-of-sample extension to directly capture a kernel matrix for outside-training data without the need of relearning. The final experiments verify that our developments are more efficient, more error-tolerant and also comparably effective in comparison with the state-of-the-art algorithm.

Semisupervised Kernel Matrix Learning by Kernel Propagation

The goal of semisupervised kernel matrix learning (SS-KML) is to learn a kernel matrix on all the given samples on which just a little supervised information, such as class label or pairwise constraint, is provided. Despite extensive research, the performance of SS-KML still leaves some space for improvement in terms of effectiveness and efficiency. For example, a recent pairwise constraints propagation (PCP) algorithm has formulated SS-KML into a semidefinite programming (SDP) problem, but its computation is very expensive, which undoubtedly restricts PCPs scalability in practice. In this paper, a novel algorithm, called kernel propagation (KP), is proposed to improve the comprehensive performance in SS-KML. The main idea of KP is first to learn a small-sized sub-kernel matrix (named seed-kernel matrix) and then propagate it into a larger-sized full-kernel matrix. Specifically, the implementation of KP consists of three stages: 1) separate the supervised sample (sub)set X(l) from the full sample set X; 2) learn a seed-kernel matrix on X(l) through solving a small-scale SDP problem; and 3) propagate the learnt seed-kernel matrix into a full-kernel matrix on X . Furthermore, following the idea in KP, we naturally develop two conveniently realizable out-of-sample extensions for KML: one is batch-style extension, and the other is online-style extension. The experiments demonstrate that KP is encouraging in both effectiveness and efficiency compared with three state-of-the-art algorithms and its related out-of-sample extensions are promising too.

Reduced Support Vector Machines: A Statistical Theory

In dealing with large data sets, the reduced support vector machine (RSVM) was proposed for the practical objective to overcome some computational difficulties as well as to reduce the model complexity. In this paper, we study the RSVM from the viewpoint of sampling design, its robustness, and the spectral analysis of the reduced kernel. We consider the nonlinear separating surface as a mixture of kernels. Instead of a full model, the RSVM uses a reduced mixture with kernels sampled from certain candidate set. Our main results center on two major themes. One is the robustness of the random subset mixture model. The other is the spectral analysis of the reduced kernel. The robustness is judged by a few criteria as follows: 1) model variation measure; 2) model bias (deviation) between the reduced model and the full model; and 3) test power in distinguishing the reduced model from the full one. For the spectral analysis, we compare the eigenstructures of the full kernel matrix and the approximation kernel matrix. The approximation kernels are generated by uniform random subsets. The small discrepancies between them indicate that the approximation kernels can retain most of the relevant information for learning tasks in the full kernel. We focus on some statistical theory of the reduced set method mainly in the context of the RSVM. The use of a uniform random subset is not limited to the RSVM. This approach can act as a supplemental algorithm on top of a basic optimization algorithm, wherein the actual optimization takes place on the subset-approximated data. The statistical properties discussed in this paper are still valid.