Nystrom Approximation Research Articles

Bridging the Gap Between Artificial Neural Networks and Kernel Regressions for Vector-Valued Problems in Microwave Applications

Thanks to their convex formulation, kernel regressions have shown an improved accuracy with respect to artificial neural network (ANN) structures in regression problems where a reduced set of training samples are available. However, despite the above interesting features, kernel regressions are inherently less flexible than ANN structures since their implementations are usually limited to scalar-output regression problems. This article presents a vector-valued (multioutput) formulation of the kernel ridge regression (KRR) aimed at bridging the gap between multioutput ANN structures and scalar kernel-based approaches. The proposed vector-valued KRR relies on a generalized definition of the reproducing kernel Hilbert space (RKHS) and on a new multioutput kernel structure. The mathematical background of the proposed vector-valued formulation is extensively discussed together with different matrix kernel functions and training schemes. Moreover, a compression strategy based on the Nystrom approximation is presented to reduce the computational complexity of the model training. The effectiveness and the performance of the proposed vector-valued KRR are discussed on an illustrative example consisting of a high-speed link and on the optimization of a Doherty amplifier.

Diversity Sampling is an Implicit Regularization for Kernel Methods

Kernel methods have achieved very good performance on large scale regression and classification problems by using the Nystrom method and preconditioning techniques. The Nystrom approximation---base...

Efficient Coherent Direction-of-Arrival Estimation and Realization Using Digital Signal Processor

A novel efficient coherent direction-of-arrival (DOA) estimation method is devised in this article. First, a new cost function without the knowledge of source number is developed exploiting the Toeplitz matrices’ joint diagonalization structure. Then, the revised steering vectors are used in the place of projection weights of the steering vectors to reconstruct the power spectrum in both noise and signal subspaces. The coherent DOAs are estimated using the 1-D search. Furthermore, the computational complexity of the proposed method is significantly reduced using the Nystrom approximation. Finally, the developed theoretical model is implemented on the TMS320C6678 digital signal processor (DSP) to exemplify the efficacy of the novel method.

A new Nystrom approximation based efficient coherent DOA estimator for radar applications

In this work, we present a new method for the direction-of-arrival (DOA) estimation of coherent sources using the symmetric uniform linear array (ULA) without prior source number information. The proposed method constructs a new cost function based on the joint diagonalization structure of the Toeplitz matrix. The estimation of DOAs is made by developing a new spatial spectrum via a 1-D search. Furthermore, we exploit the Nystrom method to diminish the computational complexity of the proposed method. The efficacy of the new estimator is theoretically investigated and the complexity of the computation is also addressed. To examine the efficacy of the proposed method, various experiments are conducted and comparisons with the recent methods are carried out.

Scalable Spectral Clustering With Nyström Approximation: Practical and Theoretical Aspects

Spectral clustering techniques are valuable tools in signal processing and machine learning for partitioning complex data sets. The effectiveness of spectral clustering stems from constructing a non-linear embedding based on creating a similarity graph and computing the spectral decomposition of the Laplacian matrix. However, spectral clustering methods fail to scale to large data sets because of high computational cost and memory usage. A popular approach for addressing these problems utilizes the Nystrom method, an efficient sampling-based algorithm for computing low-rank approximations to large positive semi-definite matrices. This paper demonstrates how the previously popular approach of Nystrom-based spectral clustering has severe limitations. Existing time-efficient methods ignore critical information by prematurely reducing the rank of the similarity matrix associated with sampled points. Also, current understanding is limited regarding how utilizing the Nystrom approximation will affect the quality of spectral embedding approximations. To address the limitations, this work presents a principled spectral clustering algorithm that exploits spectral properties of the similarity matrix associated with sampled points to regulate accuracy-efficiency trade-offs. We provide theoretical results to reduce the current gap and present numerical experiments with real and synthetic data. Empirical results demonstrate the efficacy and efficiency of the proposed method compared to existing spectral clustering techniques based on the Nystrom method and other efficient methods. The overarching goal of this work is to provide an improved baseline for future research directions to accelerate spectral clustering.

Kernel Kalman Filtering With Conditional Embedding and Maximum Correntropy Criterion

The Hilbert space embedding provides a powerful and flexible tool for dealing with the nonlinearity and high-order statistics of random variables in a dynamical system. The kernel Kalman filtering based on the conditional embedding operator (KKF-CEO) shows significant performance improvements over the traditional Kalman filters in the noisy nonlinear time-series prediction. However, KKF-CEO based on the minimum mean-square-error (MMSE) criterion is sensitive to the outliers or heavy-tailed noises. In contrast to the MMSE criterion, the maximum correntropy criterion (MCC) can achieve more robust performance in the presence of outliers. In this paper, we develop a novel kernel Kalman-type filter based on MCC, referred to kernel Kalman filtering with conditional embedding operator and maximum correntropy criterion (KKF-CEO-MCC). The proposed KKF-CEO-MCC can capture higher order statistics of errors and is robust to outliers. In addition, two simplified versions of KKF-CEO-MCC are developed, namely, KKF-CEO-MCC-O and KKF-CEO-MCC-NA. The former is an online approach and the latter is based on Nystrom approximation. Simulations on noisy nonlinear time-series prediction confirm the desirable accuracy and robustness of the new filters.

Constrained spectral clustering via multi–layer graph embeddings on a grassmann manifold

Abstract We present two algorithms in which constrained spectral clustering is implemented as unconstrained spectral clustering on a multi-layer graph where constraints are represented as graph layers. By using the Nystrom approximation in one of the algorithms, we obtain time and memory complexities which are linear in the number of data points regardless of the number of constraints. Our algorithms achieve superior or comparative accuracy on real world data sets, compared with the existing state-of-the-art solutions. However, the complexity of these algorithms is squared with the number of vertices, while our technique, based on the Nyström approximation method, has linear time complexity. The proposed algorithms efficiently use both soft and hard constraints since the time complexity of the algorithms does not depend on the size of the set of constraints.

Structured Quasi-Newton Methods for Optimization with Orthogonality Constraints

In this paper, we study structured quasi-Newton methods for optimization problems with orthogonality constraints. Note that the Riemannian Hessian of the objective function requires both the Euclidean Hessian and the Euclidean gradient. In particular, we are interested in applications that the Euclidean Hessian itself consists of a computational cheap part and a significantly expensive part. Our basic idea is to keep these parts of lower computational costs but substitute those parts of higher computational costs by the limited-memory quasi-Newton update. More specically, the part related to Euclidean gradient and the cheaper parts in the Euclidean Hessian are preserved. The initial quasi-Newton matrix is further constructed from a limited-memory Nystrom approximation to the expensive part. Consequently, our subproblems approximate the original objective function in the Euclidean space and preserve the orthogonality constraints without performing the so-called vector transports. When the subproblems are solved to sufficient accuracy, both global and local q-superlinear convergence can be established under mild conditions. Preliminary numerical experiments on the linear eigenvalue problem and the electronic structure calculation show the effectiveness of our method compared with the state-of-art algorithms.

Supervised low rank indefinite kernel approximation using minimum enclosing balls

Abstract Indefinite similarity measures can be frequently found in bio-informatics by means of alignment scores, but are also common in other fields like shape measures in image retrieval. Lacking an underlying vector space, the data are given as pairwise similarities only. The few algorithms available for such data do not scale to larger datasets. Focusing on probabilistic batch classifiers, the Indefinite Kernel Fisher Discriminant (iKFD) and the Probabilistic Classification Vector Machine (PCVM) are both effective algorithms for this type of data but, with cubic complexity. Here we propose an extension of iKFD and PCVM such that linear runtime and memory complexity is achieved for low rank indefinite kernels. Employing the Nystrom approximation for indefinite kernels, we also propose a new almost parameter free approach to identify the landmarks, restricted to a supervised learning problem. Evaluations at several larger similarity data from various domains show that the proposed methods provides similar generalization capabilities while being easier to parametrize and substantially faster for large scale data.

Low-Complexity Nonlinear Analysis of Synchrophasor Measurements for Events Detection and Localization

This paper is concerned with the computational efficient nonlinear analysis of phasor measurement unit data and presents a Nystrom principal components analysis-based algorithm for events detection and localization in an electrical power system. Based on the properly chosen sample subset of every moving window data, the Nystrom approximation is carried out to obtain the principal eigenvalues and related eigenvectors of a mapped kernel matrix. Then, the $T^{2}$ statistic is constructed to detect the abnormal states of an electrical power system. In addition, the contribution of each variable to the $T^{2}$ statistic is derived to determine the location of the fault bus. Compared with the previous works, the novel version proposed in this paper efficiently reduces the computational burden, and accurately locates the fault bus. Computer simulations using both realistic data, collected from the China power system, and simulated voltage and phase-angle data, validate the reliability of the proposed algorithm.

Efficient Multiple Kernel Classification Using Feature and Decision Level Fusion

Kernel methods for classification is a well-studied area in which data are implicitly mapped from a lower-dimensional space to a higher dimensional space to improve classification accuracy. However, for most kernel methods, one must still choose a kernel to use for the problem. Since there is, in general, no way of knowing which kernel is the best, multiple kernel learning (MKL) is a technique used to learn the aggregation of a set of valid kernels into a single (ideally) superior kernel. The aggregation can be done using weighted sums of the precomputed kernels, but determining the summation weights is not a trivial task. Furthermore, MKL does not work well with large datasets because of limited storage space and prediction speed. In this paper, we address all three of these multiple kernel challenges. First, we introduce a new linear feature level fusion technique and learning algorithm, GAMKLp. Second, we put forth three new algorithms, DeFIMKL, DeGAMKL, and DeLSMKL, for nonlinear fusion of kernels at the decision level. To address MKL's storage and speed drawbacks, we apply the Nystrom approximation to the kernel matrices. We compare our methods to a successful and state-of-the-art technique called MKL-group lasso (MKLGL), and experiments on several benchmark datasets show that some of our proposed algorithms outperform MKLGL when applied to support vector machine (SVM)-based classification. However, to no surprise, there does not seem to be a global winner but instead different strategies that a user can employ. Experiments with our kernel approximation method show that we can routinely discard most of the training data and at least double prediction speed without sacrificing classification accuracy. These results suggest that MKL-based classification techniques can be applied to big data efficiently, which is confirmed by an experiment using a large dataset.

Memory efficient kernel approximation

Scaling kernel machines to massive data sets is a major challenge due to storage and computation issues in handling large kernel matrices, that are usually dense. Recently, many papers have suggested tackling this problem by using a low-rank approximation of the kernel matrix. In this paper, we first make the observation that the structure of shift-invariant kernels changes from low-rank to block-diagonal (without any low-rank structure) when varying the scale parameter. Based on this observation, we propose a new kernel approximation framework - Memory Efficient Kernel Approximation (MEKA), which considers both low-rank and clustering structure of the kernel matrix. We show that the resulting algorithm outperforms state-of-the-art low-rank kernel approximation methods in terms of speed, approximation error, and memory usage. As an example, on the covtype dataset with half a million samples, MEKA takes around 70 seconds and uses less than 80 MB memory on a single machine to achieve 10% relative approximation error, while standard Nystrom approximation is about 6 times slower and uses more than 400MB memory to achieve similar approximation. We also present extensive experiments on applying MEKA to speed up kernel ridge regression.

Multi-view kernel completion

In this paper, we introduce the first method that (1) can complete kernel matrices with completely missing rows and columns as opposed to individual missing kernel values, with help of information from other incomplete kernel matrices. Moreover, (2) the method does not require any of the kernels to be complete a priori, and (3) can tackle non-linear kernels. The kernel completion is done by finding, from the set of available incomplete kernels, an appropriate set of related kernels for each missing entry. These aspects are necessary in practical applications such as integrating legacy data sets, learning under sensor failures and learning when measurements are costly for some of the views. The proposed approach predicts missing rows by modelling both within-view and between-view relationships among kernel values. For within-view learning, we propose a new kernel approximation that generalizes and improves Nystrom approximation. We show, both on simulated data and real case studies, that the proposed method outperforms existing techniques in the settings where they are available, and extends applicability to new settings.

A Spectral Clustering Image Segmentation Algorithm Based on Nyström Approximation

In the image segmentation, the spectral clustering algorithm are facing the problems, that the similarity matrix and Laplace matrix requires a lot of computing and storage, which limits its expansion in the large-scale image. Aiming at these problems, a semi-supervised spectrum clustering algorithm based on the Nystrom approximation was proposed. In the algorithm, adopting the Nystrom approximation method to estimate the similarity matrix, which effectively reduces the amount of computation and storage of the spectral clustering algorithm. Finally, on the Brodatz texture library, the image segmentation experiments showed the feasibility and effectiveness of the algorithm.

Unsupervised multi-graph cross-modal hashing for large-scale multimedia retrieval

With the advance of internet and multimedia technologies, large-scale multi-modal representation techniques such as cross-modal hashing, are increasingly demanded for multimedia retrieval. In cross-modal hashing, three essential problems should be seriously considered. The first is that effective cross-modal relationship should be learned from training data with scarce label information. The second is that appropriate weights should be assigned for different modalities to reflect their importance. The last is the scalability of training process which is usually ignored by previous methods. In this paper, we propose Multi-graph Cross-modal Hashing (MGCMH) by comprehensively considering these three points. MGCMH is unsupervised method which integrates multi-graph learning and hash function learning into a joint framework, to learn unified hash space for all modalities. In MGCMH, different modalities are assigned with proper weights for the generation of multi-graph and hash codes respectively. As a result, more precise cross-modal relationship can be preserved in the hash space. Then Nystrom approximation approach is leveraged to efficiently construct the graphs. Finally an alternating learning algorithm is proposed to jointly optimize the modality weights, hash codes and functions. Experiments conducted on two real-world multi-modal datasets demonstrate the effectiveness of our method, in comparison with several representative cross-modal hashing methods.

Image Denoising using Nystrom Approximation with Glide Framework

Image denoising is an important image processing task, both as a process itself and as a component in other processes. Images are often corrupted by noise during the acquisition process. Denoising aims at eliminating this measurement noise while trying to preserve important signal features such as texture and edges. Over the past few decades, a large variety of algorithms has been developed for that purpose. Image denoising is an important process in image processing. In previous approach there are two denoising approach has been implemented such as NLM and Block- Matching and 3D (BM3D).In Non Local Means (NLM) each pixel has been estimated separately and fusing other “similar” neighborhood pixel. Whereas Block-Matching and 3D filtering (BM3D) is performed on group of similar patches together. In this paper, our proposed approach is a Global denoising algorithm (GLIDE) which is a non-local patch-based Processing where each pixel is estimated from all pixels in the image. Approximated spectral decomposition is computed using Nystrom Method. Eigen decomposition of the approximated image is estimated by Sinkhorn method for getting orthogonal eigenvectors of the obtained Eigen values with the help of Orthogonalization process. Finally Denoised image will be obtained using this proposed strategy. Hence, our approach can effectively enhance the performance of existing filters. Key Words: Image denoising; Block- Matching and 3D (BM3D); NLM (Non-local means) filters; BM3D; Nystrom extension;

Computationally Efficient Spectrum Sensing With Nyström Feature Template Matching

Kernel feature template matching (KFTM) was developed in 2014 for spectrum sensing in cognitive radio. However, the high computational burden involved in the kernel matrices construction and the related eigenvalues decomposition (EVD) limits its practical applications. This letter addresses this issue and proposes a Nystrom feature template matching (NFTM) solution. Based on the randomly chosen subset of the whole data, the Nystrom approximation is carried out to obtain the principle eigenvector of a given kernel matrix. With the approximated feature templates, we then modify the similarity (inner product) as a novel detection rule. Simulations using the real-world measurements of digital television (DTV) signal confirm that the proposed NFTM algorithm with about one-third samples provides almost the same performance as the KFTM method.

Region-based approach for the spectral clustering Nyström approximation with an application to burn depth assessment

Image segmentation methods based on spectral graph theory, although capable of overcoming some of the drawbacks of the so-called "central"-grouping methods, are computationally expensive and quickly become infeasible to solve as the size of the image grows. As a counter measure, the Nystrom approximation allows to extrapolate the complete grouping solution for these methods using only a proportionally smaller set of samples instead of the whole pixels that compose the image. In this correspondence, we further explore the Nystrom approximation by taking the concept of "regions", pixels of the image previously grouped by a central method, to both reduce the computational resources required and provide a finer segmentation of the image by combining the strengths of both methods. We apply the proposed approach to the segmentation of images of burns where we attempt to extract regions that would roughly correspond to the different degrees of the lesion.

Nystrom Approximation for Sparse Kernel Methods: Theoretical Analysis and Empirical Evaluation

Nystrom approximation is an effective approach to accelerate the computation of kernel matrices in many kernel methods. In this paper, we consider the Nystrom approximation for sparse kernel methods. Instead of relying on the low-rank assumption of the original kernels, which sometimes does not hold in some applications, we take advantage of the restricted eigenvalue condition, which has been proved to be robust for sparse kernel methods. Based on the restricted eigenvalue condition, we have provided not only the approximation bound for the original kernel matrix but also the recovery bound for the sparse solutions of sparse kernel regression. In addition to the theoretical analysis, we also demonstrate the good performance of the Nystrom approximation for sparse kernel regression on real world data sets.

Kernel Based CSP Algorithm for Classifying Multi-class EEG Data by Combining K-means Clustering and Nystrom Approximation

Because Common Spatial Pattern (CSP) algorithm in brain-computer interface assumes that the relation between the observed EEG signal and the brain source signal is strictly linear and very limited nonlinear kernel functions exist, both linear CSP and nonlinear kernel CSP can not characterize the brain pattern accurately. In this paper, a novel hybrid kernel CSP algorithm is proposed in order to improve the performance of pattern classification, which is based on the combination of linear and nonlinear kernel function. K-means clustering and Nystrom approximation were used to relieve the computational complexity and the requirement for memory in eigenvalue decomposition of kernel matrix. The band pass filtered EEG data were clustered for dimensionality reduction and then Nystrom approximation was performed to ensure the validity of low rank decomposition of kernel matrix in a high dimensional feature space. The performance of the algorithm was tested on a four-class dataset and compared with that of linear and kernel CSP. Using a pairwise classification strategy, the proposed algorithm achieved superior performance in terms of classification accuracy.