Kernel Matrix Approximation Research Articles

Voice Vista

Blind image restoration involves leveraging prior information within an image to restore the sharpness of edges. De-blurring, on the other hand, aims to eliminate blurring artifacts caused by factors like defocus aberration or motion blur, which manifests as apparent streaking in still images of rapidly moving objects. Gaussian blur results from applying a Gaussian function to blur an image. Recent advancements in single-image methods owe their success in part to the utilization of various sparse priors for either latent images or motion blur kernels. KSR introduces an efficient kernel matrix approximation to accelerate blurring processes and achieve notable de-blur performance on digital datasets. License plate recognition serves as a critical tool for identifying over-speed vehicles or those involved in hit-and-run incidents. However, surveillance camera snapshots of speeding vehicles often suffer from motion blur, rendering them unrecognizable to the human eye and presenting a challenge to existing blind deblurring techniques. To address this, we propose a novel sparse representation-based scheme for identifying motion blur kernels. By analyzing sparse representation coefficients of the recovered image, we determine the kernel angle, and estimate the length of the motion kernel using Radon transform in Fourier domain. Our approach effectively handles large motion blur, even when license plates are unidentifiable by humans. Experimental evaluations on real-world images demonstrate the superiority of our method over several state-of-the-art blind image de-blurring algorithms.

Fixed-budget approximation of the inverse kernel matrix for identification of nonlinear dynamic processes

The paper considers the identification of nonlinear dynamic processes using kernel algorithms. Kernel algorithms rely on a nonlinear transformation of the input data points into a high-dimensional space that allows solving nonlinear problems through the construction of kernelized counterparts of linear methods by replacing the inner products with kernels. A key feature of the kernel algorithms is high complexity of the inverse kernel matrix calculation. Nowadays, there are two approaches to this problem. The first one is based on using a reduced training data sample instead of a full one. In case of kernel methods, this approach could cause model misspecification, since kernel methods are directly based on training data. The second one is based on the reduced-rank approximations of the kernel matrix. A major limitation of this approach is that the rank of the approximation is either unknown until approximation is done or it is predefined by the user, both of which are not efficient enough. In this paper, we propose a new regularized kernel least squares algorithm based on the fixed-budget approximation of the kernel matrix. The proposed algorithm allows regulating the computational burden of the identification algorithm and obtaining the least approximation error. We have shown some simulations results illustrating the efficiency of the proposed algorithm compared to other algorithms. The application of the proposed algorithm is considered on the identification problem of the input and output pressure of the pump station.

Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

Generating low-rank approximations of kernel matrices that arise in nonlinear machine learning techniques holds the potential to significantly alleviate the memory and computational burdens. A compelling approach centers on finding a concise set of exemplars or landmarks to reduce the number of similarity measure evaluations from quadratic to linear concerning the data size. However, a key challenge is to regulate tradeoffs between the quality of landmarks and resource consumption. Despite the volume of research in this area, current understanding is limited regarding the performance of landmark selection techniques in the presence of class-imbalanced data sets that are becoming increasingly prevalent in many applications. Hence, this paper provides a comprehensive empirical investigation using several real-world imbalanced data sets, including scientific data, by evaluating the quality of approximate low-rank decompositions and examining their influence on the accuracy of downstream tasks. Furthermore, we present a new landmark selection technique called Distance-based Importance Sampling and Clustering (DISC), in which the relative importance scores are computed for improving accuracy-efficiency tradeoffs compared to existing works that range from probabilistic sampling to clustering methods. The proposed landmark selection method follows a coarse-to-fine strategy to capture the intrinsic structure of complex data sets, allowing us to substantially reduce the computational complexity and memory footprint with minimal loss in accuracy.

Approximate Kernel Selection via Matrix Approximation.

Kernel selection is of fundamental importance for the generalization of kernel methods. This article proposes an approximate approach for kernel selection by exploiting the approximability of kernel selection and the computational virtue of kernel matrix approximation. We define approximate consistency to measure the approximability of the kernel selection problem. Based on the analysis of approximate consistency, we solve the theoretical problem of whether, under what conditions, and at what speed, the approximate criterion is close to the accurate one, establishing the foundations of approximate kernel selection. We introduce two selection criteria based on error estimation and prove the approximate consistency of the multilevel circulant matrix (MCM) approximation and Nyström approximation under these criteria. Under the theoretical guarantees of the approximate consistency, we design approximate algorithms for kernel selection, which exploits the computational advantages of the MCM and Nyström approximations to conduct kernel selection in a linear or quasi-linear complexity. We experimentally validate the theoretical results for the approximate consistency and evaluate the effectiveness of the proposed kernel selection algorithms.

Interpolative Decomposition via Proxy Points for Kernel Matrices

In the construction of rank-structured matrix representations of dense kernel matrices, a heuristic compression method, called the proxy point method, has been used in practice to efficiently compute the low-rank approximation of certain kernel matrix blocks in the form of an interpolative decomposition. We present a long overdue error analysis for the proxy point method, rigorously proving the effectiveness of the method under specific conditions. The analysis also generalizes the method, allowing it to be applied to the construction of different types of rank-structured matrices with general kernel functions in low-dimensional spaces. Based on the analysis, a systematic and adaptive scheme for selecting the proxy points used in the method is developed, which can guarantee that the method is effective for any given kernel function under specific conditions.

Approximate Kernel Selection with Strong Approximate Consistency

Kernel selection is fundamental to the generalization performance of kernel-based learning algorithms. Approximate kernel selection is an efficient kernel selection approach that exploits the convergence property of the kernel selection criteria and the computational virtue of kernel matrix approximation. The convergence property is measured by the notion of approximate consistency. For the existing Nyström approximations, whose sampling distributions are independent of the specific learning task at hand, it is difficult to establish the strong approximate consistency. They mainly focus on the quality of the low-rank matrix approximation, rather than the performance of the kernel selection criterion used in conjunction with the approximate matrix. In this paper, we propose a novel Nyström approximate kernel selection algorithm by customizing a criterion-driven adaptive sampling distribution for the Nyström approximation, which adaptively reduces the error between the approximate and accurate criteria. We theoretically derive the strong approximate consistency of the proposed Nyström approximate kernel selection algorithm. Finally, we empirically evaluate the approximate consistency of our algorithm as compared to state-of-the-art methods.

Mixture of Gaussian Blur Kernel Representation for Blind Image Restoration

The use of blind image restoration, sharpness of edges may frequently be restored using previous information from a picture. De-blurring is the technique of taking out blurring flaws of the steady photographs, including motion or defocus aberration-related blur. the appearance of fast-moving the appearance of fast-moving entities flashing in still images flashing in a still photograph is known as motion blur. When an image is blurred using a Gaussian function, the result is a Gaussian blur. The employment of different sparse priors, either for the implicit photos or the motion blur kernels, contributes to the success of contemporary single-image approaches. De-blurring is the technique of taking out blurring flaws of the steady photographs, including motion or defocus aberration-related blur. The apparent flashing of quickly moving item in a static photograph is known as motion blur. When a picture is blurred utilizing a Gaussian function, the result is a Gaussian blur. The employment of different sparse priors, either for the latent photos or the motion blur kernels, contributes to the success of contemporary single-image approaches. On digital datasets, KSR also discovers effective kernel matrix approximation to hasten blurring and provide effective de-blur performances. The licence plate, which serves as the vehicle's distinctive identifier, is an important indicator of speeding or hit-and-run cars. However, the image of a fast-moving car taken by a security camera is usually blurred and not even humanly discernible.These observed plate pictures are frequently poor resolution and have significant edge information lost, which presents a significant challenge to the current blind deblurring techniques. The blur kernel may be thought of as a linear uniform convolution and parametrically modelled with angle and length for licence plate picture blurring brought on by rapid movement. This research suggests unique technique for locating the blur kernel based on sparse representation. We determine the angle of the kernel by looking at the sparse representation coefficients of the restored picture because the retrieved photo has the highest sparse representation when the kernel angle coincides with the real motion angle. Afterwards, using Radon transform in Fourier domain, we estimate the size of the motion kernel. Even when the licence plate is impossible for a person to read, our system handles big motion blur rather effectively. We assess our method using actual photographs and contrast it with a number of well-known cutting-edge blind image deblurring techniques. Experimental findings show that our suggested technique is superior in terms of efficiency and resilience.

Kernel K-Means Sampling for Nyström Approximation.

A fundamental problem in Nyström-based kernel matrix approximation is the sampling method by which training set is built. In this paper, we suggest to use kernel -means sampling, which is shown in our works to minimize the upper bound of a matrix approximation error. We first propose a unified kernel matrix approximation framework, which is able to describe most existing Nyström approximations under many popular kernels, including Gaussian kernel and polynomial kernel. We then show that, the matrix approximation error upper bound, in terms of the Frobenius norm, is equal to the -means error of data points in kernel space plus a constant. Thus, the -means centers of data in kernel space, or the kernel -means centers, are the optimal representative points with respect to the Frobenius norm error upper bound. Experimental results, with both Gaussian kernel and polynomial kernel, on real-world data sets and image segmentation tasks show the superiority of the proposed method over the state-of-the-art methods.

Bounding the difference between RankRC and RankSVM and application to multi-level rare class kernel ranking

Rapid explosion in data accumulation has yielded large scale data mining problems, many of which have intrinsically unbalanced or rare class distributions. Standard classification algorithms, which focus on overall classification accuracy, often perform poorly in these cases. Recently, Tayal et al. (IEEE Trans Knowl Data Eng 27(12):3347---3359, 2015) proposed a kernel method called RankRC for large-scale unbalanced learning. RankRC uses a ranking loss to overcome biases inherent in standard classification based loss functions, while achieving computational efficiency by enforcing a rare class hypothesis representation. In this paper we establish a theoretical bound for RankRC by establishing equivalence between instantiating a hypothesis using a subset of training points and instantiating a hypothesis using the full training set but with the feature mapping equal to the orthogonal projection of the original mapping. This bound suggests that it is optimal to select points from the rare class first when choosing the subset of data points for a hypothesis representation. In addition, we show that for an arbitrary loss function, the Nystrom kernel matrix approximation is equivalent to instantiating a hypothesis using a subset of data points. Consequently, a theoretical bound for the Nystrom kernel SVM can be established based on the perturbation analysis of the orthogonal projection in the feature mapping. This generally leads to a tighter bound in comparison to perturbation analysis based on kernel matrix approximation. To further illustrate computational effectiveness of RankRC, we apply a multi-level rare class kernel ranking method to the Heritage Health Provider Network's health prize competition problem and compare the performance of RankRC to other existing methods.

Pure endmember extraction using robust kernel archetypoid analysis for hyperspectral imagery

A robust kernel archetypoid analysis (RKADA) method is proposed to extract pure endmembers from hyperspectral imagery (HSI). The RKADA assumes that each pixel is a sparse linear mixture of all endmembers and each endmember corresponds to a real pixel in the image scene. First, it improves the re8gular archetypal analysis with a new binary sparse constraint, and the adoption of the kernel function constructs the principal convex hull in an infinite Hilbert space and enlarges the divergences between pairwise pixels. Second, the RKADA transfers the pure endmember extraction problem into an optimization problem by minimizing residual errors with the Huber loss function. The Huber loss function reduces the effects from big noises and outliers in the convergence procedure of RKADA and enhances the robustness of the optimization function. Third, the random kernel sinks for fast kernel matrix approximation and the two-stage algorithm for optimizing initial pure endmembers are utilized to improve its computational efficiency in realistic implementations of RKADA, respectively. The optimization equation of RKADA is solved by using the block coordinate descend scheme and the desired pure endmembers are finally obtained. Six state-of-the-art pure endmember extraction methods are employed to make comparisons with the RKADA on both synthetic and real Cuprite HSI datasets, including three geometrical algorithms vertex component analysis (VCA), alternative volume maximization (AVMAX) and orthogonal subspace projection (OSP), and three matrix factorization algorithms the preconditioning for successive projection algorithm (PreSPA), hierarchical clustering based on rank-two nonnegative matrix factorization (H2NMF) and self-dictionary multiple measurement vector (SDMMV). Experimental results show that the RKADA outperforms all the six methods in terms of spectral angle distance (SAD) and root-mean-square-error (RMSE). Moreover, the RKADA has short computational times in offline operations and shows significant improvement in identifying pure endmembers for ground objects with smaller spectrum differences. Therefore, the RKADA could be an alternative for pure endmember extraction from hyperspectral images.

Scalable Gaussian Kernel Support Vector Machines with Sublinear Training Time Complexity

Gaussian kernel Support Vector Machines (SVMs) deliver state-of-the-art generalization performance for non-linear classification, but the time complexity of their training process is at least quadratic w.r.t. the size of training set, preventing them from scaling up on large datasets. To address this issue, we propose a novel approach to large-scale kernel SVMs in sublinear time w.r.t. the size of training set, which combines three well-known and efficient techniques with theoretical guarantees. First, we subsample massive samples to reduce the sample size. Then, we use the random Fourier feature mapping on the subsamples to construct explicit random feature space where we can train a linear SVM to approximate the corresponding Gaussian kernel SVM. Finally, we use parallel algorithms to make our approach more scalable. Deriving the upper bounds of kernel matrix approximation error, hypothesis error and excess risk w.r.t. the size of training set and the dimension of random feature space, we establish the theoretical foundation of our proposed approach. In this way, we can reduce the time complexity of training kernel SVMs without sacrificing much accuracy. Our proposed approach achieves high accuracy and has sublinear training time complexity, which exhibits good scalability theoretically and empirically.

Predictive Nyström method for kernel methods

Nyström method is a widely used matrix approximation method for scaling up kernel methods, and existing sampling strategies for Nyström method are proposed to improve the matrix approximation accuracy, but leaving approximation independent of learning, which can result in poor predictive performance of kernel methods. In this paper, we propose a novel predictive sampling strategy (PRESS) for Nyström method that guarantees the predictive performance of kernel methods. PRESS adaptively updates the sampling distribution via the discrepancy between approximate and accurate solutions of kernel methods caused by kernel matrix approximation, and samples informative columns from the kernel matrix according to the sampling distribution to reduce the predictive performance loss of kernel methods. We prove upper error bounds on the approximate solutions of kernel methods produced by Nyström method with PRESS, whose convergence shows that approximate solutions of kernel methods are identical to accurate ones for large enough samples. Experimental results indicate that integrating learning into approximation is necessary for delivering better predictive performance, and PRESS significantly outperforms existing sampling strategies while preserving low computational cost.

An Approximate Approach to Automatic Kernel Selection

Kernel selection is a fundamental problem of kernel-based learning algorithms. In this paper, we propose an approximate approach to automatic kernel selection for regression from the perspective of kernel matrix approximation. We first introduce multilevel circulant matrices into automatic kernel selection, and develop two approximate kernel selection algorithms by exploiting the computational virtues of multilevel circulant matrices. The complexity of the proposed algorithms is quasi-linear in the number of data points. Then, we prove an approximation error bound to measure the effect of the approximation in kernel matrices by multilevel circulant matrices on the hypothesis and further show that the approximate hypothesis produced with multilevel circulant matrices converges to the accurate hypothesis produced with kernel matrices. Experimental evaluations on benchmark datasets demonstrate the effectiveness of approximate kernel selection.

A robust non-rigid point set registration method based on asymmetric gaussian representation

Point set registration problem confronts with the challenge of large degree of degradations, such as deformation, noise, occlusion and outlier. In this paper, we present a novel robust method for non-rigid point set registration, and it includes four important parts are as follows: First, we used a mixture of asymmetric Gaussian model (MoAG) Kato et al. (2002) [1], a new probability model which can capture spatially asymmetric distributions, to represent each point set. Second, based on the representation of point set by MoAG, we used soft assignment technique to recover the correspondences, and correlation-based method to estimate the transformation parameters between two point sets. Point set registration is formulated as an optimization problem. Third, we solved the optimization problem under regularization theory in a feature space, i.e., Reproducing Kernel Hilbert Space (RKHS). Finally, we chose control points to build a kernel using low-rank kernel matrix approximation. Thus the computational complexity can be reduced down to O(N) approximately. Experimental results on 2D, 3D non-rigid point set, and real image registration demonstrate that our method is robust to a large degree of degradations, and it outperforms several state-of-the-art methods in most tested scenarios.

Sparse Representation With Kernels

Recent research has shown the initial success of sparse coding (Sc) in solving many computer vision tasks. Motivated by the fact that kernel trick can capture the nonlinear similarity of features, which helps in finding a sparse representation of nonlinear features, we propose kernel sparse representation (KSR). Essentially, KSR is a sparse coding technique in a high dimensional feature space mapped by an implicit mapping function. We apply KSR to feature coding in image classification, face recognition, and kernel matrix approximation. More specifically, by incorporating KSR into spatial pyramid matching (SPM), we develop KSRSPM, which achieves a good performance for image classification. Moreover, KSR-based feature coding can be shown as a generalization of efficient match kernel and an extension of Sc-based SPM. We further show that our proposed KSR using a histogram intersection kernel (HIK) can be considered a soft assignment extension of HIK-based feature quantization in the feature coding process. Besides feature coding, comparing with sparse coding, KSR can learn more discriminative sparse codes and achieve higher accuracy for face recognition. Moreover, KSR can also be applied to kernel matrix approximation in large scale learning tasks, and it demonstrates its robustness to kernel matrix approximation, especially when a small fraction of the data is used. Extensive experimental results demonstrate promising results of KSR in image classification, face recognition, and kernel matrix approximation. All these applications prove the effectiveness of KSR in computer vision and machine learning tasks.

Parameter Tuning via Kernel Matrix Approximation for Support Vector Machine

Parameter tuning is essential to generalization of support vector machine (SVM). Previous methods usually adopt a nested two-layer framework, where the inner layer solves a convex optimization problem, and the outer layer selects the hyper-parameters by minimizing either cross validation or other error bounds. In this paper, we propose a novel parameter tuning approach for SVM via kernel matrix approximation, based on the observation that approximate computation is sufficient for parameter tuning. We first develop a preliminary approximate computation theory of parameter tuning for SVM. We present a kernel matrix approximation algorithm MoCIC. We design an approximate parameter tuning algorithm APT, which applies MoCIC to compute a low-dimension and low-rank approximation of the kernel matrix, and uses this approximate matrix to efficiently solve the quadratic programming of SVM, then selects the optimal candidate parameter through the approximate cross validation error (ACVE). Finally, we verify and compare the feasibility and efficiency of APT on 10 artificial and benchmark datasets. Experimental results show that this new algorithm can dramatically reduce time consumption of parameter tuning and at the same time guarantee the effectiveness of the selected parameters. It comes to the conclusion that the approximate parameter tuning approach is sound, efficient, and promising.

On Learning and Cross-Validation with Decomposed Nyström Approximation of Kernel Matrix

The high computational costs of training kernel methods to solve nonlinear tasks limits their applicability. However, recently several fast training methods have been introduced for solving linear learning tasks. These can be used to solve nonlinear tasks by mapping the input data nonlinearly to a low-dimensional feature space. In this work, we consider the mapping induced by decomposing the Nystrom approximation of the kernel matrix. We collect together prior results and derive new ones to show how to efficiently train, make predictions with and do cross-validation for reduced set approximations of learning algorithms, given an efficient linear solver. Specifically, we present an efficient method for removing basis vectors from the mapping, which we show to be important when performing cross-validation.

Development of a personal-computer-based intelligent tutoring system

Gaussian kernel Support Vector Machines (SVMs) deliver state-of-the-art generalization performance for non-linear classification, but the time complexity of their training process is at least quadratic w.r.t. the size of training set, preventing them from scaling up on large datasets. To address this issue, we propose a novel approach to large-scale kernel SVMs in sublinear time w.r.t. the size of training set, which combines three well-known and efficient techniques with theoretical guarantees. First, we subsample massive samples to reduce the sample size. Then, we use the random Fourier feature mapping on the subsamples to construct explicit random feature space where we can train a linear SVM to approximate the corresponding Gaussian kernel SVM. Finally, we use parallel algorithms to make our approach more scalable. Deriving the upper bounds of kernel matrix approximation error, hypothesis error and excess risk w.r.t. the size of training set and the dimension of random feature space, we establish the theoretical foundation of our proposed approach. In this way, we can reduce the time complexity of training kernel SVMs without sacrificing much accuracy. Our proposed approach achieves high accuracy and has sublinear training time complexity, which exhibits good scalability theoretically and empirically.