Sparse kernel feature extraction via support vector learning

Abstract

‘Kernel’ principal component analysis (PCA) generalizes the standard PCA to its nonlinear counterpart while retaining the elegance by solving a kernel eigenvalue problem. Unfortunately, the fact that each component is spanned by every training patterns leads to computational problems for feature extractors, as well as large storage requirements especially for datasets such as images and gene expression data: a weakness for PCA as a completely non-parametric algorithm. Inspired by the sparse solutions obtained from support vector machine (SVM), this paper exploits the possibility to introduce the merits of SVM into kernel PCA. The geometric interpretation of PCA as estimating the best-fit ellipsoid provides a way to parameterize kernel PCA. The associated optimal ellipsoid turns out to be a variant of SVM. Instead of computing the principal axes from the sample covariance, the proposed method diagonalizes a parametric covariance in feature space, consisting of support vectors. The resulting expansion for each principal component is sparse in that only support objects have nonzero weights. Experiments show that the proposed sparse kernel PCA (SKPCA) provides similar features and classification performance to the full non-sparse kernel PCA.