Kernel-based Dimensionality Reduction Research Articles

Palm-print recognition based on quality estimation and feature dimension

Human identification exploitation biometric traits are more and more in style in recent years. Among the widely used biometric traits, palm-print is a vital one because of its acquisition convenience and comparatively high recognition results. The paper proposes a palm-print recognition system based on quality estimation and feature dimensions. Initially, a quality assessment is applied on the extracted region of interest (ROI) images. Gabor filter is employed to extract the palm-print features having various scales and orientations. The kernel-based dimensionality reduction is applied in the full space that reduces the high-dimensional Gabor features. The experiments are conducted on the PolyU, IIT-Delhi and CASIA palm-print databases. The best recognition performance in terms of an equal error rate (EER) of 0.051% and recognition rate (RR) of 98.34% was achieved on PolyU database. Experimental results prove the effectiveness of the proposed approach.

Class mean vector component and discriminant analysis

• We propose a component analysis method for kernel-based subspace learning. • We give extensive analysis of the method and its connection with KPCA and KDA. • A discriminant analysis method is proposed based on properties of KDA projections. The kernel matrix used in kernel methods encodes all the information required for solving complex nonlinear problems defined on data representations in the input space using simple, but implicitly defined, solutions. Spectral analysis on the kernel matrix defines an explicit nonlinear mapping of the input data representations to a subspace of the kernel space, which can be used for directly applying linear methods. However, the selection of the kernel subspace is crucial for the performance of the proceeding processing steps. We propose a new optimization criterion , leading to a new component analysis method for kernel-based dimensionality reduction that optimally preserves the pair-wise distances of the class means in the feature space. This leads to efficient kernel subspace learning, which is crucial for kernel-based machine learning solutions. We provide extensive analysis on the connections and differences between the proposed criterion and the criteria used in kernel Principal Component Analysis, kernel Entropy Component Analysis and Kernel Discriminant Analysis, leading to a discriminant analysis version of the proposed method. Our theoretical analysis also provides more insights on the properties of the feature spaces obtained by applying these methods. Results on a variety of visual classification problems illustrate the properties of the proposed methods.

Kernel methods for detecting coherent structures in dynamical data.

We illustrate relationships between classical kernel-based dimensionality reduction techniques and eigendecompositions of empirical estimates of reproducing kernel Hilbert space operators associated with dynamical systems. In particular, we show that kernel canonical correlation analysis (CCA) can be interpreted in terms of kernel transfer operators and that it can be obtained by optimizing the variational approach for Markov processes score. As a result, we show that coherent sets of particle trajectories can be computed by kernel CCA. We demonstrate the efficiency of this approach with several examples, namely, the well-known Bickley jet, ocean drifter data, and a molecular dynamics problem with a time-dependent potential. Finally, we propose a straightforward generalization of dynamic mode decomposition called coherent mode decomposition. Our results provide a generic machine learning approach to the computation of coherent sets with an objective score that can be used for cross-validation and the comparison of different methods.

Multiple Kernel Dimensionality Reduction via Ratio-Trace and Marginal Fisher Analysis

Traditional supervised multiple kernel learning (MKL) for dimensionality reduction is generally an extension of kernel discriminant analysis (KDA), which has some restrictive assumptions. In addition, they generally are based on graph embedding framework. A more general multiple kernel-based dimensionality reduction algorithm, called multiple kernel marginal Fisher analysis (MKL-MFA), is presented for supervised nonlinear dimensionality reduction combined with ratio-race optimization problem. MKL-MFA aims at relaxing the restrictive assumption that the data of each class is of a Gaussian distribution and finding an appropriate convex combination of several base kernels. To improve the efficiency of multiple kernel dimensionality reduction, the spectral regression frameworks are incorporated into the optimization model. Furthermore, the optimal weights of predefined base kernels can be obtained by solving a different convex optimization. Experimental results on benchmark datasets demonstrate that MKL-MFA outperforms the state-of-the-art supervised multiple kernel dimensionality reduction methods.

Kernel-based dimensionality reduction using Renyi's α-entropy measures of similarity

Dimensionality reduction (DR) aims to reveal salient properties of high-dimensional (HD) data in a low-dimensional (LD) representation space. Two elements stipulate success of a DR approach: definition of a notion of pairwise relations in the HD and LD spaces, and measuring the mismatch between these relationships in the HD and LD representations of data. This paper introduces a new DR method, termed Kernel-based entropy dimensionality reduction (KEDR), to measure the embedding quality that is based on stochastic neighborhood preservation, involving a Gram matrix estimation of Renyi's α-entropy. The proposed approach is a data-driven framework for information theoretic learning, based on infinitely divisible matrices. Instead of relying upon regular Renyi's entropies, KEDR also computes the embedding mismatch through a parameterized mixture of divergences, resulting in an improved the preservation of both the local and global data structures. Our approach is validated on both synthetic and real-world datasets and compared to several state-of-the-art algorithms, including the Stochastic Neighbor Embedding-like techniques for which DR approach is a data-driven extension (from the perspective of kernel-based Gram matrices). In terms of visual inspection and quantitative evaluation of neighborhood preservation, the obtained results show that KEDR is competitive and promising DR method.

Face recognition using supervised probabilistic principal component analysis mixture model in dimensionality reduction without loss framework

In this study, first a supervised version for probabilistic principal component analysis mixture model is proposed. Using this model, local linear underlying manifolds of data samples are obtained. These underlying manifolds are used in a dimensionality reduction without loss framework, for face recognition application. In this framework, the benefits of dimensionality reduction are used in the predictive model, while using the projection penalty idea, the loss of useful information will be minimised. The authors use support vector machine (SVM) and k‐nearest neighbour (KNN) classifiers as the predictive models in this framework. To train and evaluate the proposed method, the well‐known face databases are used. The experimental results show that the proposed method with SVM as the predictive model have the most average classification accuracy compared with many traditional methods which use predictive model SVM after dimensionality reduction, and also compared with the projection penalty idea used for linear and non‐linear kernel‐based dimensionality reduction methods. Moreover, their experiments show that the proposed method with KNN as predictive model is superior to the case that dimensionality reduction is performed, and then the KNN classifier is applied.

A novel dimensionality reduction technique based on kernel optimization through graph embedding

In this paper, we propose a new method for kernel optimization in kernel-based dimensionality reduction techniques such as kernel principal component analysis and kernel discriminant analysis. The main idea is to use the graph embedding framework for these techniques and, therefore, by formulating a new minimization problem to simultaneously optimize the kernel parameters and the projection vectors of the chosen dimensionality reduction method. Experimental results are conducted in various datasets, varying from real-world publicly available databases for classification benchmarking to facial expressions and face recognition databases. Our proposed method outperforms other competing ones in classification performance. Moreover, our method provides a systematic way to deal with kernel parameters whose calculation was treated rather superficially so far and/or experimentally, in most of the cases.

Multiple kernel dimensionality reduction via spectral regression and trace ratio maximization

The performance of kernel-based dimensionality reduction heavily relies on the selection of kernel functions. Multiple kernel learning for dimensionality reduction (MKL-DR) has been recently proposed to learn a convex combination from a set of base kernels. But this method relaxes a nonconvex quadratically constrained quadratic programming (QCQP) problem into a semi-definite programming (SDP) problem to specify the kernel weights, which might lead to its performance degradation. Although a trace ratio maximization approach to multiple-kernel based dimensionality reduction (MKL-TR) has been presented to avoid convex relaxation, it has to compute a generalized eigenvalue problem in each iteration of its algorithm, which is expensive in both time and memory. To improve the performance of these methods further, this paper proposes a novel multiple kernel dimensionality reduction method by virtue of spectral regression and trace ratio maximization, termed as MKL-SRTR. The proposed approach aims at learning an appropriate kernel from the multiple base kernels and a transformation into a lower dimensionality space efficiently and effectively. The experimental results demonstrate the effectiveness of the proposed method in benchmark datasets for supervised, unsupervised as well as semi-supervised scenarios.

Kernel optimization using nonparametric Fisher criterion in the subspace

Kernel optimization plays an important role in kernel-based dimensionality reduction algorithms, such as kernel principal components analysis (KPCA) and kernel discriminant analysis (KDA). In this paper, a nonparametric Fisher criterion is proposed as the objective function to find the optimized kernel parameters. Unlike other criterions that rooted in the kernel feature space, the proposed criterion works in the low-dimensional subspace to measure the separability of different patterns. Experiments on 13 different benchmark datasets show the effectiveness of the proposed method, in comparison with other criterions and the kernel space methods.

Kernelized Bayesian Transfer Learning

Transfer learning considers related but distinct tasks defined on heterogenous domains and tries to transfer knowledge between these tasks to improve generalization performance. It is particularly useful when we do not have sufficient amount of labeled training data in some tasks, which may be very costly, laborious, or even infeasible to obtain. Instead, learning the tasks jointly enables us to effectively increase the amount of labeled training data. In this paper, we formulate a kernelized Bayesian transfer learning framework that is a principled combination of kernel-based dimensionality reduction models with task-specific projection matrices to find a shared subspace and a coupled classification model for all of the tasks in this subspace. Our two main contributions are: (i) two novel probabilistic models for binary and multiclass classification, and (ii) very efficient variational approximation procedures for these models. We illustrate the generalization performance of our algorithms on two different applications. In computer vision experiments, our method outperforms the state-of-the-art algorithms on nine out of 12 benchmark supervised domain adaptation experiments defined on two object recognition data sets. In cancer biology experiments, we use our algorithm to predict mutation status of important cancer genes from gene expression profiles using two distinct cancer populations, namely, patient-derived primary tumor data and in-vitro-derived cancer cell line data. We show that we can increase our generalization performance on primary tumors using cell lines as an auxiliary data source.

A framework for optimal kernel-based manifold embedding of medical image data

Kernel-based dimensionality reduction is a widely used technique in medical image analysis. To fully unravel the underlying nonlinear manifold the selection of an adequate kernel function and of its free parameters is critical. In practice, however, the kernel function is generally chosen as Gaussian or polynomial and such standard kernels might not always be optimal for a given image dataset or application. In this paper, we present a study on the effect of the kernel functions in nonlinear manifold embedding of medical image data. To this end, we first carry out a literature review on existing advanced kernels developed in the statistics, machine learning, and signal processing communities. In addition, we implement kernel-based formulations of well-known nonlinear dimensional reduction techniques such as Isomap and Locally Linear Embedding, thus obtaining a unified framework for manifold embedding using kernels. Subsequently, we present a method to automatically choose a kernel function and its associated parameters from a pool of kernel candidates, with the aim to generate the most optimal manifold embeddings. Furthermore, we show how the calculated selection measures can be extended to take into account the spatial relationships in images, or used to combine several kernels to further improve the embedding results. Experiments are then carried out on various synthetic and phantom datasets for numerical assessment of the methods. Furthermore, the workflow is applied to real data that include brain manifolds and multispectral images to demonstrate the importance of the kernel selection in the analysis of high-dimensional medical images.

A trace ratio maximization approach to multiple kernel-based dimensionality reduction

Most dimensionality reduction techniques are based on one metric or one kernel, hence it is necessary to select an appropriate kernel for kernel-based dimensionality reduction. Multiple kernel learning for dimensionality reduction (MKL-DR) has been recently proposed to learn a kernel from a set of base kernels which are seen as different descriptions of data. As MKL-DR does not involve regularization, it might be ill-posed under some conditions and consequently its applications are hindered. This paper proposes a multiple kernel learning framework for dimensionality reduction based on regularized trace ratio, termed as MKL-TR. Our method aims at learning a transformation into a space of lower dimension and a corresponding kernel from the given base kernels among which some may not be suitable for the given data. The solutions for the proposed framework can be found based on trace ratio maximization. The experimental results demonstrate its effectiveness in benchmark datasets, which include text, image and sound datasets, for supervised, unsupervised as well as semi-supervised settings.

Kernel Sparse Representation-Based Classifier

Sparse representation-based classifier (SRC), a combined result of machine learning and compressed sensing, shows its good classification performance on face image data. However, SRC could not well classify the data with the same direction distribution. The same direction distribution means that the sample vectors belonging to different classes distribute on the same vector direction. This paper presents a new classifier, kernel sparse representation-based classifier (KSRC), based on SRC and the kernel trick which is a usual technique in machine learning. KSRC is a nonlinear extension of SRC and can remedy the drawback of SRC. To make the data in an input space separable, we implicitly map these data into a high-dimensional kernel feature space by using some nonlinear mapping associated with a kernel function. Since this kernel feature space has a very high (or possibly infinite) dimensionality, or is unknown, we have to avoid working in this space explicitly. Fortunately, we can indeed reduce the dimensionality of the kernel feature space by exploiting kernel-based dimensionality reduction methods. In the reduced subspace, we need to find sparse combination coefficients for a test sample and assign a class label to it. Similar to SRC, KSRC is also cast into an ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -minimization problem or a quadratically constrained ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -minimization problem. Extensive experimental results on UCI and face data sets show KSRC improves the performance of SRC.