Kernel Subspace Learning Research Articles

Multiple Kernel Subspace Learning for Clustering and Classification

In the face of high-dimensional and complex data, effective subspace can preserve specific statistical properties and provide an appropriate representation of data, which generally facilitates the underlying tasks such as clustering or classification. Meanwhile, multiple kernel learning is a technique to combine multiple kernels from different feature spaces effectively. Thus, by incorporating multiple kernels into the process of subspace learning, different feature spaces can be projected into a unified subspace. This paper proposes the Multiple Kernel Subspace Learning (MKSL) for embedding the original space into a unified subspace. Multiple kernels of different feature spaces are combined by MKSL in the process of learning, which can extend the suitability for various applications. Moreover, to generate the optimal combination kernel of subspace learning, we propose a two-step iteration strategy to learn the appropriate kernel weights and transformation matrix of projecting simultaneously. Furthermore, our proposed formulation of MKSL can introduce different prior knowledge such as class information and neighborhood relationships. Thus it is competent to the unsupervised learning, semi-supervised learning, and supervised learning. Extensive experiments are conducted on diverse datasets, and the performances are comprehensively evaluated on different tasks. The experimental results indicate that the proposed algorithm is outstanding in unsupervised clustering task and effective in supervised and semi-supervised classification tasks.

Class mean vector component and discriminant analysis

• We propose a component analysis method for kernel-based subspace learning. • We give extensive analysis of the method and its connection with KPCA and KDA. • A discriminant analysis method is proposed based on properties of KDA projections. The kernel matrix used in kernel methods encodes all the information required for solving complex nonlinear problems defined on data representations in the input space using simple, but implicitly defined, solutions. Spectral analysis on the kernel matrix defines an explicit nonlinear mapping of the input data representations to a subspace of the kernel space, which can be used for directly applying linear methods. However, the selection of the kernel subspace is crucial for the performance of the proceeding processing steps. We propose a new optimization criterion , leading to a new component analysis method for kernel-based dimensionality reduction that optimally preserves the pair-wise distances of the class means in the feature space. This leads to efficient kernel subspace learning, which is crucial for kernel-based machine learning solutions. We provide extensive analysis on the connections and differences between the proposed criterion and the criteria used in kernel Principal Component Analysis, kernel Entropy Component Analysis and Kernel Discriminant Analysis, leading to a discriminant analysis version of the proposed method. Our theoretical analysis also provides more insights on the properties of the feature spaces obtained by applying these methods. Results on a variety of visual classification problems illustrate the properties of the proposed methods.

Structured Kernel Subspace Learning for Autonomous Robot Navigation

This paper considers two important problems for autonomous robot navigation in a dynamic environment, where the goal is to predict pedestrian motion and control a robot with the prediction for safe navigation. While there are several methods for predicting the motion of a pedestrian and controlling a robot to avoid incoming pedestrians, it is still difficult to safely navigate in a dynamic environment due to challenges, such as the varying quality and complexity of training data with unwanted noises. This paper addresses these challenges simultaneously by proposing a robust kernel subspace learning algorithm based on the recent advances in nuclear-norm and -norm minimization. We model the motion of a pedestrian and the robot controller using Gaussian processes. The proposed method efficiently approximates a kernel matrix used in Gaussian process regression by learning low-rank structured matrix (with symmetric positive semi-definiteness) to find an orthogonal basis, which eliminates the effects of erroneous and inconsistent data. Based on structured kernel subspace learning, we propose a robust motion model and motion controller for safe navigation in dynamic environments. We evaluate the proposed robust kernel learning in various tasks, including regression, motion prediction, and motion control problems, and demonstrate that the proposed learning-based systems are robust against outliers and outperform existing regression and navigation methods.

A Two-Dimensional Framework of Multiple Kernel Subspace Learning for Recognizing Emotion in Speech

As a highly active topic in computational paralinguistics, speech emotion recognition (SER) aims to explore ideal representations for emotional factors in speech. In order to improve the performance of SER, multiple kernel learning (MKL) dimensionality reduction has been utilized to obtain effective information for recognizing emotions. However, the solution of MKL usually provides only one nonnegative mapping direction for multiple kernels; this may lead to loss of valuable information. To address this issue, we propose a two-dimensional framework for multiple kernel subspace learning. This framework provides more linear combinations on the basis of MKL without nonnegative constraints, which preserves more information in the learning procedures. It also leverages both of MKL and two-dimensional subspace learning, combining them into a unified structure. To apply the framework to SER, we also propose an algorithm, namely generalised multiple kernel discriminant analysis (GMKDA), by employing discriminant embedding graphs in this framework. GMKDA takes advantage of the additional mapping directions for multiple kernels in the proposed framework. In order to evaluate the performance of the proposed algorithm a wide range of experiments is carried out on several key emotional corpora. These experimental results demonstrate that the proposed methods can achieve better performance compared with some conventional and subspace learning methods in dealing with SER.

Nyström-based approximate kernel subspace learning

In this paper, we describe a method for the determination of a subspace of the feature space in kernel methods, which is suited to large-scale learning problems. Linear model learning in the obtained space corresponds to a nonlinear model learning process in the input space. Since the obtained feature space is determined only by exploiting properties of the training data, this approach can be used for generic nonlinear pattern recognition. That is, nonlinear data mapping can be considered to be a pre-processing step exploiting nonlinear relationships between the training data. Linear techniques can be subsequently applied in the new feature space and, thus, they can model nonlinear properties of the problem at hand. In order to appropriately address the inherent problem of kernel learning methods related to their time and memory complexities, we follow an approximate learning approach. We show that the method can lead to considerable operation speed gains and achieve very good performance. Experimental results verify our analysis.

Kernel propagation strategy: A novel out-of-sample propagation projection for subspace learning

Kernel matrix optimization (KMO) aims at learning appropriate kernel matrices by solving a certain optimization problem rather than using empirical kernel functions. Since KMO is difficult to compute out-of-sample projections for kernel subspace learning, we propose a kernel propagation strategy (KPS) based on data distribution similar principle to effectively extract out-of-sample low-dimensional features for subspace learning with KMO. With KPS, we further present an example algorithm, i.e., kernel propagation canonical correlation analysis (KPCCA), which naturally fuses semi-supervised kernel matrix learning and canonical correlation analysis by means of kernel propagation projections. In KPCCA, the extracted correlation features of out-of-sample data not only incorporate integral data distribution information but also supervised information. Extensive experimental results have demonstrated the superior performance of our proposed method.

Large-scale image recognition based on parallel kernel supervised and semi-supervised subspace learning

Kernel discriminant subspace learning technique is effective to exploit the structure of image dataset in the high-dimensional nonlinear space. However, for large-scale image recognition applications, this technique usually suffers from large computational burden. Although some kernel accelerating methods have been presented, how to greatly reduce computing time and simultaneously keep favorable recognition accuracy is still challenging. In this paper, we introduce the idea of parallel computing into kernel subspace learning and build a parallel kernel discriminant subspace learning framework. In this framework, we firstly design a random non-overlapping equal data division strategy to divide the whole training set into several subsets and assign each computational node a subset. Then, we separately learn kernel discriminant subspaces from these subsets without mutual communications and finally select the most appropriate subspace to classify test samples. Under the built framework, we propose two novel kernel subspace learning approaches, i.e., parallel kernel discriminant analysis (PKDA) and parallel kernel semi-supervised discriminant analysis (PKSDA). We show the superiority of the proposed approaches in terms of time complexity as compared with related methods, and provide the fundamental supports for our framework. For experiment, we establish a parallel computing environment and employ three public large-scale image databases as experiment data. Experimental results demonstrate the efficiency and effectiveness of the proposed approaches.

FACE REPRESENTATION WITH GRADIENT ORIENTATIONS AND EULER MAPPING: APPLICATION TO FACE RECOGNITION

This paper proposes a simple, yet very powerful local face representation, called the Gradient Orientations and Euler Mapping (GOEM). GOEM consists of two stages: gradient orientations and Euler mapping. In the first stage, we calculate gradient orientations of a central pixel and get the corresponding orientation representations by performing convolution operator. These representation results display spatial locality and orientation properties. To encompass different spatial localities and orientations, we concatenate all these representation results and derive a concatenated orientation feature vector. In the second stage, we define an explicit Euler mapping which maps the space of the concatenated orientation into a complex space. For a mapping image, we find that the imaginary part and the real part characterize the high frequency and the low frequency components, respectively. To encompass different frequencies, we concatenate the imaginary part and the real part and derive a concatenated mapping feature vector. For a given image, we use the two stages to construct a GOEM image and derive an augmented feature vector which resides in a space of very high dimensionality. In order to derive low-dimensional feature vector, we present a class of GOEM-based kernel subspace learning methods for face recognition. These methods, which are robust to changes in occlusion and illumination, apply the kernel subspace learning model with explicit Euler mapping to an augmented feature vector derived from the GOEM representation of face images. Experimental results show that our methods significantly outperform popular methods and achieve state-of-the-art performance for difficult problems such as illumination and occlusion-robust face recognition.