Sparse large-margin nearest neighbor embedding via greedy dyad functional optimization

Abstract

We consider the sparse subspace learning problem where the intrinsic subspace is assumed to be low-dimensional and formed by sparse basis vectors. Confined to a few sparse bases, projecting data to the learned subspace essentially has an effect of feature selection by taking a small number of the most salient features while suppressing the rest as noise. Unlike existing sparse dimensionality reduction methods, however, we exploit the class labels to impose maximal margin data separation in the subspace, which was previously shown to yield improved prediction accuracy often times in non-sparse models. We first formulate an optimization problem with constraints on the matrix rank and the sparseness of the basis vectors. Instead of computationally demanding gradient-based learning strategies used in previous large-margin embedding, we propose an efficient greedy functional optimization algorithm over the infinite set of the sparse dyadic products. Each iteration in the proposed algorithm, after some shifting operations, effectively reduces to the famous sparse eigenvalue problem, and can be solved quickly by the recent truncated power method. We demonstrate the improved prediction performance of the proposed approach on several image/text classification datasets, especially characterized by high-dimensional noisy data samples with small training sets.