A Feasible Direction Method for Optimization Problem with Orthogonal Constraint in Feature Selection

Abstract

Feature selection, as a fundamental component of building robust models, plays an important role in many machine learning and data mining tasks. Since acquiring labeled data is particularly expensive in both time and effort, unsupervised feature selection on unlabeled data has recently gained considerable attention. Without label information, unsupervised feature selection needs alternative criteria to define feature relevance. We propose a novel unsupervised feature selection model, which embeds feature selection into nonnegative spectral clustering. A tailored optimization algorithm based on Alternating Direction Method of Multipliers (ADMM) is designed to solve the proposed model. Many previous unsupervised feature selection methods used singular value decompose (SVD) to handle the subproblem with orthogonal constraint. Generally, the scale of the matrix in feature selection is significantly big, the computation of SVD will be very slow or even infeasible. To address this issue, we propose to use a feasible direction method to efficiently solve the subproblem with orthogonal constraint. The experimental study shows that we can obtain better performance compared with the state of the art methods.