Incremental relevance sample-feature machine: A fast marginal likelihood maximization approach for joint feature selection and classification

Abstract

The recently proposed Relevance Sample-Feature Machine (RSFM) performs joint feature selection and classification with state-of-the-art performance in terms of accuracy and sparsity. However, it suffers from high computational cost for large training sets. To accelerate its training procedure, we introduce a new variant of this algorithm named Incremental Relevance Sample-Feature Machine (IRSFM). In IRSFM, the marginal likelihood maximization approach is changed such that the model learning follows a constructive procedure (starting with an empty model, it iteratively adds or omits basis functions to construct the learned model). Our extensive experiments on various data sets and comparison with various competing algorithms demonstrate the effectiveness of the proposed IRSFM in terms of accuracy, sparsity and run-time. While the IRSFM achieves almost the same classification accuracy as the RSFM, it benefits from sparser learned model both in sample and feature domains and much less training time than RSFM especially for large data sets.