Simultaneous Safe Feature and Sample Elimination for Sparse Support Vector Regression

Abstract

Sparse support vector regression (SSVR) is an effective regression technique. It has been successfully applied to many practical problems. However, it remains challenging to handle the large-scale problems. A nice property of SSVR is double sparsity in the sense that most irrelevant features and samples have no effect on the regressor. Inspired by it, we propose a simultaneous safe feature and sample screening rule based on the strong convexity of objective function (FSSR1) to accelerate SSVR, including the linear and nonlinear cases. Most inactive features and samples can be simultaneously discarded before training SSVR. Only one reduced SSVR (RSSVR) needs to be solved. Moreover, to further speed up RSSVR, the screening rule based on duality gap (FSSR2) is applied to continuously discard the remaining inactive variables during the reduced model training process. Combining the rule with the grid-search method, the framework of our final method (FSSR–SSVR) is to alternatively conduct FSSR1 and FSSR2, which leads to substantial savings in both the memory usage and the computational cost. There are two appealing advantages of our FSSR-SSVR: first, it is safe, i.e., the features and samples discarded by FSSR are guaranteed to be inactive; second, it has synergy effect (in other words, the results of the previous feature screening can improve the performance of the next sample screening, and vice versa). Furthermore, the stochastic dual coordinate ascent (SDCA) method is employed as an efficient solver. Experiments on three synthetic datasets and 11 real-world datasets demonstrate the efficiency of our method.