Bi-sparse optimization-based least squares regression

Abstract

For forecasting by regression, more and more instances and features are collected and added to the regression models. When there are many noisy and redundant instances and features, these models often give the poor predictive accuracy and interpretability owing to overfitting and computational complexity. Besides, least squares support vector regression (LSSVR) can hardly obtain sparse solutions and identify important instances and features from data. In this paper, a novel bi-sparse optimization-based least squares regression (BSOLSR) method is proposed in the framework of LSSVR. Based on the new row and column kernel matrices, the ℓ0-norm sparsification function is introduced to the LSSVR model. By alternatively solving two unconstrained quadratic programming problems or two systems of linear equations, BSOLSR can predict output values for given input points and provide interpretable results by simultaneous selecting relevant and important instances and features. As shown in the experimental results on real data sets and comparison with SVR, ℓ1-norm SVR (L1SVR), LSSVR, and multiple kernel learning SVR (MKLSVR), the proposed BSOLSR can effectively increase predictive accuracy, discover representative instances and important features, and gain the interpretable results, which are very critical for many real-world applications.