Simultaneous feature selection and clustering based on square root optimization

Abstract

The fused least absolute shrinkage and selection operator (LASSO) simultaneously pursuing the joint sparsity of coefficients and their successive differences has attracted significant attention for analytics purposes. Although it is extensively used, especially when the number of features exceeds the sample size, tuning the regularization parameters, which depends on noise level σ, is a challenging task since σ is difficult to estimate accurately. To tackle this problem, in this paper, we propose and study square root fused LASSO, which combines the square root loss function and joint penalty functions. In theory, we show that the proposed method can achieve the same error rate as that of fused LASSO by proving its estimation and prediction error bounds. In addition, the error rate of square root fused LASSO is lower than those of LASSO and square root LASSO via simultaneous feature selection and clustering. The choices of the regularization parameters are also shown to be free of σ. In terms of computation, this work develops a novel algorithm based on the alternating direction method of multipliers algorithm with theoretical guarantee of its convergence. Experiments on simulation and real-world datasets demonstrate the superiority of square root fused LASSO over fused LASSO and other state-of-the-art feature selection methods.