A globally convergent approximate Newton method for non-convex sparse learning

Abstract

Newton-type greedy pursuit methods have been shown to work favorably for cardinality-constrained sparse learning problems. The appealing sparsity recovery performance of the existing Newton-type greedy pursuit methods, however, is typically guaranteed within a local neighborhood around the target solution. To address this limitation, we present in this paper a novel approximate Newton pursuit method for sparse learning with linear models. The computation procedure of our method iterates between constructing an inexact Newton-type quadratic majorization to the global empirical risk and solving the quadratic approximation via iterative hard thresholding. Provable global guarantees on mean squared prediction error, which is less understood for prior methods, are provided for our method. Numerical evidence is provided to show the advantages of our approach over the prior methods.