Accelerated Double-Sketching Subspace Newton

Abstract

This paper proposes a second-order stochastic algorithm called Accelerated Double-Sketching Subspace Newton (ADSSN) to solve large-scale optimization problems with high dimensional feature spaces and substantial sample sizes. The proposed ADSSN has two computational superiority. First, ADSSN achieves a fast local convergence rate by exploiting Nesterov’s acceleration technique. Second, by taking full advantage of the double sketching strategy, ADSSN provides a lower computational cost for each iteration than competitive approaches. Moreover, these advantages hold for actually all sketching techniques, which enables practitioners to design custom sketching methods for specific applications. Finally, numerical experiments are carried out to demonstrate the efficiency of ADSSN compared with accelerated gradient descent and two single sketching counterparts.