Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

Abstract

AbstractWe propose a new first-order method for minimizing nonconvex functions with Lipschitz continuous gradients and Hölder continuous Hessians. The proposed algorithm is a heavy-ball method equipped with two particular restart mechanisms. It finds a solution where the gradient norm is less than $$\varepsilon $$ ε in $$O(H_{\nu }^{\frac{1}{2 + 2 \nu }} \varepsilon ^{- \frac{4 + 3 \nu }{2 + 2 \nu }})$$ O ( H ν 1 2 + 2 ν ε - 4 + 3 ν 2 + 2 ν ) function and gradient evaluations, where $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] and $$H_{\nu }$$ H ν are the Hölder exponent and constant, respectively. This complexity result covers the classical bound of $$O(\varepsilon ^{-2})$$ O ( ε - 2 ) for $$\nu = 0$$ ν = 0 and the state-of-the-art bound of $$O(\varepsilon ^{-7/4})$$ O ( ε - 7 / 4 ) for $$\nu = 1$$ ν = 1 . Our algorithm is $$\nu $$ ν -independent and thus universal; it automatically achieves the above complexity bound with the optimal $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] without knowledge of $$H_{\nu }$$ H ν . In addition, the algorithm does not require other problem-dependent parameters as input, including the gradient’s Lipschitz constant or the target accuracy $$\varepsilon $$ ε . Numerical results illustrate that the proposed method is promising.