An adaptive high order method for finding third-order critical points of nonconvex optimization

Abstract

Recently, the optimization methods for computing higher-order critical points of nonconvex problems attract growing research interest (Anandkumar Conference on Learning Theory 81-102, 2016), (Cartis Found Comput Math 18:1073-1107, 2018), (Cartis SIAM J Optim 30:513-541, 2020), (Chen Math Program 187:47-78, 2021) , as they are able to exclude the so-called degenerate saddle points and reach a solution with better quality. Despite theoretical developments in (Anandkumar Conference on Learning Theory 81-102, 2016), (Cartis Found Comput Math 18:1073-1107, 2018), (Cartis SIAM J Optim 30:513-541, 2020), (Chen Math Program 187:47-78, 2021) , the corresponding numerical experiments are missing. This paper proposes an implementable higher-order method, named adaptive high order method (AHOM), to find the third-order critical points. AHOM is achieved by solving an “easier” subproblem and incorporating the adaptive strategy of parameter-tuning in each iteration of the algorithm. The iteration complexity of the proposed method is established. Some preliminary numerical results are provided to show that AHOM can escape from the degenerate saddle points, where the second-order method could possibly get stuck.