Complexity analysis of inexact cubic-regularized primal-dual methods for finding second-order stationary points

Abstract

Motivated by recent developments of using cubic regularization to escape saddle points of unconstrained optimization, in this paper we explore its potential in pursuing second-order stationary points of nonconvex constrained optimization whose exact objective function information may be hard to obtain. We first propose an algorithmic framework of inexact cubic-regularized primal-dual methods for equality constrained optimization, named as ICPD. To update the primal variable at each iteration, we construct a cubic regularized model relying on inexact first- and second-order derivatives of the objective function together with information of constraint functions. By allowing an inexact solution to each subproblem under certain conditions, we establish the iteration complexity of ICPD to find an ϵ \epsilon -approximate first- and second-order stationary point, respectively. We then consider a stochastic variant of algorithm, stochastic cubic-regularized primal-dual algorithm (SCPD) for equality constrained optimization whose objective takes an expectation form. Through a proper sampling strategy to calculate stochastic gradients and Hessians, we address the oracle complexity of SCPD to reach approximate stationary points with high probability. We also investigate the behavior of the standard gradient descent when solving each subproblem with a random perturbation. We provide a detailed analysis on how to fulfill the required conditions on an inexact subproblem solution with high probability at each iteration. Additionally, we present an analysis of an adaptive variant of ICPD which updates penalty parameters dynamically and discuss the applicability of adaptive cubic regularization parameters. Finally, preliminary numerical results are reported to showcase the performances of our proposed algorithms.