Implicit Runge-Kutta Methods for Accelerated Unconstrained Convex Optimization

Abstract

Accelerated gradient methods have the potential of achieving optimal convergence rates and have successfully been used in many practical applications. Despite this fact, the rationale underlying these accelerated methods remain elusive. In this work, we study gradient-based accelerated optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the Bregman Lagrangian. We show that for sufficiently smooth objectives, the acceleration can be achieved by discretizing the proposed ODE using $s$ -stage $q$ -order implicit Runge-Kutta integrators. In particular, we prove that under the assumption of convexity and sufficient smoothness, the sequence of iteration generated by the proposed accelerated method stably converges to the optimal solution at a rate of ${O}\left({\left({1-\tilde {C}_{p,q} \cdot \frac {\mu }{L}}\right)^{N}N^{-p}}\right)$ , where $p \geq 2$ is the parameter in the second-order ODE and $\tilde {C}_{p,q}$ is a constant depending on $p$ and $q$ . Several numerical experiments are given to verify the convergence results.