A Contracting BFGS Update in Quasi-Newton Methods for Unconstrained Optimization

Abstract

Unconstrained optimization problems, arise in many practical applications. Especially, significant improvement in deep learning training came from the Quasi-Newton methods. They exploit the idea of building up curvature information as the iterations of the training method are progressing. BFGS update in Quasi-Newton methods is the most commonly used update rule for training deep neural networks. The accuracy of computed search direction depends largely on how sensitive the Hessian approximation matrix Bk+1 is to small changes. The larger distribution of eigenvalues of the matrix will cause more sensation. In this paper, we propose a contracting BFGS update (C-BFGS), in order to contract the interval of distribution. The new update retains the Hessian approximation matrix positive definiteness, so that it makes sure the search direction down. For a quadratic positive definite function, the search directions generated by C-BFGS under the exact line search are Gconjugate, and the Quasi-Newton method with C-BFGS update satisfies quadratic termination property.