Choose Your Path Wisely: Gradient Descent in a Bregman Distance Framework

Martin Benning,Matthias J Ehrhardt,Carola-Bibiane Schönlieb,Marta M Betcke

doi:10.1137/20m1357500

Abstract

We propose an extension of a special form of gradient descent---in the literature known as linearized Bregman iteration---to a larger class of nonconvex functions. We replace the classical (squared) two norm metric in the gradient descent setting with a generalized Bregman distance, based on a proper, convex, and lower semicontinuous function. The algorithm's global convergence is proven for functions that satisfy the Kurdyka--Łojasiewicz property. Examples illustrate that features of different scale are being introduced throughout the iteration, transitioning from coarse to fine. This coarse-to-fine approach with respect to scale allows us to recover solutions of nonconvex optimization problems that are superior to those obtained with conventional gradient descent, or even projected and proximal gradient descent. The effectiveness of the linearized Bregman iteration in combination with early stopping is illustrated for the applications of parallel magnetic resonance imaging, blind deconvolution, as well as image classification with neural networks.

Highlights

Nonconvex optimization methods are indispensable mathematical tools for a large variety of applications [62]
In Remark 4 we have seen an example for which the subgradients of R diverge, but the primal iterates still converge, just not to a critical point of E. This leaves us with two open questions: (1) could we prove convergence of the primal iterates without boundedness of the dual iterates and (2) would the limit be a critical point of some other energy? It might be possible to answer the first question by slightly modifying Definition A.11 and Lemma A.12 in the appendix, as well as Lemma 4.7 to accommodate the fact that the surrogate function is constant on the set of limiting points that only depends on the primal variable (which we denote by ω(u0) for convenience)
We have presented a generalization of gradient descent that allows the incorporation of nonsmooth Bregman distances and can be seen as an extension of the linearized Bregman iteration to nonconvex functions

Summary

Introduction

Nonconvex optimization methods are indispensable mathematical tools for a large variety of applications [62]. Throughout the last decade, there has been an increasing interest in first-order methods for nonconvex and nonsmooth objectives. We present a direct generalization of gradient descent, CHOOSE YOUR PATH WISELY first introduced in [10], where the usual squared two-norm metric that penalizes the gap of two subsequent iterates is being replaced by a potentially nonsmooth distance term. This distance term is given in form of a generalized Bregman distance [20, 22, 66], where the underlying function is proper, lower semicontinuous, and convex, but not necessarily smooth. In the more general case, the proposed method is a generalization of the so-called linearized Bregman iteration [33, 83, 25, 24] to nonconvex data fidelities

Objectives

Conclusion