Lipschitz Gradient Research Articles

Forward-Backward Algorithm for Functions with Locally Lipschitz Gradient: Applications to Mean Field Games

Forward-Backward Algorithm for Functions with Locally Lipschitz Gradient: Applications to Mean Field Games

First-Order Algorithms Without Lipschitz Gradient: A Sequential Local Optimization Approach

Most first-order methods rely on the global Lipschitz continuity of the objective gradient, which fails to hold in many problems. This paper develops a sequential local optimization (SLO) framework for first-order algorithms to optimize problems without Lipschitz gradient. Operating on the assumption that the gradient is locally Lipschitz continuous over any compact set, SLO develops a careful scheme to control the distance between successive iterates. The proposed framework can easily adapt to the existing first-order methods, such as projected gradient descent (PGD), truncated gradient descent (TGD), and a parameter-free variant of Armijo linesearch. We show that SLO requires [Formula: see text] gradient evaluations to find an ϵ-stationary point, where Y is certain compact set with [Formula: see text] radius, and [Formula: see text] denotes the Lipschitz constant of the i-th order derivatives in Y. It is worth noting that our analysis provides the first nonasymptotic convergence rate for the (slight variant of) Armijo linesearch algorithm without globally Lipschitz continuous gradient or convexity. As a generic framework, we also show that SLO can incorporate more complicated subroutines, such as a variant of the accelerated gradient descent (AGD) method that can harness the problem’s second-order smoothness without Hessian computation, which achieves an improved [Formula: see text] complexity. Funding: J. Zhang is supported by the MOE AcRF [Grant A-0009530-04-00], from Singapore Ministry of Education. M. Hong is supported by NSF [Grants CIF-1910385 and EPCN-2311007]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoo.2021.0029 .

Optimal Output Consensus of Heterogeneous Linear Multiagent Systems Over Weight-Unbalanced Directed Networks.

This article investigates the distributed optimal output consensus problem of heterogeneous linear multiagent systems over weight-unbalanced directed networks. A novel distributed continuous-time state feedback controller is proposed to steer the outputs of all the agents to converge to the optimal solution of the global cost function. Under the standard condition that the unbalanced digraph is strongly connected and the local cost functions are strongly convex with global Lipschitz gradients, the exponential convergence of the closed-loop multiagent system is established. Then, the proposed state feedback control law is extended to an observer-based output feedback setting. Two examples are finally provided to illustrate the effectiveness of the proposed control schemes.

Distributed Event-Triggered Algorithms for Multi-Objective Resource Allocation Problem

This paper considers a multi-objective resource allocation problem for a multi-agent network where each agent has multiple local objective functions. The goal is to obtain the Pareto optimum by exchanging information between agents. To this end, first, we introduce the weighted [Formula: see text] preference index to reformulate this problem into a single-objective resource allocation problem, in which the weighting factor of each objective depends on its relative importance. Moreover, in order to reduce the communication burden, we propose distributed event-triggered algorithms to solve the reformulated problem. When local objective functions are strongly convex and have Lipschitz gradients, we prove that the proposed algorithms are free of Zeno behavior and achieve exponential convergence. Finally, we demonstrate the effectiveness of the proposed algorithms by a microgrid network example.

A novel stepsize for gradient descent method

We propose a novel adaptive stepsize for the gradient descent scheme to solve unconstrained nonlinear optimization problems. With the convex and smooth objective satisfying locally Lipschitz gradient we obtain the complexity O(1k) of f(xk)−f⁎ at most. By using the idea of the new stepsize, we propose another new algorithm based on the projected gradient for solving a class of nonconvex optimization problems over a closed convex set. The computational experiments show the efficiency of the new method.

Accelerated Projection Algorithm Based on Smoothing Approximation for Distributed Nonsmooth Optimization

In this paper, a distributed smoothing accelerated projection algorithm (DSAPA) is proposed to address constrained non-smooth convex optimization problems over undirected multi-agent networks in a distributed manner, where the objective function is free of the assumption of Lipschitz gradient or strong convexity. First, based on a distributed exact penalty method, the original optimization problem is translated to a problem of standard assignment without consensus constraints. Then, a novel DSAPA by combining the smoothing approximation with Nesterov's accelerated schemes is proposed. In addition, we provide a systematic analysis to derive an upper bound on the convergence rate in terms of the objective function based on penalty function and to choose the optimal step size accordingly. Our results demonstrate that the proposed DSAPA can reach <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(\frac{\log (k)}{k})$</tex-math></inline-formula> when the optimal step size is chosen. Finally, the effectiveness and correctness of the proposed algorithm are verified by numerical and practical application examples.

On the Convergence Rate of Quasi-Newton Methods on Strongly Convex Functions with Lipschitz Gradient

The main results of the study of the convergence rate of quasi-Newton minimization methods were obtained under the assumption that the method operates in the region of the extremum of the function, where there is a stable quadratic representation of the function. Methods based on the quadratic model of the function in the extremum area show significant advantages over classical gradient methods. When solving a specific problem using the quasi-Newton method, a huge number of iterations occur outside the extremum area, unless there is a stable quadratic approximation of the function. In this paper, we study the convergence rate of quasi-Newton-type methods on strongly convex functions with a Lipschitz gradient, without using local quadratic approximations of a function based on the properties of its Hessian. We proved that quasi-Newton methods converge on strongly convex functions with a Lipschitz gradient with the rate of a geometric progression, while the estimate of the convergence rate improves with the increasing number of iterations, which reflects the fact that the learning (adaptation) effect accumulates as the method operates. Another important fact discovered during the theoretical study is the ability of quasi-Newton methods to eliminate the background that slows down the convergence rate. This elimination is achieved through a certain linear transformation that normalizes the elongation of function level surfaces in different directions. All studies were carried out without any assumptions regarding the matrix of second derivatives of the function being minimized.

Convergence of the Momentum Method for Semialgebraic Functions with Locally Lipschitz Gradients

Convergence of the Momentum Method for Semialgebraic Functions with Locally Lipschitz Gradients

Difference-of-Convex Algorithm with Extrapolation for Nonconvex, Nonsmooth Optimization Problems

In this paper, we focus on the problem of minimizing the sum of a nonconvex differentiable function and a difference of convex (DC) function, where the differentiable function is not restricted to the global Lipschitz gradient continuity assumption. This problem covers a broad range of applications in machine learning and statistics, such as compressed sensing, signal recovery, sparse dictionary learning, matrix factorization, etc. We first take inspiration from the Nesterov acceleration technique and the DC algorithm to develop a novel algorithm for the considered problem. We then study the subsequential convergence of our algorithm to a critical point. Furthermore, we justify the global convergence of the whole sequence generated by our algorithm to a critical point and establish its convergence rate under the Kurdyka–Łojasiewicz condition. Numerical experiments on the nonnegative matrix completion problem are performed to demonstrate the efficiency of our algorithm and its superiority over well-known methods.

Distributed Optimization of Nonlinear Multiagent Systems via Event-Triggered Communication

This brief is concerned with a distributed optimization problem over an undirected multi-agent network, where each agent is assigned a private cost function which is strongly convex with a Lipschitz gradient. First, an event-triggered communication strategy is adopted to reduce network communication, and an event-based auxiliary system is constructed to estimate the optimal solution. The estimation is used to generate a reference signal and its high-order derivatives, which is an approximation of the optimal solution. Then, a backstepping control algorithm is developed to drive all agents’ output to the reference signal. In terms with the Lyapunov stability theory and the algebraic graph theory, it can be proved that the distributed optimization problem is solved by the developed control algorithm. Different from the existing algorithms, eigenvalues of the Laplacian matrix are not used in our proposed control design. Finally, a simulation example is presented to validate the theoretical result.

Properties of the Quadratic Transformation of Dual Variables

We investigate a solution of a convex programming problem with a strongly convex objective function based on the dual approach. A dual optimization problem has constraints on the positivity of variables. We study the methods and properties of transformations of dual variables that enable us to obtain an unconstrained optimization problem. We investigate the previously known method of transforming the components of dual variables in the form of their modulus (modulus method). We show that in the case of using the modulus method, the degree of the degeneracy of the function increases as it approaches the optimal point. Taking into account the ambiguity of the gradient in the boundary regions of the sign change of the new dual function variables and the increase in the degree of the function degeneracy, we need to use relaxation subgradient methods (RSM) that are difficult to implement and that can solve non-smooth non-convex optimization problems with a high degree of elongation of level surfaces. We propose to use the transformation of the components of dual variables in the form of their square (quadratic method). We prove that the transformed dual function has a Lipschitz gradient with a quadratic method of transformation. This enables us to use efficient gradient methods to find the extremum. The above properties are confirmed by a computational experiment. With a quadratic transformation compared to a modulus transformation, it is possible to obtain a solution of the problem by relaxation subgradient methods and smooth function minimization methods (conjugate gradient method and quasi-Newtonian method) with higher accuracy and lower computational costs. The noted transformations of dual variables were used in the program module for calculating the maximum permissible emissions of enterprises (MPE) of the software package for environmental monitoring of atmospheric air (ERA-AIR).

Global convergence of the gradient method for functions definable in o-minimal structures

We consider the gradient method with variable step size for minimizing functions that are definable in o-minimal structures on the real field and differentiable with locally Lipschitz gradients. We prove that global convergence holds if continuous gradient trajectories are bounded, with the minimum gradient norm vanishing at the rate $o(1/k)$ if the step sizes are greater than a positive constant. If additionally the gradient is continuously differentiable, all saddle points are strict, and the step sizes are constant, then convergence to a local minimum holds almost surely over any bounded set of initial points.

S-NEAR-DGD: A Flexible Distributed Stochastic Gradient Method for Inexact Communication

We present and analyze a stochastic distributed method (S-NEAR-DGD) that can tolerate inexact computation and inaccurate information exchange to alleviate the problems of costly gradient evaluations and bandwidth-limited communication in large-scale systems. Our method is based on a class of flexible, distributed first-order algorithms that allow for the tradeoff of computation and communication to best accommodate the application setting. We assume that the information exchanged between nodes is subject to random distortion and that only stochastic approximations of the true gradients are available. Our theoretical results prove that the proposed algorithm converges linearly in expectation to a neighborhood of the optimal solution for strongly convex objective functions with Lipschitz gradients. We characterize the dependence of this neighborhood on algorithm and network parameters, the quality of the communication channel and the precision of the stochastic gradient approximations used. Finally, we provide numerical results to evaluate the empirical performance of our method.

Bregman iterative regularization using model functions for nonconvex nonsmooth optimization

In this paper, we propose a new algorithm called ModelBI by blending the Bregman iterative regularization method and the model function technique for solving a class of nonconvex nonsmooth optimization problems. On one hand, we use the model function technique, which is essentially a first-order approximation to the objective function, to go beyond the traditional Lipschitz gradient continuity. On the other hand, we use the Bregman iterative regularization to generate solutions fitting certain structures. Theoretically, we show the global convergence of the proposed algorithm with the help of the Kurdyka-Łojasiewicz property. Finally, we consider two kinds of nonsmooth phase retrieval problems and propose an explicit iteration scheme. Numerical results verify the global convergence and illustrate the potential of our proposed algorithm.

On the Divergence of Decentralized Nonconvex Optimization

We study a generic class of decentralized algorithms in which $N$ agents jointly optimize the non-convex objective $f(u):=1/N\sum_{i=1}^{N}f_i(u)$, while only communicating with their neighbors. This class of problems has become popular in modeling many signal processing and machine learning applications, and many efficient algorithms have been proposed. However, by constructing some counter-examples, we show that when certain local Lipschitz conditions (LLC) on the local function gradient $\nabla f_i$'s are not satisfied, most of the existing decentralized algorithms diverge, even if the global Lipschitz condition (GLC) is satisfied, where the sum function $f$ has Lipschitz gradient. This observation raises an important open question: How to design decentralized algorithms when the LLC, or even the GLC, is not satisfied? To address the above question, we design a first-order algorithm called Multi-stage gradient tracking algorithm (MAGENTA), which is capable of computing stationary solutions with neither the LLC nor the GLC. In particular, we show that the proposed algorithm converges sublinearly to certain $\epsilon$-stationary solution, where the precise rate depends on various algorithmic and problem parameters. In particular, if the local function $f_i$'s are $Q$th order polynomials, then the rate becomes $\mathcal{O}(1/\epsilon^{Q-1})$. Such a rate is tight for the special case of $Q=2$ where each $f_i$ satisfies LLC. To our knowledge, this is the first attempt that studies decentralized non-convex optimization problems with neither the LLC nor the GLC.

Revisiting linearized Bregman iterations under Lipschitz-like convexity condition

The linearized Bregman iterations (LBreI) and its variants have received considerable attention in signal/image processing and compressed sensing. Recently, LBreI has been extended to a larger class of nonconvex functions, along with several theoretical issues left for further investigation. In particular, the Lipschitz gradient continuity assumption precludes its use in many practical applications. In this study, we propose a generalized algorithmic framework to unify LBreI-type methods. Our main discovery is that the Lipschitz gradient continuity assumption can be replaced by a Lipschitz-like convexity condition in both convex and nonconvex cases. As a by-product, a class of bilevel optimization problems can be solved in the proposed framework, which extends the main result made by Cai et al. [Math. Comp. 78 (2009), pp. 2127–2136]. At last, provably convergent iterative schemes on modified linear/quadratic inverse problems illustrate our finding.

An Adaptive Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn toward the ADMM in nonconvex settings. Recent studies of minimization problems for nonconvex functions include various combinations of assumptions on the objective function including, in particular, a Lipschitz gradient assumption. We consider the case where the objective is the sum of a strongly convex function and a weakly convex function. To this end, we present and study an adaptive version of the ADMM which incorporates generalized notions of convexity and penalty parameters adapted to the convexity constants of the functions. We prove convergence of the scheme under natural assumptions. To this end, we employ the recent adaptive Douglas–Rachford algorithm by revisiting the well-known duality relation between the classical ADMM and the Douglas–Rachford splitting algorithm, generalizing this connection to our setting. We illustrate our approach by relating and comparing to alternatives, and by numerical experiments on a signal denoising problem.

Convergence Rates of the Heavy Ball Method for Quasi-strongly Convex Optimization

In this paper, we study the behavior of solutions of the ODE associated to the Heavy Ball method. Since the pioneering work of B.T. Polyak [25], it is well known that such a scheme is very efficient for C2 strongly convex functions with Lipschitz gradient. But much less is known when the C2 assumption is dropped. Depending on the geometry of the function to minimize, we obtain optimal convergence rates for the class of convex functions with some additional regularity such as quasi-strong convexity or strong convexity. We perform this analysis in continuous time for the ODE, and then we transpose these results for discrete optimization schemes. In particular, we propose a variant of the Heavy Ball algorithm which has the best state of the art convergence rate for first order methods to minimize strongly, composite non smooth convex functions.

Proximal Gradient Algorithms Under Local Lipschitz Gradient Continuity

Composite optimization offers a powerful modeling tool for a variety of applications and is often numerically solved by means of proximal gradient methods. In this paper, we consider fully nonconvex composite problems under only local Lipschitz gradient continuity for the smooth part of the objective function. We investigate an adaptive scheme for PANOC-type methods (Stella et al. in Proceedings of the IEEE 56th CDC, 2017), namely accelerated linesearch algorithms requiring only the simple oracle of proximal gradient. While including the classical proximal gradient method, our theoretical results cover a broader class of algorithms and provide convergence guarantees for accelerated methods with possibly inexact computation of the proximal mapping. These findings have also significant practical impact, as they widen scope and performance of existing, and possibly future, general purpose optimization software that invoke PANOC as inner solver.

Synthesis of accelerated gradient algorithms for optimization and saddle point problems using Lyapunov functions and LMIs

This paper considers the problem of designing accelerated gradient-based algorithms for optimization and saddle-point problems. The class of objective functions is defined by a generalized sector condition. This class of functions contains strongly convex functions with Lipschitz gradients but also non-convex functions, which allows not only to address optimization problems but also saddle-point problems. The proposed design procedure relies on a suitable class of Lyapunov functions and, for a fixed convergence rate, is a convex semi-definite program in all but one scalar parameter. The proposed synthesis allows the design of algorithms that reach the performance of state-of-the-art accelerated gradient methods and beyond.