Lipschitz Continuous Gradient Research Articles

On generalized Jacobians in the sense of Clarke for the inverse of a bi‐Lipschitz map and applications in relaxation theory

AbstractRelaxed variational problems in the scalar case involving functions are frequently characterized by the convex envelope of the energy density . For functions with superlinear growth and Lipschitz continuous gradient, a formula for is presented in an open neighborhood of any maximal phase simplex provided that is and strictly convex close to the vertices of the phase simplex. This formula involves a carefully chosen parameterization of which identifies those subsets in in which may be written as a convex combination of a prescribed number of function values.

SBL-LCGL: sparse Bayesian learning based on Laplace distribution for robust cone-beam x-ray luminescence computed tomography

Objective. To address the quality and accuracy issues in the distribution of nanophosphors (NPs) using Cone-beam x-ray luminescence computed tomography (CB-XLCT) by proposing a novel reconstruction strategy.Approach. This paper introduces a sparse Bayesian learning reconstruction method termed SBL-LCGL, which is grounded in the Lipschitz continuous gradient condition and the Laplace prior to overcome the ill-posed inverse problem inherent in CB-XLCT.Main results. The SBL-LCGL method has demonstrated its effectiveness in capturing the sparse features of NPs and mitigating the computational complexity associated with matrix inversion. Both numerical simulation andin vivoexperiments confirm that the method yields satisfactory imaging results regarding the position and shape of the targets.Significance. The advancements presented in this work are expected to enhance the clinical applicability of CB-XLCT, contributing to its broader adoption in medical imaging and diagnostics.

Universal heavy-ball method for nonconvex optimization under Hölder continuous Hessians

AbstractWe propose a new first-order method for minimizing nonconvex functions with Lipschitz continuous gradients and Hölder continuous Hessians. The proposed algorithm is a heavy-ball method equipped with two particular restart mechanisms. It finds a solution where the gradient norm is less than $$\varepsilon $$ ε in $$O(H_{\nu }^{\frac{1}{2 + 2 \nu }} \varepsilon ^{- \frac{4 + 3 \nu }{2 + 2 \nu }})$$ O ( H ν 1 2 + 2 ν ε - 4 + 3 ν 2 + 2 ν ) function and gradient evaluations, where $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] and $$H_{\nu }$$ H ν are the Hölder exponent and constant, respectively. This complexity result covers the classical bound of $$O(\varepsilon ^{-2})$$ O ( ε - 2 ) for $$\nu = 0$$ ν = 0 and the state-of-the-art bound of $$O(\varepsilon ^{-7/4})$$ O ( ε - 7 / 4 ) for $$\nu = 1$$ ν = 1 . Our algorithm is $$\nu $$ ν -independent and thus universal; it automatically achieves the above complexity bound with the optimal $$\nu \in [0, 1]$$ ν ∈ [ 0 , 1 ] without knowledge of $$H_{\nu }$$ H ν . In addition, the algorithm does not require other problem-dependent parameters as input, including the gradient’s Lipschitz constant or the target accuracy $$\varepsilon $$ ε . Numerical results illustrate that the proposed method is promising.

Efficiency of Stochastic Coordinate Proximal Gradient Methods on Nonseparable Composite Optimization

This paper deals with composite optimization problems having the objective function formed as the sum of two terms; one has a Lipschitz continuous gradient along random subspaces and may be nonconvex, and the second term is simple and differentiable but possibly nonconvex and nonseparable. Under these settings, we design a stochastic coordinate proximal gradient method that takes into account the nonseparable composite form of the objective function. This algorithm achieves scalability by constructing at each iteration a local approximation model of the whole nonseparable objective function along a random subspace with user-determined dimension. We outline efficient techniques for selecting the random subspace, yielding an implementation that has low cost per iteration, also achieving fast convergence rates. We present a probabilistic worst case complexity analysis for our stochastic coordinate proximal gradient method in convex and nonconvex settings; in particular, we prove high-probability bounds on the number of iterations before a given optimality is achieved. Extensive numerical results also confirm the efficiency of our algorithm. Funding: This work was supported by Norway Grants 2014-2021 [Grant ELO-Hyp 24/2020]; Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii [Grants PN-III-P4-PCE-2021-0720, L2O-MOC, nr 70/2022]; and the ITN-ETN project TraDE-OPT funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement [Grant 861137].

Convergence Rate of the (1+1)-ES on Locally Strongly Convex and Lipschitz Smooth Functions

Evolution strategy (ES) is one of promising classes of algorithms for black-box continuous optimization. Despite its broad successes in applications, theoretical analysis on the speed of its convergence is limited on convex quadratic functions and their monotonic transformation. In this study, an upper bound and a lower bound of the rate of linear convergence of the (1+1)-ES on locally $L$-strongly convex functions with $U$-Lipschitz continuous gradient are derived as $\exp\left(-\Omega_{d\to\infty}\left(\frac{L}{d\cdot U}\right)\right)$ and $\exp\left(-\frac1d\right)$, respectively. Notably, any prior knowledge on the mathematical properties of the objective function such as Lipschitz constant is not given to the algorithm, whereas the existing analyses of derivative-free optimization algorithms require them.

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates

We consider the finite-sum optimization problem, where each component function is strongly convex and has Lipschitz continuous gradient and Hessian. The recently proposed incremental quasi-Newton method is based on BFGS update and achieves a local superlinear convergence rate that is dependent on the condition number of the problem. This paper proposes a more efficient quasi-Newton method by incorporating the symmetric rank-1 update into the incremental framework, which results in the condition-number-free local superlinear convergence rate. Furthermore, we can boost our method by applying the block update on the Hessian approximation, which leads to an even faster local convergence rate. The numerical experiments show the proposed methods significantly outperform the baseline methods.

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

In this paper, we explore a specific optimization problem that involves the combination of a differentiable nonconvex function and a nondifferentiable function. The differentiable component lacks a global Lipschitz continuous gradient, posing challenges for optimization. To address this issue and accelerate the convergence, we propose a Bregman proximal stochastic gradient method with extrapolation (BPSGE), which only requires smooth adaptivity of the differentiable part. Under variance reduction framework, we not only analyze the subsequential and global convergence of the proposed algorithm under certain conditions, but also analyze the sublinear convergence rate of the subsequence, and the complexity of the algorithm, revealing that the BPSGE algorithm requires at most O(epsilon\^\,(-2)) iterations in expectation to attain an epsilon-stationary point. To validate the effectiveness of our proposed algorithm, we conduct numerical experiments on three real-world applications: graph regularized nonnegative matrix factorization (NMF), matrix factorization with weakly-convex regularization, and NMF with nonconvex sparsity constraints. These experiments demonstrate that BPSGE is faster than the baselines without extrapolation. The code is available at: https://github.com/nothing2wang/BPSGE-Algorithm.

Trust Region Methods for Nonconvex Stochastic Optimization beyond Lipschitz Smoothness

In many important machine learning applications, the standard assumption of having a globally Lipschitz continuous gradient may fail to hold. This paper delves into a more general (L0, L1)-smoothness setting, which gains particular significance within the realms of deep neural networks and distributionally robust optimization (DRO). We demonstrate the significant advantage of trust region methods for stochastic nonconvex optimization under such generalized smoothness assumption. We show that first-order trust region methods can recover the normalized and clipped stochastic gradient as special cases and then provide a unified analysis to show their convergence to first-order stationary conditions. Motivated by the important application of DRO, we propose a generalized high-order smoothness condition, under which second-order trust region methods can achieve a complexity of O(epsilon(-3.5)) for convergence to second-order stationary points. By incorporating variance reduction, the second-order trust region method obtains an even better complexity of O(epsilon(-3)), matching the optimal bound for standard smooth optimization. To our best knowledge, this is the first work to show convergence beyond the first-order stationary condition for generalized smooth optimization. Preliminary experiments show that our proposed algorithms perform favorably compared with existing methods.

First-Order Algorithms Without Lipschitz Gradient: A Sequential Local Optimization Approach

Most first-order methods rely on the global Lipschitz continuity of the objective gradient, which fails to hold in many problems. This paper develops a sequential local optimization (SLO) framework for first-order algorithms to optimize problems without Lipschitz gradient. Operating on the assumption that the gradient is locally Lipschitz continuous over any compact set, SLO develops a careful scheme to control the distance between successive iterates. The proposed framework can easily adapt to the existing first-order methods, such as projected gradient descent (PGD), truncated gradient descent (TGD), and a parameter-free variant of Armijo linesearch. We show that SLO requires [Formula: see text] gradient evaluations to find an ϵ-stationary point, where Y is certain compact set with [Formula: see text] radius, and [Formula: see text] denotes the Lipschitz constant of the i-th order derivatives in Y. It is worth noting that our analysis provides the first nonasymptotic convergence rate for the (slight variant of) Armijo linesearch algorithm without globally Lipschitz continuous gradient or convexity. As a generic framework, we also show that SLO can incorporate more complicated subroutines, such as a variant of the accelerated gradient descent (AGD) method that can harness the problem’s second-order smoothness without Hessian computation, which achieves an improved [Formula: see text] complexity. Funding: J. Zhang is supported by the MOE AcRF [Grant A-0009530-04-00], from Singapore Ministry of Education. M. Hong is supported by NSF [Grants CIF-1910385 and EPCN-2311007]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/ijoo.2021.0029 .

Lp quasi-norm minimization: algorithm and applications

Sparsity finds applications in diverse areas such as statistics, machine learning, and signal processing. Computations over sparse structures are less complex compared to their dense counterparts and need less storage. This paper proposes a heuristic method for retrieving sparse approximate solutions of optimization problems via minimizing the ℓp\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _{p}$$\\end{document} quasi-norm, where 0<p<1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$0<p<1$$\\end{document}. An iterative two-block algorithm for minimizing the ℓp\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _{p}$$\\end{document} quasi-norm subject to convex constraints is proposed. The proposed algorithm requires solving for the roots of a scalar degree polynomial as opposed to applying a soft thresholding operator in the case of ℓ1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _{1}$$\\end{document} norm minimization. The algorithm’s merit relies on its ability to solve the ℓp\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _{p}$$\\end{document} quasi-norm minimization subject to any convex constraints set. For the specific case of constraints defined by differentiable functions with Lipschitz continuous gradient, a second, faster algorithm is proposed. Using a proximal gradient step, we mitigate the convex projection step and hence enhance the algorithm’s speed while proving its convergence. We present various applications where the proposed algorithm excels, namely, sparse signal reconstruction, system identification, and matrix completion. The results demonstrate the significant gains obtained by the proposed algorithm compared to other ℓp\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _{p}$$\\end{document} quasi-norm based methods presented in previous literature.

Lipschitz continuity of the metric projection operator and convergence of gradient methods

Various support conditions for a closed subset of a real Hilbert space $\mathcal H$ at a boundary point of this set are considered. These conditions ensure certain local Lipschitz continuity of the metric projection operator as a function of a point. The local Lipschitz continuity of the metric projection as a function of the set in the Hausdorff metric is also proved. This Lipschitz property is used to verify the linear convergence of some gradient methods (the gradient projection method and the conditional gradient method) without assuming that the function must be strongly convex (or even convex) and for not necessarily convex sets. The function is assumed to be differentiable with Lipschitz continuous gradient. Bibliography: 29 titles.

Communication Efficient Curvature Aided Primal-Dual Algorithms for Decentralized Optimization

This paper presents a family of algorithms for decentralized convex composite problems. We consider the setting of a network of agents that cooperatively minimize a global objective function composed of a sum of local functions plus a regularizer. Through the use of intermediate consensus variables, we remove the need for inner communication loops between agents when computing curvature-guided updates. A general scheme is presented which unifies the analysis for a plethora of computing choices, including gradient descent, Newton updates, and BFGS updates. Our analysis establishes sublinear convergence rates under convex objective functions with Lipschitz continuous gradients, as well as linear convergence rates when the local functions are further assumed to be strongly convex. Moreover, we explicitly characterize the acceleration due to curvature information. Last but not the least, we present an asynchronous implementation for the proposed algorithms, which removes the need for a central clock, with linear convergence rates established in expectation under strongly convex objectives. We ascertain the effectiveness of the proposed methods with numerical experiments on benchmark datasets.

Accelerated First-Order Methods for Convex Optimization with Locally Lipschitz Continuous Gradient

Accelerated First-Order Methods for Convex Optimization with Locally Lipschitz Continuous Gradient

An accelerated exact distributed first-order algorithm for optimization over directed networks

Distributed optimization over networked agents has emerged as an advanced paradigm to address large-scale control, optimization, and signal-processing problems. In the last few years, the distributed first-order gradient methods have witnessed significant progress and enrichment due to the simplicity of using only the first derivatives of local functions. An exact first-order algorithm is developed in this work for distributed optimization over general directed networks with only row-stochastic weighted matrices. It employs the rescaling gradient method to address unbalanced information diffusion among agents, where the weights on the received information can be arbitrarily assigned. Moreover, uncoordinated step-sizes are employed to magnify the autonomy of agents, and an error compensation term and a heavy-ball momentum are incorporated to accelerate convergency. A linear convergence rate is rigorously proven for strongly-convex objective functions with Lipschitz continuous gradients. Explicit upper bounds of step-size and momentum parameter are provided. Finally, simulations illustrate the performance of the proposed algorithm.

A proximal subgradient algorithm with extrapolation for structured nonconvex nonsmooth problems

In this paper, we consider a class of structured nonconvex nonsmooth optimization problems, in which the objective function is formed by the sum of a possibly nonsmooth nonconvex function and a differentiable function with Lipschitz continuous gradient, subtracted by a weakly convex function. This general framework allows us to tackle problems involving nonconvex loss functions and problems with specific nonconvex constraints, and it has many applications such as signal recovery, compressed sensing, and optimal power flow distribution. We develop a proximal subgradient algorithm with extrapolation for solving these problems with guaranteed subsequential convergence to a stationary point. The convergence of the whole sequence generated by our algorithm is also established under the widely used Kurdyka–Łojasiewicz property. To illustrate the promising numerical performance of the proposed algorithm, we conduct numerical experiments on two important nonconvex models. These include a compressed sensing problem with a nonconvex regularization and an optimal power flow problem with distributed energy resources.

Time Rescaling of a Primal-Dual Dynamical System with Asymptotically Vanishing Damping

In this work, we approach the minimization of a continuously differentiable convex function under linear equality constraints by a second-order dynamical system with an asymptotically vanishing damping term. The system under consideration is a time rescaled version of another system previously found in the literature. We show fast convergence of the primal-dual gap, the feasibility measure, and the objective function value along the generated trajectories. These convergence rates now depend on the rescaling parameter, and thus can be improved by choosing said parameter appropriately. When the objective function has a Lipschitz continuous gradient, we show that the primal-dual trajectory asymptotically converges weakly to a primal-dual optimal solution to the underlying minimization problem. We also exhibit improved rates of convergence of the gradient along the primal trajectories and of the adjoint of the corresponding linear operator along the dual trajectories. We illustrate the theoretical outcomes and also carry out a comparison with other classes of dynamical systems through numerical experiments.

A Bregman stochastic method for nonconvex nonsmooth problem beyond global Lipschitz gradient continuity

ABSTRACT In this paper, we consider solving a broad class of large-scale nonconvex and nonsmooth minimization problems by a Bregman proximal stochastic gradient (BPSG) algorithm. The objective function of the minimization problem is the composition of a differentiable and a nondifferentiable function, and the differentiable part does not admit a global Lipschitz continuous gradient. Under some suitable conditions, the subsequential convergence of the proposed algorithm is established. And under expectation conditions with the Kurdyka-Łojasiewicz (KL) property, we also prove that the proposed method converges globally. We also apply the BPSG algorithm to solve sparse nonnegative matrix factorization (NMF), symmetric NMF via non-symmetric relaxation, and matrix completion problems under different kernel generating distances, and numerically compare it with other algorithms. The results demonstrate the robustness and effectiveness of the proposed algorithm.

The Proxy Step-Size Technique for Regularized Optimization on the Sphere Manifold.

We give an effective solution to the regularized optimization problem g (x) + h (x), where x is constrained on the unit sphere ||x ||2 = 1. Here g (·) is a smooth cost with Lipschitz continuous gradient within the unit ball {x : ||x ||2 ≤ 1 } whereas h (·) is typically non-smooth but convex and absolutely homogeneous, e.g., norm regularizers and their combinations. Our solution is based on the Riemannian proximal gradient, using an idea we call proxy step-size - a scalar variable which we prove is monotone with respect to the actual step-size within an interval. The proxy step-size exists ubiquitously for convex and absolutely homogeneous h(·), and decides the actual step-size and the tangent update in closed-form, thus the complete proximal gradient iteration. Based on these insights, we design a Riemannian proximal gradient method using the proxy step-size. We prove that our method converges to a critical point, guided by a line-search technique based on the g(·) cost only. The proposed method can be implemented in a couple of lines of code. We show its usefulness by applying nuclear norm, l1 norm, and nuclear-spectral norm regularization to three classical computer vision problems. The improvements are consistent and backed by numerical experiments. available at https://bitbucket.org/FangBai/proxystepsize-pgs.

Global Convergence of Policy Gradient Primal–Dual Methods for Risk-Constrained LQRs

While the techniques in optimal control theory are often model-based, the policy optimization (PO) approach directly optimizes the performance metric of interest. Even though it has been an essential approach for reinforcement learning problems, there is little theoretical understanding of its performance. In this paper, we focus on the risk-constrained linear quadratic regulator (RC-LQR) problem via the PO approach, which requires addressing a challenging non-convex constrained optimization problem. To solve it, we first build on our earlier result that an optimal policy has a time-invariant affine structure to show that the associated Lagrangian function is coercive, locally gradient dominated, and has local Lipschitz continuous gradient, based on which we establish strong duality. Then, we design policy gradient primal-dual methods with global convergence guarantees in both model-based and sample-based settings. Finally, we use samples of system trajectories in simulations to validate our methods.

Stochastic Composition Optimization of Functions Without Lipschitz Continuous Gradient

In this paper, we study stochastic optimization of two-level composition of functions without Lipschitz continuous gradient. The smoothness property is generalized by the notion of relative smoothness which provokes the Bregman gradient method. We propose three stochastic composition Bregman gradient algorithms for the three possible relatively smooth compositional scenarios and provide their sample complexities to achieve an $$\epsilon $$ -approximate stationary point. For the smooth of relatively smooth composition, the first algorithm requires $$\mathcal {O}(\epsilon ^{-2})$$ calls to the stochastic oracles of the inner function value and gradient as well as the outer function gradient. When both functions are relatively smooth, the second algorithm requires $$\mathcal {O}(\epsilon ^{-3})$$ calls to the inner function value stochastic oracle and $$\mathcal {O}(\epsilon ^{-2})$$ calls to the inner and outer functions gradients stochastic oracles. We further improve the second algorithm by variance reduction for the setting where just the inner function is smooth. The resulting algorithm requires $$\mathcal {O}(\epsilon ^{-5/2})$$ calls to the inner function value stochastic oracle, $$\mathcal {O}(\epsilon ^{-3/2})$$ calls to the inner function gradient, and $$\mathcal {O}(\epsilon ^{-2})$$ calls to the outer function gradient stochastic oracles. Finally, we numerically evaluate the performance of these three algorithms over two different examples.