Lipschitz Continuous Gradient Research Articles

A unified enrichment approach of the standard three-node triangular element

The aim of this paper is to unify the ideas and to extend to a more general setting the work done in (Dell'Accio et al., 2023 [14]) for a polynomial enrichment of the standard three-node triangular element (triangular linear element) using line integrals and quadratic polynomials. More precisely, we introduce a new class of nonconforming finite elements by enriching the class of linear polynomial functions with additional functions which are not necessarily polynomials. We provide a simple condition on the enrichment functions, which is both necessary and sufficient, that guarantees the existence of a family of such enriched elements. Several sets of admissible enriched functions that satisfy the admissibility condition are also provided, together with the explicit expression of the related approximation error. Our main result shows that the approximation error can be decomposed into two parts: the first one is related to the linear triangular element while the second one depends on the enrichment functions. This representation of the approximation error allows us to derive sharp error bounds in both L∞ and L1 norms, with explicit constants, for continuously differentiable functions with Lipschitz continuous gradients. These bounds are proportional to the second and the fourth power of the circumcircle radius of the triangle, respectively. We also provide explicit expressions of these bounds in terms of the circumcircle diameter and the sum of squares of the triangle edge lengths.

A Primal-Dual Forward-Backward Splitting Algorithm for Distributed Convex Optimization

Motivated by modern large-scale information processing problems in engineering, this paper concentrates on studying distributed constrained convex optimization problems over a connected undirected network. The problem involves a sum of a differentiable convex function with Lipschitz continuous gradient and two non-smooth convex functions with a linear operator. To solve such a problem, we propose a novel distributed primal-dual forward-backward splitting algorithm, called D-PDFBS. Each agent locally computes the Lipschitz continuous gradient and two proximal operators, and exchanges information with its neighbors. D-PDFBS adopts non-identical stepsizes, and we reveal the relationship between selection of stepsizes and parameters of objective functions. The simulation results verify the feasibility of D-PDFBS and the correctness of theoretical findings.

Event-Triggered Optimal Consensus of Second-Order MASs With Disturbances and Cyber Attacks on Communications Edges

This paper investigates the distributed optimal consensus problem (DOCP) for second-order multi-agent systems (MASs) in the presence of system disturbances and cyber attacks on communication edges. In optimization, the global objective function is the sum of a class of local private objective functions, which are assumed to be strongly convex and continuously differentiable with Lipschitz continuous gradients. Unlike most optimization problems for MASs studied in the literature, in this study, the dynamics of agents are second-order and disturbed, considering cyber attacks among neighboring agents. In order to achieve DOC with low online resource consumption, novel distributed event-triggered controllers are designed for agents. Not only the communication and computation resources can be saved, but also the frequency of controller updates can be reduced. Moreover, the controllers can prevent the information of the states of second-order MASs from leaking to neighbors. Under restrictions on cyber attacks, sufficient conditions are derived, under which all controlled agents can achieve consensus on the optimal solution of the global objective function exponentially. Furthermore, the Zeno-behavior is excluded on triggering time sequences. Finally, three simulation examples are shown to illustrate the effectiveness of the designed controllers in saving online resources.

A variable metric and nesterov extrapolated proximal DCA with backtracking for a composite DC program

In this paper, we consider a composite difference-of-convex (DC) program, whose objective function is the sum of a smooth convex function with Lipschitz continuous gradient, a proper closed and convex function, and a continuous concave function. This problem has many applications in machine learning and data science. The proximal DCA (pDCA), a special case of the classical difference-of-convex algorithm (DCA), as well as two Nesterov-type extrapolated DCA – ADCA (Phan et al. IJCAI:1369–1375, 2018) and pDCAe (Wen et al. Comput. Optim. Appl. 69:297–324, 2018) – can solve this problem. The algorithmic stepsizes of pDCA, pDCAe, and ADCA are fixed and determined by estimating a prior the smoothness parameter of the loss function. However, such an estimate may be hard to obtain or poor in some real-world applications. Motivated by this difficulty, we propose a variable metric and Nesterov extrapolated proximal DCA with backtracking (SPDCAe), which combines the backtracking line search procedure (not necessarily monotone) and the Nesterov's extrapolation for potential acceleration; moreover, the variable metric method is incorporated for better local approximation. Numerical simulations on sparse binary logistic regression and compressed sensing with Poisson noise demonstrate the effectiveness of our proposed method.

LSOS: Line-search second-order stochastic optimization methods for nonconvex finite sums

We develop a line-search second-order algorithmic framework for minimizing finite sums. We do not make any convexity assumptions, but require the terms of the sum to be continuously differentiable and have Lipschitz-continuous gradients. The methods fitting into this framework combine line searches and suitably decaying step lengths. A key issue is a two-step sampling at each iteration, which allows us to control the error present in the line-search procedure. Stationarity of limit points is proved in the almost-sure sense, while almost-sure convergence of the sequence of approximations to the solution holds with the additional hypothesis that the functions are strongly convex. Numerical experiments, including comparisons with state-of-the art stochastic optimization methods, show the efficiency of our approach.

Relaxed Variable Metric Primal-Dual Fixed-Point Algorithm with Applications

In this paper, a relaxed variable metric primal-dual fixed-point algorithm is proposed for solving the convex optimization problem involving the sum of two convex functions where one is differentiable with the Lipschitz continuous gradient while the other is composed of a linear operator. Based on the preconditioned forward–backward splitting algorithm, the convergence of the proposed algorithm is proved. At the same time, we show that some existing algorithms are special cases of the proposed algorithm. Furthermore, the ergodic convergence and linear convergence rates of the proposed algorithm are established under relaxed parameters. Numerical experiments on the image deblurring problems demonstrate that the proposed algorithm outperforms some existing algorithms in terms of the number of iterations.

Surplus-based accelerated algorithms for distributed optimization over directed networks

This paper investigates a distributed optimization problem based on the framework of a multi-agent system over a directed communication network, where the global cost function is the sum of the local cost functions of agents. The communication network is abstracted as a weight-unbalanced directed graph. First, a surplus-based accelerated algorithm with a fixed stepsize (SAAFS) is proposed by integrating the gradient tracking strategy into the surplus-based consensus protocol to address the problem considered. The matrix norm argument and matrix perturbation theory are employed to prove the linear convergence of SAAFS under the assumption that each local cost function is strongly convex with the Lipschitz continuous gradient. Second, the limitation of the stepsize, which is common to all agents, is relaxed in the cases of different stepsizes for each agent, such that the surplus-based accelerated algorithm with an uncoordinated stepsize (SAAUS) is proposed. It is proven that SAAUS also has a linear convergence rate if the upper bound of the uncoordinated stepsize at each agent is restricted by a sufficiently small positive number. Finally, two simulation examples are provided to evaluate the proposed algorithms and illustrate that both SAAFS and SAAUS achieve acceleration, particularly for ill-conditioned optimization problems.

Distributed Optimization Algorithms for MASs With Network Attacks: From Continuous-Time to Event-Triggered Communication

In this paper, the distributed optimization problem for multi-agent systems under the existence of cyber attacks is researched. Each agent has a private local cost function which is assumed to be continuously differentiable and strongly convex with the Lipschitz continuous gradient. The global objective function is the sum of these local cost functions, and the communication topologies among agents are balanced and directed. Unlike most multi-agent systems in the existing literature, the cyber attacks in networks, which may result in the failure of information transmission, are considered. Two novel distributed optimization algorithms with the continuous-time communication are studied for such an optimization problem. In addition, in order to reduce computation and communication consumption efficiently, we further enhance these algorithms with an event-triggered communication mechanism. Under some assumptions and restrictions on attacks, sufficient yet efficient conditions for the convergence of the proposed algorithms are derived. Furthermore, the Zeno-behavior is shown to be excluded for the event-triggered distributed optimization algorithms. Finally, some numerical simulation examples are provided to illustrate and validate the efficiency of the proposed optimization algorithms.

Proximal stochastic recursive momentum algorithm for nonsmooth nonconvex optimization problems

In this paper, we mainly consider a class of nonconvex nonsmooth optimization problems, whose objective function is the sum of a smooth function with a Lipschitz continuous gradient and a convex nonsmooth function. We first propose a proximal stochastic recursive momentum algorithm(ProxSTORM) with mini-batch for solving the optimization problems and consider its convergence behaviour. Then, based on the Polyak–Łojasiewicz inequality, we establish the global linear convergence rate of ProxSTORM. Finally, some numerical experiments have been conducted to illustrate the efficiency of our method.

Blind deconvolution with non-smooth regularization via Bregman proximal DCAs

Blind deconvolution is a technique to recover an original signal without knowing a convolving filter. It is naturally formulated as a minimization of a quartic objective function under some assumption. Because its differentiable part does not have a Lipschitz continuous gradient, existing first-order methods are not theoretically supported. In this paper, we employ the Bregman-based proximal methods, whose convergence is theoretically guaranteed under the L-smooth adaptable (L-smad) property. We first reformulate the objective function as a difference of convex (DC) functions and apply the Bregman proximal DC algorithm (BPDCA). This DC decomposition satisfies the L-smad property. The method is extended to the BPDCA with extrapolation (BPDCAe) for faster convergence. When our regularizer has a sufficiently simple structure, each iteration is solved in a closed-form expression, and thus our algorithms solve large-scale problems efficiently. We also provide the stability analysis of the equilibriums and demonstrate the proposed methods through numerical experiments on image deblurring. The results show that BPDCAe successfully recovered the original image and outperformed other existing algorithms.

Fast Augmented Lagrangian Method in the convex regime with convergence guarantees for the iterates

This work aims to minimize a continuously differentiable convex function with Lipschitz continuous gradient under linear equality constraints. The proposed inertial algorithm results from the discretization of the second-order primal-dual dynamical system with asymptotically vanishing damping term addressed by Boţ and Nguyen (J. Differential Equations 303:369–406, 2021), and it is formulated in terms of the Augmented Lagrangian associated with the minimization problem. The general setting we consider for the inertial parameters covers the three classical rules by Nesterov, Chambolle–Dossal and Attouch–Cabot used in the literature to formulate fast gradient methods. For these rules, we obtain in the convex regime convergence rates of order {mathcal {O}}left( 1/k^{2} right) for the primal-dual gap, the feasibility measure, and the objective function value. In addition, we prove that the generated sequence of primal-dual iterates converges to a primal-dual solution in a general setting that covers the two latter rules. This is the first result which provides the convergence of the sequence of iterates generated by a fast algorithm for linearly constrained convex optimization problems without additional assumptions such as strong convexity. We also emphasize that all convergence results of this paper are compatible with the ones obtained in Boţ and Nguyen (J. Differential Equations 303:369–406, 2021) in the continuous setting.

Bregman Proximal Point Algorithm Revisited: A New Inexact Version and Its Inertial Variant

We study a general convex optimization problem, which covers various classic problems in different areas and particularly includes many optimal transport related problems arising in recent years. To solve this problem, we revisit the classic Bregman proximal point algorithm (BPPA) and introduce a new inexact stopping condition for solving the subproblems, which can circumvent the underlying feasibility difficulty often appearing in existing inexact conditions when the problem has a complex feasible set. Our inexact condition also covers several existing inexact conditions as special cases and hence makes our inexact BPPA (iBPPA) more flexible to fit different scenarios in practice. As an application to the standard optimal transport (OT) problem, our iBPPA with the entropic proximal term can bypass some numerical instability issues that usually plague the popular Sinkhorn's algorithm in the OT community, since our iBPPA does not require the proximal parameter to be very small for obtaining an accurate approximate solution. The iteration complexity of $O(1/k)$ and the convergence of the sequence are also established for our iBPPA under some mild conditions. Moreover, inspired by Nesterov's acceleration technique, we develop an inertial variant of our iBPPA, denoted by V-iBPPA, and establish the iteration complexity of $O(1/k^{\lambda})$, where $\lambda\geq1$ is a quadrangle scaling exponent of the kernel function. In particular, when the proximal parameter is a constant and the kernel function is strongly convex with Lipschitz continuous gradient (hence $\lambda=2$), our V-iBPPA achieves a faster rate of $O(1/k^2)$ just as existing accelerated inexact proximal point algorithms. Some preliminary numerical experiments for solving the standard OT problem are conducted to show the convergence behaviors of our iBPPA and V-iBPPA under different inexactness settings. The experiments also empirically verify the potential of our V-iBPPA for improving the convergence speed.

SRKCD: A stabilized Runge–Kutta method for stochastic optimization

We introduce a family of stochastic optimization methods based on the Runge–Kutta–Chebyshev (RKC) schemes. The RKC methods are explicit methods originally designed for solving stiff ordinary differential equations by ensuring that their stability regions are of maximal size. In the optimization context, this allows for larger step sizes (learning rates) and better robustness compared to e.g. the popular stochastic gradient descent method. Our main contribution is a convergence proof for essentially all stochastic Runge–Kutta optimization methods. This shows convergence in expectation with an optimal sublinear rate under standard assumptions of strong convexity and Lipschitz-continuous gradients. For non-convex objectives, we get convergence to zero in expectation of the gradients. The proof requires certain natural conditions on the Runge–Kutta coefficients, and we further demonstrate that the RKC schemes satisfy these. Finally, we illustrate the improved stability properties of the methods in practice by performing numerical experiments on both a small-scale test example and on a problem arising from an image classification application in machine learning.

DPCD: Discrete Principal Coordinate Descent for Binary Variable Problems

Binary optimization, a representative subclass of discrete optimization, plays an important role in mathematical optimization and has various applications in computer vision and machine learning. Generally speaking, binary optimization problems are NP-hard and difficult to solve due to the binary constraints, especially when the number of variables is very large. Existing methods often suffer from high computational costs or large accumulated quantization errors, or are only designed for specific tasks. In this paper, we propose an efficient algorithm, named Discrete Principal Coordinate Descent (DPCD), to find effective approximate solutions for general binary optimization problems. The proposed algorithm iteratively solves optimization problems related to the linear approximation of loss functions, which leads to updating the binary variables that most impact the value of the loss functions at each step. Our method supports a wide range of empirical objective functions with/without restrictions on the numbers of 1s and -1s in the binary variables. Furthermore, the theoretical convergence of our algorithm is proven, and the explicit convergence rates are derived for objective functions with Lipschitz continuous gradients, which are commonly adopted in practice. Extensive experiments on binary hashing tasks and large-scale datasets demonstrate the superiority of the proposed algorithm over several state-of-the-art methods in terms of both effectiveness and efficiency.

Constrained, Global Optimization of Unknown Functions with Lipschitz Continuous Gradients

We present two first-order, sequential optimization algorithms to solve constrained optimization problems. We consider a black-box setting with a priori unknown, nonconvex objective and constraint functions that have Lipschitz continuous gradients. The proposed algorithms balance the exploration of the a priori unknown feasible space with the pursuit of global optimality within a prespecified finite number of first-order oracle calls. The first algorithm accommodates an infeasible start and provides either a near-optimal global solution or establishes infeasibility. However, the algorithm may produce infeasible iterates during the search. For a strongly convex constraint function and a feasible initial solution guess, the second algorithm returns a near-optimal global solution without any constraint violation. At each iteration, the algorithms identify the next query point by solving a nonconvex, quadratically constrained, quadratic program, characterized by the Lipschitz continuous gradient property of the functions and the oracle responses. In contrast to existing methods, the algorithms compute global suboptimality bounds at every iteration. They can also satisfy user-specified tolerances in the computed solution with near-optimal complexity in oracle calls for a large class of optimization problems. We implement the proposed algorithms using GUROBI, a commercial off-the-shelf solver.

Global Linear Convergence of Evolution Strategies on More than Smooth Strongly Convex Functions

Evolution strategies (ESs) are zeroth-order stochastic black-box optimization heuristics invariant to monotonic transformations of the objective function. They evolve a multivariate normal distribution, from which candidate solutions are generated. Among different variants, CMA-ES is nowadays recognized as one of the state-of-the-art zeroth-order optimizers for difficult problems. Despite ample empirical evidence that ESs with a step-size control mechanism converge linearly, theoretical guarantees of linear convergence of ESs have been established only on limited classes of functions. In particular, theoretical results on convex functions are missing, where zeroth-order and also first-order optimization methods are often analyzed. In this paper, we establish almost sure linear convergence and a bound on the expected hitting time of an ES family, namely, the $(1+1)_\kappa$-ES, which includes the (1+1)-ES with (generalized) one-fifth success rule and an abstract covariance matrix adaptation with bounded condition number, on a broad class of functions. The analysis holds for monotonic transformations of positively homogeneous functions and of quadratically bounded functions, the latter of which particularly includes monotonic transformation of strongly convex functions with Lipschitz continuous gradient. As far as the authors know, this is the first work that proves linear convergence of ES on such a broad class of functions.

Accelerated inexact composite gradient methods for nonconvex spectral optimization problems

This paper presents two inexact composite gradient methods, one inner accelerated and another doubly accelerated, for solving a class of nonconvex spectral composite optimization problems. More specifically, the objective function for these problems is of the form $f_1 + f_2 + h$ where $f_1$ and $f_2$ are differentiable nonconvex matrix functions with Lipschitz continuous gradients, $h$ is a proper closed convex matrix function, and both $f_2$ and $h$ can be expressed as functions that operate on the singular values of their inputs. The methods essentially use an accelerated composite gradient method to solve a sequence of proximal subproblems involving the linear approximation of $f_1$ and the singular value functions underlying $f_2$ and $h$. Unlike other composite gradient-based methods, the proposed methods take advantage of both the composite and spectral structure underlying the objective function in order to efficiently generate their solutions. Numerical experiments are presented to demonstrate the practicality of these methods on a set of real-world and randomly generated spectral optimization problems.

A multivariate adaptive gradient algorithm with reduced tuning efforts

Large neural networks usually perform well for executing machine learning tasks. However, models that achieve state-of-the-art performance involve arbitrarily large number of parameters and therefore their training is very expensive. It is thus desired to implement methods with small per-iteration costs, fast convergence rates, and reduced tuning. This paper proposes a multivariate adaptive gradient descent method that meets the above attributes. The proposed method updates every element of the model parameters separately in a computationally efficient manner using an adaptive vector-form learning rate, resulting in low per-iteration cost. The adaptive learning rate computes the absolute difference of current and previous model parameters over the difference in subgradients of current and previous state estimates. In the deterministic setting, we show that the cost function value converges at a linear rate for smooth and strongly convex cost functions. Whereas in both the deterministic and stochastic setting, we show that the gradient converges in expectation at the order of O(1/k) for a non-convex cost function with Lipschitz continuous gradient. In addition, we show that after T iterates, the cost function of the last iterate scales as O(log(T)/T) for non-smooth strongly convex cost functions. Effectiveness of the proposed method is validated on convex functions, smooth non-convex function, non-smooth convex function, and four image classification data sets, whilst showing that its execution requires hardly any tuning unlike existing popular optimizers that entail relatively large tuning efforts. Our empirical results show that our proposed algorithm provides the best overall performance when comparing it to tuned state-of-the-art optimizers.

A Proximal Quasi-Newton Trust-Region Method for Nonsmooth Regularized Optimization

We develop a trust-region method for minimizing the sum of a smooth term (f) and a nonsmooth term (h), both of which can be nonconvex. Each iteration of our method minimizes a possibly nonconvex model of (f + h) in a trust region. The model coincides with (f + h) in value and subdifferential at the center. We establish global convergence to a first-order stationary point when (f) satisfies a smoothness condition that holds, in particular, when it has a Lipschitz-continuous gradient, and (h) is proper and lower semicontinuous. The model of (h) is required to be proper, lower semi-continuous and prox-bounded. Under these weak assumptions, we establish a worst-case (O(1/\epsilon^2)) iteration complexity bound that matches the best known complexity bound of standard trust-region methods for smooth optimization. We detail a special instance, named TR-PG, in which we use a limited-memory quasi-Newton model of (f) and compute a step with the proximal gradient method, resulting in a practical proximal quasi-Newton method. We establish similar convergence properties and complexity bound for a quadratic regularization variant, named R2, and provide an interpretation as a proximal gradient method with adaptive step size for nonconvex problems. R2 may also be used to compute steps inside the trust-region method, resulting in an implementation named TR-R2. We describe our Julia implementations and report numerical results on inverse problems from sparse optimization and signal processing. Both TR-PG and TR-R2 exhibit promising performance and compare favorably with two linesearch proximal quasi-Newton methods based on convex models.

Second Order Splitting Dynamics with Vanishing Damping for Additively Structured Monotone Inclusions

In the framework of a real Hilbert space, we address the problem of finding the zeros of the sum of a maximally monotone operator A and a cocoercive operator B. We study the asymptotic behaviour of the trajectories generated by a second order equation with vanishing damping, attached to this problem, and governed by a time-dependent forward–backward-type operator. This is a splitting system, as it only requires forward evaluations of B and backward evaluations of A. A proper tuning of the system parameters ensures the weak convergence of the trajectories to the set of zeros of A+B\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$A + B$$\\end{document}, as well as fast convergence of the velocities towards zero. A particular case of our system allows to derive fast convergence rates for the problem of minimizing the sum of a proper, convex and lower semicontinuous function and a smooth and convex function with Lipschitz continuous gradient. We illustrate the theoretical outcomes by numerical experiments.