Accelerated Gradient Method Research Articles

A feasible smoothing accelerated projected gradient method for nonsmooth convex optimization

Smoothing accelerated gradient methods achieve faster convergence rates than that of the subgradient method for some nonsmooth convex optimization problems. However, Nesterov's extrapolation may require gradients at infeasible points, and thus they cannot be applied to some structural optimization problems. We introduce a variant of smoothing accelerated projected gradient methods where every variable is feasible. The O(k−1log⁡k) convergence rate is obtained using the Lyapunov function. We conduct a numerical experiment on the robust compliance optimization of a truss structure.

Learning Hidden Markov Models with Structured Transition Dynamics

The hidden Markov model (HMM) provides a natural framework for modeling the dynamic evolution of latent diseases. The unknown probability matrices of HMMs can be learned through the well-known Baum–Welch algorithm, a special case of the expectation-maximization algorithm. In many disease models, the probability matrices possess nontrivial properties that may be represented through a set of linear constraints. In these cases, the traditional Baum–Welch algorithm is no longer applicable because the maximization step cannot be solved by an explicit formula. In this paper, we propose a novel approach to efficiently solve the maximization step problem under linear constraints by providing a Lagrangian dual reformulation that we solve by an accelerated gradient method. The performance of this approach critically depends on devising a fast method to compute the gradient in each iteration. For this purpose, we employ dual decomposition and derive Karush–Kuhn–Tucker conditions to reduce our problem into a set of single variable equations, solved using a simple bisection method. We apply this method to a case study on sports-related concussion and provide an extensive numerical study using simulation. We show that our approach is in orders of magnitude computationally faster and more accurate than other alternative approaches. Moreover, compared with other methods, our approach is far less sensitive with respect to increases in problem size. Overall, our contribution lies in the advancement of accurately and efficiently handling HMM parameter estimation under linear constraints, which comprises a wide range of applications in disease modeling and beyond. History: Accepted by Paul Brooks, Area Editor for Applications in Biology, Medicine, & Healthcare. Funding: This research was funded by [Grant R01DE028283] from the National Institute of Dental and Craniofacial Research, National Institutes of Health. G.-G. Garcia was also funded by the Georgia Clinical & Translational Science Alliance National Institutes of Health award [Grant UL1-TR002378]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0342 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0342 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Accelerated optimization through time-scale analysis of inertial dynamics with asymptotic vanishing and Hessian-driven dampings

ABSTRACT Gradient optimization algorithms can be effectively studied from the perspective of ordinary differential equations (ODEs), where these algorithms are obtained through the temporal discretization of the continuous dynamics. This approach provides a powerful tool to comprehend acceleration phenomena in optimization, thanks to time-scale techniques and Lyapunov analysis. In the context of a Hilbert setting, our study focuses on the rapid optimization properties of inertial dynamics, which combine asymptotically vanishing viscous damping with Hessian-driven damping. These dynamics are derived as high-resolution ODEs from both the Nesterov accelerated gradient method and the Ravine method. For a general differentiable convex function f, choosing the viscous damping coefficient in the form of α / t with 3 $ ]]> α > 3 guarantees an inverse quadratic convergence rate of the values, i.e. o ( 1 / t 2 ) , the weak convergence of trajectories towards optimal solutions, and the fast convergence of gradients towards zero. By carefully scaling the dynamics temporally and based on the tuning of the damping coefficient in front of the Hessian term, we identify the limit dynamic as α becomes large. In particular, we examine the case where the limit dynamic corresponds to the Levenberg-Marquardt regularization of Newton's continuous method. This explanation accounts for the aforementioned fast convergence properties and provides fresh insights into the complexity of these methods, which play a central role in optimization for high-dimensional problems. The interplay between numerical algorithms and continuous-time ODEs plays a fundamental role in our analysis for the design and understanding of accelerated optimization algorithms. Numerical experiments are presented to illustrate and support the theoretical results.

Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution

In a Hilbertian framework, for the minimization of a general convex differentiable function f, we introduce new inertial dynamics and algorithms that generate trajectories and iterates that converge fastly towards the minimizer of f with minimum norm. Our study is based on the non-autonomous version of the Polyak heavy ball method, which, at time t, is associated with the strongly convex function obtained by adding to f a Tikhonov regularization term with vanishing coefficient ε(t)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varepsilon (t)$$\\end{document}. In this dynamic, the damping coefficient is proportional to the square root of the Tikhonov regularization parameter ε(t)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varepsilon (t)$$\\end{document}. By adjusting the speed of convergence of ε(t)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varepsilon (t)$$\\end{document} towards zero, we will obtain both rapid convergence towards the infimal value of f, and the strong convergence of the trajectories towards the element of minimum norm of the set of minimizers of f. In particular, we obtain an improved version of the dynamic of Su-Boyd-Candès for the accelerated gradient method of Nesterov. This study naturally leads to corresponding first-order algorithms obtained by temporal discretization. In the case of a proper lower semicontinuous and convex function f, we study the proximal algorithms in detail, and show that they benefit from similar properties.

Accelerated Gradient Approach For Deep Neural Network-Based Adaptive Control of Unknown Nonlinear Systems.

Recent connections in the adaptive control literature to continuous-time analogs of Nesterov's accelerated gradient method have led to the development of new real-time adaptation laws based on accelerated gradient methods. However, previous results assume that the system's uncertainties are linear-in-the-parameters (LIP). To compensate for non-LIP uncertainties, our preliminary results developed a neural network (NN)-based accelerated gradient adaptive controller to achieve trajectory tracking for nonlinear systems; however, the development and analysis only considered single-hidden-layer NNs. In this article, a generalized deep NN (DNN) architecture with an arbitrary number of hidden layers is considered, and a new DNN-based accelerated gradient adaptation scheme is developed to generate estimates of all the DNN weights in real-time. A nonsmooth Lyapunov-based analysis is used to guarantee the developed accelerated gradient-based DNN adaptation design achieves global asymptotic tracking error convergence for general nonlinear control affine systems subject to unknown (non-LIP) drift dynamics and exogenous disturbances. A comprehensive set of simulation studies are conducted on a two-state nonlinear system, a robotic manipulator, and a complex 20-D nonlinear system to demonstrate the improved performance of the developed method. Our simulation studies demonstrate enhanced tracking and function approximation performance from both DNN architectures and accelerated gradient adaptation.

Distributed accelerated gradient methods with restart under quadratic growth condition

Distributed accelerated gradient methods with restart under quadratic growth condition

Accelerated analysis on the triple momentum method for a two-layer ReLU neural network

The momentum method has become the workhorse in the deep learning community. To theoretical understand its success, researchers put efforts in demystifying its convergence properties when optimizing neural networks. For the convex problem, it is well-known that the triple momentum (TM) method owns the fastest theoretical convergence rate among all first-order methods. However, there exists no theoretical convergence results about the TM method in solving the non-convex neural networks training problem, let alone its acceleration guarantee. In this paper, we focus on the training process of the TM method for a two-layer ReLU neural network. Inspired by the accurate characterization of the high-resolution dynamical system, we consider the high-resolution ordinary differential equation (ODE) of the TM method. Under the over-parameterized assumption, we derive that the original non-convex optimization problem can be transformed to a strongly convex task. By applying an appropriate Lyapunov function, we prove that the TM method can linearly converge to a global minimum. Compared to the heavy ball method and Nesterov’s accelerated gradient method, our result provides the first guarantee for the acceleration of the TM method in training neural networks. Through empirical experiments, the accelerated convergence of the TM method and the effect of over-parameterization are validated.

Faster algorithm and sharper analysis for constrained Markov decision process

The problem of constrained Markov decision process (CMDP) is investigated, where an agent aims to maximize the expected accumulated reward subject to constraints on its utilities/costs. We propose a new primal-dual approach with a novel integration of entropy regularization and Nesterov's accelerated gradient method. The proposed approach is shown to converge to the global optimum with a complexity of O˜(1/ϵ) in terms of the optimality gap and the constraint violation, which improves the complexity of the existing primal-dual approaches by a factor of O(1/ϵ).

Hybrid Modified Accelerated Gradient Method for Optimization Processes

This research reveals a hybrid variant of the modified accelerated gradient method. We prove that derived iteration is linearly convergent on the set of uniformly convex functions. Performance profiles of the introduced hybrid method were numerically compared with its non-hybrid version. The analyzed characteristics were CPU time, the number of iterations and the number of function evaluations. The results of the numerical experiments show a better performance in favor of the derived hybrid accelerated model compared with its forerunner.

Descent Properties of an Anderson Accelerated Gradient Method with Restarting

Descent Properties of an Anderson Accelerated Gradient Method with Restarting

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

Accelerated gradient methods for sparse statistical learning with nonconvex penalties

Fractional Order Euler-Lagrange Model for Accelerated Gradient Methods

In this study, a fractional order equation of motion as a continuous limit model for a family of gradient descent algorithms is discussed based on fractional order Euler-Lagrange equation. The aim of this proposed scheme is to search the ability to go beyond Nesterov scheme by introducing the potential of fractional calculus. The discretized version of the ”designed” fractional order equation of motion (FO- EOM) forms new gradient descent algorithm that has been tested on some optimization benchmark functions to fairly assess the performance in comparison with the standard and accelerated form of gradient descent algorithms. Promising results have been obtained with more rigorous mathematical analysis has to be carried out as our future work.

Iteration Complexity of Fixed-Step Methods by Nesterov and Polyak for Convex Quadratic Functions

This note considers the momentum method by Polyak and the accelerated gradient method by Nesterov, both without line search but with fixed step length applied to strongly convex quadratic functions assuming that exact gradients are used and appropriate upper and lower bounds for the extreme eigenvalues of the Hessian matrix are known. Simple 2-d-examples show that the Euclidean distance of the iterates to the optimal solution is non-monotone. In this context, an explicit bound is derived on the number of iterations needed to guarantee a reduction of the Euclidean distance to the optimal solution by a factor ϵ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\epsilon $$\\end{document}. For both methods, the bound is optimal up to a constant factor, it complements earlier asymptotically optimal results for the momentum method, and it establishes another link of the momentum method and Nesterov’s accelerated gradient method.

Accelerated gradient methods with absolute and relative noise in the gradient

In this paper, we investigate accelerated first-order methods for smooth convex optimization problems under inexact information on the gradient of the objective. The noise in the gradient is considered to be additive with two possibilities: absolute noise bounded by a constant, and relative noise proportional to the norm of the gradient. We investigate the accumulation of the errors in the convex and strongly convex settings with the main difference with most of the previous works being that the feasible set can be unbounded. The key to the latter is to prove a bound on the trajectory of the algorithm. We also give a stopping criterion for the algorithm and consider extensions to the cases of stochastic optimization and composite nonsmooth problems.

Distributed optimization method for economic dispatch of active distribution networks via momentum with historical information and forecast gradient

In the context of vigorously developing new energy sources, the economic dispatch(ED) of active distribution network (ADN) is essential. Due to the single fault affecting the global and high communication cost in centralized scheduling, we propose a distributed gradient method, namely the fast Nesterov accelerated gradient method(FNAGM), which can solve the economic dispatch problem(EDP) in ADN. The distributed architecture does not need to collect global information and only uses a sparse communication network to complete communication exchanges. It is the distributed architecture that can protect users’ private information and reduce communication pressure. By constructing the acceleration matrix based on the upper limit of the second derivative, the convergence speed can be effectively improved while satisfying the equality constraints. Eventually, the FNAGM combined with the historical momentum information and the forecast gradient, which is simulated in the bi-layer model of ADN via MATLAB. The verification results show that the algorithm with the historical momentum information and the forecast gradient can complete the optimal scheduling of controllable distributed generator(DG). What is more, the convergence efficiency performance is greatly improved.

Accelerated Gradient Methods Combining Tikhonov Regularization with Geometric Damping Driven by the Hessian

Accelerated Gradient Methods Combining Tikhonov Regularization with Geometric Damping Driven by the Hessian

Efficient multi-task learning with adaptive temporal structure for progression prediction

In this paper, we propose a novel efficient multi-task learning formulation for the class of progression problems in which its state will continuously change over time. To use the shared knowledge information between multiple tasks to improve performance, existing multi-task learning methods mainly focus on feature selection or optimizing the task relation structure. The feature selection methods usually fail to explore the complex relationship between tasks and thus have limited performance. The methods centring on optimizing the relation structure of tasks are not capable of selecting meaningful features and have a bi-convex objective function which results in high computation complexity of the associated optimization algorithm. Unlike these multi-task learning methods, motivated by a simple and direct idea that the state of a system at the current time point should be related to all previous time points, we first propose a novel relation structure, termed adaptive global temporal relation structure (AGTS). Then we integrate the widely used sparse group Lasso, fused Lasso with AGTS to propose a novel convex multi-task learning formulation that not only performs feature selection but also adaptively captures the global temporal task relatedness. Since the existence of three non-smooth penalties, the objective function is challenging to solve. We first design an optimization algorithm based on the alternating direction method of multipliers (ADMM). Considering that the worst-case convergence rate of ADMM is only sub-linear, we then devise an efficient algorithm based on the accelerated gradient method which has the optimal convergence rate among first-order methods. We show the proximal operator of several non-smooth penalties can be solved efficiently due to the special structure of our formulation. Experimental results on four real-world datasets demonstrate that our approach not only outperforms multiple baseline MTL methods in terms of effectiveness but also has high efficiency.

Bregman dynamics, contact transformations and convex optimization

Recent research on accelerated gradient methods of use in optimization has demonstrated that these methods can be derived as discretizations of dynamical systems. This, in turn, has provided a basis for more systematic investigations, especially into the geometric structure of those dynamical systems and their structure-preserving discretizations. In this work, we introduce dynamical systems defined through a contact geometry which are not only naturally suited to the optimization goal but also subsume all previous methods based on geometric dynamical systems. As a consequence, all the deterministic flows used in optimization share an extremely interesting geometric property: they are invariant under contact transformations. In our main result, we exploit this observation to show that the celebrated Bregman Hamiltonian system can always be transformed into an equivalent but separable Hamiltonian by means of a contact transformation. This in turn enables the development of fast and robust discretizations through geometric contact splitting integrators. As an illustration, we propose the Relativistic Bregman algorithm, and show in some paradigmatic examples that it compares favorably with respect to standard optimization algorithms such as classical momentum and Nesterov’s accelerated gradient.

Distributed data-driven distributionally robust Volt/Var control for distribution network via an accelerated alternating optimization procedure

This paper proposes a distributed data-driven distributionally robust volt/var control (DDDR-VVC) approach which schedules on-load-tap changer (OLTC), capacitor banks (CBs) and Photovoltaic (PV) inverters coordinately. With integration of distributed optimization and data-driven distributionally robust optimization, the DDDR-VVC model is constructed to reduce power losses and maintain voltage within allowable range under uncertainty. To fully address the uncertainty impacts, construction of uncertain set is improved through that the probability distribution sets and trapezoidal fuzzy functions are developed to model PV supply and load demand respectively. An accelerated alternating optimization procedure which integrates the alternating direction method of multipliers and the column-and-constraint generation algorithm is proposed. This method improves the dual information update via Nesterov’s accelerated gradient method, thus solving the DDDR-VVC model at a high convergence rate. The effectiveness of proposed method is verified through numerical simulations using IEEE 123 bus test system.

Nesterov’s acceleration for level set-based topology optimization using reaction-diffusion equations

This paper discusses level set-based structural optimization. Level set-based structural optimization is a method used to determine an optimal configuration for minimizing objective functionals by updating level set functions characterized as solutions to partial differential equations (PDEs) (e.g., Hamilton-Jacobi and reaction-diffusion equations). In this study, based on Nesterov’s accelerated gradient method, a nonlinear (damped) wave equation will be derived as a PDE satisfied by level set functions and applied to minimum mean compliance problems. Numerically, the method developed in this study will yield convergence to an optimal configuration faster than methods using only a reaction-diffusion equation, and moreover, its FreeFEM++ code will also be described.