Accelerated Gradient Method Research Articles

Combining fast inertial dynamics for convex optimization with Tikhonov regularization

In a Hilbert space setting H, we study the convergence properties as t→+∞ of the trajectories of the second-order differential equation(AVD)α,ϵx¨(t)+αtx˙(t)+∇Φ(x(t))+ϵ(t)x(t)=0, where ∇Φ is the gradient of a convex continuously differentiable function Φ:H→R, α is a positive parameter, and ϵ(t)x(t) is a Tikhonov regularization term, with ϵ(t) positive, and limt→∞⁡ϵ(t)=0. In this damped inertial system, the damping coefficient αt vanishes asymptotically, but not too quickly, a key property to obtain rapid convergence of the values. In the case ϵ(⋅)≡0, this dynamic has been highlighted recently by Su, Boyd, and Candès as a continuous version of the Nesterov accelerated gradient method. Depending on the speed of convergence of ϵ(t) to zero, we analyze the convergence properties of the trajectories of (AVD)α,ϵ. We obtain results ranging from the rapid convergence of Φ(x(t)) to min⁡Φ when ϵ(t) decreases rapidly to zero, up to the strong convergence of the trajectories to the element of minimum norm of the set of minimizers of Φ, when ϵ(t) tends slowly to zero. When ϵ(t)=1tr, the critical value of r separating the two above cases is r=2.

A variational perspective on accelerated methods in optimization

Accelerated gradient methods play a central role in optimization, achieving optimal rates in many settings. Although many generalizations and extensions of Nesterov's original acceleration method have been proposed, it is not yet clear what is the natural scope of the acceleration concept. In this paper, we study accelerated methods from a continuous-time perspective. We show that there is a Lagrangian functional that we call the Bregman Lagrangian, which generates a large class of accelerated methods in continuous time, including (but not limited to) accelerated gradient descent, its non-Euclidean extension, and accelerated higher-order gradient methods. We show that the continuous-time limit of all of these methods corresponds to traveling the same curve in spacetime at different speeds. From this perspective, Nesterov's technique and many of its generalizations can be viewed as a systematic way to go from the continuous-time curves generated by the Bregman Lagrangian to a family of discrete-time accelerated algorithms.

Achieving Low-Delay and Fast-Convergence in Stochastic Network Optimization

Due to the rapid growth of mobile data demands, there have been significant interests in stochastic resource control and optimization for wireless networks. Although significant advances have been made in stochastic network optimization theory, to date, most of the existing approaches are plagued by either slow convergence or unsatisfactory delay performances. To address these challenges, in this paper, we develop a new stochastic network optimization framework inspired by the Nesterov accelerated gradient method. We show that our proposed Nesterovian approach offers utility-optimality, fast-convergence, and significant delay reduction in stochastic network optimization. Our contributions in this paper are three-fold: i) we propose a Nesterovian joint congestion control and routing/scheduling framework for both single-hop and multi-hop wireless networks; ii) we establish the utility optimality and queueing stability of the proposed Nesterovian method, and analytically characterize its delay reduction and convergence speed; and iii) we show that the proposed Nesterovian approach offers a three-way performance control between utility-optimality, delay, and convergence.

A differential equation for modeling Nesterov's accelerated gradient method

We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus c...

骨組の幾何学的非線形解析に対する加速勾配法

A new method of geometrically nonlinear analysis of large-scale framed structures are proposed in this paper by using accelerated-gradient-method (AGM). AGM has attracted attention in recent years as an effective method for large-scale unconstrained single-objective optimization problem especially in the field of machine learning. A geometrically nonlinear analysis can be formulated as a total potential energy minimization problem. In this study, AGM is adopted for solving the total potential energy minimization problem efficiently in order to solve a geometrically nonlinear analysis at higher speed.

Distributed Nonlinear Model Predictive Control by Sequential Linearization and Accelerated Gradient Method

A suboptimal approach to distributed NMPC for nonlinear interconnected systems subject to constraints is proposed. The objective is to develop a computationally efficient approach. The suggested method is based on a sequential linearization of the nonlinear system dynamics and finding a suboptimal solution of the resulting Quadratic Programming problem by using distributed iterations of the dual accelerated gradient method. The benefits of the approach are reduced complexity of the on-line computations, and simple software implementation, which makes it appropriate for embedded distributed convex NMPC. The proposed method is illustrated with simulations on the model of a quadruple-tank system.

The Log-Exponential Smoothing Technique and Nesterov’s Accelerated Gradient Method for Generalized Sylvester Problems

The Sylvester or smallest enclosing circle problem involves finding the smallest circle enclosing a finite number of points in the plane. We consider generalized versions of the Sylvester problem in which the points are replaced by sets. Based on the log-exponential smoothing technique and Nesterov's accelerated gradient method, we present an effective numerical algorithm for solving these problems.

A Fast Low Complexity Method for Optimal Zero-Forcing Beamformer MU-MIMO System

This letter proposes a new algorithm for solving the optimal zero-forcing beamforming problem that maximizes the user achievable rate with restriction on the per-antenna element power constraints. An accelerated gradient method with step size search is proposed for solving the problem. For each iteration of the gradient approach, a quick one dimensional search is employed to obtain the step size. The advantage of the step-size search is that it is relatively fast and requires only a few calculations of the objective function. Design examples show that the proposed algorithm converges faster than the gradient approach and the accelerated gradient approach with a constant step size while achieving a low computational complexity.

Fast Desynchronization for Decentralized Multichannel Medium Access Control

Distributed desynchronization algorithms are key to wireless sensor networks as they allow for medium access control in a decentralized manner. In this paper, we view desynchronization primitives as iterative methods that solve optimization problems. In particular, by formalizing a well established desynchronization algorithm as a gradient descent method, we establish novel upper bounds on the number of iterations required to reach convergence. Moreover, by using Nesterov's accelerated gradient method, we propose a novel desynchronization primitive that provides for faster convergence to the steady state. Importantly, we propose a novel algorithm that leads to decentralized time-synchronous multichannel TDMA coordination by formulating this task as an optimization problem. Our simulations and experiments on a densely-connected IEEE 802.15.4-based wireless sensor network demonstrate that our scheme provides for faster convergence to the steady state, robustness to hidden nodes, higher network throughput and comparable power dissipation with respect to the recently standardized IEEE 802.15.4e-2012 time-synchronized channel hopping (TSCH) scheme.

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

In this paper, we generalize the well-known Nesterov's accelerated gradient (AG) method, originally designed for convex smooth optimization, to solve nonconvex and possibly stochastic optimization problems. We demonstrate that by properly specifying the stepsize policy, the AG method exhibits the best known rate of convergence for solving general nonconvex smooth optimization problems by using first-order information, similarly to the gradient descent method. We then consider an important class of composite optimization problems and show that the AG method can solve them uniformly, i.e., by using the same aggressive stepsize policy as in the convex case, even if the problem turns out to be nonconvex. We demonstrate that the AG method exhibits an optimal rate of convergence if the composite problem is convex, and improves the best known rate of convergence if the problem is nonconvex. Based on the AG method, we also present new nonconvex stochastic approximation methods and show that they can improve a few existing rates of convergence for nonconvex stochastic optimization. To the best of our knowledge, this is the first time that the convergence of the AG method has been established for solving nonconvex nonlinear programming in the literature.

Accelerating Block-Decomposition First-Order Methods for Solving Composite Saddle-Point and Two-Player Nash Equilibrium Problems

This article considers the (two-player) composite Nash equilibrium (CNE) problem with a separable nonsmooth part, which is known to include the composite saddle-point (CSP) problem as a special case. Due to its two-block structure, this problem can be solved by any algorithm belonging to the block-decomposition hybrid proximal-extragradient (BD-HPE) framework proposed in [R. D. C. Monteiro and B. F. Svaiter, SIAM J. Optim., 23 (2013), pp. 475--507]. The framework consists of a family of inexact proximal point methods for solving a more general two-block structured monotone inclusion problem which, at every iteration, solves two prox subinclusions according to a certain relative error criterion. By exploiting the fact that the two prox subinclusions in the context of the CNE problem are equivalent to two composite convex programs, this article proposes a new instance of the BD-HPE framework that approximately solves them using an accelerated gradient method. It is shown that the new instance is able to take significantly larger prox stepsizes than other instances from this framework that perform single composite gradient steps for solving the subinclusions. As a result, it is shown that the first instance has better iteration-complexity than the latter ones. Finally, it is also shown that the new accelerated BD-HPE instance computationally outperforms several state-of-the-art algorithms on many relevant classes of CSP and CNE instances.

An Inertial Forward-Backward Algorithm for Monotone Inclusions

In this paper, we propose an inertial forward backward splitting algorithm to compute a zero of the sum of two monotone operators, with one of the two operators being co-coercive. The algorithm is inspired by the accelerated gradient method of Nesterov, but can be applied to a much larger class of problems including convex-concave saddle point problems and general monotone inclusions. We prove convergence of the algorithm in a Hilbert space setting and show that several recently proposed first-order methods can be obtained as special cases of the general algorithm. Numerical results show that the proposed algorithm converges faster than existing methods, while keeping the computational cost of each iteration basically unchanged.

Accelerated gradient methods for the x-ray imaging of solar flares

In this paper we present new optimization strategies for the reconstruction of x-ray images of solar flares by means of the data collected by the Reuven Ramaty high energy solar spectroscopic imager. The imaging concept of the satellite is based on rotating modulation collimator instruments, which allow the use of both Fourier imaging approaches and reconstruction techniques based on the straightforward inversion of the modulated count profiles. Although in the last decade, greater attention has been devoted to the former strategies due to their very limited computational cost, here we consider the latter model and investigate the effectiveness of different accelerated gradient methods for the solution of the corresponding constrained minimization problem. Moreover, regularization is introduced through either an early stopping of the iterative procedure, or a Tikhonov term added to the discrepancy function by means of a discrepancy principle accounting for the Poisson nature of the noise affecting the data.

An Approach to Distributed Robust Model Predictive Control of Discrete-Time Polytopic Systems

A suboptimal approach to distributed robust MPC for uncertain systems consisting of polytopic subsystems with coupled dynamics subject to both state and input constraints is proposed. The approach applies the dynamic dual decomposition method and reformulates the original centralized robust MPC problem into a distributed robust MPC problem. It is based on distributed on-line optimization by applying an accelerated gradient method. The suggested approach is illustrated on a simulation example of an uncertain system consisting of two interconnected polytopic subsystems.

A distributed accelerated gradient algorithm for distributed model predictive control of a hydro power valley

A distributed model predictive control (DMPC) approach based on distributed optimization is applied to the power reference tracking problem of a hydro power valley (HPV) system. The applied optimization algorithm is based on accelerated gradient methods and achieves a convergence rate of O(1/k2), where k is the iteration number. Major challenges in the control of the HPV include a nonlinear and large-scale model, nonsmoothness in the power-production functions, and a globally coupled cost function that prevents distributed schemes to be applied directly. We propose a linearization and approximation approach that accommodates the proposed the DMPC framework and provides very similar performance compared to a centralized solution in simulations. The provided numerical studies also suggest that for the sparsely interconnected system at hand, the distributed algorithm we propose is faster than a centralized state-of-the-art solver such as CPLEX.

Incremental accelerated gradient methods for SVM classification: study of the constrained approach

We investigate constrained first order techniques for training support vector machines (SVM) for online classification tasks. The methods exploit the structure of the SVM training problem and combine ideas of incremental gradient technique, gradient acceleration and successive simple calculations of Lagrange multipliers. Both primal and dual formulations are studied and compared. Experiments show that the constrained incremental algorithms working in the dual space achieve the best trade-off between prediction accuracy and training time. We perform comparisons with an unconstrained large scale learning algorithm (Pegasos stochastic gradient) to emphasize that our choice can remain competitive for large scale learning due to the very special structure of the training problem.

Sparse Multi-Task Learning for Detecting Influential Nodes in an Implicit Diffusion Network

How to identify influential nodes is a central research topic in information diffusion analysis. Many existing methods rely on the assumption that the network structure is completely known by the model. However, in many applications, such a network is either unavailable or insufficient to explain the underlying information diffusion phenomena. To address this challenge, we develop a multi-task sparse linear influence model (MSLIM), which can simultaneously predict the volume for each contagion and automatically identify sets of the most influential nodes for different contagions. Our method is based on the linear influence model with two main advantages: 1) it does not require the network structure; 2) it can detect different sets of the most influential nodes for different contagions. To solve the corresponding convex optimization problem for learning the model, we adopt the accelerated gradient method (AGM) framework and show that there is an exact closed-form solution for the proximal mapping. Therefore, the optimization procedure achieves the optimal first-order convergence rate and can be scaled to very large datasets. The proposed model is validated on a set of 2.6 millions tweets from 1000 users of Twitter. We show that MSLIM can efficiently select the most influential users for specific contagions. We also present several interesting patterns of the selected influential users.

Accelerated gradient methods and dual decomposition in distributed model predictive control

We propose a distributed optimization algorithm for mixed L1/L2-norm optimization based on accelerated gradient methods using dual decomposition. The algorithm achieves convergence rate O(1k2), where k is the iteration number, which significantly improves the convergence rates of existing duality-based distributed optimization algorithms that achieve O(1k). The performance of the developed algorithm is evaluated on randomly generated optimization problems arising in distributed model predictive control (DMPC). The evaluation shows that, when the problem data is sparse and large-scale, our algorithm can outperform current state-of-the-art optimization software CPLEX and MOSEK.

Efficient sparse modeling with automatic feature grouping.

For high-dimensional data, it is often desirable to group similar features together during the learning process. This can reduce the estimation variance and improve the stability of feature selection, leading to better generalization. Moreover, it can also help in understanding and interpreting data. Octagonal shrinkage and clustering algorithm for regression (OSCAR) is a recent sparse-modeling approach that uses a l1 -regularizer and a pairwise l∞-regularizer on the feature coefficients to encourage such feature grouping. However, computationally, its optimization procedure is very expensive. In this paper, we propose an efficient solver based on the accelerated gradient method. We show that its key proximal step can be solved by a highly efficient simple iterative group merging algorithm. Given d input features, this reduces the empirical time complexity from O(d(2) ~ d(5)) for the existing solvers to just O(d). Experimental results on a number of toy and real-world datasets demonstrate that OSCAR is a competitive sparse-modeling approach, but with the added ability of automatic feature grouping.

Modeling Disease Progression via Fused Sparse Group Lasso.

Alzheimer's Disease (AD) is the most common neurodegenerative disorder associated with aging. Understanding how the disease progresses and identifying related pathological biomarkers for the progression is of primary importance in the clinical diagnosis and prognosis of Alzheimer's disease. In this paper, we develop novel multi-task learning techniques to predict the disease progression measured by cognitive scores and select biomarkers predictive of the progression. In multi-task learning, the prediction of cognitive scores at each time point is considered as a task, and multiple prediction tasks at different time points are performed simultaneously to capture the temporal smoothness of the prediction models across different time points. Specifically, we propose a novel convex fused sparse group Lasso (cFSGL) formulation that allows the simultaneous selection of a common set of biomarkers for multiple time points and specific sets of biomarkers for different time points using the sparse group Lasso penalty and in the meantime incorporates the temporal smoothness using the fused Lasso penalty. The proposed formulation is challenging to solve due to the use of several non-smooth penalties. One of the main technical contributions of this paper is to show that the proximal operator associated with the proposed formulation exhibits a certain decomposition property and can be computed efficiently; thus cFSGL can be solved efficiently using the accelerated gradient method. To further improve the model, we propose two non-convex formulations to reduce the shrinkage bias inherent in the convex formulation. We employ the difference of convex (DC) programming technique to solve the non-convex formulations. We have performed extensive experiments using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Results demonstrate the effectiveness of the proposed progression models in comparison with existing methods for disease progression. We also perform longitudinal stability selection to identify and analyze the temporal patterns of biomarkers in disease progression.