Enhanced ADMM for solving the Lasso problem: integrating the Nesterov accelerated gradient method

We derive a second-order ordinary differential equation (ODE) which is the limit of Nesterov's accelerated gradient method. This ODE exhibits approximate equivalence to Nesterov's scheme and thus c...

This paper studies an acceleration technique for incremental aggregated gradient ({\sf IAG}) method through the use of \emph{curvature} information for solving strongly convex finite sum optimization problems. These optimization problems of interest arise in large-scale learning applications. Our technique utilizes a curvature-aided gradient tracking step to produce accurate gradient estimates incrementally using Hessian information. We propose and analyze two methods utilizing the new technique, the curvature-aided IAG ({\sf CIAG}) method and the accelerated CIAG ({\sf A-CIAG}) method, which are analogous to gradient method and Nesterov's accelerated gradient method, respectively. Setting $\kappa$ to be the condition number of the objective function, we prove the $R$ linear convergence rates of $1 - \frac{4c_0 \kappa}{(\kappa+1)^2}$ for the {\sf CIAG} method, and $1 - \sqrt{\frac{c_1}{2\kappa}}$ for the {\sf A-CIAG} method, where $c_0,c_1 \leq 1$ are constants inversely proportional to the distance between the initial point and the optimal solution. When the initial iterate is close to the optimal solution, the $R$ linear convergence rates match with the gradient and accelerated gradient method, albeit {\sf CIAG} and {\sf A-CIAG} operate in an incremental setting with strictly lower computation complexity. Numerical experiments confirm our findings. The source codes used for this paper can be found on \url{http://github.com/hoitowai/ciag/}.

We present two classes of differentially private optimization algorithms derived from the well-known accelerated first-order methods. The first algorithm is inspired by Polyak's heavy ball method and employs a smoothing approach to decrease the accumulated noise on the gradient steps required for differential privacy. The second class of algorithms are based on Nesterov's accelerated gradient method and its recent multi-stage variant. We propose a noise dividing mechanism for the iterations of Nesterov's method in order to improve the error behavior of the algorithm. The convergence rate analyses are provided for both the heavy ball and the Nesterov's accelerated gradient method with the help of the dynamical system analysis techniques. Finally, we conclude with our numerical experiments showing that the presented algorithms have advantages over the well-known differentially private algorithms.

Network utility maximization (NUM) is a well-studied problem for network traffic management and resource allocation. Because of the inherent decentralization and complexity of networks, most researches develop decentralized NUM algorithms. In recent years, the Software Defined Networking (SDN) architecture has been widely used, especially in cloud networks and inter-datacenter networks managed by large enterprises, promoting the design of centralized NUM algorithms. To cope with the large and increasing number of flows in such SDN networks, existing researches about centralized NUM focus on the scalability of the algorithm with respect to the number of flows, however the efficiency is ignored. In this paper, we focus on the SDN scenario, and derive a centralized, efficient and scalable algorithm for the NUM problem. By the designing of a smooth utility function and a smooth penalty function, we formulate the NUM problem with a smooth objective function, which enables the use of Nesterov's accelerated gradient method. We prove that the proposed method has <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$O(d/t^{2})$</tex> convergence rate, which is the fastest with respect to the number of iterations <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$t$</tex> , and our method is scalable with respect to the number of flows <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$d$</tex> in the network. Experiments show that our method obtains accurate solutions with less iterations, and achieves close-to-optimal network utility.

Recent connections in the adaptive control literature to continuous-time analogs of Nesterov's accelerated gradient method have led to the development of new real-time adaptation laws based on accelerated gradient methods. However, previous results assume that the system's uncertainties are linear-in-the-parameters (LIP). To compensate for non-LIP uncertainties, our preliminary results developed a neural network (NN)-based accelerated gradient adaptive controller to achieve trajectory tracking for nonlinear systems; however, the development and analysis only considered single-hidden-layer NNs. In this article, a generalized deep NN (DNN) architecture with an arbitrary number of hidden layers is considered, and a new DNN-based accelerated gradient adaptation scheme is developed to generate estimates of all the DNN weights in real-time. A nonsmooth Lyapunov-based analysis is used to guarantee the developed accelerated gradient-based DNN adaptation design achieves global asymptotic tracking error convergence for general nonlinear control affine systems subject to unknown (non-LIP) drift dynamics and exogenous disturbances. A comprehensive set of simulation studies are conducted on a two-state nonlinear system, a robotic manipulator, and a complex 20-D nonlinear system to demonstrate the improved performance of the developed method. Our simulation studies demonstrate enhanced tracking and function approximation performance from both DNN architectures and accelerated gradient adaptation.

The proximal point method (PPM) is a fundamental method in optimization that is often used as a building block for designing optimization algorithms. In this work, we use the PPM method to provide conceptually simple derivations along with convergence analyses of different versions of Nesterov's accelerated gradient method (AGM). The key observation is that AGM is a simple approximation of PPM, which results in an elementary derivation of the update equations and stepsizes of AGM. This view also leads to a transparent and conceptually simple analysis of AGM's convergence by using the analysis of PPM. The derivations also naturally extend to the strongly convex case. Ultimately, the results presented in this paper are of both didactic and conceptual value; they unify and explain existing variants of AGM while motivating other accelerated methods for practically relevant settings.

ABSTRACTThe nonconvex regularization, which has superiority on sparsity-inducing over the convex counterparts, has been proposed in many areas of engineering and science. In this paper, we present an accelerated regularization thresholding algorithm for sparse signal recovery, which can be viewed as an extension of the well-known Nesterov's accelerated gradient method from convex optimization to nonconvex case. It has shown numerically that the proposed algorithm keeps fast convergence, and also maintains high recovery precision. Extensive numerical experiments have been to demonstrate the effectiveness of the proposed algorithm. It is also mentioned that the proposed algorithm has much faster convergence and higher recovery precision in sparse signal recovery over the commonly non-accelerated thresholding algorithm.

In this work, we approach the minimization of a continuously differentiable convex function under linear equality constraints by a second-order dynamical system with asymptotically vanishing damping term. The system is formulated in terms of the augmented Lagrangian associated to the minimization problem. We show fast convergence of the primal-dual gap, the feasibility measure, and the objective function value along the generated trajectories. In case the objective function has Lipschitz continuous gradient, we show that the primal-dual trajectory asymptotically weakly converges to a primal-dual optimal solution of the underlying minimization problem. To the best of our knowledge, this is the first result which guarantees the convergence of the trajectory generated by a primal-dual dynamical system with asymptotic vanishing damping. Moreover, we will rediscover in case of the unconstrained minimization of a convex differentiable function with Lipschitz continuous gradient all convergence statements obtained in the literature for Nesterov's accelerated gradient method.

Sparse representation (SR)‐based inverse synthetic aperture radar (ISAR) imaging method can achieve super‐resolution image of a target. However, it is computationally expensive and sensitive to noise. To overcome these two drawbacks, the authors propose the coupled Nesterov linearised Bregman iteration algorithm based on two–dimensional (2D) real‐valued dictionaries (CNLBI‐TRD) for ISAR imaging. First, the ISAR echoes are taken as a 2D joint SR model in the range frequency‐azimuth Doppler domain. Then the 2D complex‐valued dictionaries are converted into real‐valued ones through unitary transformations. The computational complexity is thus decreased by a factor of at least four. Finally, the CNLBI algorithm is proposed to reconstruct the 2D SR model directly. It combines the Nesterov's accelerated gradient method with the condition number optimisation of sensing matrices. An adaptive‐adjustment strategy of the soft threshold parameter is presented. Thus the total iteration numbers can be greatly reduced. The simulation results and real data experiments verify the effectiveness of the proposed imaging algorithm.

The introduction of the Hessian damping in the continuous version of Nesterov's accelerated gradient method provides, by temporal discretization, fast proximal gradient algorithms where the oscillations are significantly attenuated. We will extend these results to the maximally monotone case. We rely on the technique introduced by Attouch and Peypouquet [Math. Program., 174 (2019), pp. 391--432], where the maximally monotone operator is replaced by its Yosida approximation with an appropriate adjustment of the regularization parameter. In a general Hilbert framework, we obtain the weak convergence of the iterates to equilibria, and the rapid convergence of the discrete velocities to zero. By specializing these algorithms to convex minimization, we obtain the convergence rate $o(1/k^2)$ of the values, and the rapid convergence of the gradients toward zero.

This paper describes a novel quasi-Newton (QN) based accelerated technique for training of neural networks. Recently, Nesterov's accelerated gradient method has been utilized for training the neural networks. In this paper the acceleration of the QN training algorithm is realized using Momentum term in Nesterov's method. It is shown that the proposed algorithm has a similar convergence property with the conventional QN method. Neural network training for a simple function approximation and a microwave circuit modeling is presented to demonstrate the proposed algorithm. The proposed algorithm drastically improves the convergence speed of the QN algorithm.

We present a unifying framework for adapting the update direction in gradient-based iterative optimization methods. As natural special cases we re-derive classical momentum and Nesterov's accelerated gradient method, lending a new intuitive interpretation to the latter algorithm. We show that a new algorithm, which we term Regularised Gradient Descent, can converge more quickly than either Nesterov's algorithm or the classical momentum algorithm.

This paper investigates first-order optimization algorithms derived by discretizing two high-resolution ordinary differential equations. For sufficiently smooth objective functions, we demonstrate that acceleration can be achieved via Runge-Kutta integrators. Specifically, for convex objective functions f satisfying the assumptions of Lipschitz continuous gradients and ( s + 2 ) -th order differentiability, we prove that the objective function converges to its minimum value with a rate of O ( N − 2 s s + 1 ) , where s denotes the order of the Runge-Kutta integrator. For strongly convex objective functions f, under the same assumptions, we further prove that the proposed algorithms achieve a faster convergence rate than the algorithms generated by symplectic Euler discretization. Numerical experiments validate the effectiveness and superiority of the proposed algorithms through comparisons with classical benchmark methods, including Nesterov's accelerated gradient method and the symplectic Euler scheme.

There have been many recent efforts to study accelerated optimization algorithms from the perspective of dynamical systems. In this paper, we focus on the robustness properties of the time-varying continuous-time version of these dynamics. These properties are critical for the implementation of accelerated algorithms in feedback-based control and optimization architectures. We show that a family of dynamics related to the continuous-time limit of Nesterov's accelerated gradient method can be rendered unstable under arbitrarily small bounded disturbances. Indeed, while solutions of these dynamics may converge to the set of optimizers, in general, this set may not be uniformly asymptotically stable. To induce uniformity, and robustness as a byproduct, we propose a framework where we regularize the dynamics by using resetting mechanisms that are modeled by well-posed hybrid dynamical systems. For these hybrid dynamics, we establish uniform asymptotic stability and robustness properties, as well as convergence rates that are similar to those of the non-hybrid dynamics. We finish by characterizing a family of discretization mechanisms that retain the main stability and robustness properties of the hybrid algorithms.

Fuzzy co-clustering schemes including Fuzzy Co-Clustering induced by Multinomial Mixture models (FCCMM) are promising approaches for analyzing object-item cooccur-renee information such as document-keyword frequencies and customer-product purchase history transactions. However, such cooccurrence datasets are generally maintained as very large matrices and cannot be dealt with conventional batch algorithms. In order to deal with such problems, online FCCMM (OFC-CMM) that sequentially loads a single object has been proposed. Conventional OFCCMM uses stochastic gradient descent (SGD) to update parameters. SGD generally has drawbacks that convergence is slow and it is susceptible to vibration state and a saddle point. Many improvements on SGD have been proposed such as Momentum SGD, Nesterov's accelerated gradient method, AdaGrad, and Adam. In this study, we introduce various gradient descent methods into OFCCMM and observe their characteristics and performance through numerical experiments.