Accelerated Gradient Algorithm Research Articles

Comparative analysis of accelerated gradient algorithms for convex optimization: high and super resolution ODE approach

We investigate convex differentiable optimization and explore the temporal discretization of damped inertial dynamics driven by the gradient of the objective function. This leads to three accelerated gradient algorithms: Nesterov Accelerated Gradient (NAG), Ravine Accelerated Gradient (RAG), and (IGAHD). The latter was introduced by discretizing inertial dynamics with Hessian-driven damping to attenuate inherent oscillations in inertial methods. By analysing the high-resolution ODEs of order p = 0,1,2 for these algorithms, we gain insights into their similarities and differences. All three algorithms share the same low-resolution ODE of order 0, which is the dynamic proposed as a continuous surrogate for (NAG). To differentiate Nesterov from Ravine, we refine the comparison and demonstrate distinct high-resolution ODEs of order 2 in h (termed super-resolution). The corresponding Taylor expansions in h reveal matching terms of order 1 but differing terms of order 2. To the best of our knowledge, this result is completely new and emphasizes the need to avoid confusion between the Ravine and Nesterov methods in the literature. We present numerical experiments to illustrate our theoretical results. Performance profiles, measuring the number of iterations, indicate that (IGAHD) outperforms both (NAG) and (RAG) methods. (RAG) exhibits a slight advantage over (NAG) in terms of the average number of iterations. When considering CPU-time, both (RAG) and (NAG) outperform (IGAHD). All three algorithms exhibit similar behaviour when evaluating based on gradient norms.

The Nesterov accelerated gradient algorithm for Auto-Regressive Exogenous models with random lost measurements: Interpolation method and auxiliary model method

For ARX models with random lost measurements, this article proposes an interpolation and auxiliary model based Nesterov accelerated gradient algorithm. This algorithm utilizes the interpolation and auxiliary model methods to fill in lost measurements. The parameters are estimated with better accuracy based on the interpolated input and predicted output data. Two simulation examples are presented to verify the effective of the proposed algorithm.

Accelerated Learning Control for Point-to-Point Tracking Systems.

In this study, we investigate the accelerated learning control schemes for point-to-point tracking systems (PTSs) with measurement noise. The asymptotic convergence of the generated input sequence has been a long-standing open issue for point-to-point tracking problems because there are infinite possible input candidates that can drive the system dynamics to track the desired reference at specified time instants. An accelerated gradient algorithm and its generalized version with a novel direction regulation matrix are proposed, with the learning gain is adaptively triggered by the practical tracking errors. The learning gain remains constant at the early stage and begins to decrease after a certain number of iterations. The input sequence generated by the proposed scheme converges to a specified limit for any fixed initial input, with the limit being closest to the initial input, in a certain sense. Numerical simulations are provided to verify the theoretical results.

Research on Accelerated Gradient Algorithm Based on Ordinary Differential Equations with Uncertain Coefficients

Research on Accelerated Gradient Algorithm Based on Ordinary Differential Equations with Uncertain Coefficients

Improved Accelerated Gradient Algorithms with Line Search for Smooth Convex Optimization Problems

For smooth convex optimization problems, the optimal convergence rate of first-order algorithm is [Formula: see text] in theory. This paper proposes three improved accelerated gradient algorithms with the gradient information at the latest point. For the step size, to avoid using the global Lipschitz constant and make the algorithm converge faster, new adaptive line search strategies are adopted. By constructing a descent Lyapunov function, we prove that the proposed algorithms can preserve the convergence rate of [Formula: see text]. Numerical experiments demonstrate that our algorithms perform better than some existing algorithms which have optimal convergence rate.

Generating Nesterov’s accelerated gradient algorithm by using optimal control theory for optimization

Nesterov’s accelerated gradient algorithm is derived from first principles. The first principles are founded on the recently-developed optimal control theory for optimization. This theory frames a generic optimization problem as an optimal control problem. The necessary conditions for optimal control produce a controllable dynamical system. Stabilizing this system by dissipating a simple, quadratic “control” Lyapunov function generates an equation that produces Nesterov’s algorithm. In this context, the proposed theory solves the long-standing mystery surrounding the algorithm.

Synthesis of accelerated gradient algorithms for optimization and saddle point problems using Lyapunov functions and LMIs

This paper considers the problem of designing accelerated gradient-based algorithms for optimization and saddle-point problems. The class of objective functions is defined by a generalized sector condition. This class of functions contains strongly convex functions with Lipschitz gradients but also non-convex functions, which allows not only to address optimization problems but also saddle-point problems. The proposed design procedure relies on a suitable class of Lyapunov functions and, for a fixed convergence rate, is a convex semi-definite program in all but one scalar parameter. The proposed synthesis allows the design of algorithms that reach the performance of state-of-the-art accelerated gradient methods and beyond.

Multiply Accelerated Value Iteration for NonSymmetric Affine Fixed Point Problems and Application to Markov Decision Processes

We analyze a modified version of the Nesterov accelerated gradient algorithm, which applies to affine fixed point problems with non-self-adjoint matrices, such as the ones appearing in the theory of Markov decision processes with discounted or mean payoff criteria. We characterize the spectra of matrices for which this algorithm does converge with an accelerated asymptotic rate. We also introduce a $d$th-order algorithm and show that it yields a multiply accelerated rate under more demanding conditions on the spectrum. We subsequently apply these methods to develop accelerated schemes for nonlinear fixed point problems arising from Markov decision processes. This is illustrated by numerical experiments.

Relationship between employees' career maturity and career planning of edge computing and cloud collaboration from the perspective of organizational behavior.

A new IoT (Internet of Things) analysis platform is designed based on edge computing and cloud collaboration from the perspective of organizational behavior, to fundamentally understand the relationship between enterprise career maturity and career planning, and meet the actual needs of enterprises. The performance of the proposed model is further determined according to the characteristic of the edge near data sources, with the help of factor analysis, and through the study and analysis of relevant enterprise data. The model is finally used to analyze the relationship between enterprise career maturity and career planning through simulation experiments. The research results prove that career maturity positively affects career planning, and vocational delay of gratification plays a mediating role in career maturity and career planning. Besides, the content of career choice in career maturity is influenced by mental acuity, result acuity and loyalty. The experimental results indicate that when the load at both ends of the edge and cloud exceeds 80%, the edge delay of the IoT analysis platform based on edge computing and cloud collaboration is 10s faster than that of other models. Meanwhile, the system slowdown is reduced by 36% while the stability is increased when the IoT analysis platform analyzes data. The results of the edge-cloud collaboration scheduling scheme are similar to all scheduling to the edge end, which saves 19% of the time compared with cloud computing to the cloud end. In Optical Character Recognition and Aeneas, compared with the single edge-cloud coordination mode, the model with the Nesterov Accelerated Gradient algorithm achieves the optimal performance. Specifically, the communication delay is reduced by about 25% on average, and the communication time decreased by 61% compared with cloud computing to the edge end. This work has significant reference value for analyzing the relationship between enterprise psychology, behavior, and career planning.

Robust cost-sensitive kernel method with Blinex loss and its applications in credit risk evaluation

Credit risk evaluation is a crucial yet challenging problem in financial analysis. It can not only help institutions reduce risk and ensure profitability, but also improve consumers’ fair practices. The data-driven algorithms such as artificial intelligence techniques regard the evaluation as a classification problem and aim to classify transactions as default or non-default. Since non-default samples greatly outnumber default samples, it is a typical imbalanced learning problem and each class or each sample needs special treatment. Numerous data-level, algorithm-level and hybrid methods are presented, and cost-sensitive support vector machines (CSSVMs) are representative algorithm-level methods. Based on the minimization of symmetric and unbounded loss functions, CSSVMs impose higher penalties on the misclassification costs of minority instances using domain specific parameters. However, such loss functions as error measurement cannot have an obvious cost-sensitive generalization. In this paper, we propose a robust cost-sensitive kernel method with Blinex loss (CSKB), which can be applied in credit risk evaluation. By inheriting the elegant merits of Blinex loss function, i.e., asymmetry and boundedness, CSKB not only flexibly controls distinct costs for both classes, but also enjoys noise robustness. As a data-driven decision-making paradigm of credit risk evaluation, CSKB can achieve the “win-win” situation for both the financial institutions and consumers. We solve linear and nonlinear CSKB by Nesterov accelerated gradient algorithm and Pegasos algorithm respectively. Moreover, the generalization capability of CSKB is theoretically analyzed. Comprehensive experiments on synthetic, UCI and credit risk evaluation datasets demonstrate that CSKB compares more favorably than other benchmark methods in terms of various measures.

A FISTA-type accelerated gradient algorithm for solving smooth nonconvex composite optimization problems

In this paper, we describe and establish iteration-complexity of two accelerated composite gradient (ACG) variants to solve a smooth nonconvex composite optimization problem whose objective function is the sum of a nonconvex differentiable function f with a Lipschitz continuous gradient and a simple nonsmooth closed convex function h. When f is convex, the first ACG variant reduces to the well-known FISTA for a specific choice of the input, and hence the first one can be viewed as a natural extension of the latter one to the nonconvex setting. The first variant requires an input pair (M, m) such that f is m-weakly convex, \(\nabla f\) is M-Lipschitz continuous, and \(m \le M\) (possibly \(m<M\)), which is usually hard to obtain or poorly estimated. The second variant on the other hand can start from an arbitrary input pair (M, m) of positive scalars and its complexity is shown to be not worse, and better in some cases, than that of the first variant for a large range of the input pairs. Finally, numerical results are provided to illustrate the efficiency of the two ACG variants.

Accelerated gradient algorithm for RBF neural network

Gradient-based algorithms are commonly used for training radial basis function neural network (RBFNN). However, one of the challenges in the training process is determining how to avoid vanishing gradient. To solve this problem, an accelerated gradient algorithm (AGA) is designed to improve the learning performance of RBFNN in this paper. First, an indirect detection mechanism, based on the instantaneous gradient decay rate (IGDR) and instantaneous convergence rate (ICR), is developed to identify the vanishing gradient in learning process. Second, an amplification gradient strategy (AGS), which can increase the gradient value of learning parameters, is designed to accelerate the learning speed of RBFNN. Third, the analysis of AGA-based RBFNN (AGA-RBFNN) is given to guarantee the successful application. Finally, some benchmark and real problems are used to illustrate the effectiveness of AGA-RBFNN. The results demonstrate the effectiveness of AGA-RBFNN.

On stochastic accelerated gradient with non-strongly convexity

&lt;abstract&gt;&lt;p&gt;In this paper, we consider stochastic approximation algorithms for least-square and logistic regression with no strong-convexity assumption on the convex loss functions. We develop two algorithms with varied step-size motivated by the accelerated gradient algorithm which is initiated for convex stochastic programming. We analyse the developed algorithms that achieve a rate of $ O(1/n^{2}) $ where $ n $ is the number of samples, which is tighter than the best convergence rate $ O(1/n) $ achieved so far on non-strongly-convex stochastic approximation with constant-step-size, for classic supervised learning problems. Our analysis is based on a non-asymptotic analysis of the empirical risk (in expectation) with less assumptions that existing analysis results. It does not require the finite-dimensionality assumption and the Lipschitz condition. We carry out controlled experiments on synthetic and some standard machine learning data sets. Empirical results justify our theoretical analysis and show a faster convergence rate than existing other methods.&lt;/p&gt;&lt;/abstract&gt;

Convex Synthesis of Accelerated Gradient Algorithms

Related DatabasesWeb of Science You must be logged in with an active subscription to view this.Article DataHistorySubmitted: 12 February 2021Accepted: 31 May 2021Published online: 14 December 2021Keywordsoptimization algorithms, robust control, algorithm synthesisAMS Subject Headings93D09, 93D15, 93D25, 90C22, 90C25Publication DataISSN (print): 0363-0129ISSN (online): 1095-7138Publisher: Society for Industrial and Applied MathematicsCODEN: sjcodc

Stochastic optimization with momentum: Convergence, fluctuations, and traps avoidance

In this paper, a general stochastic optimization procedure is studied, unifying several variants of the stochastic gradient descent such as, among others, the stochastic heavy ball method, the Stochastic Nesterov Accelerated Gradient algorithm (S-NAG), and the widely used Adam algorithm. The algorithm is seen as a noisy Euler discretization of a non-autonomous ordinary differential equation, recently introduced by Belotto da Silva and Gazeau, which is analyzed in depth. Assuming that the objective function is non-convex and differentiable, the stability and the almost sure convergence of the iterates to the set of critical points are established. A noteworthy special case is the convergence proof of S-NAG in a non-convex setting. Under some assumptions, the convergence rate is provided under the form of a Central Limit Theorem. Finally, the non-convergence of the algorithm to undesired critical points, such as local maxima or saddle points, is established. Here, the main ingredient is a new avoidance of traps result for non-autonomous settings, which is of independent interest.

Research on three-step accelerated gradient algorithm in deep learning

Gradient descent (GD) algorithm is the widely used optimisation method in training machine learning and deep learning models. In this paper, based on GD, Polyak's momentum (PM), and Nesterov accelerated gradient (NAG), we give the convergence of the algorithms from an initial value to the optimal value of an objective function in simple quadratic form. Based on the convergence property of the quadratic function, two sister sequences of NAG's iteration and parallel tangent methods in neural networks, the three-step accelerated gradient (TAG) algorithm is proposed, which has three sequences other than two sister sequences. To illustrate the performance of this algorithm, we compare the proposed algorithm with the three other algorithms in quadratic function, high-dimensional quadratic functions, and nonquadratic function. Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning. For conveniently facilitate the proposed algorithms, we rewite the R package ‘neuralnet’ and extend it to ‘supneuralnet’. All kinds of deep learning algorithms in this paper are included in ‘supneuralnet’ package. Finally, we show our algorithms are superior to other algorithms in four case studies.

Applications of the accelerated gradient algorithm for pooling problem

Applications of the accelerated gradient algorithm for pooling problem

Sparse deep nonnegative matrix factorization

Nonnegative matrix factorization is a powerful technique to realize dimension reduction and pattern recognition through single-layer data representation learning. Deep learning, however, with its carefully designed hierarchical structure, is able to combine hidden features to form more representative features for pattern recognition. In this paper, we proposed sparse deep nonnegative matrix factorization models to analyze complex data for more accurate classification and better feature interpretation. Such models are designed to learn localized features or generate more discriminative representations for samples in distinct classes by imposing $L_1$-norm penalty on the columns of certain factors. By extending one-layer model into multi-layer one with sparsity, we provided a hierarchical way to analyze big data and extract hidden features intuitively due to nonnegativity. We adopted the Nesterov's accelerated gradient algorithm to accelerate the computing process with the convergence rate of $O(1/k^2)$ after $k$ steps iteration. We also analyzed the computing complexity of our framework to demonstrate their efficiency. To improve the performance of dealing with linearly inseparable data, we also considered to incorporate popular nonlinear functions into this framework and explored their performance. We applied our models onto two benchmarking image datasets, demonstrating our models can achieve competitive or better classification performance and produce intuitive interpretations compared with the typical NMF and competing multi-layer models.

An accelerated gradient algorithm for well control optimization

The simulation-assisted optimization of injection and production control settings can be considered as a way for improved management of hydrocarbon reservoirs. Although well control optimization problems are expected to have a relatively smooth landscape, they suffer from having a high number of dimensions. As a result, gradient-based algorithms are typically the preferred choice over derivative-free algorithms. Since the calculation of the exact gradient is not always feasible, recent studies into gradient-based algorithms have focused on ameliorating the accuracy of gradient approximation methods. Less attention has been paid to the framework in which these gradient approximations are utilized. The most common framework is the steepest descent with a backtracking line-search. This study aims to provide an improved framework using an adaptive moment estimation technique. The proposed method is compared to the steepest descent framework in two case studies of increasing complexity. In both frameworks, the gradient is approximated by the simultaneous perturbation stochastic approximation. The first case study is a two-dimensional heterogeneous reservoir model produced under water-flooding. The second case study investigates the injection and production settings of the three-dimensional benchmark case, the Brugge model, whilst incorporating geological uncertainty. The proposed framework showed improvements of up to 91% in convergence speeds and up to 5% in optimal value for the first case study. In addition, the proposed framework showed an improvement of up to 81% in convergence speeds and up to 2% in optimal value for the second case study over the steepest descent framework. For both case studies, the effect of the gradient approximation accuracy (perturbation number) was also investigated. The results indicated that the proposed algorithm is less sensitive to the gradient approximation accuracy than the steepest descent framework. In addition, this study investigated the effect of two popular bound constraint handling techniques in both frameworks. The results indicated that the projection method outperforms the logarithmic transformation method when applied to production optimization problem.

Energy-Optimal Dynamic Computation Offloading for Industrial IoT in Fog Computing

Fog computing is emerging as a promising mode to meet the stringent requirement of low latency in industrial Internet of Things (IIoT). By dynamically offloading part of the computation-intensive tasks from a fog node to a cloud server, the computation experience of users can be further improved in fog computing systems. In this paper, we develop an energy-optimal dynamic computation offloading scheme (EDCO) for IIoT in a fog computing scenario. The purpose is to minimize energy consumption when computation tasks are accomplished within a desired energy overhead and delay. Specifically, we first formulate an energy minimization computation offloading problem with delay, energy and other network resource constraints. To address this optimization problem, an accelerated gradient algorithm with joint optimization of the offloading ratio and transmission time is proposed; it can find the optimal value with a fast speed that improves the convergence speed of traditional methods. Subsequently, to better meet the stringent energy and latency requirements of IIoT applications, the dynamic voltage scaling (DVS) technique is integrated into the above solution, and we develop an alternating minimization algorithm to achieve energy-optimal fog computation offloading by jointly optimizing the offloading ratio, transmission power, local CPU computation speed and transmission time. Finally, the numerical results reveal that the proposed offloading scheme is superior to the local computing, full offloading and partial offloading with fixed computation speed schemes in terms of energy consumption and completion time. We also confirm the convergence rate advantage of the accelerated algorithm.