Momentum Acceleration Research Articles

A Unified Analysis of AdaGrad With Weighted Aggregation and Momentum Acceleration.

Integrating adaptive learning rate and momentum techniques into stochastic gradient descent (SGD) leads to a large class of efficiently accelerated adaptive stochastic algorithms, such as AdaGrad, RMSProp, Adam, AccAdaGrad, and so on. In spite of their effectiveness in practice, there is still a large gap in their theories of convergences, especially in the difficult nonconvex stochastic setting. To fill this gap, we propose weighted AdaGrad with unified momentum and dubbed AdaUSM, which has the main characteristics that: 1) it incorporates a unified momentum scheme that covers both the heavy ball (HB) momentum and the Nesterov accelerated gradient (NAG) momentum and 2) it adopts a novel weighted adaptive learning rate that can unify the learning rates of AdaGrad, AccAdaGrad, Adam, and RMSProp. Moreover, when we take polynomially growing weights in AdaUSM, we obtain its O(log(T)/√T) convergence rate in the nonconvex stochastic setting. We also show that the adaptive learning rates of Adam and RMSProp correspond to taking exponentially growing weights in AdaUSM, thereby providing a new perspective for understanding Adam and RMSProp. Finally, comparative experiments of AdaUSM against SGD with momentum, AdaGrad, AdaEMA, Adam, and AMSGrad on various deep learning models and datasets are also carried out.

Dynamically accelerating the power iteration with momentum

AbstractIn this article, we propose, analyze and demonstrate a dynamic momentum method to accelerate power and inverse power iterations with minimal computational overhead. The method can be applied to real diagonalizable matrices, is provably convergent with acceleration in the symmetric case, and does not require a priori spectral knowledge. We review and extend background results on previously developed static momentum accelerations for the power iteration through the connection between the momentum accelerated iteration and the standard power iteration applied to an augmented matrix. We show that the augmented matrix is defective for the optimal parameter choice. We then present our dynamic method which updates the momentum parameter at each iteration based on the Rayleigh quotient and two previous residuals. We present convergence and stability theory for the method by considering a power‐like method consisting of multiplying an initial vector by a sequence of augmented matrices. We demonstrate the developed method on a number of benchmark problems, and see that it outperforms both the power iteration and often the static momentum acceleration with optimal parameter choice. Finally, we present and demonstrate an explicit extension of the algorithm to inverse power iterations.

An efficient ADAM-type algorithm with finite elements discretization technique for random elliptic optimal control problems

We consider an optimal control problem governed by an elliptic partial differential equation (PDE) with random coefficient, and introduce an efficient numerical method for the problem under concern. Based on a finite element discretization of the constraint PDE and objective functional, an ADAM-type iteration scheme is proposed to compute the approximated optimal control, which equips the stochastic gradient iteration with modified momentum acceleration and adaptive step size. We provide theoretical bounds on both the discretization and iteration error of the proposed numerical method. Our method is tested on two examples, and the numerical results show the efficiency and performance of the method.

Iterative algorithm for a generalized matrix equation with momentum acceleration approach and its convergence analysis

This article discusses a numerical gradient-based method for solving a generalized matrix equation. The iterative method mentioned includes two positive parameters, for which a range is determined to ensure the convergence of the introduced method. It has been shown that the optimal parameters of this method satisfy a constrained optimization problem. Then, specific solutions of this equation, such as the reflexive solution, are examined. Furthermore, to increase the rate of convergence of the proposed method, the idea of momentum is utilized, and a range for the momentum parameter is obtained to ensure the convergence. Finally, the efficiency of this method is investigated through numerical simulations. Additionally, in numerical results section, applications of mentioned matrix equation in image restoration and anti-linear systems are addressed.

Accelerated Stochastic Variance Reduction Gradient Algorithms for Robust Subspace Clustering.

Robust face clustering enjoys a wide range of applications for gate passes, surveillance systems and security analysis in embedded sensors. Nevertheless, existing algorithms have limitations in finding accurate clusters when data contain noise (e.g., occluded face clustering and recognition). It is known that in subspace clustering, the ℓ1- and ℓ2-norm regularizers can improve subspace preservation and connectivity, respectively, and the elastic net regularizer (i.e., the mixture of the ℓ1- and ℓ2-norms) provides a balance between the two properties. However, existing deterministic methods have high per iteration computational complexities, making them inapplicable to large-scale problems. To address this issue, this paper proposes the first accelerated stochastic variance reduction gradient (RASVRG) algorithm for robust subspace clustering. We also introduce a new momentum acceleration technique for the RASVRG algorithm. As a result of the involvement of this momentum, the RASVRG algorithm achieves both the best oracle complexity and the fastest convergence rate, and it reaches higher efficiency in practice for both strongly convex and not strongly convex models. Various experimental results show that the RASVRG algorithm outperformed existing state-of-the-art methods with elastic net and ℓ1-norm regularizers in terms of accuracy in most cases. As demonstrated on real-world face datasets with different manually added levels of pixel corruption and occlusion situations, the RASVRG algorithm achieved much better performance in terms of accuracy and robustness.

Stationary Landweber method with momentum acceleration for solving least squares problems

In this article, we proposed an enhancement to the convergence rate of Landweber’s method by incorporating the concept of momentum acceleration. Landweber’s method is commonly used to solve least squares problems of the form minx‖Ax−b‖. Our approach is based on Landweber’s method, which is acknowledged as a particular case of the methodologies outlined in Ding and Chen (2006). Through optimizing the momentum parameter, we were able to demonstrate the superior performance of the momentum-accelerated Landweber method. Specifically, we established that when A is a nonsquare m×n matrix with Rank(A)=n and σmin(A)≠σmax(A), the momentum-accelerated Landweber method with the optimal parameter consistently outperforms the standard Landweber method. Our numerical experiments have confirmed the theoretical findings, demonstrating a notable improvement in the convergence rate of the Landweber method.

Improved Analytic Learned Iterative Shrinkage Thresholding Algorithm and Its Application to Tomographic Synthetic Aperture Radar Building Object Height Inversion

Tomographic Synthetic Aperture Radar (TomoSAR) building object height inversion is a sparse reconstruction problem that utilizes the data obtained from several spacecraft passes to invert the scatterer position in the height direction. In practical applications, the number of passes is often small, and the observation data are also small due to the objective conditions, so this study focuses on the inversion under the restricted observation data conditions. The Analytic Learned Iterative Shrinkage Thresholding Algorithm (ALISTA) is a kind of deep unfolding network algorithm, which is a combination of the Iterative Shrinkage Thresholding Algorithm (ISTA) and deep learning technology, and it has the advantages of both. The ALISTA is one of the representative algorithms for TomoSAR building object height inversion. However, the structure of the ALISTA algorithm is simple, which has neither the excellent connection structure of a deep learning network nor the acceleration format combined with the ISTA algorithm. Therefore, this study proposes two directions of improvement for the ALISTA algorithm: firstly, an improvement in the inter-layer connection of the network by introducing a connection similar to residual networks obtains the Extragradient Analytic Learned Iterative Shrinkage Thresholding Algorithm (EALISTA) and further proves that the EALISTA achieves linear convergence; secondly, there is an improvement in the iterative format of the intra-layer iteration of the network by introducing the Nesterov momentum acceleration, which obtains the Fast Analytic Learned Iterative Shrinkage Thresholding Algorithm (FALISTA). We first performed inversion experiments on simulated data, which verified the effectiveness of the two proposed algorithms. Then, we conducted TomoSAR building object height inversion experiments on limited measured data and used the deviation metric P to measure the robustness of the algorithms to invert under restricted observation data. The results show that both proposed algorithms have better robustness, which verifies the superior performance of the two algorithms. In addition, we further analyze how to choose the most suitable algorithms for inversion in engineering practice applications based on the results of the experiments on measured data.

A Fast Total-Variation Driven Network for Sparse Aperture ISAR Imaging

Conventional Fourier transform-based Inverse Synthetic Aperture Radar (ISAR) imaging cannot deal with the cases of incomplete radar echoes, in which the data needs to be specially processed by Sparse Aperture (SA) imaging methods. Most current SA-ISAR imaging employs sparsity-driven optimization methods. However, the assumption that the target reflectivity is sparse in the spatial domain is insufficient to capture the surface and edge features of the target if no sparse representations are found for the target and well incorporated in the imaging process. In view of the edge-preserving capability of Total Variation (TV) and the strong learning ability of the Deep Neural Network (DNN), we propose a TV-driven network to improve the SA ISAR imaging performance. We first develop a fast TV regularization method to perform the imaging where the Gradient Descent (GD) along with the Momentum Acceleration (MA) are incorporated to increase the computational efficiency. Then, we unwrap the iterative fast TV regularization into a cascaded neural network to make the key imaging parameters learnable, leading to improved imaging performance, which we refer to as FGDTV-Net. Experiments under a variety of scenarios show that the proposed FGDTV-Net for SA-ISAR imaging is superior to existing SA imaging algorithms in preserving surface and edge features and is more robust in low signal-to-noise scenarios.

Deep unrolling networks with recurrent momentum acceleration for nonlinear inverse problems

Combining the strengths of model-based iterative algorithms and data-driven deep learning solutions, deep unrolling networks (DuNets) have become a popular tool to solve inverse imaging problems. Although DuNets have been successfully applied to many linear inverse problems, their performance tends to be impaired by nonlinear problems. Inspired by momentum acceleration techniques that are often used in optimization algorithms, we propose a recurrent momentum acceleration (RMA) framework that uses a long short-term memory recurrent neural network (LSTM-RNN) to simulate the momentum acceleration process. The RMA module leverages the ability of the LSTM-RNN to learn and retain knowledge from the previous gradients. We apply RMA to two popular DuNets—the learned proximal gradient descent (LPGD) and the learned primal-dual (LPD) methods, resulting in LPGD-RMA and LPD-RMA, respectively. We provide experimental results on two nonlinear inverse problems: a nonlinear deconvolution problem, and an electrical impedance tomography problem with limited boundary measurements. In the first experiment we have observed that the improvement due to RMA largely increases with respect to the nonlinearity of the problem. The results of the second example further demonstrate that the RMA schemes can significantly improve the performance of DuNets in strongly ill-posed problems.

On the Polyak momentum variants of the greedy deterministic single and multiple row‐action methods

AbstractFor solving a consistent system of linear equations, the classical row‐action method, such as Kaczmarz method, is a simple while really effective iteration solver. Based on the greedy index selection strategy and Polyak's heavy‐ball momentum acceleration technique, we propose two deterministic row‐action methods and establish the corresponding convergence theory. We show that our algorithm can linearly converge to a least‐squares solution with minimum Euclidean norm. Several numerical studies have been presented to corroborate our theoretical findings. Real‐world applications, such as data fitting in computer‐aided geometry design, are also presented for illustrative purposes.

Momentum accelerated unfolding network with spectral–spatial prior for computational spectral imaging

Edge computing is a key technology in computational imaging, where the algorithms determine reconstruction quality and reconstruction speed. Recovering spectral information from ill-conditioned data of coded aperture snapshots is a novel but challenging solution in spectral imaging for dynamic scenes. Further study is necessary to enhance the edge computing power of computational spectral programs. Deep unfolding networks have made considerable headway in this direction, but the existing methods still have two drawbacks: (1) The 3D spectral cube exhibits long-range dependency and non-local self-similarity in both spatial and spectral dimensions, yet learn the global features of spatial and spectral is difficult, which remains to be investigated. (2) The global features degenerate as the number of iterations increases. To solve the above problems, this paper proposes a Momentum Accelerated Unfolding Network with Spectral–Spatial prior (MAUNSS). Specifically, a denoising module based on a spectral–spatial transformer is designed, which uses both sparse spectral features to recover global information and dense spatial features to enhance detail texture. Within such a framework, cross-stage feature transmission channels transmitting features across spectral and spatial transformers at different stages are built to avoid feature degeneration. In order to increase the training speed and save the time and cost associated with the iterative approach, we improve it by means of the momentum acceleration module. Supplemented by a re-projection loss, a technique introduced from 3D measurement, the accuracy is further improved. To the best of our knowledge, the proposed method is an attempt to use momentum to expedite the convergence of unfolding networks. Extensive experiments demonstrate that our proposed method outperforms previous state-of-the-art methods by more than 1.94 dB.

AdaSAM: Boosting sharpness-aware minimization with adaptive learning rate and momentum for training deep neural networks

Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of AdaSAM in the stochastic non-convex setting. We theoretically show that AdaSAM admits a O(1/bT) convergence rate, which achieves linear speedup property with respect to mini-batch size b. Specifically, to decouple the stochastic gradient steps with the adaptive learning rate and perturbed gradient, we introduce the delayed second-order momentum term to decompose them to make them independent while taking an expectation during the analysis. Then we bound them by showing the adaptive learning rate has a limited range, which makes our analysis feasible. To the best of our knowledge, we are the first to provide the non-trivial convergence rate of SAM with an adaptive learning rate and momentum acceleration. At last, we conduct several experiments on several NLP tasks and the synthetic task, which show that AdaSAM could achieve superior performance compared with SGD, AMSGrad, and SAM optimizers.

Multi-Innovation Nesterov accelerated gradient parameter identification method for autoregressive exogenous models

This paper proposed a multi-innovation Nesterov accelerated gradient (MNAG) parameter identification method for the autoregressive exogenous (ARX) model. First, a momentum acceleration term is added stochastic gradient descent (SGD) algorithm to increase the convergence rate of the SGD. Second, the parameter updating process is expanded from a single batch of current information iteration to the multiple batches of both previous and current information iteration, which extended the algorithm from single-innovation Nesterov accelerated gradient (NAG) to multi-innovation NAG parameter identification method. That enhances the algorithm’s anti-noise and anti-abnormal data abilities, and its data utilization rate. Then, the convergence of the MNAG parameter identification method is proven. The effectiveness of the MNAG parameter identification method is verified by numerical simulation and the rotational speed system of a ring-pendulum double-sided polisher.

Hierarchical Federated Learning With Momentum Acceleration in Multi-Tier Networks

Hierarchical Federated Learning With Momentum Acceleration in Multi-Tier Networks

Low-Mach number asymptotic analysis of fluid–structure-interaction (FSI) pressure waves inside an elastic tube

We consider the pressure wave having velocity cp inside an elastic tube of internal radius R0, thickness e, length L, shear modulus G and density ρs0, filled with a fluid with kinematic viscosity νf. We theoretically analyze the fluid–structure coupling between: (i) the elastic sheath, (ii) the fluid boundary layer, and (iii) the core acoustic pressure and velocity fields. Our analysis provides an asymptotic derivation of the fluid–structure-interactions (FSI) model that recovers known pulse-wave velocities and provides a new theoretical prediction for the exponential time decay of the wave longitudinal attenuation envelope. Taking advantage of highly distinct time-scales between the viscous radial diffusion τd=R02/νf compared with wave-convective time τc=L/cp as well as the elastic relaxation time τe=eρs/G, such that τe∼τc≪τd we perform a two time-scale asymptotic analysis based on a small parameter δ=τc/τd. Said parameter is obtained by balancing the momentum acceleration and the viscous damping rate in the inner unsteady boundary layer, the thickness of which being δR0. The resulting asymptotic sequence provides a unique consistent scaling for solid deformation and velocity fields, with the secularity condition associated with the leading-order slow-time scale envelope attenuation obtained by extending the analysis to investigate the first-order corrections.On the one hand our approach reconciles both predictions for the precursive elastic wave and the pulse velocities obtained when considering solid deformation only, and, on the other hand, predictions for the longitudinal attenuation resulting from the effect of boundary layers only. Our analysis also permits the derivation of a new convoluted model for the wall shear stress, which is (FSI)–consistent. The theoretical results are successfully compared with experimental measurements.

The relationship between physical characteristics and match collision performance among elite international female rugby union players

ABSTRACT This study investigated whether anthropometric and physical abilities explained variance in match collision performance among international female rugby union players. Physical performance and anthropometric data for fifty-one international female rugby union players, and collision actions categorised as “effort” or “performance” variables, from 20 international matches, were analysed using partial least squares regression. Among forwards, variance in carries/min was explained (R2 = .22) by a combination of; body mass, skinfolds, acceleration momentum and negative associations with mean aerobic speed and single-leg isometric squat relative force (SLISO/kgBM). Variance in collision dominance among forwards was explained (R2 = .21) by lower skinfolds and higher acceleration momentum, while tackles/min (R2 = .19) were explained by greater jumping power and single-leg isometric squat (SLISO). Among backs, variance in tackles/min (R2 = .54) was explained by greater bench press, SLISO and SLISO/kgBM. Variance in collision dominance among backs was explained (R2 = .23) by negative and positive associations with body mass and SLISO/kgBM, respectively. These findings suggest the development of physical characteristics, such as body mass and composition, strength and power contribute towards successful collision actions among international female rugby union players. The contribution of different physical characteristics towards collision events is dependent on position, and whether the collision event is categorised by “performance” or “effort”. It is suggested that physical training programmes should reflect this level of specificity. Highlights Among elite female rugby union forwards, acceleration momentum, body mass and skinfolds are positively associated with winning collisions and carrying the ball into contact more frequently, whilst tackle frequency is positively associated with relative leg strength and power output. Among elite female backs, the ability to win collisions is positively associated with relative leg power output, and negatively associated with body mass. Tackle frequency is associated with maximum upper- and lower-body strength in this group. Physical characteristics account for some of the variability in collision performance, but interpretation of these findings should consider that factors such as technique during collision events may account for a larger proportion of total variance. Sports science practitioners can improve collision performance, to varying degrees, by enhancing specific gross physical characteristics, according to a player’s position and the tactical role they are expected to fulfil.

Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization

Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization

On the Origin of the Solar Cycle Modulation of the Winter North Atlantic Oscillation

AbstractUsing data from recent 64 years, we found that the zonal wind signal associated with the winter‐mean North Atlantic Oscillation (NAO) tends to extend into the upper stratosphere when solar activity is high, but the vertical extension is limited when solar activity is low. To examine the physical source of this phenomenon, the momentum and wave‐energy budget are analyzed using the 63‐winter record of the Japanese Reanalysis data set from 1958 to 2021. These analyses show that (a) the zonal‐mean momentum acceleration is driven by eddies throughout the entire depth of the atmosphere around 60°N, (b) the barotropic energy conversion from eddies to zonal‐mean flow is centered in the middle stratosphere around 55°N, and (c) the baroclinic energy conversion from zonal‐mean flow to eddies from the surface to the middle stratosphere around 60°N tends to be enhanced from December to February associated with the positive winter‐mean NAO index in High Solar (HS) activity years. These energy transfers are closely related to enhanced zonal wind. These analyses suggest that the structural modulation of the NAO due to the solar cycle comes from the enhanced sensitivity of the wave‐mean flow interaction in the stratosphere in HS years, which is broadly shared with the solar cycle modulation of the late‐winter Southern Annular Mode (SAM).

Triangularized Orthogonalization-Free Method for Solving Extreme Eigenvalue Problems

A novel orthogonalization-free method together with two specific algorithms is proposed to address extreme eigenvalue problems. On top of gradient-based algorithms, the proposed algorithms modify the multicolumn gradient such that earlier columns are decoupled from later ones. Locally, both algorithms converge linearly with convergence rates depending on eigengaps. Momentum acceleration, exact linesearch, and column locking are incorporated to accelerate algorithms and reduce their computational costs. We demonstrate the efficiency of both algorithms on random matrices with different spectrum distributions and matrices from computational chemistry.

Radial basis network estimator of oxygen content in the flue gas of debutanizer reboiler

<span>The energy efficiency in the debutanizer reboiler combustion can be monitored from the oxygen content of the flue gas of the reboiler. The measurement of the oxygen content can be conducted in situ using an oxygen sensor. However, soot that may appear around the sensor due to the combustion process in the debutanizer reboiler can obstruct the sensor’s function. In-situ redundancy sensors’ unavailability is a significant problem when the sensor is damaged, so measures must be made directly by workers using portable devices. On the other hand, worker safety is a primary concern when working in high-risk work areas. In this paper, we propose a software-based measurement or soft sensor to overcome the problems. The radial basis function network model makes soft sensors adapt to data updates because of their advantage as a universal approximator. The estimation of oxygen content with a soft sensor has been successfully carried out. The soft sensor generates an estimated mean square error of 0.216% with a standard deviation of 0.0242%. Stochastics gradient descent algorithm with momentum acceleration and dimension reduction using principal component analysis successfully improves the soft sensors’ performance.</span>