Step Size Selection Research Articles

Reachability of optimal convergence rate estimates for high-order numerical convex optimization methods

The Monteiro-Svaiter accelerated hybrid proximal extragradient method (2013) with one step of Newton’s method used at every iteration for the approximate solution of an auxiliary problem is considered. The Monteiro-Svaiter method is optimal (with respect to the number of gradient and Hessian evaluations for the optimized function) for sufficiently smooth convex optimization problems in the class of methods using only the gradient and Hessian of the optimized function. An optimal tensor method involving higher derivatives is proposed by replacing Newton’s step with a step of Yu.E. Nesterov’s recently proposed tensor method (2018) and by using a special generalization of the step size selection condition in the outer accelerated proximal extragradient method. This tensor method with derivatives up to the third order inclusive is found fairly practical, since the complexity of its iteration is comparable with that of Newton’s one. Thus, a constructive solution is obtained for Nesterov’s problem (2018) of closing the gap between tight lower and overstated upper bounds for the convergence rate of existing tensor methods of order p ≥ 3.

Conjugate Gradient Hard Thresholding Pursuit Algorithm for Sparse Signal Recovery

We propose a new iterative greedy algorithm to reconstruct sparse signals in Compressed Sensing. The algorithm, called Conjugate Gradient Hard Thresholding Pursuit (CGHTP), is a simple combination of Hard Thresholding Pursuit (HTP) and Conjugate Gradient Iterative Hard Thresholding (CGIHT). The conjugate gradient method with a fast asymptotic convergence rate is integrated into the HTP scheme that only uses simple line search, which accelerates the convergence of the iterative process. Moreover, an adaptive step size selection strategy, which constantly shrinks the step size until a convergence criterion is met, ensures that the algorithm has a stable and fast convergence rate without choosing step size. Finally, experiments on both Gaussian-signal and real-world images demonstrate the advantages of the proposed algorithm in convergence rate and reconstruction performance.

Accelerated stochastic gradient descent with step size selection rules

Abstract Accelerated stochastic gradient descent (ASGD) methods, which incorporate accelerated proximal gradient (APG) and stochastic gradient (SG), have received considerable attention recently for solving regularized risk minimization problems in signal/image processing, statistics and machine learning. However, there has been a paucity of practical guidance proposed for resolving one of the major issues in ASGD: how to choose an appropriate step size. To solve this problem, we propose to use the Barzilai-Borwein (BB) method to automatically compute step size for the accelerated mini-batch Prox-SVRG (Acc-Prox-SVRG) method (the state of the art ASGD method), thereby obtaining a new accelerated method: Acc-Prox-SVRG-BB. We prove the convergence of Acc-Prox-SVRG-BB and show that its complexity is comparable with the best known stochastic gradient methods. In addition, we incorporate Beck and Teboulle’s APG (FISTA) and Prox-SVRG in a mini-batch setting and obtain another new accelerated gradient descent method, FISTA-Prox-SVRG, which requires the selection of fewer unknown parameters than those required in Acc-Prox-SVRG. Finally, we introduce the BB method into FISTA-Prox-SVRG to further show the efficacy of the BB method. Numerical results demonstrate the advantage of our algorithms.

On the Barzilai‐Borwein basic scheme in FFT‐based computational homogenization

SummaryBuilding upon the equivalence of the basic scheme in the work of Moulinec and Suquet with gradient descent methods, we investigate the effect of using the celebrated Barzilai‐Borwein step size selection technique in this context. We provide an overview of recent convergence theory and present efficient implementations in the context of computational micromechanics, with and without globalization. In contrast to polarization schemes and fast gradient methods, no lower bound on the eigenvalues of the material tangent is necessary for the Barzilai‐Borwein scheme. We demonstrate the power of the proposed method for linear elastic and inelastic large scale problems with finite and infinite material contrast.

Distributed Energy Resource Coordination Over Time-Varying Directed Communication Networks

In this paper, we consider the optimal coordination problem for distributed energy resources (DERs), including distributed generators and energy storages. We first propose an algorithm based on the push-sum and gradient method to solve the optimal DER coordination problem in a distributed manner. In the proposed algorithm, each DER only maintains a set of variables and updates them through information exchange with a few neighboring DERs over a time-varying directed communication network. We show that the proposed distributed algorithm with appropriately chosen diminishing step sizes solves the optimal DER coordination problem if the time-varying directed communication network is uniformly jointly strongly connected. Moreover, in order to improve the convergence speed and to reduce the communication burden, we propose an accelerated distributed algorithm with a fixed step size. We show that the new proposed algorithm exponentially solves the optimal DER coordination problem if the cost functions satisfy an additional assumption and the selected step size is less than a certain critical value. Both proposed distributed algorithms are validated and evaluated using the IEEE 39-bus system.

Design of High-Order Decoupled Multirate GARK Schemes

Multirate time integration methods apply different step sizes to resolve different components of a system based on local activity levels. This local selection of step sizes allows increased computational efficiency while achieving the desired solution accuracy. While the multirate idea is elegant and has been around for decades, multirate methods are not yet widely used in applications. This is due, in part, to the difficulties created by the construction of high-order multirate schemes. Seeking to overcome these challenges, this work focuses on the design of practical high-order multirate methods using the theoretical framework of multirate general-structure additive Runge--Kutta (MrGARK) methods [M. Günther and A. Sandu, Numer. Math., 133 (2016), pp. 497--524], which provides generic order conditions and linear and nonlinear stability analyses. A set of design criteria for practical multirate methods is defined herein: method coefficients should be generic in the step size ratio but should not depend strongly on this ratio, unnecessary coupling between the fast and the slow components should be avoided, and the step size controllers should adjust both the micro- and the macrosteps. Using these criteria, we develop MrGARK schemes of up to order four that are explicit-explicit (both the fast and slow component are treated explicitly), implicit-explicit (implicit in the fast component and explicit in the slow one), and explicit-implicit (explicit in the fast component and implicit in the slow one). Numerical experiments verify the accuracy of these new schemes.

Nonlocal Means Filtering Based Speckle Removal Utilizing the Maximum a Posteriori Estimation and the Total Variation Image Prior

In this paper, the problem of image speckle removal is addressed. To alleviate the pepper–salt remainder in the speckled image, we propose to utilize the nonlocal means filtering, where the weighting coefficients are derived based on the maximum a posteriori estimation with the total variation image prior. As a result, the objective function of the pixel fitting term plus the total variation regularizer is formulated, and it is solved with the majorization–minimization approach. To avoid the computationally intractable step size selection in the huge-scale gradient-based optimization, we split and solve the variables in the pixel fitting term and regularizer by means of the alternating direction method of multipliers. Performance analysis is performed for the Rayleigh and Gamma distributed signal models. The simulation and experimental results show the superior performance compared with other image despeckling methods in terms of various metrics and visual perception.

A posteriori error estimation for Magnus-type integrators

We study high-order Magnus-type exponential integrators for large systems of ordinary differential equations defined by a time-dependent skew-Hermitian matrix. We construct and analyze defect-based local error estimators as the basis for adaptive stepsize selection. The resulting procedures provide a posteriori information on the local error and hence enable the accurate, efficient, and reliable time integration of the model equations. The theoretical results are illustrated on two numerical examples .

Research on Multivariable Adaptive Control of HVAC System

HVAC system need be quickly diagnosed the location and location of the fault and find the cause of the failure, and reduce the failure to improve the comfort of users, improve the energy efficiency of buildings. Firstly, features of HVAC system failures were introduced in this paper. Failure correlation and transitivity and hierarchy of faults and parameter alignment were analyzed. On this basis, step size selection of depth neural network model was constructed by statistical-based machine learning. Adaptive depth neural network algorithm was built by weight updating formula. Finally, by an application case, multivariable adaptive control of HVAC System can strengthen the fault prediction and monitoring, can reduce the occurrence of fault, and prolong the life of equipment.

Prediction-correction method with BB step sizes

In the prediction-correction method for variational inequality (VI) problems, the step size selection plays an important role for its performance. In this paper, we employ the Barzilai-Borwein (BB) strategy in the prediction step, which is efficient for many optimization problems from a computational point of view. To guarantee the convergence, we adopt the line search technique, and relax the conditions to accept the BB step sizes as large as possible. In the correction step, we utilize a longer step length to calculate the next iteration point. Finally, we present some preliminary numerical results to show the efficiency of the algorithms.

Advanced multi-sector P&O maximum power point tracking technique for wind energy conversion system

This article suggests a new solution for the limitations of conventional fixed step-size perturb and observe (PO) maximum power point tracking (MPPT) of the wind energy conversion systems (WECSs). The proposed algorithm offers a new fast MPPT technique and it keeps the controller searching in the correct region on the power-speed curve. For this issue, a variable-step perturb and observe (VS-PO) MPPT technique based on multisector operation is proposed. The proposed technique divides the operating region into four sectors by comparing the power-speed curve and a special synthesized curve. Hence, the selection of the appropriate step-size is related to the operating sector. For the two sectors close to the maximum power point (MPP), a small step-size is applied. Otherwise, the controller utilizes a large step-size. The large-scale grid-tied WECS is composed of the back-to-back (BTB) converter. The BTB converter is divided into dual sides; the machine side converter and the grid side converter. The proposed technique has been validated over a large scale of 1.5 MW grid-tied PMSG-WECS using MATLAB/SIMULINK. The results prove the efficient operation of the proposed technique based on the improved efficiency and the fast system response compared to conventional P&O techniques.

Adaptive Step-Size Control for Dynamic Relaxation Using Continuous Kinetic Damping

Dynamic relaxation (DR) is a widely used numerical method to determine the static equilibrium of a dynamic system. However, it is difficult to apply conventional DR methods to nonlinear models because they require estimation of a stiffness matrix of the model. To resolve the forementioned problem, a new dynamic relaxation method using continuous kinetic damping (CKDR) was proposed in previous research. The CKDR method does not require any model parameters including the stiffness matrix, and it possesses absolute stability and a second-order convergence rate. However, the convergence rate is proportional to square of the step size, and this may result in a low convergence rate if the selected step size is excessively small. This problem leads to difficulties in the practical use of CKDR. Thus, an adaptive step-size method is proposed in this paper to control the convergence rate of CKDR. The proposed method estimates natural frequency of the model and determines adaptive step size. Static equilibrium simulations were performed for three different models to verify the method. The results revealed that the computational cost of CKDR with a variable step size was very efficient when compared to fixed step sizes and that the convergence rate was also controlled as intended. In addition, the lowest natural frequencies of models in static equilibrium were accurately estimated.

A New Gamma Scaling Maximum Power Point Tracking Method for Solar Photovoltaic Panel Feeding Energy Storage System

This paper analyzes the variable step size selection methods applied to the incremental conductance based maximum power point tracking technique (MPPT) for the solar photovoltaic system. The delta P/delta V (DpDv) variable step size method and the novel variable step size method, namely the gamma scaling method, are developed in Matlab/Simulink. Extensive results are obtained using these step size methods and compared. The proposed gamma scaling method is found to have higher efficiency with lower peak-peak ripple in load current as compared to DpDv, when sudden irradiance of 1000 W/m2 is applied to the PV panel. There are numerous other performance advantages of the gamma scaling method compared to the DpDv method under various irradiation conditions. The hardware prototype of the inverse sepic converter is used which is supplemented by the gamma scaling and DpDv methods. The simulated and experimental results of gamma scaling and DpDv methods are compared. In terms of performance, the proposed gamma scaling method has been found to be better than the DpDv method.

A Novel Model and ADMM Algorithm for MR Image Reconstruction

Motivated by the ideas from the LOT model and its deformations, we propose a coupling model for the MR image reconstruction and apply the split Bregman iterative method on the proposed model by utilizing the augmented Lagrangian technique. The related minimization problem is then divided into four subproblems by means of the alternating minimization method. And on this basis, by combining the Barzilai-Borwein step size selection scheme, generalized shrinkage formulas, and the shrink operator, we propose an ADMM type algorithm to solve the proposed model. Several numerical examples are implemented; the experimental results demonstrate the feasibility and effectiveness of the proposed model and algorithm.

Distributed Subgradient-Based Multiagent Optimization With More General Step Sizes

A wider selection of step sizes is explored for the distributed subgradient algorithm for multi-agent optimization problems, for both time-invariant and time-varying communication topologies. The square summable requirement of the step sizes commonly adopted in the literature is removed. The step sizes are only required to be positive, vanishing and non-summable. It is proved that in both unconstrained and constrained optimization problems, the agents' estimates reach consensus and converge to the optimal solution with the more general choice of step sizes. The idea is to show that a weighted average of the agents' estimates approaches the optimal solution, but with different approaches. In the unconstrained case, the optimal convergence of the weighted average of the agents' estimates is proved by analyzing the distance change from the weighted average to the optimal solution and showing that the weighted average is arbitrarily close to the optimal solution. In the constrained case, this is achieved by analyzing the distance change from the agents' estimates to the optimal solution and utilizing the boundedness of the constraints. Then the optimal convergence of the agents' estimates follows because consensus is reached in both cases. These results are valid for both a strongly connected time-invariant graph and time-varying balanced graphs that are jointly strongly connected.

Inversion of Surface Deformation Data for Rapid Estimates of Source Parameters and Uncertainties: A Bayesian Approach

AbstractNew satellite missions (e.g., the European Space Agency's Sentinel‐1 constellation), advances in data downlinking, and rapid product generation now provide us with the ability to access space‐geodetic data within hours of their acquisition. To truly take advantage of this opportunity, we need to be able to interpret geodetic data in a prompt and robust manner. Here we present a Bayesian approach for the inversion of multiple geodetic data sets that allows a rapid characterization of posterior probability density functions (PDFs) of source model parameters. The inversion algorithm efficiently samples posterior PDFs through a Markov chain Monte Carlo method, incorporating the Metropolis‐Hastings algorithm, with automatic step size selection. We apply our approach to synthetic geodetic data simulating deformation of magmatic origin and demonstrate its ability to retrieve known source parameters. We also apply the inversion algorithm to interferometric synthetic aperture radar data measuring co‐seismic displacements for a thrust‐faulting earthquake (2015 Mw 6.4 Pishan earthquake, China) and retrieve optimal source parameters and associated uncertainties. Given its robustness and rapidity in estimating deformation source parameters and uncertainties, our Bayesian framework is capable of taking advantage of real‐time geodetic measurements. Thus, our approach can be applied to geodetic data to study magmatic, tectonic, and other geophysical processes, especially in rapid‐response operational settings (e.g., volcano observatories). Our algorithm is fully implemented in a MATLAB®‐based software package (Geodetic Bayesian Inversion Software) that we make freely available to the scientific community.

An adaptive step size controller for iterative implicit methods

The automatic selection of an appropriate time step size has been considered extensively in the literature. However, most of the strategies developed operate under the assumption that the computational cost (per time step) is independent of the step size. This assumption is reasonable for non-stiff ordinary differential equations and for partial differential equations where the linear systems of equations resulting from an implicit integrator are solved by direct methods. It is, however, usually not satisfied if iterative (for example, Krylov) methods are used.In this paper, we propose a step size selection strategy that adaptively reduces the computational cost (per unit time step) as the simulation progresses, constraint by the tolerance specified. We show that the proposed approach yields significant improvements in performance for a range of problems (diffusion–advection equation, Burgers' equation with a reaction term, porous media equation, viscous Burgers' equation, Allen–Cahn equation, and the two-dimensional Brusselator system). While traditional step size controllers have emphasized a smooth sequence of time step sizes, we emphasize the exploration of different step sizes which necessitates relatively rapid changes in the step size.

Adjoint- and Hybrid-Based Hessians for Optimization Problems in System Identification

An adjoint sensitivity-based approach to determine the gradient and Hessian of cost functions for system identification of dynamical systems is presented. The motivation is the development of a computationally efficient approach relative to the direct differentiation (DD) technique and which overcomes the challenges of the step-size selection in finite difference (FD) approaches. An optimization framework is used to determine the parameters of a dynamical system which minimizes a summation of a scalar cost function evaluated at the discrete measurement instants. The discrete time measurements result in discontinuities in the Lagrange multipliers. Two approaches labeled as the Adjoint and the Hybrid are developed for the calculation of the gradient and Hessian for gradient-based optimization algorithms. The proposed approach is illustrated on the Lorenz 63 model where part of the initial conditions and model parameters are estimated using synthetic data. Examples of identifying model parameters of light curves of type 1a supernovae and a two-tank dynamic model using publicly available data are also included.

Single cardiac ventricular myosins are autonomous motors.

Myosin transduces ATP free energy into mechanical work in muscle. Cardiac muscle has dynamically wide-ranging power demands on the motor as the muscle changes modes in a heartbeat from relaxation, via auxotonic shortening, to isometric contraction. The cardiac power output modulation mechanism is explored in vitro by assessing single cardiac myosin step-size selection versus load. Transgenic mice express human ventricular essential light chain (ELC) in wild- type (WT), or hypertrophic cardiomyopathy-linked mutant forms, A57G or E143K, in a background of mouse α-cardiac myosin heavy chain. Ensemble motility and single myosin mechanical characteristics are consistent with an A57G that impairs ELC N-terminus actin binding and an E143K that impairs lever-arm stability, while both species down-shift average step-size with increasing load. Cardiac myosin in vivo down-shifts velocity/force ratio with increasing load by changed unitary step-size selections. Here, the loaded in vitro single myosin assay indicates quantitative complementarity with the in vivo mechanism. Both have two embedded regulatory transitions, one inhibiting ADP release and a second novel mechanism inhibiting actin detachment via strain on the actin-bound ELC N-terminus. Competing regulators filter unitary step-size selection to control force-velocity modulation without myosin integration into muscle. Cardiac myosin is muscle in a molecule.

Adaptive Step Size Gradient Ascent ICA Algorithm for Wireless MIMO Systems

Independent component analysis (ICA) is a technique of blind source separation (BSS) used for separation of the mixed received signals. ICA algorithms are classified into adaptive and batch algorithms. Adaptive algorithms perform well in time-varying scenario with high-computational complexity, while batch algorithms have better separation performance in quasistatic channels with low-computational complexity. Amongst batch algorithms, the gradient-based ICA algorithms perform well, but step size selection is critical in these algorithms. In this paper, an adaptive step size gradient ascent ICA (ASS-GAICA) algorithm is presented. The proposed algorithm is free from selection of the step size parameter with improved convergence and separation performance. Different performance evaluation criteria are used to verify the effectiveness of the proposed algorithm. Performance of the proposed algorithm is compared with the FastICA and optimum block adaptive ICA (OBAICA) algorithms for quasistatic and time-varying wireless channels. Simulation is performed over quadrature amplitude modulation (QAM) and binary phase shift keying (BPSK) signals. Results show that the proposed algorithm outperforms the FastICA and OBAICA algorithms for a wide range of signal-to-noise ratio (SNR) and input data block lengths.