Gradient Optimization Method Research Articles

Mini-Batch Adaptive Random Search Method for the Parametric Identification of Dynamic Systems

A possible method for estimating the unknown parameters of dynamic models described by differential-algebraic equations is considered. The parameters are estimated using the observations of a mathematical model. The parameter values are found by minimizing a criterion written as the sum of the squared deviations of the values of the state vector’s coordinates from their exact counterparts obtained through measurements at different time instants. Parallelepiped-type constraints are imposed on the parameter values. For solving the optimization problem, a mini-batch method of adaptive random search is proposed, which further develops the ideas of optimization methods used in machine learning. This method is applied for solving three model problems, and the results are compared with those obtained by gradient optimization methods of machine learning procedures and also with those obtained by metaheuristic algorithms.

Optimizing the Efficiency of First-Order Methods for Decreasing the Gradient of Smooth Convex Functions

This paper optimizes the step coefficients of first-order methods for smooth convex minimization in terms of the worst-case convergence bound (i.e., efficiency) of the decrease in the gradient norm. This work is based on the performance estimation problem approach. The worst-case gradient bound of the resulting method is optimal up to a constant for large-dimensional smooth convex minimization problems, under the initial bounded condition on the cost function value. This paper then illustrates that the proposed method has a computationally efficient form that is similar to the optimized gradient method.

A New Family of Hybrid Conjugate Gradient Methods for Unconstrained Optimization

The conjugate gradient method is a very efficient iterative technique for solving large-scale unconstrainedoptimization problems. Motivated by recent modifications of some variants of the method and construction of hybrid methods, this study proposed four hybrid methods that are globally convergent as well as computationally efficient. The approach adopted for constructing the hybrid methods entails projecting ten recently modified conjugate gradient methods. Each of the hybrid methods is shown to satisfy the descent property independent of any line search technique and globally convergent under the influence of strong Wolfe line search. Results obtained from numerical implementation of these methods and performance profiling show that the methods are very competitive with well-known traditional methods.

A Deformable Configuration Planning Framework for a Parallel Wheel-Legged Robot Equipped with Lidar.

The wheel-legged hybrid robot (WLHR) is capable of adapting height and wheelbase configuration to traverse obstacles or rolling in confined space. Compared with legged and wheeled machines, it can be applied for more challenging mobile robotic exercises using the enhanced environment adapting performance. To make full use of the deformability and traversability of WHLR with parallel Stewart mechanism, this paper presents an optimization-driven planning framework for WHLR with parallel Stewart mechanism by abstracting the robot as a deformable bounding box. It will improve the obstacle negotiation ability of the high degree-of-freedoms robot, resulting in a shorter path through adjusting wheelbase of support polygon or trunk height instead of using a fixed configuration for wheeled robots. In the planning framework, we firstly proposed a pre-calculated signed distance field (SDF) mapping method based on point cloud data collected from a lidar sensor and a KD -tree-based point cloud fusion approach. Then, a covariant gradient optimization method is presented, which generates smooth, deformable-configuration, as well as collision-free trajectories in confined narrow spaces. Finally, with the user-defined driving velocity and position as motion inputs, obstacle-avoidancing actions including expanding or shrinking foothold polygon and lifting trunk were effectively testified in realistic conditions, demonstrating the practicability of our methodology. We analyzed the success rate of proposed framework in four different terrain scenarios through deforming configuration rather than bypassing obstacles.

Pointwise‐constrained optimal control of a semiactive vehicle suspension

SummaryThis article presents an optimal control strategy (OCS) for semiactive vehicle suspensions with road profile sensors. The suspension is modeled as a quarter‐car model with a magnetorheological (MR) damper. The OCS main objective is to minimize the fourth‐power acceleration of the sprung mass. In addition, three pointwise constraints of the model are taken into account when the optimal control problem is solved: suspension travel limits (upper and lower) and tyre vertical force. In order to deal with a large number of constraints, we implement the gradient optimization method based on the method of moving asymptotes routine, which shows very good performance reaching optimal controls while satisfying the constrains. The solution has been compared with two passive MR damper configurations (low and high damping) as well as Skyhook and Balance control strategies for three different road inputs. Results show that OCS fulfills the constraints and reduces the sprung mass acceleration peak and the root‐mean‐quad acceleration up to 59%, in comparison to passive strategies.

Two-step conjugate gradient method for unconstrained optimization

Using Taylor’s series, we propose a modified secant relation to get a more accurate approximation of the second curvature of the objective function. Then, using this relation and an approach introduced by Dai and Liao, we present a conjugate gradient algorithm to solve unconstrained optimization problems. The proposed method makes use of both gradient and function values, and utilizes information from the two most recent steps, while the usual secant relation uses only the latest step information. Under appropriate conditions, we show that the proposed method is globally convergent without needing convexity assumption on the objective function. Comparative results show computational efficiency of the proposed method in the sense of the Dolan–More performance profiles.

Gradient-based adaptive modeling for IoT data transmission reduction

Spatial and temporal correlation between sensor observations in an Internet of Things environment can be exploited to eliminate unnecessary transmissions. Transmitting less data certainly contributes to meet the growing need for energy-saving and robust transmissions, thus prolong the lifespan of the entire WSN. Spatiotemporal correlation-based dual prediction (DP) and data compression (DC) schemes aim to reduce the amount of data transmission while ensuring data accuracy. In practice, however, the existing methods restrict the stability of the system when the model hyper-parameters are uncertain. Thus adaptive model has lately attracted extensive attention for the development of resource-constrained WSN. In this paper, we propose a gradient-based adaptive model that implements both schemes in a two-tier data reduction framework. To the best of our knowledge, the proposed scheme is the first attempt to introduce adaptiveness into both the DP and DC schemes by using a simple gradient optimization method. Gradient-based Optimal Step-size LMS (GO-LMS) is introduced to make the DP aspects adaptive, while a Gradient-based Adaptive PCA (GA-PCA) approach is used for the DC aspects. The Barzilai–Borwein method is incorporated into the gradient optimization to enable adaptive computation of the step-size for each iteration. Through extensive simulations, the developed framework was found to outperform other state-of-the-art schemes in terms of both the transmission reduction ratio and data recovery accuracy.

Estimation of parameters of various damping models in planar motion of a pendulum

In this work, planar free vibrations of a single physical pendulum are investigated both experimentally and numerically. The laboratory experiments are performed with pendula of different lengths, for a wide range of initial configurations, beyond the small angle regime. In order to approximate the air resistance, three models of damping are considered—involving the three components of the resistive force: linear (proportional to velocity), quadratic (velocity-squared) and acceleration-dependent (proportional to acceleration). A series of numerical experiments is discussed, in which the damping coefficients are estimated by means of several computational methods. Based on the observed efficiency, a gradient method for optimization is treated as the main tool for determination of a single set of damping parameters, independent of the system’s initial position. In the model of resistive force, the term proportional to acceleration, associated with the empirical Morison equation, seems to be indispensable for the successful approximation of the real pendulum motion.

A globally convergent projection method for a system of nonlinear monotone equations

ABSTRACT In this paper, a new conjugate gradient-based projection method for solving a system of nonlinear monotone equations is proposed. The method can be viewed as an extension of a family of conjugate gradient methods for unconstrained optimization by Li et al. [A new family of conjugate gradient methods for unconstrained optimization, J. Appl. Math. Comput. 58 (2018), pp. 219–234]. The proposed method is derivative-free which makes it suitable for large-scale nonlinear monotone equations. We show that the method satisfies the descent condition independent of line searches and that the method is globally convergent. Numerical results indicate that the proposed method is efficient.

Allocation of unequally-weighted wastewater discharge loads using a simulation-optimization approach

In this study, a new simulation–optimization approach is proposed to solve the multi-pollutant waste load allocation (WLA) problems in surface water systems. The proposed approach simulates the fate and transport of multiple pollutants utilizing the AQUATOX model. Since AQUATOX is an independent model and its integration into the optimization process is computationally not efficient, a multi-pollutant concentration–response matrix (mpCRM) is developed by using the results of the AQUATOX model. This mpCRM is then integrated into an optimization model where a nonlinear generalized reduced gradient (GRG) optimization method is used. Unlike in previously conducted studies, a new optimization formulation is proposed wherein the multiple pollutant loads are allocated among the source locations depending on their pre-assigned load allocation weights. This formulation allows for an equal or variable pollutant load allocation plan among source locations depending on the water management strategy for the watershed. The applicability of the proposed approach is evaluated on a sub-watershed of the Küçük Menderes River Basin (KMRB) in Turkey by considering different load allocation scenarios. Furthermore, a detailed sensitivity analysis is conducted to evaluate the model results for different problem parameters. Identified results suggest that the proposed simulation–optimization approach is an effective way to solve the multi-pollutant WLA problem.

A Modified Bat Algorithm with Conjugate Gradient Method for Global Optimization

Metaheuristic algorithms are used to solve many optimization problems. Firefly algorithm, particle swarm improvement, harmonic search, and bat algorithm are used as search algorithms to find the optimal solution to the problem field. In this paper, we have investigated and analyzed a new scaled conjugate gradient algorithm and its implementation, based on the exact Wolfe line search conditions and the restart Powell criterion. The new spectral conjugate gradient algorithm is a modification of the Birgin and Martínez method, a manner to overcome the lack of positive definiteness of the matrix defining the search direction. The preliminary computational results for a set of 30 unconstrained optimization test problems show that this new spectral conjugate gradient outperforms a standard conjugate gradient in this field and we have applied the newly proposed spectral conjugate gradient algorithm in bat algorithm to reach the lowest possible goal of bat algorithm. The newly proposed approach, namely, the directional bat algorithm (CG-BAT), has been then tested using several standard and nonstandard benchmarks from the CEC’2005 benchmark suite with five other algorithms and has been then tested using nonparametric statistical tests and the statistical test results show the superiority of the directional bat algorithm, and also we have adopted the performance profiles given by Dolan and More which show the superiority of the new algorithm (CG-BAT).

N-hidden layer artificial neural network architecture computer code: geophysical application example

We provide a MATLAB computer code for training artificial neural network (ANN) with N+1 layer (N-hidden layer) architecture. Currently, the ANN application to solving geophysical problems have been confined to the 2-layer, i.e. 1-hidden layer, architecture because there are no open source software codes for higher numbered layer architecture. The restriction to the 2-layer architecture comes with the attendant model error due to insufficient hidden neurons to fully define the ANN machines. The N-hidden layer ANN has a general architecture whose sensitivity is the accumulation of the backpropagation of the error between the feedforward output and the target patterns. The trained ANN machine can be retrieved by the gradient optimization method namely: Levenberg-Marquardt, steepest descent or conjugate gradient methods. Our test results on the Poisson's ratio (as a function of compressional and shear wave velocities) machines with 2-, 3- and 4-layer ANN architectures reveal that the machines with higher number of layers outperform those with lower number of layers. Specifically, the 3- and 4-layer ANN machines have ≥97% accuracy, predicting the lithology and fluid identification in the oil and gas industry by means of the Poisson's ratio, whereas the 2-layer ANN machines poorly predict the results with as large error as 20%. These results therefore reinforce our belief that this open source code will facilitate the training of accurate N-hidden layer ANN sophisticated machines with high performance and quality delivery of geophysical solutions. Moreover, the easy portability of the functions of the code into other software will enhance a versatile application and further research to improve its performance.

Sensor placement for RSSD-based localization: Optimal angular placement and sequential sensor placement

Energy-based localization has received great interest due to its low cost and simple implementation. It is well-known that sensor placement around the source plays a significant role in localization performance. This paper considers source localization based on the received signal strength difference (RSSD). Then, we propose two methods for sensor placement using maximization of the determinant of Fisher information matrix (FIM). The first one is based on the Gradient optimization method in which our optimization metric is a function of angular locations of all sensors, and the output of this method is optimal angular sensor separation which is called the optimal angular placement (OAP). In the second approach, we obtain the optimization metric as a function of single variable for each sensor in which the sensors are arranged in step-by-step manner, this method is called sequential sensor placement (SSP) in our study. At the end of this paper, simulation results reveal the ability of the proposed sensor placements.

Two New Conjugate Gradient Methods for Unconstrained Optimization

The conjugate gradient method is very effective in solving large-scale unconstrained optimal problems. In this paper, on the basis of the conjugate parameter of the conjugate descent (CD) method and the second inequality in the strong Wolfe line search, two new conjugate parameters are devised. Using the strong Wolfe line search to obtain the step lengths, two modified conjugate gradient methods are proposed for general unconstrained optimization. Under the standard assumptions, the two presented methods are proved to be sufficient descent and globally convergent. Finally, preliminary numerical results are reported to show that the proposed methods are promising.

Maximum likelihood identification of dual‐rate Hammerstein output‐error moving average system

This study discusses the parameter estimation of the Hammerstein output-error moving average system using the dual-rate sampled data. The polynomial transformation technique is used to obtain the identification model of the discussed dual-rate sampled systems. The stochastic gradient optimisation method is an effective optimisation method. Compared with the Newton optimisation, it only needs to calculate the first derivative during the optimisation and the amount of calculation is relatively small. It is a good choice to use the stochastic gradient algorithm for the identification of Hammerstein dual-rate model after using the polynomial transformation technique. In order to improve the convergence speed, a maximum likelihood forgetting factor stochastic gradient identification algorithm is proposed by combining the maximum likelihood principle and the gradient search method. The convergence of the algorithm is analysed by using the stochastic process theory. Furthermore, in order to improve the estimation accuracy of the identification algorithm, a maximum likelihood multi-innovation forgetting factor stochastic gradient algorithm is proposed by using the multi-innovation identification theory. The effectiveness of the proposed algorithms is illustrated by a numerical simulation example and a water tank system.

A linearly convergent stochastic recursive gradient method for convex optimization

The stochastic recursive gradient algorithm (SARAH) (Nguyen et al. in: Neural information processing systems, pp 2613–2621, 2017) attracts much interest recently. It admits a simple recursive framework for updating stochastic gradient estimates. Motivated by this, in this paper we propose a new stochastic recursive gradient method, called SARAH-I. Different from SARAH, SARAH-I incorporates importance sampling strategy and computes the full gradient at the last iterate in each inner iteration. We show that the sequence of distances between iterates and the optima set is linearly convergent under both strong convexity and non-strong convexity conditions. Furthermore, we propose to use the Barzilai–Borwein (BB) approach to adaptively compute step sizes for SARAH-I, and name the resulting method as SARAH-I-BB. We establish its convergence and complexity properties in different cases. Finally numerical tests are reported to indicate promising performances of proposed algorithms.

Gradient methods exploiting spectral properties

We propose a new stepsize for the gradient method. It is shown that this new stepsize will converge to the reciprocal of the largest eigenvalue of the Hessian, when Dai-Yang's asymptotic optimal gradient method (Computational Optimization and Applications, 2006, 33(1): 73–88) is applied for minimizing quadratic objective functions. Based on this spectral property, we develop a monotone gradient method that takes a certain number of steps using the asymptotically optimal stepsize by Dai and Yang, and then follows by some short steps associated with this new stepsize. By employing one step retard of the asymptotic optimal stepsize, a nonmonotone variant of this method is also proposed. Under mild conditions, R-linear convergence of the proposed methods is established for minimizing quadratic functions. In addition, by combining gradient projection techniques and adaptive nonmonotone line search, we further extend those methods for general bound constrained optimization. Two variants of gradient projection methods combining with the Barzilai-Borwein stepsizes are also proposed. Our numerical experiments on both quadratic and bound constrained optimization indicate that the new proposed strategies and methods are very effective.

Issues of implementing neural network algorithms on memristor crossbars

The property of natural parallelization of matrix-vector operations inherent in memristor crossbars creates opportunities for their effective use in neural network computing. Analog calculations are orders of magnitude faster in comparison to calculations on the central processor and on graphics accelerators. Besides, mathematical operations energy costs are significantly lower. The essential feature of analog computing is its low accuracy. In this regard, studying the dependence of neural network quality on the accuracy of setting its weights is relevant. The paper considers two convolutional neural networks trained on the MNIST (handwritten digits) and CIFAR_10 (airplanes, boats, cars, etc.) data sets. The first convolutional neural network consists of two convolutional layers, one subsample layer and two fully connected layers. The second one consists of four convolutional layers, two subsample layers and two fully connected layers. Calculations in convolutional and fully connected layers are performed through matrix-vector operations that are implemented on memristor crossbars. Sub-sampling layers imply the operation of finding the maximum value from several values. This operation can be implemented at the analog level. The process of training a neural network runs separately from data analysis. As a rule, gradient optimization methods are used at the training stage. It is advisable to perform calculations using these methods on CPU. When setting the weights, 3—4 precision bits are required to obtain an acceptable recognition quality in the case the network is trained on MNIST. 6-10 precision bits are required if the network is trained on CIFAR_10.

New Investigation for the Liu-Story Scaled Conjugate Gradient Method for Nonlinear Optimization

This article considers modified formulas for the standard conjugate gradient (CG) technique that is planned by Li and Fukushima. A new scalar parameter θkNew for this CG technique of unconstrained optimization is planned. The descent condition and global convergent property are established below using strong Wolfe conditions. Our numerical experiments show that the new proposed algorithms are more stable and economic as compared to some well-known standard CG methods.

On the line-search gradient methods for stochastic optimization

We consider several line-search based gradient methods for stochastic optimization: a gradient and accelerated gradient methods for convex optimization and gradient method for non-convex optimization. The methods simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. The focus of this paper is to numerically compare such methods with state-of-the-art adaptive methods which are based on a different idea of taking norm of the stochastic gradient to define the stepsize, e.g., AdaGrad and Adam.